Scrolling bar is automatically triggered in a DataGridView

I'm using a DataGridView for a VSPackage tool window, WinForms style.
I'm using virtual mode since I have a very large data-set to display (`CellValueNeeded` is subscribed and handled appropriately).

The number of rows and columns to be displayed is determined only at run-time, so I use a paging mechanism to display only small parts of the data, e.g.

    // initialization of the view

    DataGridView dgv = new DataGridView();
    dgv.RowCount = 100;    
    dgv.ColumnCount = 50;

So the user sees the first 100 rows of the data, and when he scrolls down/right, I have an event handling method that increments the RowsCount and ColumnCount to display more data:

private void dataGridView_Scroll(object sender, ScrollEventArgs e)
{
dataGridView.ColumnCount += getNumColsLeft();
dataGridView.RowCount += getNumRowsLeft();
// ...
}

But, the problem is that the scrolling bar goes crazy after I draw it to the bottom, it repeatedly scrolls-down on its own behalf even though I dragged it only once and I was idle with the mouse.

It's like i'm in an infinite loop of scrolling that triggers another scrolling, feels like a big events mess.

How can I prevent it?