Hi guys,
I am using the OpenXMLReader to read a large Excel file with circa 500,000 rows of 20 columns. The file size is not that big (35MB).
I am reading through all elements and gathering together data from 512 rows at a time into a List. Then after processing the data the routine continues to process the rows and get the next batch. This continues until all batches have been read.
Memory consumption is absolutely fine and when the first few batches start to be read it is reasonably fast to read a batch of 512 rows (e.g. 100-200 milliseconds). As this continues this gradually slows down to seconds and then by the end of the file it's reached about 17 seconds to read 512 rows.
Is this expected behavior because it sure came as a surprise to me? With a streaming approach I would have expected that reading the last few batches would be roughly the same as reading the first few batches.
I have added detailed logging to make sure the time is spent in the function that is parsing the elements and there really is nothing complicated going on in there. Just reading the cell values and writing the strings into a List for each row.
Is there anything I need to do to prevent this behavior? All input very welcome.
Kind regards,.
Ian
