Skip to content

Speed up for big file reading

2011/02/27

That helps parsing logs with power shell:

“You can dramatically speed up reading large text files by adding the parameter -ReadCount 0 to Get-Content”

You have to set read-count to 0 .

What I didn’t find till now: The default is ???. I’ve not done some testing by my self but everything should be faster than the default ;-)

Update:

done some testing. Powershell itself helps here with the measure-command cmdlet which keeps track of the execution time for an expression. With a 30 mb text file a one time test shows a factor of 10:

PS C:\> Measure-Command -Expression {$a=get-content C:\_allUserAccounts.txt}


TotalSeconds      : 4,5549688
TotalMilliseconds : 4554,9688

PS C:\> Measure-Command -Expression {$a=get-content C:\_allUserAccounts.txt -ReadCount 0}

..
TotalSeconds      : 0,4323658
TotalMilliseconds : 432,3658

That is not bad. but what happens if the files get bigger . Here a 120 mb text file test. It shows that there seem to be internal differences. The first invocation , looking at the task manager,  blows up the memory usage of the powershell process to 1.5 gig, whereas the second invocation with “read-count 0” only consumes around 360 mb of memory. The factor of execution time changes to 1:16. The bigger the file, the more profit you get by using read-count parameter:

PS C:\> Measure-Command -Expression {$a=get-content "C:\AUTN\out\20110406-TEST_P.txt"}
..
TotalSeconds      : 54,0904235
TotalMilliseconds : 54090,4235

PS C:\> Measure-Command -Expression {$a=get-content "C:\AUTN\out\20110406-TEST_P.txt" -readcount 0}
..
TotalSeconds      : 3,3633212
TotalMilliseconds : 3363,3212
Advertisements

From → file, powershell

Leave a Comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: