I have a very large files -- some reaching up to 10GB -- that contains mostly structured data (e.g. 99% of it a tab-separated values for each line of text I'm interested in). I need to extract from these files very specific pieces of data that I can easily find via regex. However, my concern is that I'm going to run into all sorts of problems if I try to, say, convert the file into a string and then regex that string.
What's a good strategy for regex parsing very large files?
Read the file line by line (fgets) and process it in chunks.