I have huge problem.
I have a task to put in database information about products. All information's except price are in XML file that is big 100mb.
In another file there are information's about price of that product. That is CSV file with size about 200mb
In simple logic
But i Cannot do that. Problem is in SIZE.
I manage to parse XML and put it in array, but I cannot handle CSV file. If there is 1000 products, for each product he need to parse CSV file (that CSV file have more than 30.000.000 lines). I cannot parse that CSV one, much less 1000.
I need suggestion how to solve this problem
Looks like a job for a database to me. Load both files into a database and then do a simple join.
Use XMLReader for the XML, SplFileObject for the CSV.
Do not keep the information in memory. Write it directly into a database like MySQL or SQlite. After all the data is imported you can use SQL to aggregate it.