I got a function that downloads lists of links in PHP (lets say about 100 000, but not at once). Would like to download data from those links only if it wasn't downloaded yet, so I need to check which of them are not in MySQL database. Database contains about 40 000 records for now. What is the proper way to do this? I can't keep all those links in array and compare to MySQL results, because it takes too much memory. And I am downloading information from those links multi-threaded (by forks). And if parent takes 10MB of RAM, 30 forks take 300MB, etc. I tried to query database for each link separately, but after short time I am getting disconnected from MySQL server, and when I try to connect again (i ping the connection to check if it is still alive) and try to select database it closes connection with error "MySQL server has gone away". How i supposed to be done?
You can "save" links in text file only for this check, its a lot faster to use this to compare if link is downloaded or not.
Have a look at this mytxt
This is not exactly an answer to your question but it might be worth your while considering saving all of the found results but store them in an associative array with the link as the key. This way duplicates will simply o rewrite previous versions.
The advantages of this approach is that you will not "waste" any time with checking but the disadvantage could be, especially if you are handling many columns, that you need too much time downloading redundant information.