I'm trying to update a large number of couch records (order of millions) using PHP's Curl interface, and running into a Curl error after processing 30,000ish records. It is throwing:
"Failed to connect to 127.0.0.1: Cannot assign requested address"
Other answers on StackOverflow and the internet have indicated this is the result of the system running out of resources, namely ports. Periodically checking the number of ports stuck in TIME_WAIT (netstat -an | grep TIME_WAIT | wc -l), this does seem to be the case.
Looking at my code, I'm calling "curl_init()" and "curl_close()" for each and every update PUT message, so this makes sense, and if I was trying to grab millions of views, I could just set a curl resource once, run my views, then close it.
However, since a CouchDB record update includes the docID as part of the URL string, every one of the millions of targets are effectively unique URLs. I can recycle and reuse a single curl resource, re-targeting it for each PUT, but this is vastly slower then just abandoning them in TIME_WAIT purgatory and making a new connection. While this does work, when updating millions of records performance matters.
SO_REUSEADDR and SO_REUSEPORT look like interesting ideas, but from what I've been able to find those only exist on the raw socket level. Since we are interfacing to Curl through PHP, I can't find any way to set these to see if they might help.
Is there some clever way to recycle a curl resource when doing Couch updates that I haven't thought of?
Is there a way to set SO_REUSEADDR/PORT for Curl in PHP?
Is there some way to expedite the release of resources after I'm done and "curl_close()"?