I have a crawler that uses cURL to scrape data from many arrays of URLs but this is rather slow and I'd like to accelerate it using multi-threading by forking into several child processes that run concurrently.
The question is how do I determine the optimal number of threads? I have a decent dedicated server, but I'm not sure how to calculate and allocate those resources to run my scripts in the least amount of time.