I have to setup a recurring invoice system for my boss and our system is built with php on a centos linux server. We have about 50,000 customers who we invoice every month, my job is to build a way to process the invoice script for all 50,000 customers each week, so the script will create a new invoice and do an insert query into the invoices table in our mysql database.
My concern is if 50,000 seperate cron jobs ran every week will result in too heavy a load for our server, and what kind of limit crontab is able to handle before causing performance problems?
Also important to note that the crontab file will need to be edited via a php script very often for editing and/or deleting individual invoices/cronjobs.
Basically is this the best route to go?
You should have a single Cron job that runs once, and loops through the 50,000 customers.
What I generally do is pick one from the database, process it, mark it as done, then move on to the next one. I make my loop drop out after running for more than 170 seconds, and set a Cron job to run every 3 minutes for the whole month - it'll get through them as it can.
Definitely not! You may have one cron job scheduled to generate the invoices. This job will fetch the list of customers and generate the invoices for them. You may consider parallel processing if it takes lot of time for one job to generate the invoices.