如何为Web应用程序构建高效的后台批处理过程？

I have a web application written in PHP using a Postgres database.

The next phase of development is for background batch processes to be built that will need to be executed once a day (or adhoc as requested) for each user of the app. The process will query, await response and process the response from third-party services to feed information into the user's account within the web application.

Are there good ways to do this?

• How would batches be triggered every day at 3am for each user?
• Given there could be a delay in the response, is this a good scenario to use something like node.js?
• Is it best to have the output of the batch process directly update the web application's database with the appropriate data? Or, is there some other way to handle the output?

Update: The process doesn't have to run at 3am. The key is that a few batch processes may need to run for each user. The execution of batches could be spread throughout the day.. I want this to be a "background" process separate to the app.

You could write a PHP script that runs through any users that need to be processed, and set up a cron job to run your script at 3am. Running as a cron job means you don't need to worry so much about how slow the third party call is. Obviously you'd need to store any necessary data in the database.

Alternatively, if the process is triggered by the user doing something on the site, you could use exec() to trigger the PHP script to process just that user, right away, without the user having to wait. The risk with this is that you can't control how rapidly the process is triggered.

Third option is to just do the request live and make the user wait. But it sounds like this is not an option for you.

It really depends on what third party you're calling and why. How long does the third party take to respond, how reliable they are, what kind of rate limits they might enforce, etc...