I need to write a server-side program that lives on the server, and is checking a database consistently for new entries.
When a new entry shows up in the database, the program should process the data and put the results somewhere else.
It is important to hi-light that the process isn't instigated by new entries showing up, but by the program checking for new entries on its own.
Some people I've spoken to brought up cron jobs, I was curious what if this is the solution for me? I see that it has limitations, it won't run less than every minute. I was hoping for the program to run every 5 seconds, would I be better off writing a shell script or is that a bootleg fix?
I'm not sure if this is conventional (?) but...
Use a database trigger on INSERT that runs an external program (PHP, Python, .. whatever). Which database are you using? I think this post is old but might be of help: http://crazytechthoughts.blogspot.co.uk/2011/12/call-external-program-from-mysql.html
There is a technique I've frequently used when dealing with queues that I've been processing.
#!/bin/sh
php -f checkDBAndAct.php
sleep 5
exec $0
The exec $0 part starts the script running again, replacing itself in memory, so it will run forever without issues. Any memory the PHP script uses is cleaned up whenever it exits, so that's not a problem either.
A simple line will start it, and put it into the background:
cd /x/y/z ; nohup ./loopToProcessDB.sh &
or it can be similarly started when the machine starts with various means (such as Cron's '@reboot ....')
-- from https://stackoverflow.com/a/2686100/6216 An extended version is on http://PHPscaling.com and https://gist.github.com/alister/1386212
Though I'd use an actual queue system, rather than a DB, as there are a number of downsides to bending a database to this task.