Similar to how google analytics sends beacons from javascript that track events, what are the most efficient ways to collect that beacon data and return back to the client in the fastest time?
For example, if I have a server to server beacon call I want to make that call as fast as possible on the clients server.
PHP to a flat files? PHP to a local queue? Java Server that logs to a queue and maintains a connection the remote queue the whole time? custom c++ server?
This would be on the order of 1000 requests per second.
There are 2 aspects to this.
1) the client's beacon call should be done as quickly as possible. This means the incoming HTTP request should respond 200 OK and exit as soon as possible, so it probably shouldn't do the actual data writing itself. It should hand that off to another process in the background, either by a background shell execution or by utilizing a queue/job mechanism like Gearman.
2) The data writing itself, if done in a background thread away from the client's attention, has a little more time luxury. 1000 writes per second should be fine for a modern hardware well tuned database with row locking that's not being SELECTed from too heavily at the same instant. Perhaps, though, this could be a good usage scenario for a key-value store for the immediate data storage. Then a separate analysis/reporting process could query the key-value store off-line for all stored data, process it, and eventually copy it into a database.