I was wondering if somebody could give me some advice. At the moment we are building a application that uses a lot of API's, such as Flickr, Facebook, Salesforge, Picasa, Twitter, LinkedIn etc. Now I was wondering what would eb the best idea to tackle the problem of data usage. First of all, requesting all that data eats away at your data capacities. Also the application should eb able to handle 1000's of users at the same time so using all those requests might get us flagged as DDOS attackers :P So my solution would be to store everything locally in MySQL, and update critical data every hour and other data once a day. How would you guys tackle this problem? One other problem is images, we will be using 1000000's of pictures. Would you guys recommend hosting those locally and updating them every so often, or just store links and send those to users? One final problem I am having, is how does MySQL perform with 1000's of users making hundreds of request every second, from complex tables. For example if a user loads a album, the database has to spit out 100 photos. Imagine that being done for 100 of users at the same time. Is mysql able to handle that all?
Thanks in advance, Sam
Most (if not all) public API's have something called ETag. This is a fingerprint of the data and you can make conditional GET requests to the API like:
"Give me the image with id 6123765 if the Etag differs from E33SDF33D234DMD!" This is done with the If-None-Match HTTP header.
If-None-Match:W/"98F96B877E576EA4E5A278D41217959117663007"
The API will then reply either with the image or it will give you a HTTP 302 (It is in your cache). If the server give you the image your cached one was outdated and the server will also give you the new Etag in the Etag HTTP response header:
ETag:W/"A4D860491DE0E51EA05F46143A41CC58F7270142"
In your cache you store the image ID, the image data and the Etag. Then when you request the image from the API with the ETag and if the server replies with 302 you can deliver the image you have cached. If the server give you an image and a 200 reply, you update your cache with the new image and the Etag the server gave you for the image. And transfer the image to the caller.