Basically I'm trying to fetch live scores of soccer matches and add them to a personal database on my server. This works nearly as expected, execpt for one thing:
I use a PHP script to automatically fetch the score and insert it into the database. I use the uefa.com website as source. Let me give you an example:
This http://www.uefa.com/under21/season=2017/matches/live/day=20/session=1/match=2016266/index.html is the path to track the current score. This page is updated every minute using this JSON file: http://www.uefa.com/livecommon/match-centre/cup=13/season=2017/day=20/session=1/match=2016266/feed/minute.json (later referred to as $json_url). When using FireBug or similar tools, one can see that the JSON responded by the HTTP GET request (initiated by the JS on the page) is "live", it's content changes frequently. However when getting the JSON through PHP (file_get_contents($json_url)), it appears that the response changes just every couple of minutes; it seems somehow cached.
I did a little testing and analysing with HTTP headers and stuff, I got the result that using linux cli cURL can fetch the JSON just perfectly without this cache-like phenomenon.
Naturally I now tried achieving this using the cURL-plugin of PHP but - no effect compared to the situation before. Now I compared what those two cURL requests do in detail and - they do exactly the same. However they get different HTTP headers back:
This is the output of curl -v $json_url:
* About to connect() to www.uefa.com port 80 (#0)
* Trying 95.100.248.114...
* connected
* Connected to www.uefa.com (95.100.248.114) port 80 (#0)
> GET /livecommon/match-centre/cup=13/season=2017/day=20/session=1/match=2016266/feed/minute.json HTTP/1.1
> User-Agent: curl/7.26.0
> Host: www.uefa.com
> Accept: */*
>
* additional stuff not fine transfer.c:1037: 0 0
* HTTP 1.1 or later with persistent connection, pipelining supported
< HTTP/1.1 200 OK
< Cache-Control: public, max-age=300, must-revalidate
< Content-Type: application/json
< Last-Modified: Wed, 07 Oct 2015 20:37:55 GMT
< ETag: "16ec79b401d11:0"
< Server: Microsoft-IIS/7.5
< X-Powered-By: ASP.NET
< Date: Wed, 07 Oct 2015 21:57:53 GMT
< Content-Length: 285
< Connection: keep-alive
<
* Connection #0 to host www.uefa.com left intact
[JSON DATA...]
* Closing connection #0
And now the one sent by PHP-cURL:
* About to connect() to www.uefa.com port 80 (#0)
* Trying 95.100.248.114...
* connected
* Connected to www.uefa.com (95.100.248.114) port 80 (#0)
> GET /livecommon/match-centre/cup=3/season=2016/day=20/session=1/match=2016266/feed/minute.json HTTP/1.1
User-Agent: curl/7.26.0
Host: www.uefa.com
Accept: */*
* additional stuff not fine transfer.c:1037: 0 0
* HTTP 1.1 or later with persistent connection, pipelining supported
< HTTP/1.1 200 OK
< Cache-Control: private
< Content-Length: 285
< Content-Type: application/json; charset=utf-8
< Expires: Wed, 07 Oct 2015 21:38:54 GMT
< Server: Microsoft-IIS/7.5
< X-AspNet-Version: 2.0.50727
< X-Accel-Cache-Control: max-age=600
< X-Powered-By: ASP.NET
< Date: Wed, 07 Oct 2015 21:28:54 GMT
< Connection: keep-alive
<
* Connection #0 to host www.uefa.com left intact
* Closing connection #0
(Note that the dates in those reponses aren't the problem - this match was finished before I tested the commands)
I think Cache-Control: private in the HTTP response header of the second block is the problem and it should be Cache-Control: public, max-age=300, must-revalidate like in the first one; but, I want to know why the headers are different at all! As far as I understand it, the request headers are exactly the same. Some notes to the php curl block: I used the most basic curl request in PHP but added curl_setopt($ch, CURLOPT_USERAGENT, 'curl/7.26.0'); otherwise no user-agent was included using curl.
I'm at my wits' end with this - does anyone know what I'm doing wrong or at least can explain where the difference comes from? I appreciate any help!