Summary :
I want to fetch all Play store search result, problems is that, Apps that show after scroll that are not show in PHP file_get_content().
Detail:
I'm trying to make a php based play store scraper.
I checked all stackoverflow answers and github example, but they all are old, and not working. This is because previously Play store use "start" parameter for more apps/Next page.. Now play store shows more apps on scroll
So after so many research, I decide to make my own scraper.
What I am doing:
file_get_contents() : to fetch the query url from play store like so:
$result = file_get_content( "https://play.google.com/store/search?q=football" );
This only returns 20 apps in the result, I want 250.
Play store shows only 20 apps on the query result, more apps are shown ONLY on scrolling.
I try to get google ajax URL from "network tab", But google use token parameter for next scroll page...
Questions:
How can I get scrolled content with PHP? If not possible, how can I get play store all search result?
Otherwise how can I fetch page content with PHP that shows with javascript?
Have a look at the store site in chrome dev tools. It has a perfect data source, but you may have trouble getting the tokens. I doubt they want this scraped.
General:
Request URL: https://play.google.com/store/apps?authuser=0
Request Method: POST
Status Code: 200
Remote Address: 216.58.220.142:443
Form Data:
start: 20
num: 5
numChildren: 10
pagTok: CBQQFA==:S:ANO1ljJ62NU
ipf: 1
xhr: 1
token: OmILlMEkRv_FCJ1tveWI8rtELHQ:1459318564260