I'm trying to get info from a website using AJAX. When I use the REST plugin for chrome I can send the request and receive the answer required. I want to write a python script but I am stuck as to how to write the request in the correct format. Currently I am getting an unauthorized page response, which I believe is due to not sending it using XHR?
Code:
import json
import requests
payload = {"stores":"3650","products":{"31445761":[{"sku":"6000197536050","upc":["4549659075"]}]},"origin":"pip","csrfToken":"d707af2ed8b79a78a669b38dff593c909f6b6262-1507764346644-ebc72845dfa30431a8f7b1c4"}
with requests.Session() as session:
session.get("https://www.walmart.ca")
r = session.post('https://www.walmart.ca/ws/en/products/availability', data=json.dumps(payload),
headers={"Content-Type": "application/x-www-form-urlencoded; charset=UTF-8"})
print(r.content)
When I am in REST Chrome extension I type:
stores=["3650"]&products={"31445761":[{"sku":"6000197536050","upc":["4549659075"]}]}&origin=pip&csrfToken=d707af2ed8b79a78a669b38dff593c909f6b6262-1507764346644-ebc72845dfa30431a8f7b1c4
which gives me the desired result.
Desired output:
{
31445761: {
"online": [
{
"maxRegularPrice": 0,
"minRegularPrice": 0,
"mapPrice": 0,
"minCurrentPrice": 99.96,
"maxCurrentPrice": 99.96,
"inventory": 0,
"sku": "6000197536050",
"clearance": false,
"offerId": "6000197536050",
"limited": false,
"reducedPrice": false,
"offerType": "1P",
"limitedStock": false,
"sellerId": "0",
"rollback": false,
"date": "",
"status": "OutOfStock",
"eligible": false,
"sellerName": "Walmart",
"asAdvertised": false
}
],
"stores": [
{
"minRegularPrice": 0,
"maxRegularPrice": 0,
"minCurrentPrice": 99.96,
"maxCurrentPrice": 99.96,
"inventory": 0,
"sku": "6000197536050",
"clearance": false,
"limited": false,
"limitedStock": false,
"rollback": false,
"date": "",
"status": "OutOfStock",
"storeNumber": "3650",
"eligible": false,
"asAdvertised": false
}
],
"onlineSummary": {
"status": "OutOfStock",
"date": "",
"eligible": false,
"clearance": false,
"rollback": false,
"asAdvertised": false,
"limited": false,
"limitedStock": false,
"minRegularPrice": 0,
"maxRegularPrice": 0,
"minCurrentPrice": 99.96,
"maxCurrentPrice": 99.96
},
"storeSummary": {
"status": "OutOfStock",
"date": "",
"eligible": false,
"clearance": false,
"rollback": false,
"asAdvertised": false,
"limited": false,
"limitedStock": false,
"minRegularPrice": 0,
"maxRegularPrice": 0,
"minCurrentPrice": 99.96,
"maxCurrentPrice": 99.96
}
}
}`
You are just missing a header :
'X-Requested-With': 'XMLHttpRequest'
Updated code :
import json
import requests
payload = {"stores":"3650","products":{"31445761":[{"sku":"6000197536050","upc":["4549659075"]}]},"origin":"pip","csrfToken":"d707af2ed8b79a78a669b38dff593c909f6b6262-1507764346644-ebc72845dfa30431a8f7b1c4"}
with requests.Session() as session:
session.get("https://www.walmart.ca")
r = session.post('https://www.walmart.ca/ws/en/products/availability', data=json.dumps(payload),
headers={"Content-Type": "application/x-www-form-urlencoded; charset=UTF-8", 'X-Requested-With': 'XMLHttpRequest'})
print(r.content)
Also, usually the CSRF token changes. So you should retrieve HTML, get the CSRF token, and then do your XHR POST request.
I would recommend beautifulsoup to find the CSRF token in the HTML.