Custom Headers
If you would like to add custom/additional headers (user agents, cookies, etc.), simply add them as parameters to the request, with the “wsa-” prefix.
Only use this feature to get customized results, do not use this feature to avoid blocks. Shifter's Scraping API has a powerful internal engine that takes care of everything for you.
On this page, you will find an example request used to scrape the URL https://httpbin.org/headers, which will mirror the headers sent.
Custom Headers example
GET https://scrape.shifter.io/v1?api_key=api_key&url=http%3A%2F%2Fhttpbin.org%2Fheaders
⇡ Input
curl --request GET --url "https://scrape.shifter.io/v1?api_key=api_key&url=http%3A%2F%2Fhttpbin.org%2Fheaders" --header "Wsa-My-header: test" --header "Wsa-User-Agent: potato"
⇣ Output
{
"headers": {
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
"Accept-Encoding": "gzip, deflate, br",
"Host": "httpbin.org",
"My-Header": "test",
"Upgrade-Insecure-Requests": "1",
"User-Agent": "potato",
"X-Amzn-Trace-Id": "Root=1-6267c064-44fa9993017cfcc32e41dfc1"
}
}
Last updated
Was this helpful?