How to get a random IP address for every HTTP request?
A frequent issue encountered by web scrapers is getting blocked by websites while scraping them. It happens because a web server can see the IP address of incoming requests. An obvious solution is to use a random IP address for every request you send to your target website.
In this blog post, you will learn how to hide your IP address.
There are many techniques to prevent getting blocked, but we are going to focus on solutions based on using proxies.
Contents:
What is a Proxy server?
First of all, we need to understand what is a proxy server and how it works.
A proxy server acts as a middle man between client and server. A server can see the IP address of incoming requests. Normally, your request will be sent directly to the server, therefore the server will be able to see your IP and detect al your future requests.
That is why we want to use a proxy server that will be sitting between you and the server. The Proxy will take your request and pass it to the target server. It means that the server receives a request from a proxy and not from you. The server even doesn't know about your existence. This is how we will stay anonymous.
Now, when we know how we can benefit from using proxies, let's learn how we can use them.
Proxies
The below example shows how to use Python-Requests to send your HTTP requests through a proxy server.
import requests
proxies = {
'http': 'http://10.10.1.10:3128',
'https': 'http://10.10.1.10:1080',
}
requests.get('http://example.org', proxies=proxies)
The httpbin service should respond with an origin property holding the IP address of the incoming request. This is how you can verify that a proxy server worked and your actual IP was hidden.
{
"origin": "10.10.1.10"
}
Where to find proxy servers?
There are plenty of websites offering free proxy lists. For example https://free-proxy-list.net
Of course, free proxies have their disadvantages. It happens because they are free and available to the public which means that a lot of people are using them at the same time. Many simultaneous connections might affect the performance and reliability of the proxies. I have also noticed that the proxies die pretty quickly. You will have to constantly check their health and ick only working proxies.
If you do care about quality and anonymity then we are highly recommending using premium proxies. Below you can see a list with the best proxy providers on the market.
If you don’t like an idea of maintaining a proxy list there is an easier solution - using proxy rotating service. The next section I will explain how you can use it.
Rotating IP service
The idea behind these services is very easy. Your request will be automatically sent through a proxy and a new random IP address will be assigned to your request. You don't have to do any setup or search for proxies, because the service will do it for you.
There are plenty of services offering similar functionality, but my favorite one is Scraperapi.
The below example demonstrates, how easy it is to use the service. You simply need to send an HTTP request and specify your target website URL.
http://api.scraperapi.com/?api_key=APIKEY&url=http://httpbin.org/ipYour request to the "http://httpbin.org/ip" will be sent through a random proxy. You can verify it by executing this request several times. Every response will contain a random IP address proving the usage of a proxy.
The service also allows you to send your request through a proxy with a specific geographic location. For example, you can pretend your request is coming from Germany or Spain. You simply need to add URl query param with country code you want to use, for example, use "country_code=de" for Germany. See the below example.
http://api.scraperapi.com/?api_key=APIKEY&url=http://httpbin.org/ip&country_code=de
Documentation will full list of available counties can be found here.
The Scraperapi is subscriptions based. In case you just need to buy a specific amount of API call you can use an alternative service called Proxybot.io