How to avoid IP address-based blocks?
Learn how you can avoid IP blocks which can be extremely useful when you are doing web scraping.
Contents:
You IP is Visible
You need to remember that your IP address can be seen when you do web-scraping.
In case you send many requests coming from the same IP address the target website might block your IP. Which means you can't access this website anymore.
That is why you need to use proxy servers to hide your IP address.
In order to hide your IP address, we are going to use Proxybot - proxy rotation service to send an HTTP request through a random proxy server.
Proxy Rotating Service
Proxybot sends requests through random proxy servers. When the target website receives an incoming request it will see the IP address of a proxy server, but not your IP.
This strategy makes it impossible for the target website to distinguish between incoming requests and link them with your server. This simple trick allows to avoid and bypass IP blocks.
Let's see how we can use the service.
Hide Your IP address
It is very easy to use Proxybot service. We simply need to send an HTTP request with the following URL:
https://proxybot.io/api/v1/API_KEY?url=www.your-target-website.comEvery request will be sent through a random proxy. For the target website, it will look like all your requests are coming from random people/servers.
You can verify it by opening http://httpbin.org/ip website which shows the IP address of the incoming requests.
If you simply open the website in your browser - you will see your actual IP.
However, if you open the website through proxybot service, then you will see a new IP address.
Pretend you are from another country
There is also a possibility to send your request through proxy servers located in a specific country.
For example, if you want your request to come from Germany, then you need to append '&geolocation_code=de' URL param to your request.
See the below request example:
https://proxybot.io/api/v1/API_KEY?url=http://httpbin.org/ip&geolocation_code=deFor people who prefer watching videos, there is a quick video showing how to your IP address using Proxybot API service.
See the full list with all available proxy geolocations here.
As you can see, it is very easy to stay anonymous and avoid IP blocks.
Have a nice day ✌️