Multiple Proxy Servers in Selenium Web-driver Python

Published in

ML Book

3 min readOct 29, 2019

Last update: December, 2020

While scraping data from a website, due to large number of requests, website stop their service for user IP address or block user for further requests. In such case Proxying is very useful when conducting intensive web crawling/scrapping or when you just want to hide your identity (anonymization).

Content:

Proxies
Install http-request-randomizer
Collect proxy list
Use proxy in Selenium Web driver
Iterate over proxy list
Use Country specific proxy
Next Totorial

1.0 Proxies

Proxies provide a way to use server P (the middleman) to contact server A and then route the response back to you. In more nefarious circles, it’s a prime way to make your presence unknown and pose as many clients to a website instead of just one client. Often times websites will block IPs that make too many requests, and proxies is a way to get around this.

2.0 Install http-request-randomizer using pip

pip install http-request-randomizer

3.0 Collect proxy list

3.1 Check IP address and country of proxy

>> proxies[0].get_address()
>> '179.127.241.199:53653'>>proxies[0].country
>>'Brazil'

4.0 Use proxy in Selenium Web driver

4.1 Select proxy

PROXY = proxies[0].get_address()
print(PROXY)
>>'179.127.241.199:53653'

4.2 Use in Firefox

4.3 Use in Chrome

4.3 Check your IP address in browser

driver.get('https://www.expressvpn.com/what-is-my-ip')

you will see proxy address which you have selected as your IP address not your actual IP address, you can see your actual IP address on https://www.expressvpn.com/what-is-my-ip

5.0 Iterate over proxy list

Now you can iterate over proxy list, use web-driver one by one with each proxy and close it. check how many requests a particular websites allows and use a single proxy for that many requests

6.0 Use Country specific proxy

Usually proxy address slows down internet speed, to overcome this problem you can use proxies of your country, which may be faster than other proxies. but this may reduce number of proxy significantly, for example if I have total 800 proxies and if I select only proxies from India then I may get less than 50 proxies out of that.

In this way you can create list of any specific country.

In this way you can user proxy, you may get proxy list other then “http-request-randomizer” Module, On internet there are lots of proxy providers.

Note: These proxies require a good internet connection, and some of these proxies may not work, so try with different proxies.

Thanks 🙂

7.0 Next Tutorial

Coming soon…

Please clap if you like this tutorial

Join our Telegram channel for more updates, study resources and discussion