Introduction to Selenium-wire
Selenium is a popular open-source web-based browser automation suit. It focuses on automating web-based applications, hence mostly used for automation testing. By extending Selenium’s Python bindings, Selenium Wire provides access to underlying browser requests.
Since selenium is open-source, which is a significant benefit over other testing tools as follows:
- Test scripts can be written in any of these programming languages.
e.g. Java, Python, C#, Ruby, Perl, Javascript, Kotlin.
- You can use any of these browsers
e.g. Google Chrome, Safari, Mozilla Firefox, Internet Explorer
- Selenium support is community-based which enables continuous support and constant updates.
- It Supports various operating systems.
This is all about a short overview of selenium, In this blog post, we are discussing the selenium-based library that is Selenium Wire.
Selenium Wire is used to give you access to the underlying requests made by the browser.
Features
- You can capture both HTTPS and HTTPS requests
- Intercept requests and responses
- Easy to modify header, parameters, and body content on the fly
- It captures WebSocket messages.
- It supports a Proxy server.
1. Installation
You can use the pip command to install the selenium wire library. Use the below pip command to install the selenium wire.
pip install selenium-wireNext, it will require any web browser driver. For the coming demonstration, I will be using a chrome web driver.
You can download the chrome browser version compatible chrome web driver from here.
Now we are ready to create the instance for the chrome driver.
driver = webdriver.Chrome('/home/user/drivers/chromedriver')If the above Chrome driver doesn't work simply use webdriver-manager.
Use the below pip command to install webdriver-manager.
pip install webdriver-managerOnce it is installed.Use ChromeDriverManager().install() instead of the chrome driver path
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
driver = webdriver.Chrome(ChromeDriverManager().install())Now no need to update the chrome driver and its path again and again for every new version of the driver.
Once installation is done we are good to go for implementation.
2. Implementation
Now let's see how to access different requests and responses using selenium wire.
- Create instance
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
driver = webdriver.Chrome(ChromeDriverManager().install())2. Next, to navigate the given webpage, use the get() function.
driver.get('https://www.python.org/')3. Use the request attribute to access requests.
for request in driver.requests:
if request.response:
passdriver.requests return all the requests there in the given webpage. we can get the URL, response code, header, and even the response body of that request and much more. The following are some examples of these methods.
- request.url — return URL of the request
- request.response.status_code — return response status code made by the request.
- request.response.headers[‘Content-Type’] — it returns the content type of all the header attributes.
- request.response.body — It returns the body of the response.
3. Application
In migration projects, it is required to migrate thousands of pages of the same type like articles, recipes, blogs, podcasts, etc. From some other platform like Salesforce, etc.
To migrate web pages from one platform to another, only the content of those pages is required. To get the content for each page we can use different tools for such APIs. But it is a manual process to get that JSON API request for that particular page and then retrieve the response and then use it for further processing.
But with the help of Selenium Wire, we can simply pass the webpage URL, filter out the request, and get the response body. We can run the same set of actions on thousands of web pages without any manual intervention.
References —
