Exploring Website’s API

Published in

Analytics Vidhya

9 min readJan 16, 2021

Nowadays, due to the increasing popularity of Javascript frameworks such as React, many websites are rendering dynamic content by loading JSON or XML from their backend, meaning that fewer sites are generated server-side and are instead being rendered client-side.

In this post, we will explore Wolt’s website by accessing their API directly using some Python code.

Wolt is a food delivery platform. It allows people to find food from their restaurant partners to either pick it up themself or have it delivered by their delivery partners.

We are going to use Chrome DevTools to monitor all the network requests Wolt’s website is making to their server and see how we can make the same requests using our own code. The data we will retrieve from the server will include information about nearby restaurants such as rating, description, address, delivery price, and more.

For this, we will use:

Please Note

Any time you are loading a web page, you are making a request to a server. When you are just a human with a browser, there is not a lot of damage you can possibly do. However, when writing code to access a server, you can send thousands of requests a second. That can cost the website owner a lot of money and even shut-down their website.

With that in mind- you want to be very careful with your code and limit the number of requests you are sending to the server to avoid causing any damage.

Let’s Get Started

First, let’s go to Wolt’s website, and go to the delivery page (as we want to find information about nearby restaurants) and click on Add an address.

Now, let’s try to figure what the server’s response when searching for nearby restaurants. Open the Chrome DevTools by right-clicking anywhere on the page, click on Inspect, go to the Network tab, and press XHR. We should now have something that looks like this:

In the search box, let’s enter a street name, Allenby for example, and click Search. We now should see a few results matching our search query. Let’s select the one in Tel-Aviv Yafo, Israel.

We now can see that the Network tab in the Chrome DevTools is populated with 3 new XHR requests. Note that there might be more than 3 requests, it requires some trial and error to find exactly what we are looking for.

The XHR will contain information about the browser’s requests to the server, such as the request URL (the API’s endpoint in this case), request method (GET, POST, etc.), headers, and params. It will also contain the server’s response to that specific request.

Let’s click on the first request and then the preview tab on the left side of the Chrome DevTools. This will show us what the response from the server looks like.

We can see that the server returns a JSON response with a list of cities that has Allenby street in them. But wait- this is the response, what about the request? To understand what is the request the website sends to the server and what it looks like, we will use curl.trillworks.com that will allow us to convert cURL to a Python script easily.

Let’s right-click on the first request and click Copy > Copy as cURL and paste it on Trillworks’ tool.

After cleaning up the code a bit, we have something that looks like this:

import requestsURL = 'https://restaurant api.wolt.com/v1/google/places/autocomplete/json'headers = {
    'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.68 Safari/537.36'
}params = [{'input': 'Allenby'}]response = requests.get(URL, 
                        headers=headers,
                        params=params)
print(response.text)

First, we are setting the headers for the request which includes a dictionary with the required headers. The User-Agent header lets the server ‘think’ that we are trying to access it from an actual browser (Chrome on a Mac in this case). When it’s not there, the server might think that an automated request is being made by a bot and might return a bad response.

Then, we are setting the params which are basically the query we are sending to the server (remember when we entered ‘Allenby’ in the search box?).

Finally, we are sending a GET request to the server using the requests.get() function with the API’s endpoint, headers, and the params we have set.

This code will basically mock the same request the website is sending to the server and will allow us to get the same response the website is receiving.

From the response we got, we will pick the first item from the list of cities the server returned, it will look something like this:

{'description': 'Allenby Street, Tel Aviv-Yafo, Israel',
                  'matched_substrings': [{'length': 7, 'offset': 0}],
                  'place_id': 'EiVBbGxlbmJ5IFN0cmVldCwgVGVsIEF2aXYtWWFmbywgSXNyYWVsIi4qLAoUChIJx0FPdoNMHRURoQbkZoqtGc8SFAoSCR98OxmmTB0VEZA_lsCicvvB',
                  'reference': 'EiVBbGxlbmJ5IFN0cmVldCwgVGVsIEF2aXYtWWFmbywgSXNyYWVsIi4qLAoUChIJx0FPdoNMHRURoQbkZoqtGc8SFAoSCR98OxmmTB0VEZA_lsCicvvB',
                  'structured_formatting': {'main_text': 'Allenby Street',
                                            'main_text_matched_substrings': [{'length': 7,
                                                                              'offset': 0}],
                                            'secondary_text': 'Tel Aviv-Yafo, '
                                                              'Israel'},
                  'terms': [{'offset': 0, 'value': 'Allenby Street'},
                            {'offset': 16, 'value': 'Tel Aviv-Yafo'},
                            {'offset': 31, 'value': 'Israel'}]
                  'types': ['route', 'geocode']}

At this point, it doesn’t tell us much. We can see that Allenby street has an attribute called ‘place_id’ along with some more related attributes.

Now let’s check the second request to the server.

Same as before, we copy the cURL of the request and pasted it on TrillWorks’ tool, and now we have something that looks like this:

import requestsURL = 'https://restaurant-api.wolt.com/v1/google/geocode/json'headers = {
    'user-agent': 
    'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.68 Safari/537.36',
}params = [
    {'place_id':
  'EiVBbGxlbmJ5IFN0cmVldCwgVGVsIEF2aXYtWWFmbywgSXNyYWVsIi4qLAoUChIJx0FPdoNMHRURoQbkZoqtGc8SFAoSCR98OxmmTB0VEZA_lsCicvvB'
}
]response = requests.get(URL, headers=headers, params=params)print(response.text)

We can see that now, the website is sending another GET request to the server with the same ‘place_id’ attribute we received from the first request. This is what the response looks like:

"results": [
    {
        "address_components": [
            {
                "long_name": "Allenby Street",
                "short_name": "Allenby St",
                "types": ["route"]
            },
            {
                "long_name": "Tel Aviv-Yafo",
                "short_name": "Tel Aviv-Yafo",
                "types": ["locality", "political"]
            },
            {
                "long_name": "Tel Aviv District",
                "short_name": "Tel Aviv District",
                "types": ["administrative_area_level_1", "political"]
            },
            {
                "long_name": "Israel",
                "short_name": "IL",
                "types": ["country", "political"]
            }
        ],
        "formatted_address": "Allenby St, Tel Aviv-Yafo, Israel",
        "geometry": {
            "bounds": {
                "northeast": {
                    "lat": 32.07381530000002,
                    "lng": 34.7739591
                },
                "southwest": {
                    "lat": 32.06058009999997,
                    "lng": 34.7650244
                }
            },
            "location": {
                "lat": 32.0676462,
                "lng": 34.7712005
            },
            "location_type": "GEOMETRIC_CENTER",
            "viewport": {
                "northeast": {
                    "lat": 32.07381530000002,
                    "lng": 34.7739591
                },
                "southwest": {
                    "lat": 32.06058009999997,
                    "lng": 34.7650244
                }
            }
        },
        "place_id": "EiVBbGxlbmJ5IFN0cmVldCwgVGVsIEF2aXYtWWFmbywgSXNyYWVsIi4qLAoUChIJx0FPdoNMHRURoQbkZoqtGc8SFAoSCR98OxmmTB0VEZA_lsCicvvB",
        "types": ["route"]
    }
],
"status": "OK"

We can see that now the response has attributes such as ‘lat’ and ‘lng’ which represent the Latitude and Longitude of Allenby street in Tel-Aviv Yafo.

Moving to the third request, copying the cURL and converting it to Python, will look like this:

import requestsURL = 'https://restaurant-api.wolt.com/v1/pages/delivery'headers = {
    'user-agent':
        'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.68 Safari/537.36',
}params = [
    {'lat', '32.0676462'},
    {'lon', '34.7712005'}
]response = requests.get(URL, headers=headers, params=params)print(response.text)

Now, we can see that the lat and lon values are being passed as the parameters in our GET request to the server (note that longitude is spelled differently now: lon).

In return, we will get a JSON response that looks like this:

Expanding the sections attribute will show a huge list of restaurants that work with Wolt and delivers food to the provided street, and their information.

I picked up the first restaurant so we could see what information is available in the response:

{image: {blurhash: "jdRYf1X;TtrcLLKYPcKXJkh3hmh4",…},…}
image: {blurhash: "jdRYf1X;TtrcLLKYPcKXJkh3hmh4",…}
blurhash: "jdRYf1X;TtrcLLKYPcKXJkh3hmh4"
url: "https://prod-wolt-venue-images-cdn.wolt.com/s/sI8s6J88YvIarmXei4Tmr0XUEC5wWJVwX_2IXER1m2k/5d25876a70bfc61476fe9774/1d3bb9c41565710c313996ab39623ee9-edits/fdec11eea2a5b5afe684b07acbbce331"
variants: ["xs", "sm", "md", "frontpage"]
link: {target: "https://wolt.com/en/isr/tel-aviv/restaurant/arte-gelateria", target_sort: "default",…}
target: "https://wolt.com/en/isr/tel-aviv/restaurant/arte-gelateria"
target_sort: "default"
target_title: ""
title: ""
type: "url"
template: "venue-large"
title: "Arte Italian Ice Cream"
track_id: "venue-arte-gelateria"
venue: {address: "נחלת בנימין 11", badges: [], city: "Tel Aviv", currency: "ILS", delivers: true,…}
address: "נחלת בנימין 11"
badges: []
city: "Tel Aviv"
currency: "ILS"
delivers: true
delivery_price: "₪10.00"
estimate: 20
estimate_range: "15-25"
franchise: ""
id: "5d25876a70bfc61476fe9774"
location: [34.770119190216064, 32.06833246901664]
name: "Arte Italian Ice Cream"
online: true
price_range: 2
product_line: "restaurant"
rating: {rating: 4, score: 9.6}
short_description: "The Art of Ice-cream"
slug: "arte-gelateria"
tag: "ice cream"
tags: ["ice cream", "sweets", "dessert"]}

We now have all the publicly available information about restaurants nearby Allenby street in Tel-Aviv Yafo. We can see attributes that will tell us if the restaurant is currently online and accepting orders, if they have a delivery option, the estimated delivery time, how expensive their menu is, and more.

Recap

We are sending a GET request to the server with a street name and receive a response containing a list of cities that have the street name we passed, and their corresponding id.
Once selecting a city, we are sending another GET request to the server with the id of the selected city (Allenby in Tel-Aviv Yafo in our case).
In return, we receive a JSON response containing the Latitude and Longitude of the street.
Finally, we are sending a GET request to the server with the street’s Latitude and Longitude and getting a JSON containing all the information about restaurants nearby the street we passed.

I have written a code that wraps up everything we went through in this post.

import json
import requests
class Wolt:
    HEADERS = {
        'user-agent':
            'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.68 Safari/537.36'
    }    PLACES_URL = 'https://restaurant-api.wolt.com/v1/google/places/autocomplete/json'
    GOOGLE_GEOCODE_URL = 'https://restaurant-api.wolt.com/v1/google/geocode/json'
    DELIVERY_URL = 'https://restaurant-api.wolt.com/v1/pages/delivery'    def get_matching_cities(self, street):
        """
        :param street: a street name
        :return: a list of dictionaries containing the all the cities that has the passed street name and its id
        """
        params = {'input': street}
        response = json.loads(requests.get(self.PLACES_URL, headers=self.HEADERS, params=params).text)
        predictions = response['predictions']        return [
            {
                'street': result['description'],
                'place_id': result['place_id']} for result in predictions] if response else None    def get_lat_lon(self, city_id):
        """
        :param city_id: the ID of the city you want to get the latitude and longitude of
        :return: dictionary containing the lat and lon of the
        """
        params = {'place_id': city_id}
        response = json.loads(requests.get(self.GOOGLE_GEOCODE_URL, headers=self.HEADERS, params=params).text)
        lat_lon = response['results'][0]['geometry']['location']
        return lat_lon    def get_nearby_restaurants(self, lat, lon):
        """
        :param lat: latitude of the street
        :param lon: longitude of the street
        :return: list of dictionaries containing all the available information about nearby restaurant
        """
        params = {'lat': lat, 'lon': lon}
        response = requests.get(self.DELIVERY_URL, headers=self.HEADERS, params=params).text
        restaurants = json.loads(response)['sections'][0]['items']
        return restaurants
if __name__ == '__main__':
    wolt = Wolt()    # Get the matching streets
    cities = wolt.get_matching_cities('Allenby')    # Select the first place (Allenby, Tel-Aviv Yafo)
    # and get the latitude and longitude of it
    city = cities[0]['place_id']
    lat_lon = wolt.get_lat_lon(city)    # Pass the latitude and longitude to get all nearby restaurants
    restaurants = wolt.get_nearby_restaurants(lat_lon['lat'], lat_lon['lng'])
    print(restaurants)

The code is also available on my GitHub.

Summary

We can use the code in various ways; create a Telegram bot that notifies us whenever our favorite restaurant is available for delivery, create an app with functionality that does not exist in Wolt’s app/website, or use it for analysis or research.

Legal Note

The information brought in this article is for study purposes only and using it will be at your own risk. Although the information from the server is publicly available, some websites might not allow accessing their public data using code. Web scraping can generally be done without asking for permission from the data owner if it does not violate the website’s terms of service.