Twitter API: Cursoring
The Twitter REST API utilizes a technique called ‘cursoring’ to paginate large result sets. Cursoring separates results into pages (the size of which are defined by the count request parameter) and provides a means to move backwards and forwards through these pages.
The next_cursor is the cursor that you should send to the endpoint to receive the next batch of responses, and the previous_cursor is the cursor that you should send to receive the previous batch. You will know that you have requested the last available page of results when the API responds with a next_cursor = 0.
The pseudocode to iterate over all responses from a cursored endpoint is:
cursor = -1
api_path = "https://api.twitter.com/1.1/endpoint.json?
url_with_cursor = api_path + "&cursor=" + cursor
responses = perform_http_get_request_for_url(url_with_cursor)
cursor = responses['next_cursor']
} while (cursor != 0)
Let’s see a real-life example of cursors in action. Consider the common scenario where we want to obtain the list of IDs who follow a user who has a large amount of followers. Instead of returning all IDs in one response set, the endpoint will instead return pages of results.
We now have a means to move forwards through our dataset, via the next_cursor. To get the next batch of at least 5,000 results, perform the same request, but set the cursor to the value of the next_cursor.
Huzzah. The next_cursor is now 0, which indicates that there are no more remaining pages. We’ve now completed iterating over the accounts’ followers.