Retrieving Full Historical Data for Every Cryptocurrency on Binance & BitMex Using the Python API

A single function to read, update, save, and gather data

Peter Nistrup
Jun 6 · 3 min read

Disclaimer: I know this isn’t what I typically post but since it’s an interest of mine — just like traditional finance, machine learning, and data science — I think some of you might find this interesting as well.


I’ve recently been investigating the relationship between data science and the cryptocurrency market for a pretty long article I’m writing. During the process, I needed to retrieve price history and other data, so I decided to write a supplementary piece about how I accomplished that.

I hope you find it useful. Make sure to follow me if you enjoy this piece and want to see more!


TL;DR

To retrieve Bitcoin prices and data (1m klines):

  1. Sign-up on Binance and/or BitMex to get API access.
  2. Import the functions I’ve created for easy-of-use and add your API details.
  3. a. Call the function: get_all_binance(“BTCUSDT”, “1m”, save = True)
    b. Call the function: get_all_bitmex(“XBTUSD”, “1m”, save = True)
An example of getting 5m historical price data of EOS on BitMex using the get_all_bitmex function
An example of getting 5m historical price data of ICX on Binance using the get_all_binance function

Why Binance and BitMex?

If you’ve had any experience with the cryptocurrency market you know that the answer to this question is obvious:

Binance is the most popular exchange on the market BitMex is the largest volume exchange on the market.

And as we know, large volume equals big data and big data (usually) equals better modeling and predictions, especially for machine learning models.

Please note that you don’t have to buy any Bitcoin or any other cryptocurrency whatsoever to perform the steps in this article. The reason we need to sign up is that once signed up you’ll have access to their APIs, which is what we’re gonna be using for getting the actual data.

You can choose either or both, the functions introduced below are flexible to both exchanges.


Retrieving and Storing the Data

Our code will do the following:

  1. Check for existing saved data
  2. Find the difference between the latest saved data (if any) and the newest data
  3. Send API calls and append an existing or new DataFrame
  4. Save and/or return full DataFrame

The only thing you need to do is add your API-key and API-secret to your code, also remember to NEVER share these with anyone. If you are trading or holding cryptocurrency on either Binance or BitMex, this could result in a loss of funds.

It looks messy but it’s very functional and fast, the only limiting factor is the number of API calls allowed by BitMex and Binance each minute.

This is incredibly easy to use, simply call the function in Python with:

data = get_all_bitmex(“XBTUSD”, “5m”, save = True)

Once downloaded, the function will return the entire DataFrame for you to use as you wish. If the argument save is passed as True, it’ll save the DataFrame on your computer with the filename “[symbol]-[kline_size]-data.csv”, i.e. “XBTUSD-5m-data.csv” in the case above.

The clever thing about this function is that it will keep your data up to date at all time.

Each time the function is called it will check the latest point of data you’ve downloaded and update with any new data!

Just call the get_all_bitmex(“XBTUSD”, “5m”, save = True) each time you’re working with your XBTUSD data from BitMex to make sure it’s up to date!


But I Want Data for All the Markets

Don’t worry, this is just as easy to do! You can retrieve the data for all markets with:

Here we’re downloading all available data for Bitcoin and its relation to Ethereum and Ripple from both BitMex and Binance.

Conclusion

Thanks for reading. Hopefully, you found this useful!

Better Programming

Advice for programmers.

Peter Nistrup

Written by

DATA SCIENCE, STATISTICS & AI … Stay up to date on LinkedIn ☞ www.linkedin.com/in/peter-nistrup/ and Twitter ☞ @PeterNistrup

Better Programming

Advice for programmers.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade