Go Undercover by Scraping Cryptocurrency Market Metrics

Data-Informed Technique To Cryptocurrency Market Speculation using Python

Published in

The Crypto

10 min readApr 28, 2020

Information on cryptocurrencies like Bitcoin, Litecoin, Ethereum, are now in the main-stream media with speculation on which will be next digital gold standard. Trust is scarce in these markets. What many of the self-proclaimed experts are missing is valid data statistics to explain the foundation of their expertise. What if we could do our own cryptocurrency market analysis?

In this article, I am going to analyze the following cryptocurrencies Bitcoin, Ethereum, Litecoin. These are known to be the foundational cryptos, covering the majority of news mentions.

I plan to show you how easy it is to pull cryptocurrency market data using Python for further examination. You will learn how to use a Python code structure to extract, transform, and load data on just about every cryptocurrency market. We will also be able to configure the Python code to extract things like the cryptocurrency name, crypto symbol / ticker, price, percentage change over 24 hours, market cap data, volume over 24 hours, number of url shares, amount of reddit posts, number of tweets, galaxy score, volatility, social volume, news (amount of news mentions by coin), and close price.

Skip to ‘Cryptocurrency Market Data API’ If you know how to set up your Python Environment with Jupyter Notebook.

Requirements

Python 3
Jupyter Notebook
LunarCRUSH Account

This is not an article explaining what cryptocurrencies are, nor is this an article of me telling you which cryptos to buy, nor which will rise or fall in market price. This is a tutorial for extracting cryptocurrency markets with Python using raw data. I am hoping this article will motivate you enough to build upon the code already written, and for you to build something even better.

Don’t know which Crypto to buy? Do your own analysis. Photo by @cryptofunny

Preparing Your Data Environment

The skills that you will need are basic understanding of cryptocurrency and enough motivation to learn the basics of Python. This tutorial is not only for developers but for those who are looking to gain deep-insights into financial markets. A completed version of the code used in this write-up and with more examples can be found here.

Install Python — Anaconda

The most straight forward way to setup your environment is to use Anaconda, a dependency manager and prepackaged data science environment, which will help with installing dependencies and keeping projects separate.

To learn more about Anaconda, I would recommend reading the official installation instructions — https://docs.anaconda.com/anaconda/install/

Setup your Anaconda Project Environment

When Anaconda is installed, we need to create a new environment to keep our projects, dependencies, and environments organized.

conda create --name analyze-cryptocurrency python=3

This will create a new Anaconda environment for the project.

source activate analyze-cryptocurrency (macOS)

activate analyze-cryptocurrency (Windows)

source activate analyze-cryptocurrency (Linux)

This will activate your environment.

Next, run:

conda install numpy pandas nb_conda jupyter

This will install the required dependencies in the environment. This will most likely take few minutes to complete.

Why use environments? If you plan on developing multiple Python projects on your computer, it is helpful to keep the dependencies (software libraries and packages) separate in order to avoid conflicts. Anaconda will create a special environment directory for the dependencies for each project to keep everything organized and separated.

Interactive Jupyter Notebooks

When the environment and dependencies are set up, in your terminal, run:

jupyter notebook

This will start Jupyter Notebook, and typically opens your browser to http://localhost:8888/. If the web page is not automatically opened, copy and paste it into your browser. Next, navigate to your desired folder and create a new Python notebook.

Import Dependencies in Jupyter Notebook

With your empty Jupyter notebook open, the first thing we’ll do is import the required dependencies.

import urllib.request
import ssl
import json
import time

Cryptocurrency Market Data API

We have our Python environment set up with Jupyter Notebook and now we are ready to start extracting cryptocurrency data for our analysis. Now, we need to get API Key from LunarCRUSH — Social Listening for Crypto.

Introduction to LunarCRUSH — Social Listening For Crypto

In order to start pulling in cryptocurrency social insights, we need to head over to LunarCRUSH.com and set up an account.

In the left-hand navigation, head to Developers and select API Docs.

Generate V2 API Key

Once you are in the API Docs, generate your V2 API Key.

Retrieve Cryptocurrency Data — Set API Key

Our ‘api_key’ is where you store the LunarCRUSH API Key, which we previously generated. You will use it to make requests to the LunarCRUSH API.

The idea of this Python code is to pull data from LunarCRUSH and display different metrics in our notebook.

Let’s add our LunarCRUSH API Keys to the code. Simply add:

api_key = "API_KEY_HERE"

We will now be authenticated when we call LunarCRUSH API. It is time to decide which cryptocurrencies you would like to pull into your project. Use coin_list to store the different crypto symbols. For instance, ‘LTC’ is Litecoin, ‘ETH’ is Ethereum, and ‘BTC’ is Bitcoin.

# Allows adding as many coins as desired
coin_list = [
    "BTC",
    "ETH",
    "LTC"
]
coins = ','.join(coin_list)

List comprehensions provide a concise way to create lists. Common applications are to make new lists where each element is the result of some operations applied to each member of another sequence or iterable, or to create a subsequence of those elements that satisfy a certain condition. — Python Data Structures Docs

Now you can map which values you would like to pull from LunarCRUSH API. As the program becomes more complex, this should be written in a more robust manner.

map = [
    {"name":""},
    {"symbol": ""},
    {"price": " Price: "},
    {"percent_change_24h": " - 24 Hour Percent Change: "},
]def final_render(asset_coin, value, key, asset):
    if key == 'symbol':
        asset_coin += " (" + asset[key] + ")"
    elif key == 'percent_change_24h':
        asset_coin += value + str(asset[key]) + "%"
    else:
        asset_coin += value + str(asset[key])
    return asset_coin

Within the map, we will be pulling the Cryptocurrency Name, Symbol, Price, and 24 Hour Percentage Change.

Now, iterate over each of the fields from LunarCRUSH which gets the value from LunarCRUSH and renders it with the field name.

def main():url = "https://api.lunarcrush.com/v2?data=assets&key=" + api_key + "&symbol=" + coins
    assets = json.loads(urllib.request.urlopen(url).read())for asset in assets['data']:
        asset_coin = ""
        for field in map:
            key = list(field.keys())[0]
            value = list(field.values())[0]
            asset_coin = final_render(asset_coin, value, key, asset)
        print(asset_coin)
        print(len(asset_coin))

If you notice the bold part of the code, assets, is a REST API Endpoint. I will expand on this concept below, but keep in mind this is not the only endpoint we can use. We have the option to customize our REST API Endpoint in order to extract more cryptocurrency market metrics.

Automating Our Code To Run Every 30 Minutes:

# Runs main() every 30 minutes
while True:
    main()
    time.sleep(1800)

Running Our Code

LunarCRUSH REST API ENDPOINTS

The LunarCRUSH API lets you access most of the data available on our public website. Our front-end application uses the same API to get and display all of the data on our site…
You can use our data for free under a Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0) but all we ask is that you credit us with a link and our logo if you use our data on your website or app. Paid plans are available for commercial use without attribution. — LUNARCRUSH API V2 DOCS

Assets: Details, overall metrics, and time series metrics for one or multiple assets. We refer to all supported cryptocurrencies as assets.
Market: Summary information for all supported assets including 5 recent time series values for some metrics.
Global: Overall aggregated metrics for all supported assets.
Feeds: Social posts, news, and shared links for one or multiple coins.
Alerts: Historical timestamped records of changes in metrics that are notable beyond typical trends including periodical highs, lows, and sudden increases or decreases.
Influencers: List of social accounts that have the most influence on different assets based on number of followers, engagements, and volume of posts.
Influencer: Individual influencer details including actual posts.

Analyzing Average Cryptocurrency Sentiment

{"average_sentiment": " Average Sentiment: "},

By simply adding the above code to our map, we can pull Average Cryptocurrency Sentiment from LunarCRUSH. Adding it to our previous code, it will look like this.

map = [
    {"name":""},
    {"symbol": ""},
    {"price": " Price: "},
    {"percent_change_24h": " - 24 Hour Percent Change: "},
    {"average_sentiment": " Average Sentiment: "},
]

Result:

Bitcoin (BTC) Price: 6799.824241 - 24 Hour Percent Change: -0.84% Average Sentiment: 3.3Ethereum (ETH) Price: 143.297439 - 24 Hour Percent Change: -0.32% Average Sentiment: 3.5Litecoin (LTC) Price: 40.528366 - 24 Hour Percent Change: 0% Average Sentiment: 3.3

Additional Cryptocurrency Market Metrics

It is possible to configure the Python code to extract things like the cryptocurrency name, symbol, price, percentage change over 24 hours, market cap data, volume over 24 hours, url shares, reddit posts, tweets, galaxy score (proprietorial measure provided by LunarCRUSH), volatility, social volume, news mentions (amount of news mentions), and close price. Let’s add the configurations to our map:

map = [
 {“name”:””},
 {“symbol”: “”},
 {“price”: “ Price: “},
 {“percent_change_24h”: “ — 24 Hour Percent Change: “},
 {“market_cap”: “ Market Cap: “},
 {“volume_24h”: “ 24 Hour Volume: “},
 {“url_shares”: “ URL Shares: “},
 {“reddit_posts”: “ Reddit Posts: “},
 {“tweets”: “ Tweets: “},
 {“galaxy_score”: “ Galaxy Score: “},
 {“volatility”: “ Volatility: “},
 {“social_volume”: “ Social Volume: “},
 {“news”: “ News: “},
 {“close”: “ Close: “},
]

Result:

Bitcoin (BTC) Price: 6783.203105 - 24 Hour Percent Change: -1.52% Market Cap: 123875196896 24 Hour Volume: 24893565282.51 URL Shares: 690 Reddit Posts: 16 Tweets: 5244 Galaxy Score: 58.5 Volatility: 0.00713268 Social Volume: 5744 News: 2 Close: 6783.203105Ethereum (ETH) Price: 142.949595 - 24 Hour Percent Change: -1.18% Market Cap: 15765467834 24 Hour Volume: 9016572873.14 URL Shares: 212 Reddit Posts: 3 Tweets: 464 Galaxy Score: 61 Volatility: 0.00844722 Social Volume: 646 News: 7 Close: 142.949595Litecoin (LTC) Price: 40.308414 - 24 Hour Percent Change: -1.03% Market Cap: 2592376506 24 Hour Volume: 2267773770.18 URL Shares: 40 Reddit Posts: 1 Tweets: 250 Galaxy Score: 60.5 Volatility: 0.00450274 Social Volume: 285 News: 1 Close: 40.308414

Full Code

import urllib.request
import ssl
import json
import time
import tweepyssl._create_default_https_context = ssl._create_unverified_contextapi_key = "API_KEY_HERE"# Allows adding as many coins as desired
coin_list = [
    "LTC",
    "ETH",
    "BTC"
]
coins = ','.join(coin_list)map = [
    {"name":""},
    {"symbol": ""},
    {"price": " Price: "},
    {"percent_change_24h": " - 24 Hour Percent Change: "},
    {"market_cap": " Market Cap: "},
    {"volume_24h": " 24 Hour Volume: "},
    {"url_shares": " URL Shares: "},
    {"reddit_posts": " Reddit Posts: "},
    {"tweets": " Tweets: "},
    {"galaxy_score": " Galaxy Score: "},
    {"volatility": " Volatility: "},
    {"social_volume": " Social Volume: "},
    {"news": " News: "},
    {"close": " Close: "},
]def final_render(asset_coin, value, key, asset):
    
    if key == 'symbol':
        asset_coin += " (" + asset[key] + ")"
    elif key == 'percent_change_24h':
        asset_coin += value + str(asset[key]) + "%"
    else:
        asset_coin += value + str(asset[key])
    return asset_coindef main():url = "https://api.lunarcrush.com/v2?data=assets&key=" + api_key + "&symbol=" + coins
    assets = json.loads(urllib.request.urlopen(url).read())for asset in assets['data']:
        asset_coin = ""
        for field in map:
            key = list(field.keys())[0]
            value = list(field.values())[0]
            asset_coin = final_render(asset_coin, value, key, asset)
        print(asset_coin)
        print(len(asset_coin))# Runs main() every 30 minutes
while True:
    main()
    time.sleep(1800)

Pandas.Dataframe

We can turn our code into Pandas Dataframe in order to export to CSV or plot the points. Pandas.DataFrame gives us the ability of high-performance array operation. Similar to lists, pandas.DataFrame makes it more simple to structure data and allows variable data types. Tabular data with row index and column index allows flexible iterations to values using integer position or index.

def create_coins_df(assets):
    global coins_df
    
    for n in range(len(coin_list)):
        data = assets['data'][n]
        required_data = [{key: data[key] for key in fields}]
        
        coins_df = coins_df.append(required_data, ignore_index = True)
        
        coins_df['time'] = pandas.to_datetime(coins_df['time'], unit='s')
        
    return coins_dfcreate_coins_df(main())

Full code & Result:

Dataframe Code & Output

Full code of the Python Jupyter Notebook is available here

nirholas/ANALYZING-CRYPTOCURRENCY-MARKETS-USING-PYTHON-REST-API

💰 DATA-DRIVEN APPROACH TO CRYPTOCURRENCY SPECULATION WITH PYTHON, LunarCRUSH, REST API …

github.com

Configuration Options

We can also add additional configurations. I will not cover these configuration options in this tutorial; however, look out for my next article on visualizing cryptocurrency markets with Python.

market_cap (Market Cap)
galaxy_score (Galaxy Score)
price_score (Price Score)
average_sentiment (Average Sentiment)
social_impact_score (Social Impact Score)
market_cap (Market Cap)
galaxy_score (Galaxy Score)
price_score (Price Score)
social_impact_score (Social Impact Score)
correlation_rank (Correlation Rank)
volatility (Volatility)
social_score (Social Volume)
social_volume (Social Volume)
twitter_volume (Twitter Volume)
reddit_volume (Reddit Volume)
news_volume (News Volume)
search_volume (Search Volume)
spam_volume (Spam Volume)
bullish_sentiment (Bullish Sentiment)
bearish_sentiment (Bearish Sentiment)

Final Thoughts

This article is just the introduction into using Python for Cryptocurrency Analysis. With what we coded today, there are tons of ideas to expand upon using our code.

Here are some ideas:

Transform your Data into table, visualize the data, find correlations.
Demonstrated correlation with Bitcoin prices compared against alt-coins.
Train a predictive machine learning model on the data to predict tomorrow’s prices. (“Correlation does not imply causation”)
Building a Twitter Bot to Automate Posting Cryptocurrency News with Python
Apply your analysis to create an automated “Cryptocurrency Trading Bot” on a trading site such as Binance or Coinbase with trading APIs. But remember: if not done correctly, a trading bot is an easy way to lose your money quickly.

Knowledge is Power! The decentralized nature of cryptocurrency makes for more free and democratic ways of accessing networks. Share your knowledge, open source your projects , participate in a community (any community!), and maybe just maybe publish a blog post about it.

Now that you have the skills to conduct your own cryptocurrency analysis, pay attention to the speculative cryptocurrency articles pushing social media sentiment in their favor, and always validate data before you trust an opinion.

None of this should be treated as investment advice and same goes for the code. Whichever investments you pursue are purely at your own discretion.

Thanks For Reading

Constructive criticism and feedback is welcomed. Nicholas Resendez can be reached on Instagram @nirholas, on Medium, and Twitter @Bothersome for updates on new articles.