Real-time REST API endpoints
There are three major ways to obtain data for AI modeling:
- Data sets from experimental collections. Kaggle datasets, Google Cloud Platform BigQuery datasets, Tensorflow datasets and many other AI modeling companies and/or research institutions (ie: IEEE DataPort) have downloadable datasets that are collected in a scientific manner.
- Webscraping websites. Web driver programs like Selenium, Puppeteer, and Playwright can be used to fetch data from “dynamic” websites/webapps; dynamic implies that the data is gathered from the deployed html form document that the user is using in real-time. And, fetching programs like fetch, requests, Axios, jQuery, XMLHttpRequest, etc are used to fetch data from static websites/webapps. XML search/extraction programs like Cheerio, BeautifulSoup, etc are used to find desired website data for data collection.
- Real-time REST API endpoints. Data collected by sensors, like temperature, position/velocity/acceleration, humidity, etc, referred to as the field of Internet of Things (IoT). The sensors are connected to a computer with an internet connection, where the data can be accessible at an Internet address and port number using the REST API protocol. Data can be obtained by anyone who sends a REST API GET request to the HTTPS URL and/or specified port number, depending on the data server configuration.
In this post, I list four major usages for REST API endpoint data: [0] Fitness/Exercise, [1] Weather/Pollution, [2] Professional Sports Monitoring, [3] News and Finance/Stock Market. REST API endpoints are extremely useful because one can obtain past, current, and future data for the respective measurement by just performing one command (REST API GET).
[0] Fitness/Exercise
Exercise applications need client/user consent to obtain data that is generated by the user’s personal device (ie: watch, phone). The user’s data is stored on the company’s server/computer (ie: Google, Fitbit, MiBand, etc), but the data can not be accessed with REST API until the user gives consent usually with the OAuth2 protocol. Therefore, when using these REST API endpoints one needs an API_KEY, which is a passcode generated to show the user gave consent for the data to be obtained.
Google Fit API
The Google Fit API instruction page (https://developers.google.com/fit/rest) outlines all the steps for obtaining fitness data (ie: footsteps, position) or a user. However, for initial users, the steps maybe difficult to follow, therefore below is a rough outline/global view of the steps to complete to connect to the API using bash.
# [0] Go to https://console.cloud.google.com/
# OR
# login using your PC
gcloud auth login
# --------------------------------
# [1] Create a project
export PROJECT_ID=$(echo "project-name-0")
gcloud projects create $PROJECT_ID
# --------------------------------
# [2] Enable project billing
# Obtain ACCOUNT_ID
gcloud alpha billing accounts list
# The ACCOUNT_ID will be listed
# Assign the ACCOUNT_ID to a variable
ACCOUNT_ID="XXXXXXXXXXX"
# Enable project billing
gcloud alpha billing accounts projects link $PROJECT_ID --billing-account=$ACCOUNT_ID
# --------------------------------
# [3] Enable Services to have access to your/client account information
# To read about each of the services: Go to https://console.cloud.google.com/ - APIs & Services - Library
# Fitness API
gcloud services enable fitness.googleapis.com
# Cloud Billing API
gcloud services enable cloudbilling.googleapis.com
# Google Workspace - Google Workspace Add-ons API
gcloud services enable gsuiteaddons.googleapis.com
# Identity and Access Management (IAM) API
gcloud services enable iam.googleapis.com
# Cloud Resource Manager API
gcloud services enable cloudresourcemanager.googleapis.com
# --------------------------------
# [4] Create an OAuth2 client
# List BRANDS to obtain the BRAND_ID
gcloud services enable gcloud alpha iap oauth-brands list
# The output will give the PROJECT_NUMBER and BRAND_ID
# Assign the PROJECT_NUMBER and BRAND_ID to variables
PROJECT_NUMBER="XXXXXXXXXXX"
BRAND_ID="XXXXXXXXXXX"
# Create an OAuth2 client
# https://cloud.google.com/sdk/gcloud/reference/iap/oauth-clients/create
DISPLAY_NAME="OAuth2_KEY"
gcloud iap oauth-clients create projects/$PROJECT_NUMBER/brands/$BRAND_ID --display_name=$DISPLAY_NAME
# Under secret the OAuth2_KEY will given
# Assign the OAuth2_KEY to a variable
ACCESS_TOKEN="OAuth2_KEY"
# --------------------------------
# [5] Call the REST API GET command to view available dataSources for
# your account (meaning your device (ie: phone, watch) that is connected to Google).
curl \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $ACCESS_TOKEN" \
https://www.googleapis.com/fitness/v1/users/me/dataSourcesStep 5, initially calling the dataSources is important because data names for each device are different, so it is necessary to find your devices first. Then, you can go to https://developers.google.com/fit/rest/v1/data-sources and use more specific REST API calls, like requesting a specific datasource, posting data, etc.
Smart Watches (Fitbit API/ Miband API)
The majority of smart watches make it mandatory for users to create an account at the manufacturer’s website to unlock the watch (ie: make the watch function). When you make an account at the manufacturer’s site, you give consent for the manufacture to manage/store your data and/or allow the data to be available for REST API usage.
One can use the manufacture API or download Google Play Apps that allow for sensor/watch data to be downloaded to the phone. For example, if one did not want to use the Miband API, one could use https://play.google.com/store/apps/details?id=com.mc.miband1&hl=en (Notify for Mi Band) to download the data to the phone.
[1] Weather/Pollution
Open-meteo API
This endpoint API is the only major weather API that DOES NOT require an account, nor an API_KEY, with it’s basic features! It is freely open like OpenStreetmap.org for latitude and longitude information. The open-meteo API requires that one enter latitude and longitude for the requested weather information, thus I recommend using the openstreetmap API to obtain latitude and longitude information without any hassle of a key or making an account. Below is the REST API to obtain latitude and longitude information using the openstreetmap API.
async function GET_LATITUDE_LONGITUDE(LOCATION) {
// LOCATION = "City, Country"
// [Step 0] Get latitude and longitude from typing in your City, Country
// https://maps.googleapis.com/maps/api/geocode/json?address=${address}&key=${GCP_API_KEY}
// OR
// https://nominatim.org/release-docs/latest/api/Search/
var url = `https://nominatim.openstreetmap.org/search?q=${LOCATION}&format=json`;
var method = 'GET'; // method: 'GET', 'POST'
var data = []; // data: required data format for url address
var headers = {};
var return_type = 'json'; // return_type: 'json', 'blob', 'blob_file_pdf', 'arrayBuffer', 'text'
var out = await fetch_CORS(url, method, data, headers, return_type);
console.log('out: ', out);
// Parse for correct location: choose the first location because
// it will be most precise
return [Number(out[0].lat), Number(out[0].lon)];
}After obtaining the latitude and longitude information, send a GET REST API request to open-meteo.com.
I find that open-meteo.com API is the best endpoint for weather and other data because it can give past, current, and future weather information with in a span of 80 years per hour, without commitment to creating an account! This is wonderful because one could do timeseries prediction analysis, without data collection (ie: storing the data in a database/file); the API already does the data collection for you and you simply just request the data you want. In addition to providing historical, current, and future weather information, it can provide climate change, marine forecast, air quality (European Air Quality Index, United States Air Quality Index), geocoding, elevation, and flood information. The site is extremely organized and they even have a GitHub page (https://github.com/open-meteo/open-meteo), it is a great choice for infrequent to moderate API users.
In the example below, I request past temperature data for the previous 2 days such that I can perform timeseries prediction/forecasting with a deep learning model in the browser. I then also requested timeseries prediction data for the next 24-hours, in order to compare my model predictions with the open-meteo.com predictions. Predictions could also be used directly to create weather applications!
async function GET_weather_timeseries_data(latitude, longitude) {
// [Step 0] Get weather data
// https://open-meteo.com/en/docs
// The API endpoint /v1/forecast accepts a geographical coordinate, a list of weather variables and responds with a JSON hourly weather forecast for 7 days. Time always starts at 0:00 today and contains 168 hours. If &forecast_days=16 is set, up to 16 days of forecast can be returned.
var method = 'GET'; // method: 'GET', 'POST'
var data = []; // data: required data format for url address
var headers = {};
var return_type = 'json'; // return_type: 'json', 'blob', 'blob_file_pdf', 'arrayBuffer', 'text'
// --------------------------------
// Obtain weather [temperature] data from the past (Prediction)
// Return temperature every hour for the [previous 2 days OR more], such that the next day can be predicted
url = `https://api.open-meteo.com/v1/forecast?latitude=${latitude}&longitude=${longitude}&hourly=temperature_2m&past_days=10&forecast_days=0`;
var open_meteo_past_data = await fetch_CORS(url, method, data, headers, return_type);
console.log('open_meteo_past_data: ', open_meteo_past_data);
// --------------------------------
// Obtain weather [temperature] predictions by open-meteo (Actual)
// Return temperature every hour for the next day
url = `https://api.open-meteo.com/v1/forecast?latitude=${latitude}&longitude=${longitude}&hourly=temperature_2m&past_days=0&forecast_days=1`;
var open_meteo_predictions = await fetch_CORS(url, method, data, headers, return_type);
console.log('open_meteo_predictions: ', open_meteo_predictions);
return [open_meteo_past_data.hourly, open_meteo_predictions.hourly];
}
OpenWeatherMap API
This weather API requires that one to create an account and pay for API information, https://openweathermap.org/api. The available API requests are similar to open-meteo.com, however making a paid account is mandatory. In addition to historical, current, and future weather information, this API provides information for fire weather, air pollution, geocoding, uv index,
WeatherAPI
Similar to OpenWeatherMap API, WeatherAPI (https://www.weatherapi.com/api.aspx) requires that API user make an account to use the API. The WeatherAPI also has APIs for Astronomy, Sports API, Air Quality Data, Solar Irradiance, Evapotranspiration, and Marine Weather and Tide Data.
IQAir AirVisual
Go to the IQAir AirVisual website (https://www.iqair.com/), select your country. Create a user account to obtain a free community account, where you can obtain an API_KEY to use the REST API.
The REST API commands for a community account only allows one to obtain ONE DATAPOINT per GET REST API call. So to use this REST API, it is necessary to store the data values in a database or file, if the goal is to perform timeseries prediction or create a dataset.
I was interested in pollution and weather data, so I used the following REST API instructions (https://api-docs.iqair.com/?version=latest).
I put my API_KEY into GitHub Secrets. Next, I created a GitHub Actions workflow that takes the API_KEY from GitHub Secrets, encrypts it, and then stores it in a file located at .github/.env. I call a GitHub REST API library that I made on npm, to GET the encrypted key from .github/.env and decrypt it. The function call run_env_decode_desalt is from the library (library_to_run_GitHub_Actions.js) [1].
async function GET_weather_pollution_data_timeseries_data(latitude, longitude) {
// [Step 0] Get nearest city data (GPS coodinates)
// https://api-docs.iqair.com/?version=latest
// Decode the encrypted key from the .github/.env file using library_to_run_GitHub_Actions.js
var RepoAobj = {};
RepoAobj.repoOwner = 'CodeSolutions2';
RepoAobj.foldername = '.github'; // foldername in RepoB
RepoAobj.filename = '.env'; // filename to create in RepoB, in foldernameobj.auth = obj.env_text; // Initialize value
RepoAobj.repoB_name = 'timeseries_analysis';
var API_KEY = await module.run_env_decode_desalt(RepoAobj);
// Call the endpoint
var method = 'GET'; // method: 'GET', 'POST'
var data = []; // data: required data format for url address
var headers = {};
var return_type = 'json'; // return_type: 'json', 'blob', 'blob_file_pdf', 'arrayBuffer', 'text'
// --------------------------------
url = `https://api.airvisual.com/v2/nearest_city?lat=${latitude}&lon=${longitude}&key=${API_KEY}`;
var current_obj = await jQuery_CORS(url, method, data, headers, return_type);
console.log('current_obj: ', current_obj);
// pollution: air_quality_index_aqicn, air_quality_index_aqius
var pollution_datapoint = [current_obj.data.current.pollution.aqicn, current_obj.data.current.pollution.aqius];
// temperature: humidity, air_pressure, temperature, wind_direction, wind_speed
var temperature_datapoint = [current_obj.data.current.weather.hu, current_obj.data.current.weather.pr, current_obj.data.current.weather.tp, current_obj.data.current.weather.wd, current_obj.data.current.weather.ws];
// --------------------------------
return [pollution_datapoint, temperature_datapoint];
}The endpoint outputs the following:
{“status”:”success”,”data”:{“city”:”YOUR_CITY”,”state”:”YOUR_STATE”,”country”:”YOUR_COUNTRY”,”location”:{“type”:”Point”,”coordinates”:[longitude,latitude]},”current”:{“pollution”:{“ts”:”2024–09–16T13:00:00.000Z”,”aqius”:40,”mainus”:”p2",”aqicn”:29,”maincn”:”o3"},”weather”:{“ts”:”2024–09–16T14:00:00.000Z”,”tp”:19,”pr”:1018,”hu”:47,”ws”:9.77,”wd”:350,”ic”:”04d”}}}}
The abbreviations for the pollution data are:
- aqicn: Air Quality Index (AQI) for China
- aqius: Air Quality Index (AQI) for the United States
- maincn: Main pollutant for China
- mainus: Main pollutant for the United States
The abbreviations for the weather data are:
- hu: Humidity
- ic: Air quality index
- pr: Air pressure
- tp: Temperature
- ts: Time stamp
- wd: Wind direction
- ws: Wind speed
[2] Professional Sports Monitoring
Sportradar API
Sports statistics and information can be obtained by REST API using https://developer.sportradar.com/getting-started/docs/make-your-first-call. It is necessary to make an account, and using the API is not free; they give a 30-day free trial usage.
[3] News and Finance/Stock Market
NewsAPI
NewsAPI (https://newsapi.org/docs/get-started#search) uses query parameters in their GET requests, which allow API users to fetch any type of news data, including specific sources (sources), topics (q), locations (country), time periods (from), and impact (sortBy). The API requires an API_KEY, so one must register to the site to obtain a key before they can use the API.
Other APIs
Other popular new APIs include Bing News Search API and The New York Times API. For stock market data, the following APIs are popular: Alpha Vantage, IEX Cloud, and Yahoo Finance API.
Subfunctions (fetch functions)
async function fetch_CORS(url, method, data, headers, return_type) {
// url: url address
// method: 'GET', 'POST'
// data: required data format for url address
var w_or_wo_proxyhandler = 'w_proxyhandler'; // w_or_wo_proxyhandler: 'w_proxyhandler', 'wo_proxyhandler'
// return_type: json, blob, arrayBuffer, text
// --------------------------------
// Determine request method
var headers_final = {
"Content-Type": "application/json",
"Referer": url,
"Origin": "https://codesolutions2.github.io",
"Connection": "keep-alive",
//"Sec-Fetch-Dest": "empty",
//"Sec-Fetch-Mode": "cors",
//"Sec-Fetch-Site": "cross-site",
//"Sec-Fetch-User": "?1",
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64; rv:120.0) Gecko/20100101 Firefox/120.0",
"Access-Control-Allow-Origin": "*",
//'Access-Control-Allow-Credentials': false,
//"Access-Control-Request-Method": method,
//"Access-Control-Request-Headers": 'content-type,x-pingother',
//'Cache-Control': 'max-age=0',
};
headers_final = Object.assign({}, headers_final, headers);
var options = {
method : method,
mode: 'cors',
headers: new Headers(headers_final),
cache: "no-cache",
// crossorigin: 'anonymous',
crossorigin: '*',
redirect: "follow"
};
if (method == 'POST') {
options.body = JSON.stringify(data);
}
// --------------------------------
// Determine the type of output to return
if (return_type == 'json') {
if (w_or_wo_proxyhandler == 'w_proxyhandler') {
const proxy3 = await define_proxy_handler();
return await proxy3(url, options)
.then(response => response.json())
.then(async function(data) { return data; })
.catch(error => console.error("error: ", error));
} else {
return await fetch(url, options)
.then(response => response.json())
.then(async function(data) { return data; })
.catch(error => console.error("error: ", error));
}
} else if (return_type == 'blob') {
if (w_or_wo_proxyhandler == 'w_proxyhandler') {
const proxy3 = await define_proxy_handler();
return await proxy3(url, options)
.then(response => response.blob())
.then(async function(blob_object) { return blob_object; })
.catch(error => console.error("error: ", error));
} else {
return await fetch(url, options)
.then(response => response.blob())
.then(async function(blob_object) { return blob_object; })
.catch(error => console.error("error: ", error));
}
} else if (return_type == 'arrayBuffer') {
if (w_or_wo_proxyhandler == 'w_proxyhandler') {
const proxy3 = await define_proxy_handler();
return await proxy3(url, options)
.then(response => response.arrayBuffer())
.then(async function(arraybuffer) { return arraybuffer; })
.catch(error => console.error("error: ", error));
} else {
return await fetch(url, options)
.then(response => response.arrayBuffer())
.then(async function(arraybuffer) { return arraybuffer; })
.catch(error => console.error("error: ", error));
}
} else if (return_type == 'text') {
if (w_or_wo_proxyhandler == 'w_proxyhandler') {
const proxy3 = await define_proxy_handler();
return await proxy3(url, options)
.then(res => res.text())
.then(str_data => { return str_data; })
.catch(error => console.error("error: ", error));
} else {
return await fetch(url, options)
.then(res => res.text())
.then(str_data => { return str_data; })
.catch(error => console.error("error: ", error));
}
} else if (return_type == 'html') {
if (w_or_wo_proxyhandler == 'w_proxyhandler') {
const proxy3 = await define_proxy_handler();
return await proxy3(url, options)
.then(res => res.text())
.then(str_data => { return str_data; })
.catch(error => console.error("error: ", error));
} else {
return await fetch(url, options)
.then(res => res.text())
.then(str_data => { return str_data; })
.catch(error => console.error("error: ", error));
}
}
}
// -----------------------------------------------
async function define_proxy_handler() {
// Parameters to influence Response Header
// The `handler` object defines a `apply` method that intercepts the `fetch` function calls and adds the "Access-Control-Allow-Origin" header with a value of "*" to the request options. The `new Proxy(fetch, handler)` creates a proxied version of the `fetch` function that applies the defined handler.
// https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Access-Control-Allow-Headers
const handler = {
apply: function(target, thisArg, args) {
// One needs to re-define the input arguments of the handler because a constant variable is used as a function (ie: await proxy3(url, options))
const [url, options] = args;
if (options && options.headers) {
options.headers['Access-Control-Allow-Origin'] = '*';
options.headers['Access-Control-Allow-Credentials'] = false;
options.headers['Access-Control-Allow-Methods'] = 'GET, POST, PUT, DELETE';
// options.headers['Access-Control-Allow-Headers'] = 'Origin, X-Requested-With, Content-Type, Accept';
options.headers['Access-Control-Allow-Headers'] = 'Content-Type, Authorization';
options.headers['Access-Control-Max-Age'] = 0;
} else {
args[1] = {
headers: {
'Access-Control-Allow-Origin': '*',
'Access-Control-Allow-Credentials': false,
'Access-Control-Allow-Methods': 'GET, POST, PUT, DELETE',
'Access-Control-Allow-Headers': 'Content-Type, Authorization',
'Access-Control-Max-Age': 0
}
};
}
// console.log('args: ', args);
return Reflect.apply(target, thisArg, args);
}
};
// Proxy sturcture works!
// Calls the function WITHOUT input variables to assign a proxy definition to a constant variable.
// The constant variable is called as a function, where it resolves promises.
// Call the constant variable proxy3 as a function to resolve the promise correctly
return await new Proxy(fetch, handler);
}
// -----------------------------------------------
async function jQuery_CORS(url, method, data, headers, return_type) {
// https://api.jquery.com/jQuery.ajax/
// url: url address
// method: 'GET', 'POST', 'PUT'
// data: required data format for url address
// return_type: json, blob, arrayBuffer, text
var headers_final = {
"Accept": "application/json",
"Referer": url,
"Origin": "https://codesolutions2.github.io",
"Connection": "keep-alive",
//"Sec-Fetch-Dest": "empty",
//"Sec-Fetch-Mode": "cors",
//"Sec-Fetch-Site": "cross-site",
//"Sec-Fetch-User": "?1",
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64; rv:120.0) Gecko/20100101 Firefox/120.0",
"Access-Control-Allow-Origin": "*",
//'Access-Control-Allow-Credentials': false,
//"Access-Control-Request-Method": method,
//"Access-Control-Request-Headers": 'content-type,x-pingother',
//'Cache-Control': 'max-age=0',
};
headers_final = Object.assign({}, headers_final, headers);
var settings = {
// url: url,
type : method,
async: true,
crossDomain: true,
beforeSend: function(xhr) {xhr.withCredentials = true;},
// headers: new Headers(headers_final),
success: function(response) { console.log('Success'); },
error: function(xhr, status, error) { console.log('Error:', error); }
};
// --------------------------------
if (method == 'POST') {
settings.body = JSON.stringify(data);
}
// --------------------------------
// Determine the type of output to return
if (return_type == 'json') {
settings.xhrFields = {responseType: 'json'};
settings.dataType = 'json'; // data type response format: 'xml', html, script, json, jsonp, text
return $.ajax(url, [,settings])
.done(function(response) { return response; });
} else if (return_type == 'blob') {
settings.xhrFields = {responseType: 'blob'};
settings.dataType = 'binary'; // data type response format: 'xml', html, script, json, jsonp, text, binary
return $.ajax(url, [,settings])
.done(function(response) { return response; });
} else if (return_type == 'arrayBuffer') {
settings.xhrFields = {responseType: 'arrayBuffer'};
settings.dataType = 'binary'; // data type response format: 'xml', html, script, json, jsonp, text, binary
return $.ajax(url, [,settings])
.done(function(response) { return response; });
} else if (return_type == 'text') {
settings.xhrFields = {responseType: 'text'};
settings.dataType = 'text'; // data type response format: 'xml', html, script, json, jsonp, text, binary
return $.ajax(url, [,settings])
.done(function(response) { return response; });
} else if (return_type == 'html') {
settings.xhrFields = {responseType: 'text'};
settings.dataType = 'html'; // data type response format: 'xml', html, script, json, jsonp, text, binary
return $.ajax(url, [,settings])
.done(function(response) { return response; });
}
// --------------------------------
}Happy Practicing! 👋
💻 GitHub | 🔔 Subscribe to Practicing Datscy @ Medium | 🌷 Practicing Datscy @ Dev.to
References
- Most updated version of library_to_run_GitHub_Actions.js: https://github.com/CodeSolutions2/library_to_run_GitHub_Actions
