Streaming Stocks with Python, Kafka, Flink and NiFi

FLaNK-Py-Stocks

finnhub stocks

Apache Flink, Apache NiFi, Apache Kafka (FLaNK Stack)

Python App

Python application receives websocket stream of JSON arrays and sends individual JSON messages with a JSON schema.

Raw Data

{"data":[{"c":["1","8","24","12"],"p":122.1,"s":"TSLA","t":1672348887195,"v":1},{"c":["1","8","24","12"],"p":122.09,"s":"TSLA","t":1672348887196,"v":4},{"c":["1","8","24","12"],"p":122.09,"s":"TSLA","t":1672348887196,"v":10},{"c":["1","8","24","12"],"p":122.1,"s":"TSLA","t":1672348887196,"v":1},{"c":["1","8","24","12"],"p":122.1,"s":"TSLA","t":1672348887196,"v":2},{"c":["1","8","24","12"],"p":122.1,"s":"TSLA","t":1672348887196,"v":10},{"c":["1","8","24","12"],"p":122.1,"s":"TSLA","t":1672348887198,"v":79},{"c":["1","24","12"],"p":129.58,"s":"AAPL","t":1672348887666,"v":1},{"c":["1","24","12"],"p":129.575,"s":"AAPL","t":1672348887785,"v":1}],"type":"trade"}

{"c":["1","8","24","12"],"p":122.1,"s":"TSLA","t":1672348887195,"v":1}

Data Description

data
List of trades or price updates.

s
Symbol.
p
Last price.
t
UNIX milliseconds timestamp.
v
Volume.
c
List of trade conditions. A comprehensive list of trade conditions code can be found here

Schema

{
 "type": "record",
 "name": "finnhubstocks",
 "fields": [
  {
   "name": "symbol",
   "type": [
    "null",
    "string"
   ]
  },
  {
   "name": "uuid",
   "type": [
    "null",
    "string"
   ]
  },
  {
   "name": "tradeconditions",
   "type": [
    "null",
    "string"
   ]
  },
  {
   "name": "volume",
   "type": [
    "double"
   ]
  },
  {
   "name": "price",
   "type": [
    "double"
   ]
  },
  {
   "name": "ts",
   "type": [
    "long"
   ],
   "default": 1,
   "logicalType": "timestamp-millis"
  },
  {
   "name": "currentts",
   "type": [
    "long"
   ],
   "default": 1,
   "logicalType": "timestamp-millis"
  }
 ]
}

References

• https://finnhub.io/docs/api/authentication
• https://finnhub.io/docs/api/websocket-trades
• https://finnhub.io/
• https://github.com/Finnhub-Stock-API/finnhub-python
• https://medium.com/@augustin.goudet/introduction-to-finnhub-97c2117dd9a9
• https://medium.com/@tspann/lets-check-our-stocks-from-finnhub-and-do-some-real-time-analytics-1b7963008e19
• https://github.com/tspannhw/FLaNK-Py-Stocks
• https://medium.com/@tspann/lets-check-our-stocks-from-finnhub-and-do-some-real-time-analytics-1b7963008e19