Trials and Triumphs of Time-Series Machine Learning with TensorFlow.js
With all the hype surrounding machine learning and AI these days, it feels like there should be a well-trod path to approach, learn about, and apply machine learning to just about anything at this point…or maybe I was just being overly optimistic.
My goal was fairly simple: I wanted to build a simple forward propagation neural network model that would analyze data over a time series and return reasonably correct results. I initially tried for a long short-term memory recurrent neural network, but that is a-whole-nother beast to set up. Anyway, since the data is readily available, I decided to try to predict Bitcoin’s price fluctuations one minute into the future with some level of “I really hope I can get it to kinda follow the general shape at least”.
As a programmer approaching machine learning for the first time, the first thing that you need to know is that you can’t just pick up a library, read some documentation, do a tutorial, and then be able to apply that library to a different kind of problem. If you want to do machine learning properly, it takes a LOT of data science, data normalization, data analysis, and just plain data. On top of that, machine learning has it’s own pseudo-language full of terminology that seems to split the difference between terms data scientists would know, and ones programmers might want to use. Hopefully, by the end of this article, you will be able to set up, structure, and apply a simple model to your own data set. Let’s get to it:
Step 1 - Prep your data inputs:
- Ensure there are no missing time segments in your data (repeat previous value or interpolate where data is missing)
- Remove the time indicator from the training data
- Scale data down to range between [0, 1] or [-1, 1]
- Break the data set into batches to speed up training
- Slice the batched data into training and testing segments
This part is both the least exciting, and the most important part of getting a model working. You will need to make sure that each of your time series is complete and scaled down to values between [0, 1] or [-1, 1]. Using large numbers causes the statistical models running under the hood to go a bit haywire during training, so keeping values in this range is preferred. Batching is important to increase the speed of training, as it can take a very long time if you have a large data set. My data set was around 57,000 points and I don’t have a particularly powerful computer, so I used a batch size of 128. It is then vital to separate a chunk of data for testing so that you can check your model against data it hasn’t seen before.
Below is a gist of scaling, batching, and segmenting the data into training and testing sets.
Step 2 - Initialize you model and add layers:
- Initialize a sequential TensorFlow model
- Build your hidden and output layers
- Add your hidden and output layers to the model
- Select your optimizer function
- Select your loss function
- Compile your model
This is the most complicated and involved step of the process. This is where you set up model in the right way so that it can actually learn based on the data you feed it. I would highly recommend working through Google’s Machine Learning Crash Course at this point if you want to understand what is actually going on here at a high level. I will give you a basic play-by-play in the below gist that should help you set up and build out a working model here. TensorFlow.js come built in with a handful of options for activation, optimizer, and loss functions, so I would stick with those at this point. Feel free to try out different versions as you see fit, but I found the SGD and meanSquaredError setup to work well for me!
Step 3 - Build your tensors and train your model:
- Build your input and output tensors for each batch
- Configure your ‘fit’ training
- Train your model on each input/output tensor batch
We are finally at the training portion! Woohoo! Now to another confusing piece. Tensors. A tensor is the data structure that TensorFlow uses to determine weights (importance) for each input, bias, and hold static values to pass between each of the different layers in the network. These tensors are 2D and contain input/output arrays for each item in a batch of inputs/outputs, but once they are built, fitting can begin! If you have followed to this point, congratulations! You are almost there!
Step 4: Panic
But don’t actually panic! When working with large data sets, tensors, and machine learning, memory requirements get quite large. You may very well run into an issue like I did where the only solution is literally just to throw more memory at it. I amped Node up to run on 4Gb of RAM and never ran into that error message again!
Step 5: Testing the model on your test data
- Build your input tensors for each batch
- Run your model on the input tensors to predict the outputs
- Format your results for graphing or use!
This is basically a repeat of step 3 with a couple idiosyncrasies mentioned in the gist below.
Step 6: Tweaking the model and re-testing again…and again…and again…and again…until finally…SUCCESS!!!
Run 1: 10 epochs, 256 neurons, 1 layer, 0.1 learning rate
Run 3: 10 epochs, 256 neurons, 2 layers, 0.1 learning rate, added momentum trading indicators to input data
Run 5: 100 epochs, 512 neurons, 1 layer, 0.1 learning rate
Final run: 25 epochs, 256 neurons, 2 layers, 0.05 learning rate, w/ indicators
I could keep improving this chart by tweaking the settings in the model, but I think I hit my target of decent trend following and results!
I hope that this intro to TensorFlow helped you get past some of the hurdles that I faced when trying to grasp what was needed to build a model of my own. If you are interested in gaining a better understanding of how to set these models up, I would highly, highly recommend looking into Google’s Machine Learning Crash Course. Yes, this course is in python, but even if you aren’t familiar with the language, the rest of the material is vitally important to learning how to approach the problem you are trying to solve and how to properly set up your data to be useful. Once that is complete and you feel at least somewhat comfortable with the more data science-y part of ML, take a look through the Coding Train — TensorFlow.js YouTube series. Daniel Shiffman does a great job bringing TensorFlow.js down to approachable level from a coding standpoint.
EDIT: Please note that since I was using charts and applying this to a UI, I ended up using the TensorFlowJS CDN for this project. In order to get it running, you will need to add the following script tag into your index.html and then can run your model through a web browser:
There are a few other steps to get it up and running with the full npm package that are detailed in the TensorflowJS Docs.