SUPERALGOS DATA MINING
Indicator Building In Superalgos: The Ultimate Guide to Tame the UI
Prior conditions to understand this article are of course to have completed all the basic tutorials included in Superalgos and to be familiar with the different concepts of the user interface (UI).
Let us demystify the process! First of all, the UI is built in a way everything is logically chained and understanding what is the role of a node and the relationships and references between nodes, are the only requirements here.
So, do you feel more confident now? Then brace yourself and take a trip with me down the rabbit hole!
Data Mine Creation
A New Data Mine
The Data Mine node is where indicators and plotters are defined.
Superalgos is provided with a set of existing data mines developed by seasoned developers. The most efficient way to start a new data mine is to reuse an existing one. Since it would have been reviewed by the Data Mining Team and built and maintained by an experienced developer following good practices, it is a good starting point.
Find the Data Mines plugin folder at…
… and drag-n-drop a data mine to your workspace:
Mark Your Territory
Hover your mouse on your data mine and left click on the title to rename it :
You can now proceed to some clean-up and remove the unwanted nodes, plotters and indicator bots.
Change the Codename of the data mine at the Configure menu:
It will be the name of the folder where the mined data will be stored. Choose something wise and avoid characters that would not be handled by your operating system.
Hit Ctrl + Shift + S in order to save your workspace. Since the data mine have been manually added to the workspace, it will be saved inside.
A Bit of Cleaning and Tiding
After the first steps, you should have ended with a data mine with only the remaining nodes of interest :
Rename both your plotter and indicator bot. To keep things clean, use the same name for both.
You can now proceed to the Codename changes
As per the data mine codename, it will be the name of the folder at the data storage, avoid spaces and choose simple names without special characters or others funny things (spoiler: or it will not work!)
When reusing existing nodes, it might be some references are tied to the original data mine, cleaning a little bit will save you some long and cold sleepless winter nights. Break all the irrelevant references
Setting up the Process Definition
The process definition node is where the indicator’s calculation process is organized. It will tell the bot when to start/end, what are the outputs to store and where to find required data like the candles.
The Output dataset is referenced to the Dataset Definition of the multi-time frame XXXX at the product definition. At least one Status dependency is self-referred to the Status report of the process definition. A reference between a status dependency and the status report of the candles is required. A reference between a data dependency and the Dataset definition of the Candles is required.
Each status dependency needs to be configured as per required by its usage:
The process definition for Market and Daily time frames does not necessitate the same references.
The Daily require both market starting and ending point whereas the Market will need only the Market ending point :
Setting up the Record Definition
The record definition literally contains the Objects that will create stored variables.
- Clean existing Record Property except for Begin and End that will be the stamp for Begin of Candle and End of Candle (time reference for the variables)
- Establish the list of variables you want to record
- Create a set of Record Property + Record formula for each variable you need to store
- For each record property, set the « codename » parameter with a name in order to remind the name of your variable (choose something simple, space and special chars are prohibited, it will be the name you will use at the Plotter)
Tip: this is where you define the data to be stored on your SA machine drive; keep it simple and only with what it is strictly necessary to minimize disk space and CPU usage
Tip : now a bit of work has been done, it should be a good time to have another level of backup ; use the clone node function and save your datamine file at a safe place. In case of a disaster, you will just have to drag’n drop it to revive your data mine
Write Down the Indicator’s Code: the Procedure Loop Node
The data building procedure node is situated under the Product definition node of the indicator bot. the procedure loop contains the main code to be executed at each iteration. The Procedure initialization contains the variable initialization: you will declare the array typed variables at this place.
The rabbit hole may sometime be dark and damp so here are 3 tools you may find useful to avoid traps.
Always code your indicators outside Superalgos : the Procedure JS Code node is handful but difficult to read. A good tool to type your code is Notepad++: it is free, lightweighted and can recognize a large variety of coding languages; if you work on a linux/debian based plateform, Geany is also a super great tool.
Sometime, a simple synthax checking is a good start before trying to execute your indicator: Esprima offer a free online version of a synthax checker: https://esprima.org/demo/validate.html. When it comes to debug, VS-Code is nearly the best tool you can find.
Finally, testing the math of your indicator outside Superalgos in a first time will save you a lot of debugging time. There is a lot of very good datamining tools available but if you can spend time to discover a new software, Knime is the best data mining and Machine Learning tool you can find at 0 cost: https://www.knime.com/knime-analytics-platform. Plus : it provides free online courses on data mining.
Write Down the Indicator’s Code: the Procedure Loop Node
0- THE INITIALIZATION RULE: the mother of all rules! The charting space is really annoyed by non-numerical values. Be sure to avoid NaN, undefined, infinity in the data.json files. In the best case, the chart will not work, but it can go up to a crash of the browser. Not to mention also the devastating effect of such a value in a live trading session. Please, manage exceptions with hard coded procedures at initialization and during execution.
1- Comment your code: there is nothing more annoying than a huge block of code without any comments to indicate what it is done. It will help you and other developers to keep track of the meaning of the code, will make it easier to maintain and to review
3- Keep it clean: when developing it happens we take different directions and want to keep track of what we have done. Most of us simply comment the lines. When you release your work, please, clean before PR
4- Do not store useless variables: you may have found interesting to follow the evolution of sub variables at the record definition but it is costly in terms of disk space. If you intend to deliver your indicator to the community, clean your record definition
5- Do not be afraid of data mining specialists: they don’t bite, at least if they are fed, and it will save you a lot of time and a lot of frustrating nights. Also, specialists from the Community Mod team are committed to help you so when we ask for a clone of your node it’s for your own good and it will be easier for us to find a bug with something running on a known configuration with a good overview!
6- Accept your own learning curve and that you will not have a perfect result at the first try. Also, since people with little knowledge on programming will have no choice but to trust the content of the indicators, it may happen your contribution is postponed due to further needs of investigation on the reliability of the code.
7- Submit to the community first and demonstrate: snapshots of the results of your work with some statistics on the stability are very appreciated by the community and will grant you trust and exposure.
Plug ‘n Pray: Pull the Trigger, Find Errors, Debug, Repeat to Exhaustion…
Settle and indicator Bot Instance for your brand-new indicator Run the data task and then the Candles tasks This first step should show you all errors that are not at the code.
If your indicator now processes funny values (NaN, all values to 0, undefined…) then you may have some bad stuffs at your code. Use the debugging procedure from SA doc using your preferred IDE.
Actually, you should not have NaN or undefined values… unless you have not respected the Mother of all Rules. Having 0s everywhere or NaN or undefined… means there is a problem.
Once you get things working, time has come to design your plotter bot.
The Plotter is the node where the graphic representation of your dataset is defined and produced. Since the Plotter of your indicator has been reclaimed from another data mine, the 1st step is to rename it and set the codename in the parameters.
Unwrap the Plotter node and proceed to the renaming of The Plotter Module and the Plotter Panel and eventually the Shapes node; don’t forget to change the codename of each node at the Configuration menu.
Unwrap the Plotter Module and check there’s a reference between the Product Definition and the Plotter Module. Each plotter module at the plotter will define and produce a chart for each product definition referred.
Each plotter module contains a Plotter Panel (display of the indicator values in a panel), and a Shapes node (plot geometry and colors)
Plotter panel gathers Panel Data. A panel data defines the indicators value to display. A Panel Data contains a Data Formula to indicate the source of the data and a Text Style node where you can define the format of the displayed value.
Set the number of decimals do display in the Configuration menu of the Panel data.
Type the formula to point at your recorded value: record.<record_property_codename>
Shapes node contains the definition of the Chart Points (x and y coordinates) and the Polygos (geometry and colors).
Chart Points node contains Point nodes defined by Point Formula nodes. To plot a simple curve, 2 points are needed to define the segment between the begin and the end of a candle.
The begin point formula refers to record.begin and must point at the record.previous.<your_record_property>
The begin point formula refers to record.end and must point at the record.<your_record_property>
The Polygon node contains the definition of the geometry of the plotted indicator. It uses Vertices to define the points and Polygon Border and Body for the border and the filling of the plotted indicator.
The chart is plotted according to the clockwise rotation around the Polygon node following the vertices.
The polygon vertices have to be referred to the Plot Nodes at the Chart Points node according to the clockwise direction around the Polygon. Begin and End will have to be addressed. Add one Polygon vertex per each and build the reference to each Point.
To build a simple curve add the Polygon Border node and define the curve style at the Style Node.
The modification in the Plotter node will not appear at the chart until you refresh the UI. Save your workspace and hit the refresh key of your browser.
You have reached the last step of the indicator building procedure. Save your workspace and clone your data mine and store the file in a safe place. You can now add the layer for your indicator at the Time Machine or use the Add product function.
If you enjoyed this article and want to participate in the most promising open-source social trading platform… come and join us !
- Superalgos Community : https://t.me/superalgoscommunity
- Superalgos Support Group : https://t.me/superalgossupport
- Superalgos Develop Group: https://t.me/superalgosdevelop
- Superalgos Data Mining : https://t.me/superalgosdatamining
- Superalgos Machine Learning Group: https://t.me/superalgosmachinelearning
- On Discord Server: https://discord.gg/CGeKC6WQQb