Running Python code in Serenytics
In some BI solutions, you can use Python, but it’s not straightforward at all. It feels like a feature that’s been added but was not planned in the initial software architecture.
When we started Serenytics, we wanted it to be fully integrated into the software and as easy as possible to use (no server setup, no configuration). With the current version, it is as easy as signing up on a website, writing python code and clicking on a button to execute it.
Benefits of Python for BI
There are two main benefits of using Python in a BI platform:
- There are a lot of data-oriented libraries (e.g. pandas, numpy, scikit-learn). These libraries are included in the environment we provide.
- There are a lot of multi-purpose features and libraries to manipulate character strings, transform dates, query a REST API, get files from an SFTP server…
Run your first Python script in Serenytics in less than a minute
In the Serenytics studio, open the Automation page (from the left bar). And click on the New Automation button, and then choose Python script:
This creates a new python script that you can edit directly from our interface:
To run the code, open the Execution tab and click on the Run now button. When your script has been executed, you obtain the execution information:
Congratulations, you have just run your first Python script in Serenytics!
Writing useful Python code with the Serenytics client
The above “Hello Serenytics!” example is not very useful. The most common use case for Python in Serenytics is to retrieve some data in a pandas data-frame, manipulate it and then make it available for a dashboard.
The Serenytics python client helps a lot to achieve these steps. The example below shows you a full script:
A few notes about this script:
- the input data is retrieved from a data source configured in Serenytics. It could be any type of source handled by Serenytics (a SQL table, a file, a REST API, a Google Analytics query…)
- the reload_data_from_dataframe() reloads a pandas data-frame into a table of the Serenytics internal data-warehouse (AWS Redshift). Then, it is straightforward to use it in a dashboard for visualisation.
The full documentation for our python client is available here: https://doc.serenytics.com/developer/api_reference/python_client/
Some other features of the Serenytics client include:
- Send an email (optionally with a dashboard attached as a PDF file). This is useful to create ad-hoc alarms on your data.
- Invalidate/warm-up the Serenytics cache engine
- Trigger another script
- Push/pop data attached to a script (for example, this can be used to store a date of last sync for an incremental update script).
Scheduling your Python script
Scheduling a Python script (or any other automation) is straightforward using our interface:
Running a Serenytics Python script outside the Serenytics platform
By default, a Serenytics script is run within our cloud architecture. But there are a few reasons why you may need to run it elsewhere:
- To use your favourite IDE (at Serenytics, we use & love PyCharm) for debugging. That’s the way we usually write scripts at Serenytics. We start by running it locally, using breakpoints, step-by-step execution, debug console. And when the script is ready, we paste the code into Serenytics to schedule it.
- To access a local database. Let’s say you have a SQL database you can’t access from our cloud IPs for security reasons. You can write a python script to get data from it and push it to the Serenytics data-warehouse. All you have to provide is a local server and you need to schedule the script with your own system (e.g. a cron tab).
To be able to run a script outside Serenytics, you just need to provide your Serenytics API KEY (available in your account page) to the function used to create the python client:
Trigger a script with an API call
Each python script can be triggered by an API call. The URL for a given script is: https://api.serenytics.com/api/script/SCRIPT_ID/run
The SCRIPT_ID is displayed on the script edition page.
You can then trigger your script with a curl call:
curl -X POST -H "Content-Type: application/json" -H "X-Api-Key: MY_API_KEY" https://api.serenytics.com/api/script/SCRIPT_ID/run
Here is an example calling such a script in Python:
In some cases, for security reasons, you don’t want to provide your API_KEY (remember that this key is highly sensitive as it can be used to access all your data). In some apps (e.g. Zapier, Slack), you just can’t provide these headers. For these situations, you can use the script web-hook. This is a URL with a token dedicated to a given script. It’s available in the webhook tab:
With this option, running the script is as simple as:
curl -X POST https://api.serenytics.com/api/script/SCRIPT_ID/run?token=MY_TOKEN
From a python code, it would be as simple as:
And if you need to, you can pass parameters to your script:
The passed parameters are available in the script from the variable client.script_args
Conclusion
If you went so far, I hope you already have created your first Serenytics Python script!
Most of what you’ve seen in this article can be achieved by setting up servers, cron-tabs, docker containers, API in Flasks… But that’s a lot more complicated to setup and maintain. If you consider your time worths $X/hour, the simplicity brought by Serenytics is often a win.
Any comment, feedback, use case? contact-us!
