How ‘Dash It Up’ Is Executing User-Written Code to Create Customized Dashboards

I am currently working on a platform, ‘Dash It Up’, where developers can create customizable dashboards as internal tools using different data sources like SQL and NoSQL databases or APIs without any maintenance and setup overhead. Team members with non-tech backgrounds can access these dashboards to monitor customers’ info without being concerned with underlying technical details. This ‘Micro-saas’ product is intended for startups at an early stage or with a small team.

I wanted to add a feature in the platform where users could write their own API code logic and create dashboards from their code’s returned payload. It allows users to not stay restricted with database connections and gives them the flexibility to do more things. I had a great time building this feature and wanted to share the thought process involved in building this feature from ideation to execution.

Feature Expectations

• Devs should be able to write code in javascript to interact with APIs.
• Execute dev’s code and return the expected data to create dashboards.
• In cases of errors, it should gracefully exit and return those errors.

Challenges Faced

1. To isolate and to run it in a containerized environment.
2. To execute asynchronous javascript code from string format which can use external libraries like ‘axios’, ‘request’, etc.

Implemented Solution

To be honest, Giving devs an option to write code to query is kind of scary. It can go wrong in many cases as we do not have control of what code is being run to create dashboards. If the user accidentally writes code that runs indefinitely or takes up all the server memory, the whole application server might crash in worst case. So the respective solution should handle all the worst cases and still gives the best user experience.

Rough implementation sketch

To isolate or containerize this feature with all its worst cases, ‘Function-as-a-Service (FaaS)’ services felt like a good choice. I was totally amazed by the whole paradigm and had a superb time coding and learning more about it.

First, What is ‘Faas’?

“FaaS is the concept of serverless computing via serverless architectures. Software developers can leverage this to deploy an individual “function”, action, or piece of business logic. They are expected to start within milliseconds and process individual requests and then the process ends.”

— https://stackify.com/function-as-a-service-serverless-architecture/

I am using ‘AWS Lambda’ for serverless computing service and ‘Serverless Framework’ to build, operate and monitor these serverless functions. Using ‘AWS Lambda’, we can have control over the execution environment which is a big plus, that is, we can set memory limit (say 512 MB) and max execution time (say 10 seconds). If query functions cross these limits, it throws an error and gracefully exit. Also, it auto-scales and has a good free tier.

To know more about ‘serverless framework’, check out this awesome blog. Cool features of this framework which I really liked:

• Easy single line command-line deployment for different kinds of environment (dev, stage or prod).

# default is dev
serverless deploy
serverless deploy --stage production

• ‘Serverless-offline’ plugin to run and test plugin locally.

npm install serverless-offline --save-dev
# in 'serverless.yml'
plugins:
  - serverless-offline
# To run function locally
serverless offline start --port 3002

• ‘Serverless Dashboard’ to track function invocations and error monitoring.

serverless dashboard

This helped in setting up and monitoring almost all the challenges related to challenge #1.

I underestimated challenge #2 at the start and it took me a lot of hit and trial methods to find a good feasible solution for it. Initially, I was using the infamous ‘eval’ method to convert string to a javascript function. Frankly, It looked a bit hacky and did not work as expected as string required a lot of formatting (stripping of newline characters, etc) plus external libraries like axios could not be used. While searching for its solution, I came across this Node.js library ‘vm2’ which solved this problem in an efficient way. ‘vm2’ is a sandbox that can run untrusted code with whitelisted Node’s built-in modules securely. So on top of the serverless framework, this is the second layer of isolation or containerization to run the user’s code safely and securely.

const {NodeVM} = require('vm2');
const vm = new NodeVM({
    require: {
        external: true
    }
});

vm.run(`
    var request = require('request');
    request('http://www.google.com', function (error, response, body) {
        console.error(error);
        if (!error && response.statusCode == 200) {
            console.log(body) // Show the HTML for the Google homepage.
        }
    })
`, 'vm.js');

Result

Previously, only database connections were supported and needed to be established to write its respective logic for the dashboard. Now, developers can write their own API logic to create dashboards for their team.

UX view

Thanks for reading. Checkout out Dash It Up to see this feature (and many more like this) in action! Don’t hesitate to correct any mistakes in the comments or provide suggestions for future posts!