Improving Data Operations with Node.js

It’s 2021 and the language of data is definitively Python. Just take a look at recent studies from Kaggle, Stack Overflow, and KDNuggets. That doesn’t mean it’s the best for everything — but it has definitely won the popularity contest.

So what becomes of teams that aren’t using Python? When they inevitably have to build out workflows for their data operations, they’re getting the short end of the stick. Tools like Airflow and Prefect have established themselves as Python “Workflow as Code” tools, forcing teams to write all of their scripts in Python.

At the end of the day, your coding language of choice is just a preference. And for some teams — that preference is Node.js for scale and flexibility.

At Shipyard, our goal is to make sure that you can launch, monitor, and share solutions that your team is building for data operations regardless of the language that it’s written in.

Making Node.js Automation Easier

Today, we’re excited to announce that Node.js is now natively supported by Shipyard.

Building a Node Vessel that uses Data Forge

Just like our Python and Bash Blueprints beforehand, Node.js Blueprints enable your data team to:

• Automate Node.js code in isolation without making any updates to your script
• Trigger Node.js code on demand, with a schedule, or with a webhook
• Automatically install packages at runtime from package.json files, or manage package installation through the UI
• Securely pass environment variables to scripts
• Build Blueprints to share and re-use Node.js scripts in a no-code fashion
• Build Fleets that connect your Node.js scripts to other scripts written in any language

Enabling New Data Efforts

The inclusion of Node.js means that your data solutions can take any form and use packages that may make it easier for your engineers to work with Data.

• Use Dataform to classify and clean your data sets in BigQuery
• Use Data Forge to transform your data with ease
• Build CLI tools that can be transformed into Shipyard Blueprints
• Write quick scripts to extract data from APIs

Building Language Agnostic Workflows

Shipyard has been designed from the ground up to be Language Agnostic. That means that you can build scripts in any language and connect them together without running into issues.

Because Vessels within a Fleet share the same files, you can string together larger solutions without having to write each component with the same language or even the same packages. All you need to do is ensure that every Vessel, regardless of it’s language, outputs a file that can be picked up by downstream Vessels.

Multi-Language Web Scraper Example

You can even build 80% of a solution with Shipyard’s no-code Blueprint Library and then fill in the gaps by scripting in whatever language you choose! Take the following example, where you could build an exhaustive website scraper that loads results into Snowflake and alerts the team via Email.

The possibilities of how you can move and manipulate data are truly endless and with the addition of Node.js, those solutions can be faster and more powerful. As always, even if the language is not natively supported by Shipyard, you can always install it at runtime and execute it using Bash.

Node.js is now available to all subscribers and can be tested with any trial account. Sign up for a free 14-day trial to start launching scripts that you’ve written with Node.js. For more information about automating your code with Vessels, view our documentation.

About Shipyard:
Shipyard is a serverless data workflow platform that helps Data Teams launch, monitor, and share their solutions 10x faster. Driven by a mission to simplify every company’s data operations, they are creating an ecosystem where organizations can break down data silos and move beyond dashboards towards a future of fully automated, data-driven actions. The founding team draws on their previous experience at top agencies and media companies, handling high-throughput digital advertising and inventory data for Fortune 500 companies. For more information, visit www.shipyardapp.com or get started with a 14-day free trial.

Originally published at https://www.shipyardapp.com on March 3, 2021.