Using Webhook Triggers for Event-Driven Automation
When you’re working on automating the entirety of your company’s data operations, you’ll quickly start running into edge cases around scheduling. Not every workflow needs to happen on a set schedule and you don’t always know exactly when something needs to run.
- Can you only run a specific workflow once you’ve been sent a file?
- Do you only need to run the workflow when code is updated?
- Does your process heavily rely on someone else’s work completing first?
- What if you need to run a solution more frequently during peak season, but much less outside of that?
If you’ve found yourself trying to build for these types of situations then we have just the thing to make it easier.
Execute from Anywhere, at Anytime
Today, we’re announcing the introduction of Webhook Triggers for Vessels and Fleets in Shipyard. This new feature will enable teams to programmatically execute solutions living in Shipyard with a simple POST request.
After you create your Vessel or Fleet, you can navigate to the Triggers tab to add a webhook. Each Vessel or Fleet can have exactly one webhook attached to it.
After you successfully run a POST request against a Vessel or Fleet webhook, it will immediately be scheduled to run. The best part is that with webhooks, you’ll still be able to run hundreds of jobs at once and have resources scale dynamically, just like you’ve come to expect from Shipyard. Plus, you’ll still be able to run everything on-demand or with your existing schedules.
With this new trigger type, you can now build your solutions to be entirely reactive, rather than trying to rely 100% on schedule-based triggers. This update makes Shipyard even easier to execute any data workflow that your team could build.
Integrate with External Tools
The introduction of webhooks opens up a world of possibilities to integrate Shipyard with other tools that you use every day. We want to build Shipyard in a way that makes it easier to “play nicely” with other tools, while still building the foundation that will help the platform become your central source for running data operations. The following examples show how easy it is to connect Shipyard to your other tools:
AWS Cloudwatch + Lamda — Use AWS Cloudwatch to monitor AWS resources, like S3, for new files. When a new file gets delivered, kick off a Lambda job that runs a POST command against Shipyard. With this setup, you can keep Shipyard running all of your heavy-duty, long-running data jobs while keeping Lambda responsible for quickly triggering Shipyard at the right moment.
Zapier — Run your existing zaps that connect SaaS apps together with small “if this, then that” rules, such as loading in new emails from a Google form to your CRM of choice. Once the job completes, run a Webhook to kick off a Fleet in Shipyard that processes and manipulates the newly loaded data in bulk.
GitHub Actions — While our GitHub Code Sync integration ensures that Shipyard is always executing up-to-date code, the platform never knows the exact moment that your code is updated. If you want your code to be executed as soon as new code exists, you can set up GitHub Actions to run a POST request to Shipyard every time you merge PRs or make commits.
This is only a short list of possibilities you can open up with the latest support for webhooks. You can just as easily integrate with:
- CI/CD tools like Jenkins, Travis and CircleCI
- Function as a Service tools like Pipedream or Cloud Functions
- Business workflow tools like Tray.io, IFTTT, and n8n.io
- Your own internal applications
While this update makes it easier for other services to trigger Shipyard, the inverse is true as well! Shipyard can always run an HTTP request (POST, GET, UPDATE, PATCH) to any URL to make updates and push changes between your favorite SaaS tools.
Building Internal References
Shipyard’s security measures prevent Vessels or Fleets located in different projects from talking to each other. This setup is ideal for data governance, preventing data from leaking between clients or teams. However, it has sometimes prevented customers from connecting Fleets across projects or organizations. This is a perfectly legitimate use case that we wanted to help support.
Now that Shipyard has webhooks, you can make one Vessel or Fleet trigger another, potentially in a completely different project or organization. In this way, you can build out a workflow that connects Fleet A -> Fleet B -> Fleet C with ease!
This setup is perfect for working as part of a larger data team, where Data Engineering may want their jobs to trigger Data Science jobs, but the teams don’t want each other having direct access to their mission-critical workflows.
This setup could also work in situations where a company needs to deliver files to an external vendor using Shipyard. The company could run their own data processing jobs on Shipyard and once the Fleet finishes running, trigger the Vendor’s Shipyard webhook to pick up the file and process it. With this process in place, data loads are tracked more specifically, rather than relying on finicky file watchers that eat up resources.
Webhooks are now available to all subscribers and can be tested with any trial account. Sign up for a free 14-day trial to start triggering any script you write via a webhook. For more information about how to set up and use these webhooks, view our documentation.
We’re looking forward to seeing how users will take advantage of this functionality to quickly develop and deploy data solutions.
About Shipyard:
Shipyard is a serverless data workflow platform that helps Data Teams launch, monitor, and share their solutions 10x faster. Driven by a mission to simplify every company’s data operations, they are creating an ecosystem where organizations can break down data silos and move beyond dashboards towards a future of fully automated, data-driven actions. The founding team draws on their previous experience at top agencies and media companies, handling high-throughput digital advertising and inventory data for Fortune 500 companies. For more information, visit www.shipyardapp.com or get started with a 14-day free trial.
Originally published at https://www.shipyardapp.com on February 23, 2021.
