Automatically Sync Your Workflows On Dockstore With Our GitHub App
There is a decent chance that if you are reading this, you work with software that is hosted on GitHub. While GitHub isn’t required to use Dockstore, Dockstore integrates closely with GitHub to leverage its version control, release management, and collaborative environment. For these reasons, we recommend you register your Dockstore entries with GitHub, particularly leveraging our new method of registering workflows, which uses the Dockstore GitHub app and a configuration file (.dockstore.yml) placed in your repo. You can easily update your existing Dockstore entries to include a .dockstore.yml file. We’ll cover the basics of the process here, but you can see our documentation for more info.
Why register using the Dockstore app?
Before we go over how, it is worth exploring why. Using our GitHub App, which works in tandem with .dockstore.yml files to configure it, allows for the best experience when using Dockstore.
Automatically get the latest commits, branches, tags, and releases
The GitHub App communicates with Dockstore every time you push a new commit to your repository. If you make a commit that updates the code of your workflow, you will find that its entry on Dockstore will be updated to reflect that change. The exact timing varies depending on GitHub API limits, but we generally see new commits showing up on Dockstore in less than five minutes.
There is a bonus for Terra users: If you import a workflow from Dockstore to Terra using the Launch With feature, you will find that your Terra workspace also keeps the latest version of the workflow. This is a great way to run tests on Terra — simply push your changes, wait a few minutes, and then go straight to Terra to run with your new changes.
New branches and tags will also appear automatically when you use a .dockstore.yml. Note that the default version of your workflow that shows when someone first goes to a workflow entry will not change — so if you make a new branch to test changes that might break everything, you do not need to worry about people using this new branch by accident, as they would have to manually select it. As with new commits, this also benefits Terra users — new branches also show up in Terra under the drop-down "version" menu on its workflow execution page.
Import only your favorite branches or tags to Dockstore
You may not want every tag or branch to show up on Dockstore. Using the filter feature, .dockstore.yml files make it easy to show only what you want. You can list out specific names of branches or tags, or use pattern strings/RegEx to select a group of branches or tags with a common naming pattern.
Easily manage multiple workflows in a single repository
If you prefer to keep multiple workflows in a single repository, a .dockstore.yml file makes managing those workflows easier. Creating a new entry for a new workflow in an existing repo is as simple as adding a few lines to the existing .dockstore.yml. Each entry can have its own JSON example files too — and you don’t need the same number of example files for each entry.
Prerequisites
Use this checklist to determine if you are ready to register your workflow with the GitHub App.
- Your code is on GitHub in a public repository
- Either the Dockstore GitHub app is already installed for the personal GitHub user or organization that the code is part of, or you have sufficient permissions to install the app
- You are working with a workflow, not a tool — registration of tools is planned, but not currently supported
- You are working with a workflow written in CWL, WDL, Galaxy, or Nextflow
Process
Installing the GitHub app
First of all, if you have not done so already, install our GitHub app on your GitHub organization. (If your repo is not on an organization but is instead connected to your personal GitHub account, the installation process is the same; just select your GitHub username instead of an organization during installation.)
GitHub permissions can quickly get complicated, so we currently recommend that the app is installed by an organization admin and is given access to all repos on that organization. This means that new repositories in the organization will automatically be “seen” by the app. If you choose to instead give the app access to only one repo at a time, you may need to have an organization admin update the app’s permissions when new repos are created.
Create the .dockstore.yml file
This blog will assume you are registering a workflow — please see the services section of our docs if you wish to register a service, as the resulting .dockstore.yml file has a different syntax than what we will be going over in this blog.
A very simple version of a .dockstore.yml can be found below. Note that the first line lists which version of .dockstore.yml syntax is being used — it does not correlate to a tag or version of the workflow itself.
This is a real example — the gist is a mirror of the .dockstore.yml for the goleft repo, and you can see the resulting Dockstore entry here, which includes the WDL itself (goleft_functions.wdl) and an example parameter file (goleft_terra.json). We could have also added a line for the name field, but because there is only one workflow in this repository, it does not need this field to differentiate it from other workflows.
Make sure your .dockstore.yml:
- Is in the top-level directory, not a subfolder
- Has a period before the filename
- Has extension yml, not yaml
For more information, including how to use the filter feature to only register specific branches or tags, see our docs for more information.
Add workflow to Dockstore
With your file all set up, it will take a short while for GitHub to communicate the changes to Dockstore — we can’t guarantee a time as it depends on GitHub, but we have seen that it generally takes less than five minutes, just enough time to check your email or make a cup of tea. You can now go into Dockstore’s ‘My Workflows’ section.
You should see your workflow(s) listed as unpublished workflows. Simply press their publish buttons, and they will be right where they need to be on Dockstore.
Last remarks
Dockstore is still in active development, so we are continuously working to improve the user experience. In-progress is allowing users to register tools with this method and clarifying the permissions side of things. We are open source and welcome contributions and suggestions on our repo. Also, check our documentation if you would like more information on the .dockstore.yml registration feature, especially our GitHub App registration FAQs if you run into trouble.
