Writing a Custom Concourse Resource — The check

Photo by JESHOOTS.COM on Unsplash

In Concourse, resources are comprised of three parts check, in, and out. Here we will be discussing the check, what it needs to do, and try to cover some questions you might have when trying to write your own Concourse resource.

The check is typically performed before anything can use the resource. This can be seen if you click on a resource block (smaller blocks on the left-hand side of a task).

My example Concourse pipeline using my mongo-resource type.

check is used to detect new versions of things where the resource is designed to look. For example the git resource type monitors for new git hashes and my apjansing/mongo-resource resource is designed to look for new ObjectIds within a MongoDB Collection using a given a query.

Things To Consider

When writing a Concourse resource there are a few things to take into consideration before you begin:

1. Which parts of the resource do you need
2. What to return
3. What input do you want from the user

The rest is your own logic. I’ll show some of this after covering these three points.

Which Parts Of The Resource Do You Need

Do you want to write something like my mongo-resource or do you want to write something that does not even need a check? There are resources like Karunamon/concourse-resource-bitbucket that only commits changes to a Bitbucket repository. And returns [] and {} under check and in, respectively. You can get away with this if all you want your resource to do is to put some data somewhere. Normally the out will perform an in after completion because of the logic of Concourse. The put in Concourse wants to make sure there is something in the location specified by the version returned by the out script.

Similarly, you can write your resource to do nothing at any stage as long as it makes sense for what you are writing. You could return empty results from a check and allow the in to do all of the heavy lifting. Although it is best to separate responsibilities when writing code. If you deviate from the recommended recipe Concourse expects your resource may not behave predictably. checks provide history, where a blind in would be harder to control.

What To Return

Conceptually the check is the first part of a resource to write, but before you jump in consider the following:

• Your check needs to output something that the in will use to fetch data.
• The out also need to return something that will be used to run an additional in to confirm the out was successful in its task.

Essentially, your first task it to decide what you are going to be using as a standard value for the rest of the components of your resource.

My mongo-resource monitors for new ObjectIDs. Since the object ObjectID is not json-serializable, I used the string representation. And I am lucky since the string representation can be used to reconstruct the original object easily.

The check of a Concourse resource with ObjectIDs shown as versions.

What Input Do You Want From The User

You may not know all the fields you are going to need right away, but it is important to know how to get custom values into your resource.

If you have written a pipeline before you have probably filled out the source of a resource. These mappings can be whatever you want. If you need or want to provide functionality in your resource, this is one of the places where you can provide that freedom.

All you need to do to take advantage of source is to write logic in your check to access the key-value pairs within the mapping (I will show how to use this mapping in the next section.)My resource can accept the following source keys:

• url
• port
• db
• collection
• find
• username
• password

I have provided default values for username and password in common.py when I parse the source from the overall payload and generate the URI used to connect to my MongoDB. Since there are default values for username and password they are not required in the resource definition in the pipeline configuration:

Note: check_every is a keyword for all resources. Its default value is 1m.

The Logic Of A Check

Core

Now that you know that the key-value pairs of the source can be anything that you need. You receive information from Concourse through standard-in and you can parse that input using the json package in Python. The entire payload looks like {“source”: {“url”: “mongo”, “port”: “27017”, “db”: “test”, “collection”: “trigger”, “find”: “”}, “version”: None}. The core logic of a check is

Core Logic for a check script.

Once you get the to ### LOGIC ### section of the code you can start accessing whatever you decided you needed from source as specified in the pipeline configuration. This a bit of a chicken-and-egg problem where you will probably add and remove things as you go.

If you decide you need a favorite_color option, include it under source like all the other variables in the pipeline configuration and then access it in your check like source[‘favorite_color’] (if you can’t decide on a favorite color yourself, be safe and just say yellow).

“What is your favorite color?” — Old Man at the Bridge of Death in Monty Python and the Holy Grail.

Troubleshooting

To provide print statements that are not meant to be a returned version, you must send whatever data you want to see to standard-error. Here the function msg is sending what it receives to standard-error that I found from cf-platform-eng/concourse-pypi-resource’s common.py and expanded upon:

Expanded standard-out printing function.

I added the pprint because I found myself just passing dicts to msg while troubleshooting and dict does not have a format function like str does.

If check exits without error it will only show you what you send to standard-out. check will only show you what you send to standard-error if it actually fails. If you need to see any intermediate output you may need to cause some of your own problems! Here is an example where I forgot to provide credentials and my resource could not reach my MongoDB:

Induced Error for a check by providing bad credentials. Useful for checking developer standard-error outputs.

Here you can see on the third line of text, I sent my own string to standard-error.

Preventing Bottleneck

As I said at the start of the post, “check is used to detect new versions.” If check returns a version that was previously reported, Concourse will not trigger* or report on any additional events it has in its history. The previous version will be held in the version field of the payload as you can see in the image below. Notice that the version reported in the last run of the check reported the same hash that is shown in the version field in the output of standard-error (also at the time of taking this screenshot, I modified the standard-error from the previous image.)

*Exceptions may be resources like the git resource. It will not always trigger new events off of a git hash it has already reported on except when you roll back commits and force-push to the branch being monitored. This may cause problems because the git resource keeps track of the order of git hashes it has seen.

Check error showing how versions are passed to future runs of the check script.

A problem I have had while writing a resource to check exclude previous versions can be seen in the image above. The ObjectID bash being provided is only a string, where I need it to be in an array. I could wrap that string in an array, but while writing the code I figured that the standard-in input my resource was receiving was the same that the in script was receiving, that meant that the version was always going to be a sting. This might not be an issue for all resource, but it would mean that even if I put version in an array I would always be receiving one element long arrays and I would only ever return the top two results results of a given query.

Instead of relying solely on the provided version, I realized that since the resource does not go away I could write to disk. So I wrote the versions json I wish was being given ({‘version’: [SOME_HASH, MAYBE_ANOTHER_HASH, …]}) to me in the check to the file system of the resource and read it back on each run. For apjansing/mongo-resource this is done so that if a find being performed against the MongoDB returns more than one result we can:

1. read in the versions.json and use the array in versions to a $nin (not in) json can concatenate it to the find,
2. grab the top result according to the modified find,
3. add the resulting entry’s ObjectId to the versions json and save it to the disk,
4. and repeat.

Important: Please test this further if you are deploying to a production environment. I do not want you to get yourself into trouble if this turns out to be bad advice. My version tracking allows me to keep querying the database without return the same results every time. If my database were significantly large I would look for a more scalable solution, but that is beyond the scope of this tutorial.

Final Thoughts

Please refer to my other post Writing a Custom Concourse Resource — Overview and my repository (both linked below) for more details. I will continue to write about how to write a Concourse resource in my next post Writing a Custom Concourse Resource — The in. And please comment below if you have any questions or highlight sections and leave me private comments if my writing is unintelligible as it sometimes tends to be.

• Writing a Custom Concourse Resource — Overview
• Writing a Custom Concourse Resource — The check
• Writing a Custom Concourse Resource — The in
• Writing a Custom Concourse Resource — The out
• apjansing/mongo-resource