DevOps Dudes
Published in

DevOps Dudes

Writing a Custom Concourse Resource — The IN

In Concourse, resources are comprised of three parts: check, in, and out.

Photo by K. Mitch Hodge on Unsplash

In Concourse, resources are comprised of three parts check, in, and out. Here we will be discussing the in, what it needs to do, and try to cover some questions you might have when trying to write your own Concourse resource.

The in is performed after a check has confirmed there is something there to pull from and has provides information that will be sent to the standard-in of the in script.

Example pipeline

Things To Consider

Before you write the in of our resource, I suggest you write the check (link) first. I went over in my previous article several points that you may want to consider before writing a check. I would suggest looking at the What Input Do You Want From The User section.

The Logic Of An In

Core

If you follow what I did in my article on how to write a check you probably wrote your output version as something like {"key": value}. In the section below I access that value as payload['version']['key']. I will refer to this simply as version from here on.

The core logic of an in script can be seen below. I will break down the logic into three parts:

  1. input
  2. logic
  3. and output
Core Logic for in.py

Input

The input of the in is slightly different than that of the check. Similarly a payload is delivered via standard-in and it can be parsed with the jsonmodule. But the in stage of a resource is provided a destination directory (the name of the resource) as an argument to the script.

Inside the _in() function (you can call it whatever you want), you have to parse the standard-in to a dictionary you can work with.

My Payload

Logic

In my mongo-resource I need each script to connect to and either find or post data to the database. To connect, I need to use the information in the source section of the input. The find data, I use the version.

I am not going to go over the specific logic and teach you how to use MongoDB and pymongo here, but once I connect and find the data I am looking for, I save that data to the resource directory, dest.

Saving results of in.py to resource disk

Output

After finishing with the logic of a resource you still need to provide a version output. If all you provide as a return statement is return {"version": "5e61d16cd6ca3b229d44eead"} Concourse will throw the following error

json: cannot unmarshal string into Go struct field VersionResult.version of type atc.Versionwhen parsing resource response:{"version": "5e61d16cd6ca3b229d44eead"}

When Concourse calls your in it is expecting a VersionResult variable back. Without going too deep into what that is, you can deduce from the error that when parsing the VersionResult it is looking for a version field. Hence your return needs to wrap your version in a dictionary like return {"version": {"key": version}} as found in the above Core Logic for in.py example code.

Troubleshooting

In my opinion the in and out scripts are much easier to troubleshoot than the checkbecause they will display messages you send to standard-error no matter what. Whereas the check will only show you standard-error if something goes wrong.

To provide print statements that are not meant to be final outputted version, you must send your text to standard-error. Here the function msg is sending what it receives to standard-error that I found from cf-platform-eng/concourse-pypi-resource's common.py and expanded upon:

I added the pprint because I found myself just passing dicts to msg while troubleshooting and dict does not have a format function like str does.

Here is an example of my in and out providing standard-error outputs on successful runs:

Showing off Standard Error Outputs

As long as you follow the pattern of the core logic, this standard-error output method should be sufficient to check your logic along the way.

Final Thoughts

Please refer to my other posts about this topic below to help answer any other questions and please comment below if you have any questions or highlight sections and leave me private comments if my writing is unintelligible as it sometimes tends to be.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Alexander Jansing

Alexander Jansing

29 Followers

Data Scientist / Software Engineer Engineer with Five Years of Experience. I love getting lost in a good problem.