The IBM Stock Trader operator, part 2: Development
Learn how to develop an umbrella operator like the one for the IBM Stock Trader sample
In my last post, I described how to use the operator I created for my IBM Stock Trader sample. Now let’s take a look at what I did to develop that operator, so that you can create such an operator too for your application.
There are multiple ways you can author your operator, including via Helm, Ansible, or Go. About a year ago, I spent a fair amount of time creating a Helm chart for my sample, so I chose the Helm route, so I could re-use all of that work, which I wrote about earlier.
I won’t go into everything involved in creating a helm chart here, but I will point out that it basically is a collection of yaml files that get run. For example, here are the contents of the helm chart for IBM Stock Trader (from the https://github.com/IBMStockTrader/stocktrader-helm repo):
As you can see, there are some global yaml files in the root of the helm chart archive, and then the individual yaml files you want run are in the templates directory. I have one there for each microservice in my application, plus one for my config map and one for my secret. Note there is no code anywhere; I wrote nothing in Java or Node.js or Go or anything; it’s just a collection of standard Kubernetes yaml files.
One nice thing about helm is that you can use substitution variables, rather than hard-coding values, in the yaml it contains. So if the helm chart asks how many replicas (pods) you want per microservice, you can reference that in the yaml, as in this snippet from the portfolio.yaml shown above (which also parameterizes the container name and image, and checks a boolean for whether to include some lines in the yaml for monitoring):
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: {{ .Release.Name }}-portfolio
labels:
app: {{ .Release.Name }}-stock-trader
annotations:
prism.subkind: Liberty
spec:
replicas: {{ .Values.global.replicas }}
template:
metadata:
labels:
app: portfolio
{{- if .Values.global.monitoring }}
annotations:
prometheus.io/scrape: 'true'
prometheus.io/port: "9080"
{{- end }}
spec:
containers:
- name: portfolio
image: "{{ .Values.portfolio.image.repository }}:{{ .Values.portfolio.image.tag }}"So if I want the logic for my operator updated, like to conditionally deploy something new like a Route or a HorizontalPodAutoscaler, I have to go add it to my helm chart first, then the operator will inherit that. OK, enough about Helm — let’s move on to how to “wrap” that helm chart inside of an operator. First, you’ll want to install the Operator SDK. Since I develop on a MacBook, I used homebrew to install this:
brew install operator-sdkThis puts the operator-sdk executable in my PATH, so I’m ready to start using it. Note I’m essentially describing a concrete example of following the steps that Red Hat explains in the OpenShift documentation here: https://docs.openshift.com/container-platform/4.3/operators/operator_sdk/osdk-helm.html
Then I just run the following command:
operator-sdk new stocktrader-operator --api-version=operators.ibm.com/v1 --kind StockTrader --type helm --helm-chart ../stocktrader-helm/stocktrader-0.2.0.tgzThat assumes you have cloned the stocktrader-helm repo into a sibling directory of the one where you are developing your operator, and have done a helm package stocktrader in that sibling directory, which packages up the .tgz file. If you don’t have helm setup, you can grab the tgz file from where I self-host the helm repository in GitHub.
The above command generates various files in various subdirectories of a stocktrader-operator directory. The most important of these is the CustomResourceDefinition (CRD) for your type of application. It also generates a sample CustomResource (CR), which is essentially a copy of the values.yaml from the helm chart. I like to think of it as the CRD being to an XSD as a CR is to an XML that conforms to that XSD. You’ll also see the helm chart is copied in, as are the deployment yaml files you saw/ran in part 1. Note the olm-catalog subdirectory shown below won’t be there initially; I’ll talk about how to generate that in part 3.
In the generated stocktrader-operator directory is the watches.yaml file, that says what type of Kubernetes object this operator acts against, and the helm chart that it wraps. Its contents don’t need any further editing, but I’ll show them here for completeness/understanding:
- version: v1
group: operators.ibm.com
kind: StockTrader
chart: helm-charts/stocktraderAnother file that gets generated is the Dockerfile in the build subdirectory, for the Docker image that gets run as a deployment/pod for the operator. You can build this image by changing into the generated stocktrader-operator directory and running the following command:
operator-sdk build stocktrader-operatorThis produces a stocktrader-operator:latest Docker image (containing the helm chart, at the location specified in the watches.yaml), which I then pushed to DockerHub via the following usual commands (if building yourself, you'll need to push to somewhere that you have authority, and will need to update the operator.yaml to reference that location):
docker tag stocktrader-operator:latest ibmstocktrader/stocktrader-operator:latest
docker push ibmstocktrader/stocktrader-operator:latestNote that if you were to pursue getting your operator certified to be placed into OperatorHub, it would have to be pushed to Red Hat’s quay.io instead. I’ve not personally done that, since I don’t think a sample like mine makes sense in the main repository of operators for real products and services.
For my helm-based operator, the structure/contents of the CRD basically needs to match what’s in my helm chart’s values.yaml. So every field the helm chart asks for — which now the operator will ask for — needs to be described in that CRD.
Sadly, the operator SDK does NOT generate all of this for you (it seems like it would have enough info to have at least given it a try, similar to generating an XSD from an example XML), so I had to spent a LOT of time hand-editing the CRD yaml, mostly in the openAPIV3Schema section. Having the Open Liberty operator available to look at in public GitHub was quite helpful, I will say, for working on the CRD and some of the other yaml files. Here’s a snippet of what the CRD looks like, basically describing the fields and their data types (I picked a variety of data types to show here):
spec:
description: StockTraderSpec defines the desired state of StockTrader
type: object
properties:
global:
description: Global Settings
type: object
properties:
auth:
description: Authentication type
enum:
- basic
- ldap
- oidc
type: string
replicas:
description: Initial number of pods per microservice
format: int32
minimum: 1
type: integer
autoscale:
description: Enable Auto-scaling
type: boolean
monitoring:
description: Enable Prometheus monitoring
type: booleanSo I had to do that for each section in my helm chart’s values.yaml, which was tedious, but not difficult. Note the nesting of fields underneath sections, like my “global” and “database” sections from my helm chart.
I also had to edit the generated role.yaml. It gives you a good start at each of the apiGroups that need to be defined, where basically you enumerate what kind of resources (like Deployments, Services, etc.) you will be creating, and what verbs (like get, create, update, etc) you want to allow. You can use a “*” wildcard for the resources and/or verbs, if desired. I had to add the following apiGroups, to avoid permission denied problems when deploying a StockTrader via my operator, and to be allowed to generate a Route:
- apiGroups:
- operators.ibm.com
resources:
- '*'
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- apiGroups:
- route.openshift.io
attributeRestrictions: null
resources:
- routes
verbs:
- '*'You might need to get an administrator to run such a yaml for you, since your ID may not have authority to grant such permissions (I have the luxury of being the administrator for the cluster I provisioned in the IBM Cloud for this work — note that OCP 4.3 support in IBM Cloud has exited beta and is fully GA now — you need to be using at least version 4.3 to use this operator).
Once you have deployed the operator, as described in part 1, you’ll be able to deploy instances of the StockTrader CRD type via a standard oc create -f command against a yaml that contains the values of all of the fields in that CRD (you can omit any fields where the default is acceptable to you). So you can edit the example CR yaml mentioned earlier, putting in your actual values of things like the host/port/id/password for the database (the one mandatory prereq where you can’t just get by with defaults), and for optional stuff like the Watson Tone Analyzer:
apiVersion: operators.ibm.com/v1
kind: StockTrader
metadata:
name: blog
spec:
# Default values copied from <project_dir>/helm-charts/stocktrader/values.yaml
database:
kind: db2
db: BLUDB
host: dashdb-txn-flex-yp-dal10-68.services.dal.bluemix.net
id: BLUADMIN
password: <your password>
port: 50000
watson:
id: apikey
passwordOrApiKey: <your API key>
url: https://gateway.watsonplatform.net/tone-analyzer/api/v3/tone?version=2017-09-21&sentences=falseWith the StockTrader CRD being a fully functional type in your OpenShift environment now, you can act against instances of it, such as the blog CR described above, via commands like oc get StockTraders, or oc get StockTrader blog -o yaml, or oc describe StockTrader blog, or even oc delete StockTrader blog. It’s nice being able to work against a StockTrader deployment as a whole, rather than having to individually worry about the dozens of resources that it creates.
So once again, congratulations, you have now learned how to develop an operator via its SDK, and how to use that operator from the oc CLI. We’ll wrap up part 2 here. Stay tuned for the final part, where I’ll talk about how I integrated it into the OpenShift console, and some final thoughts on operators and their future directions.
