Enforce TLS on OpenTelemetry Operator
Background
As we are working on Vulnerability Assessment Testing on the k8s cluster, we found out that OTel Operator v0.59.0 is using default TLSv1.0 and default TLS cipher suites on the port where it communicates to kube-api server. Using TLSv1.0 and default cipher suites is vulnerable to SWEET32 attack as it allows DES and 3DES cipher.
How can we replicate this in the local environment?
First, we can set up a local environment using kind and install the OpenTelemetry Operator via helm.
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update
helm install opentelemetry-operator open-telemetry/opentelemetry-operator --version 0.12.0Next, we can port forward port 9443 for opentelemetry-operator-controller for testing purposes.
The tool we choose to test the TLS setting is via sslscan.
sslscan localhost:<port>
Admission Webhook
The opentelemetry operator implements mutating admission webhook that is invoked when the Pod object is created or updated.
The webhook inject auto-instrumentation libraries into the application container and it configures OpenTelemetry SDK and runtime.
Why did this happen?
Under Otel-Operator Controller Manager main.go, there is no option to set the TLS setting.
var (
metricsAddr string
probeAddr string
enableLeaderElection bool
collectorImage string
targetAllocatorImage string
autoInstrumentationJava string
autoInstrumentationNodeJS string
autoInstrumentationPython string
autoInstrumentationDotNet string
labelsFilter []string
webhookPort int
)Hence by default, when the controller manager starts up, it will start up with the default option.
How to patch this?
Since we found the root cause, we can apply the patches.
First, we need to import two packages,
import (
"crypto/tls"
k8sapiflag "k8s.io/component-base/cli/flag"
)“crypto/tls” implement TLSv1.2 and 1.3, whereas k8sapiflag “k8s.io/component-base/cli/flag” is the common core implementation on k8s api flag.
Next, we can implement command line flags for the controller manager. This includes the setting of default value for the TLS version and TLS cipher suites.
type tlsConfig struct {
minVersion string
cipherSuites []string
}
...
func main() {
...
var (
metricsAddr string
probeAddr string
...
webhookPort int
tlsOpt tlsConfig
)
...
pflag.StringVar(&tlsOpt.minVersion, "tls-min-version", "VersionTLS12", "Minimum TLS version supported. Value must match version names from https://golang.org/pkg/crypto/tls/#pkg-constants.")
pflag.StringSliceVar(&tlsOpt.cipherSuites, "tls-cipher-suites", nil, "Comma-separated list of cipher suites for the server. Values are from tls package constants (https://golang.org/pkg/crypto/tls/#pkg-constants). If omitted, the default Go cipher suites will be used")
...
}Next adding the setting for the mgrOption, where optionsTLSOptsFuncs is an array of a type function pointers to set the configuration.
func main() {
...
optionsTlSOptsFuncs := []func(*tls.Config){
func(config *tls.Config) { tlsConfigSetting(config, tlsOpt) },
}
mgrOptions := ctrl.Options{
Scheme: scheme,
MetricsBindAddress: metricsAddr,
Port: webhookPort,
TLSOpts: optionsTlSOptsFuncs,
...
}
...
}
And lastly, we can add the function below:
func tlsConfigSetting(cfg *tls.Config, tlsOpt tlsConfig) {
// TLSVersion helper function returns the TLS Version ID for the version name passed.
version, err := k8sapiflag.TLSVersion(tlsOpt.minVersion)
if err != nil {
setupLog.Error(err, "TLS version invalid")
}
cfg.MinVersion = version
// TLSCipherSuites helper function returns a list of cipher suite IDs from the cipher suite names passed.
cipherSuiteIDs, err := k8sapiflag.TLSCipherSuites(tlsOpt.cipherSuites)
if err != nil {
setupLog.Error(err, "Failed to convert TLS cipher suite name to ID")
}
cfg.CipherSuites = cipherSuiteIDs
}This function gets the option from the command argument (tlsConfig), checks the validity through k8sapiflag, and sets the config for the webhook server.
We have made our changes, and time to compile and test it locally using makefile.
make manager
make prepare-e2e
make e2eAfter the deployment is successful, port forward port 9443 opentelemetry-operator-controller-manager to localhost for testing.
Voila!!! We successfully patch this port with TLSv1.2 and with the list of “safe” cipher suites!
Since we found this issue, we have been actively working with otel-operator community.
and we also submitted PR for this fix.
Reference:
https://kubernetes.io/docs/concepts/extend-kubernetes/operator/
https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/
https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/controller
