A Spark Attack
Time to check your spark clusters
It is well known — or should be — that spark is not secured by default. It is right there in the docs
Security in Spark is OFF by default
So you should be well aware that you’ll need to put the effort to secure your cluster. And there are many things to consider, like the application UI, the master UI, the workers UI, data encryption, and ssl for the communication between nodes and so on. I’ll probably make another post covering the above at some point.
One thing you probably don’t have in mind is that spark has a REST API, where you can submit jobs. It is not available when running locally, or at least I haven’t managed to make it work, but it makes sense that you need a master to submit to, so, you need a cluster.
The spark setting to enable or disable it is:
spark.master.rest.enabled true
And in older spark versions it was set to true by default.
This Rest API is available through port 6066 and if open, anyone can submit a job to your cluster.
One of our clusters had been behaving strangely recently, and we noticed a job with the application name ML exp by a user called lambda had run a short time ago. It was a short lived job. But…
