Kill the Hype or Don’t Kill the Hype

If everybody is excited why tell them?

To be honest, I always hated the way scientists were portrayed in works like the MIT Technology Review. Invariably, the article was written as if the author managed to sneak in the secret lair and got a first hand account of how science gets made. Leading scientists are invariably portrayed as lone geniuses, fighting against the establishment to make sure that kernel of truth they are hunting for gets acknowledged.

This contradicted the way science really works in my experience. In real life, very few leading scientists work alone but are actually managers of large groups of PostDocs and PhD students. And while the professor is usually supervising what these people do, a lot of the actual hard work is carried out by these students.

Professors usually need to be really well connected and networked to get published or recognized. You can’t just do this alone (except probably in mathematics where you can try to prove a long outstanding theorem alone in the woods). You also need to constantly apply for grants and that means being well connected with the people handing out grants and having a good understanding what topics are considered “hot” right now.

I can understand why they like to spin the story like this, though. Somehow, it is much more interesting to reduce the whole complexity to this one person, and there is something about this truth seeking against all odds which appeals to us. But in the end it creates a false image in the public mind about how scientists work and how science happens.

Now, when I once was complaining about exactly this to a few fellow researchers they all agreed, but also said that they really couldn’t see any problem with this.

They said that at least the public gets to know about where their tax money goes, and they would rather have a few professors taking the hit and being portrayed like that, than having to fight for a more realistic depiction themselves.

So in essence: no harm done, let’s move along.

And again I can agree. Sure, this is all major BS, and people will believe in a myth, but in the end, overall perception is positive and everybody wins. Right?

And now… Big Data

Now I hadn’t gotten back to this point if it didn’t happen all the time. Take data science and big data. Sure, the hype oversells how easy that stuff is to use, and how much value you can get from it. But should we fight this when it only means more jobs and funding for data science-y topics? Shouldn’t we just wing it and take the money and in the long run everything will be fine?

Isn’t this what the hype cycle is about?

There is currently a huge push to move data scientists to Spark. It has a Python interface. Scala is sold to being the new Python with notebooks and all.

Now correct me if I’m wrong, but doesn’t PySpark work by embedding a JVM inside a Python interpreter, meaning that if you do a map, you do the roundtrip back into Python for each element in the RDD? If we are honest, isn’t Spark more like glorified SQL and compared to raw performance of in-memory matrix computation it is like 100x slower? And you can spin it like you want, Python outmatches Scala in terms of ML and visualization libraries significantly, not speaking of Scala actually being a pretty advanced programming language not easily picked up by people whose main experience is a dynamically typed scripting language.

Well still, we’re probably better off with all that attention on Spark and Big Data. And maybe some would even be calling me damaging (or at least I see very little open criticism. After all where should it come from?).

Still, at the end of the day the end users are the people buying this stuff, and if they are massively disappointed because of inflated expectations, then the damage is real.

And somehow I also think we don’t get a good exchange going about the features we really need if we all pretend it’s awesome.