Are You Ready for Vibe Analytics?
Michael Schrage of the MIT Sloan School explains how the rise of large language models is poised to transform the way organizations analyze their data.
Earlier this year Andrej Karpathy, the former senior director of AI at Tesla and a founding member of OpenAI, described a new type of software development he calls vibe coding.
“You fully give in to the vibes, embrace exponentials and forget that the code even exists,” Karpathy explained on social media. “It’s not really coding. I just see stuff, say stuff, run stuff, and copy-paste stuff — and it mostly works.”
Now Michael Schrage, a Lecturer at the MIT Sloan School, wants organizations to apply Karpathy’s “vibe” approach to analyzing their data. He calls his new approach vibe analytics; his article on this topic will appear soon in the MIT Sloan Management Review.
To explain vibe analytics and its implications for organizations, Michael spoke recently with Peter Krass, a contributing editor and writer to the MIT Initiative on the Digital Economy (IDE). The following is an edited (by humans!) version of their conversation.
Q: I’m surprised by how good Generative AI has gotten. Before we spoke, I sent you some questions, and the answers you sent back were so good, I assumed you had written them. Only later did I realize those answers were generated by an AI agent.
Yes, I was doing vibe analytics on vibe analytics! I dropped your questions in along with my prompt: “Drawing on the tone of vibe analytics here, answer these questions in my voice.”
Q: Before we get too far ahead of ourselves, what is vibe analytics?
Vibe analytics is all about getting more value from data. In the past, we used regression analysis and decision trees. Now we need a different way of engaging with data.
You can train AI for effect, you can train it for affect, and you can weight them. Tonality and affect matter. It’s not just how do we optimize around quantifiable parameters. It’s also, what flavor or emphasis do we want?
We’re entering an era where the challenge for talented knowledge workers and professionals is not just a question of how we do a more rigorous and diligent job of applying formal methods and methodologies to generate a desired output, answer or outcome. Instead, it’s what can we improvise? How can we play with possibilities and probabilities to give us greater insights and understanding around the choices that matter for what we ultimately want that answer, outcome or output to mean?
It’s like the difference between going to Julliard to play classical music and heading uptown to play jazz. Maybe for you it’s not enough to be able to follow the score really well. You also want to be able to play and engage.
Q: What are the benefits?
The issue is, how do you use AI models to get more value from human beings? So one of the things that every organization that takes vibe analytics seriously is going to get is a more engaged knowledge workforce.
This is an opportunity to allow, encourage and incent people to engage in a way that they can literally generate actionable insights that our legacy analytics methods and methodology will not generate or facilitate.
Q: Is vibe analytics still theoretical, or are there real-world examples?
This is not purely hypothetical. We’re vibe analyzing in real life. For example, I organize prompt-a-thons as a side hustle; they’re like hackathons for prompts. Organizations run them to figure out how to use prompts to get greater value from their data in conjunction with LLMs. I wanted to play a computer-security guy’s data to understand the battlespace for a prompt-a-thon. So, I asked my GenAI model some questions, got some responses and sent them to him. “Here are my prompts,” I told him. “Are these responses interesting?”
He replied that they were kind of interesting but what he really needs to know are X, Y and Z. So I took his email and dropped it in to the agent, making his response the new prompt. When I sent him the new response from the agent, this time he replied, “Now that’s interesting. I hadn’t even thought about that.” And all I did was copy and paste.
Q: Any other real-world examples?
Yes, I’m working with an organization that has call centers and contact centers, and they use all the classic metrics: NPS [net promoter score], customer satisfaction, first-call resolution, etc.
I’ve told them we could drop in the call transcripts and basically interrogate the responses: What are people unhappiest about? And when are the support people unhappier than the people calling in? Because these tools are fantastic for sentiment analysis. Then you can ask, how can we link sentiment analysis with NPS?
Q: But what about hallucinations? They seem to be a feature of GenAI, not a bug. How can we ever trust or depend on AI?
Wait! You don’t have to depend on it. It’s an input. How much you want to depend on that input is up to you. We want humans to be the good bottleneck, applying judgment, critical insight, taste, even chops. With respect to the memory of Ronald Reagan, this is a case of “trust but verify” cubed.
Further, I’d argue that an AI hallucination is like a Netflix docudrama based on a true story. It’s not a hallucination as if you’ve taken LSD; it’s a simulation. The agent decided to weight the data this way instead of that way. So, you need to review it. You need to rigorously and ruthlessly test it. You need to decide.
Q: If someone’s interested in exploring vibe analytics, how can they get started?
For one, you can do a prompt-a-thon. Get unstructured datasets, and instead of analyzing them, play with them. Ask them unusual questions.
Or you could start with either structured SQL data or unstructured data and then ask affective questions. Not just effective questions; affective. How do people feel?
Feel may be a four-letter word, but it’s a four-letter word that matters.
Want to collaborate on analytics with MIT and the IDE? Sign up to work with MIT grad students on fall semester projects thru the MIT Analytics Lab.
