The semiotics of UX and product design
Context is everything.
“Wherever I am, it’s here.”
—My dad
That’s the response I always get whenever I call my dad. The Yogi Berra reincarnate that he is, he’s full of mixed metaphor aphorisms that don’t make sense until they do. The wisdom in this one is that, as much as we want things to be objective and empirical, we are all locked in our own bodies—our own here and now. We each are always “here.”
This is the root of “context.” It’s a word I hear tossed about here and there in my UX work, but not nearly as much as it should be. We hear about being user-centered, but what does that really mean? Outside of the Cult of UX, does this word, “user-centered,” really make sense other than superficially? How is one supposed to act on it? What decisions does it help us make? What outcomes does it help us aim for?
The context about context
[T]he interrelated conditions in which something exists or occurs.
Such ontological!
“Context” literally means putting something in the reality of a given situation. In other words, here relative to who. But whose here are we talking about?
Enter semiotics
You’ve probably never heard of it. It’s an obscure science with roots in ancient philosophy, pops up as discussion here and there in the early modern period, and finally starts taking off around the turn of the 20th century.
More definitions:
Semiotics is the study of sign processes (semiosis), which are any activity, conduct, or process that involves signs, where a sign is defined as anything that communicates a meaning that is not the sign itself to the sign’s interpreter.
Ok, semiotics is the study of semiosis. What’s semiosis, then?
Semiosis, or sign process, is any form of activity, conduct, or process that involves signs, including the production of meaning. A sign is anything that communicates a meaning, that is not the sign itself, to the interpreter of the sign.
A sign is anything that is not the sign itself? Whaaa…?!
Basically, it’s a triangulation of “things.” (Thing in quotes, because it’s a rather loose idea of thingy-ness. Hi there, Plato!)
Picture for a thousand words:
- There’s a real-world cat. It exists outside of your brain. Empirically. Objectively.
- There’s also what we call the cat. It is not the cat; it’s something some human made up to be able to talk about said real-world cat. It’s not very practical to carry a cat around all the time just to have it handy to point at whenever you want to talk about it.
But whenever you do talk about that particular cat, there’s something amazing that goes on in the mind of whomever you’re talking to. Some mental picture of Yojo the cat pops into the listener’s mind. Involuntarily. A third “thing.”
Still more amazing: once you know that Yojo = that cat, you can’t un-know it. The association is there, always. (Unless you get amnesia, of course. We’ll just assume we’re talking about normal brain function from here on out.)
There is no immutable law in the universe that dictates that Yojo = a cat. The connection is made up. It’s not objective, empirical. It’s purely subjective. That is, an interpreter is involved.
As soon as the connection between Yojo and the real-world cat is made, a third “thing” appears on the scene: the interpreter creates an interpretant, which is fancy Latin for, more or less, concept. (Let’s just assume for simplicity’s sake that an interpreter is always human. For UX purposes, a user is typically a human, after all.)
To summarize. Three “things”:
- Some thing [ “First,” i.e. a symbol] that your mind takes as “pointing” to
- Another thing [“Second,” i.e. an object] that exists in the real world and together with the first thing invokes
- A thing [“Third,” i.e. a concept] in your mind. It’s all in your head.
Words as tools
The thing is, a word is itself a “thing.” As soon as you know that I know that Yojo = a cat, we can always talk about that cat. That word becomes a very useful tool. I, you, everyone can take it around everywhere we go and pull it out whenever we want to.
Only, that third “thing”—the interpretant—is an existing thing, too. Empirically.
It can exist in the mind of other people, too. As long as we both interpret that the First represents the Second, the word becomes a useful tool.
That is the act of semiosis.
Now, just for fun, abstract this act to everything you think you know. Every word you have in your vocabulary. Every thought that becomes a thing itself, and can also point to another thing or even another thought-thing, and so on. Triangles upon triangles upon triangles.
Everything is, subjectively speaking, context.
As if it wasn’t complicated enough.
Let’s play a game
Which of these is a dog?
Now, which is a hound?
What this shows us is that dog can “point” our minds to two different things. One First, two Seconds, two different Thirds. A homophone.
Equally, dog and hound can both point our minds to a single thing. Two Firsts, one Second, one Third. A synonym.
Finally, my mental picture of a dog and your mental picture of a dog may be different. One First, two Seconds, two Thirds. Or maybe even one first, one Second, two Thirds.
It’s a wonder we can communicate at all.
So what? How does this affect me?
Context.
Why are our mental images of a dog different?
Because my mental image of a dog stems from my experiences with dogs—probably influenced greatly by my first dog. The same goes for you.
But we agree — consciously or not — for the sake of agreement that dog = a general concept of a doggish thing. A non-cat. A non-bird. A non-mouse. You get the picture.
Back to “here.”
My here is not your here. Never was, never will be. Physical space, coordinates on the globe, whatever you map it with—we cannot occupy the same place at the same time. Even if we could, your mind and my mind can’t. We both come with different baggage.
On thinking machines
I once interviewed for a job at a place that was building voice apps for Amazon’s Echo. The business goal behind this and similar devices is ubiquity — put them everywhere and humans should be able to get any question they have answered. Right?
Turns out, inanimate objects — even “smart” ones — tend to be pretty bad at semiosis. Go figure. They really don’t have a sense of here.
The hiring manager for this position was keen to find someone with a background in semiotics because of precisely this conundrum.
This was the thinking behind the Echo:
Put an Echo in hotel rooms — all of them. Everywhere.
When a guest asks, “What’s the weather?” the gadget does pretty well at figuring out what its IP address is and identifying by that where it is in the world. It then requests the weather data from a weather service API for that location.
But if the guest says, “I’m out of toilet paper.” The gadget has a hard time.
First, it has to interpret from the statement that an imperative (command) is being made.
Second, it has to determine its location relative to the speaker and to their intended audience.
Third, it has to send a request for more TP.
“Oh, old dude’s on the pot and ran out of toilet paper! Help! Or come take a picture!”
The hiring manager needed someone to architect all of this to happen and turn it into something developers could make.
“See,” the hiring manager said, “programmers don’t really think about context when they’re coding.” I’m not sure about the universality of that statement, but the aphorism holds true: a computer is only as smart as its stupidest programmer. (To be fair, it’s not just programmers.)
A computer — which is so very close to being a interpreter that we’re all very often convinced that it “thinks” — cannot think anything that it hasn’t been programmed to do. (Yeah, I know, AI and self-programming computers — but society’s not quite there yet.)
Computers can certainly make associations. They’ve learned via AI to assign labels to entities in an image, telling us to greater or less accuracy what the picture contains. It’s not half bad when it comes to the objective stuff like that. Even an Echo finding its location via its IP address is an objective process.
Yet computers struggle with subjectivity — interpreting whose where (and by extension, why). It can’t put itself in another person’s shoes. It can’t come to a tacit agreement with an individual human that, yeah, let’s call that cat “Yojo.” You can tell it to call it that and it will, but it won’t agree to. It’s not a relationship of equals.
For now at least, a computer is still just a tool. It is a “thing,” (First, Second, or Third), but not quite an interpreter. For all its artificial intelligence, it still needs a human. A user.
The semiotics of products and services
What we build as humans all become tools. We use them to do work — to change our environment. Our here. Our context.
Products are only useful, even usable, in context.
This is a major thing that separates us from much of the rest of the animal world. (Not all of it, we’re finding out slowly, but darn near all of it.) Tool usage is one of the first things paleo-anthropologists look for when they find the remains of an ancient hominid. Tool usage is associated with self-awareness, i.e., contextual understanding. Tool-making = human. No tool-making = animal.
We humans build or seek out new tools when we’re faced with a problem we need to overcome.
We don’t need a drill. No, we don’t even need a hole. We want to hang a picture. Because our judgy-judge of a mother-in-law is coming to visit next week. And my wife’s freaking out!
Who? Where? Why? When? More why?
Context.
The success and fate of companies hang on context
Really, we should be focusing more on context when we say, “user-centered.”
Knowing the who and the context gives us a goal—an outcome. The raison d’etre of user experience design making a meaningful change in the context of a who. We can act on context. I’m not sure anyone knows how to act on “user-centric.”
By who and context, I’m not talking personas, though they are a valuable step. But they’re just an output — a deliverable like wireframes and mock-ups. Or they’re the result of an input — an activity like affinity mapping.
This is an area where, quite bluntly, UX simply needs to grow up. Let’s not reduce ourselves (or let ourselves be reduced) to these mere outputs.
What products need in order to be truly innovative is not more outputs. It’s not more features. It’s not more requirements.
It’s about making a meaningful change to a human’s context.
Wherever they are, it’s always here to them.
