I get around 60 non-marketing emails into my inbox a day and have to respond to at least 30 of those. If you send me an email, you’ll find that I’m much more likely to respond if your email is three sentences or less. Nothing good ever happens after the third sentence.

In fact, the most successful people are known for their brief emails. Here are some tips to keep your emails to three sentences or less:

When you’re asking for something.

Make it easy for people to help you by asking for specific things.

1. Context: Say why you need something.
2. Objective: Say what your goal state is, and how they know their contribution was meaningful.
3. Actionable steps: Say exactly what you need them to do. Make this bold.

When you did something and need to tell people.

Help people understand what you did, why you did it, and how it impacts them.

1. Context: Say what the thing you did is relevant for and why it was necessary.
2. Content: Say what you did.
3. Conclusion: Say what the impact of what you did to the recipient is.

When you have to provide constructive criticism.

Good people with the best intentions sometimes do bad things, or good things poorly.

1. Context: Say what happened and what the recipient did.
2. Effect: Say what the negative effect of their action was.
3. Actionable steps: Say how they could have done better.

Always assume good intentions. People make mistakes not out of malice and seldom out of incompetence, but often out of ignorance of their actions impact.

When you’re planning on doing something.

You are planning something with uncertain outcome and need people to be on your side.

1. Context: Say why you need to do something and state assumptions.
2. Hypothesis: Say what you will do and what you expect the effect of what you’re trying to do to be.
3. Goals: Say how you know that what you are trying to do is successful.

When you’re giving praise.

Forget brevity, never limit the praise you give.

Finally…

1. Make the subject line your spokesperson. Make sure it’s relevant and searchable.
2. You can omit context if you’re replying to an email if everybody understands and is aligned on the context. If not, provide context, even if it seems repetitive.
3. A gif says more than a thousand words. But it’s not always clear which words those are.

AWS Lambda is a service that lets you execute a single function on the AWS cloud, and only pay for the actual execution time. This is tremendously helpful for computation intensive tasks like cropping and converting images uploaded by users — you can trigger an AWS Lambda function with a file in S3, and it will dutifully perform its task independently of the rest of your architecture. It doesn’t matter whether you’re converting ten thousand pictures an hour or ten every month, there is absolutely no effort in scaling, and no architectorial differences.

That makes Lambda incredibly appealing for a lot of distributed computation tasks. However it can be a bit of a pain to set up: you have to bundle all of your dependencies into a single zip file along with your code. If you’ve ever tried that for a complex machine learning environment involving numpy and sklearn, you will have already experienced the torment and misery this will bring upon you.

Here is how to do it while maintaining your sanity.

Setting up a Machine Learning Environment on EC2

First, create an EC2 instance using Amazon Linux and login to that. There we’ll install all of our dependencies.

Remember Fortran? Yeah, we need it.

sudo yum -y update
sudo yum -y upgrade
sudo yum -y groupinstall “Development Tools”
sudo yum -y install blas
sudo yum -y install lapack
sudo yum -y install atlas-sse3-devel
sudo yum install python27-devel python27-pip gcc

Scikit-Learn won’t compile on less than 1GB RAM. If you’re using a free micro instance, create a swap file:

sudo dd if=/dev/zero of=/swapfile bs=1024 count=1500000
sudo mkswap /swapfile
sudo chmod 0600 /swapfile
sudo swapon /swapfile

Okay, let’s create a virtual environment and install everything we need:

virtualenv ~/stack 
source ~/stack/bin/activate
sudo ~/$VIRTUAL_ENV/bin/pip2.7 install numpy
sudo ~/$VIRTUAL_ENV/bin/pip2.7 install scipy
sudo ~/$VIRTUAL_ENV/bin/pip2.7 install pandas
sudo ~/$VIRTUAL_ENV/bin/pip2.7 install sklearn

Another challenge is that Lambda has a limit of 50MB for zipped files including all dependencies. Let’s use some dirty tricks to bring our bundle size down. We can use strip to remove everything from binaries we probably won’t need in both lib and lib64:

find “$VIRTUAL_ENV/lib*/python2.7/site-packages/” -name “*.so” | xargs strip

Finally, let’s bundle up all of our modules in ~/lambda.zip:

pushd $VIRTUAL_ENV/lib/python2.7/site-packages/
zip -r -9 -q ~/lambda.zip *
popd
pushd $VIRTUAL_ENV/lib64/python2.7/site-packages/
zip -r -9 -q ~/lambda.zip *
popd

Lambda will be looking for shared libraries in /var/task/lib, so let’s put everything we need into a lib folder:

cp /usr/lib64/atlas-sse3/liblapack.so.3 lib/.
cp /usr/lib64/atlas-sse3/libptf77blas.so.3 lib/.
cp /usr/lib64/atlas-sse3/libf77blas.so.3 lib/.
cp /usr/lib64/atlas-sse3/libptcblas.so.3 lib/.
cp /usr/lib64/atlas-sse3/libcblas.so.3 lib/.
cp /usr/lib64/atlas-sse3/libatlas.so.3 lib/.
cp /usr/lib64/atlas-sse3/libptf77blas.so.3 lib/.
cp /usr/lib64/libgfortran.so.3 lib/.
cp /usr/lib64/libquadmath.so.0 lib/.
cp /usr/lib64/libquadmath.2so.0 lib/.

And add this to our bundle:

zip -r -9 -q ~/lambda.zip lib/

Great! Back on your local machine, get the zip file from EC2:

scp -i pemfile.pem ec2-user@100.110.120.130:~/lambda.zip lambda.zip

Now you can add your lambda_handler.py. If you have a more complex program, I recommend putting that into a module that you import in lambda_handler.py. Let’s remove all .pyc files first though:

find my_module/ -name ‘*.pyc’ -delete
zip -9 lambda.zip lambda_handler.py
zip -9r lambda.zip my_module/

Because it’s so big, we need to upload it to an S3 bucket first before updating lambda:

aws s3 cp lambda.zip s3://my_bucket/lambda.zip
aws lambda update-function-code --region us-east-1 --function-name lambda_function --s3-bucket my_bucket --s3-key lambda.zip

This should take care of most obstacles you are going to run into. Did this work for you? Let me know in the comments.

Artificial Intelligence is on the rise, but (for a change) I’m not talking about the great AI panic of 2015. I’m talking about everyday AI that’s build into vacuum cleaner robots, Siri, web analytics, and your email. Everybody is fascinated by the technology, and it seems like AI is making huge breakthroughs.

I’m sitting at the source of cutting edge AI development (pun intended), and as in any other discipline, 95% of the work has very little to do with coming up with amazing new algorithms to outsmart humans. But more importantly, 95% of how well these AI systems perform also has nothing to do with smarter algorithms.

To explain why, here’s a little story.

Back in 2005, I took an applied machine learning class in college. We had a little competition to classify smileys drawn on surveys into various categories like “happy”, “sad”, “angry”, and “confused”. The better exemplars looked like that:

The more challenging smileys were wearing hats, eye-patches, tongues sticking out, ears, nose piercings, Marge Simpson wigs, and Braveheart war paints. I am not kidding.

Each smiley was 200⨉200 pixels large, which meant a solid 40,000 input dimensions. The various teams in the class immediately brought out the heavy artillery: support vector machines, recurrent neural networks, subspace ensemble classifiers… Their performance was well below satisfactory.

Disappointed with the absence of a quick win, our team tried something which we would now call “feature engineering”. We wrote a little script that would start at the bottom edge of the image and draw a line up until it hit the first black pixel. Then we compared the length of three of those lines, and also the ratio of black pixels in the upper left quadrants to compared to the upper right quadrant:

In the end, we ended up with only three values. We built a maximally dumb decision tree with these three values and outperformed all other teams by a large margin.

However, what I now call “feature engineering” and do every day was called “cheating” back in the class and got us disqualified, which lead me to this tendentious observation:

Manuel Ebert on Twitter

What successful #MachineLearning is made of vs. what academics think it's made of: pic.twitter.com/Ypnf0d0o7w

The dirty secret is that this kind of feature engineering — using your human intuition, domain knowledge, and reckless shortcuts to reduce 40,000 input dimensions to three— is exactly what makes most of AI applications work. The other thing, of course, is having enough, good, well-curated data.

This is precisely the reason Facebook and Google are giving away all of their machine learning and AI infrastructure for free. Algorithms, on their own, are worthless.

The real value is in what you feed these algorithms, and companies keeping a tight lid on both their data and their feature engineering. The real job of many AI engineers is using their experience massaging data into something more digestible for the algorithms.

If you had a similar feature engineering win, please share your story!

Many people have written about the toxicity of a culture where being busy is seen as an advantage. I couldn’t agree more, but underneath all the stroking-your-ego, appearing-important, and being-hard-at-it is a mystery that honestly confused the shit out of me.

Why do smart people confuse the cost of something with its value?

If someone boasts that they just pulled an all-nighter to finish a project, this is what I hear:

“Dude, last night I paid $80 for a pizza!”

I don’t need to know. I don’t even want to know. What I want to know is whether the pizza was any good. And then, maybe, whether it was worth $80.

You wouldn’t boast about how much you money you spent on dinner (unless you’re a real douche), why boast about how much time you spent on getting stuff done?

Time is your cost. Time is what you pay for creating value, doing your work, getting things done. And even if you don’t boast about your all-nighters, chances are you feel guilty if you don’t work as much as your co-workers. They might look at you funny if you go home at 3pm and declare that you’re done for today. It all boils down to the thinking that you didn’t “spend” enough on the work you did.

I believe this confusion comes from the fact that time is the only currency we all have in common. Designers are bad at judging the value of the work of engineers, engineers are comically bad at judging the value of the work done by managers, high-level managers are frighteningly bad at judging the value of work of low-paid staffers and the building cleaners. All we have to talk about is time.

Time is the only natural resource equally distributed amongst every single person on earth, and so we all know how spending time feels like. We all make the same amount of time — we have a fixed stipend of 24 hours per day which is ours to spend.

The only way out of the cultural trap of being busy is to stop talking about how much time you spent on doing things. Of course you will need to talk about time when planning your tasks — it’s a resource that needs to be allocated like any other. And it often makes sense to look back to figure out how to spend less time on what you did.

Choose being productive over being busy.
Choose talking about value over talking about cost.
Choose improving over complaining.

Don’t ever complain about how much time you spent on something, or worse, brag about it. Instead, help your peers understand the value of your work.

Unicode confuses a lot of programmers, but nobody wants to admit it. What’s the relationship between Unicode and UTF-8? Where should I use UTF-8 and where Unicode? Why do we even need the two?

Armin Ronacher has a very exhaustive answer to this question. But really, it’s simple. Let’s talk about Machine Translation for a second.

There are basically two ways of doing machine translation — that is, automatic translations from say French to German. One is a direct translation: you translate every sentence from French directly to its equivalent in German. The other method of machine translation is using something called an Interlingua, a hypothetical language that captures the meaning of a sentence without being concerned about the words to use, word order, grammar, and so on.

Back to Unicode: UTF-8 is German. Latin-1 is French. ASCII is Spanish. Unicode is — you guessed it — interlingua. The actual languages are called encodings. They are a specific way of representing the meaning of a symbol. Unicode isn’t concerned with how exactly atrocious characters like ö or ñ or ‽ or ✈︎ are represented on a storage medium — which bits in a byte should be 1 and which 0 and how many bytes to start with. Unicode cares about the symbol and nothing but the symbol.

Here’s an error that every python programmer alive has seen:

‘ascii’ codec can’t decode byte 0xc3 in position 0: ordinal not in range(128)

This and many related errors, are easy to explain. You are trying to say something in Spanish, but there’s simply no word for it. In this case, the letter Ü doesn’t exist in ASCII.

Another common error is the one you might get from reading a file that has a different encoding than you think it has. It’s like reading a book that you assume to be written in French, but the words won’t make sense — because it’s actually written in German.

Here’s a simple rule that all programmers should follow:

If it’s on disk, make it UTF-8.

If it’s flowing through your program, make it Unicode.

Specifically, as soon as you load text from a file or a database, at the earliest point, convert it to your interlingua, Unicode. Within your software, all text should always be assumed to be Unicode. Only when text leaves your software, when it’s written on disk or sent through an API or stored in a database, encode it in UTF-8 (or whatever the appropriate encoding for your case is).

Prosper.

What Design can learn from Philosophy and Neuroscience

This is a three-part blog post. The first part will be about philosophical discipline of phenomenology and what that has to do with how we use tools. To keep you from falling asleep, there will also be Nazis and Street-Fighter references. In the second part, I’ll present findings from neuroscience that basically say the philosophers got it right. Finally, I will show you that this means that we’re all, in fact, cyborgs, and what that has to do how we use and hence should design tools, interfaces, and wearables.

If you rather listen to things than read them, here’s a video of a talk I gave on the subject:

Phenomenology: let’s sip whiskey and talk about feelings.

But first, let me take you to France. Paris, the city of love. In 1942. The city is under Nazi occupation. It’s night, long past the curfew, and outside SS patrols march down dark alleys with cobblestone streets. Inside, a man sat in front of a typewriter to work on the manuscript for his philosophical opus magnum “L’Être et le Néant” — “Being and Nothingness”.

Sartre and his lifelong partner Simone de Beauvoir would later become the most talked about celebrity couple in post-war France — think the Brangelina of the 50s, except with a lot more juicy affairs on the side. In 1949, de Beauvoir would publish “The Second Sex”, kickstarting second-wave feminism in Europe, and in 1964 Sartre would be awarded — and refuse — the Nobel Prize for his literary work.

“Hell is other people’s code” — JP Sartre. Well, something along these lines anyway.

But that night in 1942, he was tired. As he was typing on the manuscript, he tried to focus on the ideas, the concepts, the world the words he was typing were creating. But then his eyes became sore and weary, and the letters started to blur. He noticed how his attention shifted from the concepts to the letters that carry them. And then soon to his tired fingers that created those letters. Finally he couldn’t think about anything but his sore eyes and had trouble keeping them open.

Most ordinary people would just have decided to call it a night. But Sartre was anything but ordinary. He was a philosopher. Particularly, he was a phenomenologist. Phenomenology is the philosophical study of experiences, or more correctly, the study of structures of experiences. In a way, it’s a scientific discipline, a way of describing the world and theorising about it. But as opposed to physics or psychology, phenomenology is a first-person approach: you’re analysing, deconstructing and describing your own experiences and consciousness. Phenomenologists don’t study things for what they are, but for how they appear to us.

Sartre, being a phenomenologist, was quick to notice a certain pattern in his experiences. He noticed how his attention shifts from one thing to another — the ideas, the words, his fingers, his eyes. What’s interesting here is that his attention shifts from the subject of perception to the medium of perception. First he perceives the ideas through the words, then the typed words themselves become centre of his attention. Finally, his own eyes become the subject of his experience, rather than the medium, the method of experiencing.

Heidegger: Philosophical Hadouken Punch

His book ”Being and Nothingness” is of course a play on the title of Martin Heidegger’s seminal work “Being and Time” — “Sein und Zeit” in German, which Sartre read in a German prison camp when he was a prisoner of war in 1941. Being and Time was a philosophical Hadouken punch, blasting away 2500 years of philosophy that was in its way to answer questions like “Why is there something instead of nothing?”.

Spoiler alert: we still don’t know.

The book was written with a brutal accuracy that is only possible in German where you can make up words like Ersatzzeitgeist and Auskunftspflicht — which loosely translates to “the obligation to inform authorities about any changes in permanent residential address or marital status in a timely fashion”.

Needless to say, these linguistic “features” render Being and Time almost unreadable in any other language than German.

Heidegger introduces a distinction that is incredibly important in all of design, although we don’t call it like that: presence-at-hand and readiness-to-hand. Or, in German: vorhandensein and zuhandensein. What’s that? In his famous example, he considers a hammer. Alright, so most philosophers probably never used a hammer in their whole lives, which makes this example somewhat amusing.

When the hammer is lying on a table, we can look at it, analyse it, describe it based on its constituents — wooden handle, heavy cast iron head — maybe even infer its function and use from the way it’s shaped. The hammer is present-at-hand.

Magic happens when we pick up the hammer:

As soon as the hammer is in my hand, when I use the hammer to drive a nail into the wall, I do not think about which angle to hold my hand to manipulate the hammer — I think about how to hold the hammer to manipulate the nail. The hammer becomes almost invisible to me, it transitions from being an object in the world to a way of interacting with the world through it. Same thing happens when I use a pen to write or a computer mouse to point to things. I won’t think about how to move my hand on the trackpad, I think about how to move the cursor on the screen. When driving a car, I don’t think about how to twist my elbows to turn the steering wheel, I just think about where I want to go. The tool becomes part of my body in that way; I interact with the world through the tool now.

That’s what Heidegger calls ready-to-hand. Only when the hammer is ready-to-hand it becomes a means of action, rather than a subject of it, only then can we achieve some fluidity of using it. When the hammer breaks, it immediately loses its readiness-to-hand and becomes merely present-at-hand, our attention will immediately shift back from what we’re trying to hammer to the hammer itself.

What we’ve seen so far two examples here of how our experience of a tool can shift from the tool itself to what we’re perceiving through the tool (In Sartre’s case) or (in Heidegger’s example) manipulating through the tool and back.

The interesting part is this transition. Moments ago the hammer was a distinct thing, an ontologically discrete entity lying there in the outside world. And as soon as I pick it up, it becomes part of me. I use it as naturally as my own hands. The boundary between “me” and the “world” shifts — now the hammer is part of me, and ceases to be part of the “outside” world.

Remember Affordances?

Designers love to talk about affordances.The flat surface of a chair affords sitting on it. The little shadow under a button affords clicking on it. Affordances are action possibilities. Here’s what James Gibson had to say about hammers:

“When in use, a tool is a sort of extension of the hand, almost an attachment to it or part of the user’s own body, and thus no longer a part of the environment of the user. But when not in use the tool is simply a detached object of the environment, graspable and portable, to be sure, but nevertheless external to the observer. This capacity to attach something to the body suggests that the boundary between the animal and the environment is not fixed at the surface of the skin but can shift. More generally it suggests that the absolute duality of ‘objective’ and ‘subjective’ is false. When we consider the affordances of things, we escape this philosophical dichotomy.”

Before Gibson, the mainstream view of perception was that there’s an outside object, the subjective observer, and some internal representation of the observer object. But that model can’t explain what happens when tools become extensions of our bodies.

Tackling this problem of perception was central to Gibson’s work when he came up with it in 1977, but has unfortunately got lost a bit when affordances entered mainstream design lingo.

So originally, affordances were meant to be an answer to this philosophical conundrum, not something to be lightly tossed at wireframes as a fancy way of saying “can you make this button pop a little more?”

I promised you cyborgs. Here’s a cliffhanger.

In this post, I demonstrated that when we inspect our own experiences of tool use, we can see that tools become part of our body. I also talked about some of the philosophical frameworks to explain that. But how do we know this is true and not just something some crackhead philosophers made up in their cozy leather armchairs on a particularly scotch-fuelled night?

Stay tuned for next week, where I’ll present some fantastic results from neuroscience studies that show what is going on in the brain while we use tools. Oh, and will also say that this means that we are, in fact, natural-born cyborgs.

“Artificial” Intelligence refers to the fact that in many obvious ways, a computer is not a brain. It doesn’t require oxygen or adenosine triphosphate, it executes one operation after another, and it’s neither wet nor gooey.

This implicitly (but rather bluntly) equates the brain with “Intelligence”. However, intelligence is not a tangible thing, it’s not an entity, it doesn’t exist in a vacuum, you can’t find it by dissecting a pickled brain. Intelligence is something we do, or more accurately, something we ascribe to actions and behaviours. There are no such things as artificial actions or behaviours. If something moves, it moves for real. If Google suggests a new route for you that avoids the traffic, it’s a real suggestion, not an artificial one.

Conversely, an action cannot be artificially or naturally intelligent. It is either intelligent, or it is not, and that depends on the context and the goals.

In that way, intelligence is like humour. Humour is not a little man sitting in your brain pushing the “joke” button every now and then — humour is if you intentionally say or do something funny. It’s funny if it makes you laugh. It is not artificially funny, it’s just funny.

It doesn’t make sense to talk about artificial intelligence.

We should talk about “machine intelligence” to signify that the action we deem intelligent (or not) was planned and executed by a machine. But the intelligence of the action is just as real.

Note: This article was featured as a guest post on the Kimono Blog.
Image credit: Black Rock City (based on this areal, CC-BY-NC-SA)

Different people have different ideas about their perfect vacation. Some day-dream of relaxing on white beaches and palm trees. Others seek the thrill of snowboarding down the rockies. And then there are those who really just want to run through the desert half-naked, entangled in EL wire and exploding things to toss anvils 100ft into the air.

That’s me. And the probably only place on earth you can do this is the Burning Man festival. Problem is, I completely failed to get a ticket in time this year, and in between a lot of travelling I woke up with the sudden realisation that there were only five days left to find a ticket. The ePlaya forum had plenty of people offering tickets at face-value, however the demand this year was so high that the average time until a ticket got sold was less than four minutes. I was still stuck in Edinburgh, UK; and there was no possible way I could monitor that page all day round to call dibs on the next ticket offered.

APIs and my Inner Nerd to the rescue.

I would personally find it unethical to write a bot to get a ticket for me, but at least the process of sitting on a computer and hitting the refresh button can be automated. The battle plan is to turn the HTML mess of the ePlaya phpBB board which only barely graduated from nested tables into clean data with Kimono and then notify my whenever that data changed.

First, let’s identify what we need in our API. Here’s what a section of the forum looks like:

We obviously need the title of the post since we’re ultimately only interested in tickets offered, not tickets needed. The date of the post might also be interesting, as are the number of replies (since we won’t care unless we’re the first). To select the date and replies, simply highlight the text by clicking and dragging your cursor over it the way you would do it in a text editor.

Opening the raw data view, we can now see the the data, the whole data, and nothing but the data:

{
  "Title": {
    "text": “Oh… my… god…”,
    "href": "https://eplaya.burningman.com/viewtopic.php?f=370&t=71750"
  },
  "Date": {
    "text": "Sat Aug 30, 2014 9:16 pm",
    "href": "https://eplaya.burningman.com/viewtopic.php?f=370&t=71750"
  },
  "Replies": "0"
}

Note how KimonoLabs neatly captures the URL to the individual thread, too. We’ll use that later.

Now, we can tell Kimono to send us an email whenever that data changes. However we’re

1. not interested in getting email every time somebody posts a reply to an old thread and
2. the lowest interval for automatic polling is 15 minutes.

So, let’s write a few lines of Python to deal with the data. We’ll use the requests module to fetch the API:

import requests
URL = "https://www.kimonolabs.com/api/6zzoaezg?apikey=<MY_API_KEY>"
response = requests.get(URL).json()
data = response['results']['collection1']

Now data will be a list of dictionaries like the one above. Next we need to need to find all new posts. Let’s have a closer look at the URLs in the title attribute:

https://eplaya.burningman.com/viewtopic.php?f=370&t=71750

Here, 370 is the id of the forum, and 71750 is the id of the topic. Before we run the script the first time, let’s create a file called latest_topic.txt and put 71750 inside. Back in Python, we load the file find find the newest topic, split every URL to find the topic id of the post, and if it’s newer than the newest topic from the last time we run the script, we remember this posts in a new_posts list. And while we’re on it, we also keep track of the maximum topic id we encounter so we can save it when we’re done finding new topics:

latest_topic = int(open('latest_topic.txt').read())
max_topic = latest_topic
new_posts = []

for post in data:
    topic = int(post['Title']['href'].split('t=')[-1])
    if topic > latest_topic:
      new_posts.append(post)
      max_topic = max(max_topic, topic)

with open('latest_topic.txt', 'w') as f:
    f.write(str(max_topic))

Great. Now let’s turn to our friends at Twilio for turning data into a concierge web notification service. Twill allows you to send text messages to any phone and takes less than 5 minutes to set up. Let’s write a short method that will take a post and send a text message to a number:

def send_message(post, number):
    URL = "https://api.twilio.com/2010-04-01/Accounts/<MY_ACCOUNT_NUMBER>/SMS/Messages.json"
    text = "{}: '{}' ({} replies) {}".format(
        post['Date']['text'],
        post['Title']['text'],
        post['Replies'],
        post['Title']['href']
    )
    params = {
       "From": "<MY_TWILIO_NUMBER>",
       "To": number, 
       "Body": text
    }
    requests.post(URL, data=params, auth=("<TWILIO_API_KEY>", "<TWILIO_API_SECRET>"))

The format operation turns the post into a string such as

"Sat Aug 30, 2014 9:16 pm: 'Oh... my... god...' (0 replies) https://eplaya.burningman.com/viewtopic.php?f=370&t=71750"

Finally, a simple POST request to the Twilio API is enough to send this string to a number. All that’s left to do is sending a text message for every post in new_posts. But wait… have you heard of the term “geospatial arbitrage”? Instead of notifying only myself, let’s send this message to friends in three different timezones in the US, the UK and Japan to maximise the chance of one of us being able to reply first:

for post in new_posts:
    for number in ('+1 415 123–4567', '+44 20 1234567' '+81 3 12345678'):
        send_message(post, number)

Done. If you’re on a free Twilio trial account, make sure all of the recipients’ numbers are verified. Last thing is to make sure this script runs, like, very often. Let’s save it as ticket_search.py and make an entry in our Crontab by typing crontab -e in our Terminal and adding a line

* * * * * python ticket_search.py

To make this script run once per minute.

Lean back.

At 7:04 PST in the morning, three phones on three different continents gently buzzed as part of our globally orchestrated concert of APIs and data, scripts and servers. I was the first one to tap the reply link, and sure enough:

Incredibly happy and exhausted, I paypal’ed the money to the seller, wrote a hand-letter postcard to say thanks, and started packing my bag for Burning Man.

You can get the full script here.

Three years ago I was working in a neuroscience lab in Barcelona, busy putting electrodes on people and teaching classes on cognitive systems. Today I design and write software for a living.

Of course back in science I wrote a lot of software — if you want to make any sense of 40 GB of brain scan data you’ll have to roll up your sleeves and write scripts to crunch those numbers, and I was always a good programmer. But it wasn’t until I quit my job (and possibly my future) in academia and started working for a small and ambitious start-up that I understood what being a software engineer — and more importantly, being in the business of software engineering — is really about. It’s not knowing more programming languages, libraries, algorithms, and design patterns. It’s a mindset.

That mindset would have made my work a lot easier if only I had known and adopted it before I started grad school. This is a note to my younger self, a list of things I learned, sometimes painfully, in the past three years.

1. Intelligence is overrated

When you’re young, being smart gets you a long way. You’re a big fish in a small pond. Doubly so if you have a knack for expressing yourself half-way eloquently. In fact, being intelligent and a smooth talker will get anyone through high-school and most of college without learning much at all (you’ll have to study for physics though. Can’t just talk an equation away.) — congratulations, you’re lucky. And also, very unlucky. Because while you were effortlessly rushing through school, picking things up as you were going, others had to learn what would be much more important later on: Diligence. Persistence. Networking. And probably some of the eight things further down the list.

Our society values intelligence beyond proportion. When I tell people that I used to work in neuroscience, the first response if often: “Wow, you must be super smart”. I’m not dumb, but I know a lot of people who are probably less intelligent than I am, but far better neuroscientists.

Intelligence is certainly still a door-opener. But it will never get the job done on its own. Diligence, rigour, a reliable network, and finally not being a dick are essential qualities of not just software engineering but any profession that’s outside the little bubble called grad school.

2. Take pride in your craft

That mantra may be over-used, but it still holds an important lesson for you, dear younger self: whatever you do, consider it to be an honourable craft. Nothing should ever be just a means to an end. We all love seeing our names on publications, but the actual craft is to come up with and invalidate dozens of hypotheses, to work with your subjects — whether human or floating in a test tube — and tend to their needs, to rigorously analyse your data and validate your statistics, and to start over again because at some point you will notice an embarrassingly stupid mistake you made earlier on. If you write software, that means planning your features, researching existing open source code, learning new paradigms and programming languages, fixing your bugs, refactoring code and maintaining it. If you take no pleasure in these steps and just consider them to be what has to get done in order to publish your paper or ship your product, then you will never become truly good at it. If you have no ambition to become truly good at your craft, then maybe being a scientist, or an engineer, or whatever you’re doing right now is a waste of your time.

A good sign that you’re honouring your craft is that you’re taking on pet projects: silly little projects that don’t necessarily serve any immediate need that you’re doing just for the project’s sake. Because you enjoy working on them. Interestingly, these seem to be fairly common in the software community — many products we use every day started as someone’s pet projects — but are much more rare in scientific circles. One of my favourite quotes comes from Konrad Lorenz:

“It is a good morning exercise for a research scientist to discard a pet hypothesis every day before breakfast.”

If that sounds stupid to you, maybe you shouldn’t be a research scientist.

3. Learn new tools

As a continuation of the last point: devote time to learning new tools. Not just to expanding your abstract knowledge, but to actually learn tools that will help you get things done. It will pay off soon enough.

A great way of learning new tools is with “pet projects” mentioned above. Every time you build something new, also build it in a new way. Remember, pet projects are about failing. You don’t invest much, you learn a bit, and if it doesn’t take off or you lose interest or you realise that the challenge was a little too much: no harm done. No ego hurt.

Great tools that I highly recommend learning if you’re in academia:

• Git and Github. Git helps you to manage your work and never worry about backups again, and there’s a ton of great code on Github so you don’t have to reinvent the wheel over and over again. Oh, and please do code reviews with your peers. Don’t ever use code to analyse data that no-one has read but you (I can’t even believe I even have to tell you this, younger self. You’ve always been a good programmer, but the mistakes I still make that would go unnoticed if it wasn’t for code reviews makes me believe that 30% of all results in science are probably bogus because of bugs).
• An illustration software: I personally prefer Inkscape, but the industry standard Adobe Illustrator or newcomer Sketch are just as good. Use it to post-process your plots and graphs; it’s often much easier than writing plotting directives in Matlab or matplotlib.
• Learn how to use your text and code editor efficiently. Sublime Text is a great editor with a much lower learning curve than Vim or Emacs. Learn the shortcuts. It will save you an enormous amount of time.
• Learn how to speak. Watch TED talks and pay attention how the more seasoned speakers can engage the audience for 15 minutes while telling a compelling story. Practice in front of a mirror. Your body and voice are tools, too.
• Knowing the fundamentals of Python, R, HTML and Javascript will get you a long way. If you’re no stranger to programming already, learn a new aspect or library. Play with computer vision. Natural language processing. Web scraping. Music synthesis. Robots!

The solutions you can see to a problem will always be limited by the tools you know. Learning new tools means looking at problems from other angles.

If you’re in college, I strongly suggest you set apart one day every week only for learning new tools. When you start doing your own research as a PhD: make that two days a week. You will save a lot of time in the long run and people will be astounded by your efficiency. If that sounds like a lot and you think that you don’t have the time and there’s too much pressure to get other things done, talk to your older peers and ask them for advice on what to really spend your time on.

4. Be a stake-holder and make your agenda known.

It’s a common assumption that your supervisor or CEO will always act in the institute’s or company’s best interest; that’s her job.

But, neither a company nor a lab is a conscious entity, and as such has no intrinsic interest. When we speak about a company’s best interest, we actually mean the best interest of the stakeholders. The real question now is: who does your CEO or supervisor think those stakeholders are, and how important are their interests?

If your boss thinks she or he is the only stakeholder (get as many publications as possible; aim for a quick and profitable exit): get out as fast as you can. You will be thrown under the bus. Who else? Your investors or grant givers? The employees? Students? Humanity? The point is: find out as soon as possible. If you’re not seen as a stakeholder, get out. As much as you may love your work, it’ll be a one-sided, abusive relationship.

5. Shipping it

“Shipping it” has become a very fashionable term in tech. It means getting your product out of your warehouse and to the consumer. But more than an action, it is a mentality. It means that your work is worthless until it ends up in the hands of the consumer, and that this should always be your main goal.

In academia, most software I wrote had to work exactly once on exactly one system. Writing production-ready code that will work for half a million users is a completely different kind of animal, and when I started to write code professionally my work often fell short of that.

But it also means that there’s no point in iterating for years until you have the perfect product: make a small point, and get it across. Write the simplest paper you can possibly get accepted. Worry about making a more complex study later. Get the basics right quickly, and get them out there as soon as possible. Just ship it.

6. Know the 80/20 rule

The 80/20 rule basically states that it will take you 20% of a project’s time to achieve 80% of the desired effect, and then the remaining 80% of time to just get the last 20% right. It’s like driving from the suburbs to the city: in 20% of the time, you’ll cover 80% of the distance. But once you hit urban traffic, the last 20% of distance will take you much longer.

Why is that important to know? Because people constantly underestimate the time a project needs. Scientists and engineers are particularly prone to that. That’s partly a matter of experience: the more you know, the better you can anticipate what will go wrong later and what the funny edge cases will be that nobody thought of when you started out.

If you don’t have that experience yet, just multiply the time you need for a project by 5, and expect to be “almost there” after a fifth of your estimated time.

7. You didn’t sell your soul.

I started my PhD for all the wrong reasons. One of them is what I now call “academic guilt”. I believed that if I didn’t pursue a PhD programme I would be wasting my talent. And I felt that I owed it to the people who went out of their way to support my academic career — professors and the people who paid for my scholarship — to do research. I didn’t. They may have invested in my academic future and may be disappointed that their investment didn’t pay off and produce a great scientist, but that’s their problem, not mine.

The same holds for any other job. People will always invest in you and it’s often in their best interest to do so. But that doesn’t mean they own your soul.

8. Leave your comfort zone.

Here’s how I look at the world:

There’s your comfort zone. You know every fish in that pond. You belong. You know how to deal with problems. Nothing new under the sun. If you want to learn something new and grow, you’ll have to leave your comfort zone. This is where learning starts. That’s where interesting things happen. That’s where you don’t immediately have a response to everything.

Of course, there’s also the point where you’re just overwhelmed. That’s the panic zone. That’s where you’ll black out. That’s where all you can do is to try to keep your head outside the water hope somebody will save you.

The sweet spot is right before your panic zone. That’s where the challenges are where you’ll learn the most, grow the most, and change the most. Go there.

“Forget safety.
Live where you fear to live.
Destroy your reputation.
Be notorious.”

- Rumi

9. Tame your monkey mind

Sit comfortably, close your eyes and just continue breathing normally. Focus on how the air you exhale through your nostrils feels on your skin above your upper lip. Nothing else. Just focus on that.

How long before your mind wanders off? Five minutes? Probably not. A minute? Good. Twenty seconds or less? Congratulations, you’re normal. Your mind is like a monkey and it will grasp whichever branch is closest. I would probably phrase that slightly differently in academic settings… the buzzword is associative thinking. Associative thinking is great if you want to do something creative, but it’s the killer of focus. Good news is: you can learn how to focus. There are a bajillion “productivity techniques” out there, but all of them just scratch on the surface. You don’t want pasta timers and distraction free writing software. You want to tame your monkey mind once and for all.

What works for me might be grossly different than what works for you. I get great results with meditating regularly (which has a number of other beneficial side effects), but even then there are so many different styles and traditions, and I can’t possibly recommend one that suits everybody. What I do recommend is keeping your mind in great shape, and taking that seriously. Think meditation is a waste of time? You go to the gym to pump up on your body. You should spent at least twice as much time on mental work-outs: Lose a few pounds of distracting thoughts. Improve your mental eye-sight. Strengthen your back to be able to keep your mind upright longer.

Thanks to Benedict, Judith, Gabriel, and Peter for feedback and discussion.