Defined by mathematician John Horton Conway, it plays out on a two dimensional grid for which each cell can be in one of two possible states. Starting from an initial grid configuration the system evolves at each unit step taking into account only the immediate preceding configuration. If for each cell we consider the eight surrounding cells as neighbors, the system transition is defines by four simple rules.

I was interested in exploring the visualization of such phenomenon with Blender. What follows are some experimental results.

Here the code if someone is interested. It’s a reusable script you can import and run directly in the Blender scripting interface. It defines the GOL logic and breaks down the porting of the GOL grid to Blender in two customizable components:

• Generator —responsible for the generation of the Blender object that will be mapped to a cell in the original GOL grid. A generator is exactly used to build the initial Blender grid with the preferred mesh (examples: cube, sphere, monkey).
• Updater — defines update behavior for a Blender object based on GOL grid value. Should specify what happens to an object depending if the corresponding grid value is 0 or 1 (examples: scale, hide).

The rest is just helper code to register the updater handler such that a frame change causes an update of the GOL grid (possibly including keyframing).
I suggest to delete the handler once you obtain a result you like, in such a way that additional frame changes won’t trigger again the update and screw up you results.

3D Version

I experimented also with a three dimensional grid which used the same set of rules while extending the count of neighbors to the new dimension.

I am planning to look more into the 3D concept to find more stable configurations. An additional interesting improvement would be to have a non-constrained grid, meaning an automaton that starting from initial configuration can grow indefinitely in space. For such approach I have to reformulate my current code logic, and probably experiment first with some alternative to Blender, cause the guy here consumed a lot of resources even for these simply renders, so any suggestions to this regard is more than welcome!

Digression

Playing around these cellular automata visualizations prompted in my mind concepts like causality/teleology and dust theory. All system states are deterministically defined, but can be computed only based on the state at the previous time-step. With the scaling/shrinking function in place, the system behaves in exactly the same way it would do with other type of updating function, what change is the behavior of cells with regards to the system state. A cell is instantly shrank by external forces to reflect a state equal to zero, but then evolves with the illusion of free will to a renewed state of full size. Practically speaking this growth is nothing more than keyframing filling. It is not the story of a cell that defines its future, but the future of the system that defines the story of all cells life.

Conway’s Game Of Life in Blender was originally published in Towards Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.

This is a light and speculative entry where I brainstorm personal ideas about the future of lifelogging (imminent or sci-fi future, depending on your optimism level). I will focus mainly on the practical aspects of such activity, while largely ignoring ethical, moral and psychological issues. Even if speculative, I tried to include as many “good” references as possible, and I as well hope that ideas here expressed can themselves inspire you or simply give you food for thought.

I will start with a questionable distinction between two otherwise intertwined, blurry-defined movements: quantified-self and life-logging. Both are about pretty much the same main point: the tracking of data related and generated by an individual to improve the life of such individual. The goal of our idealized individual is to collect (or better, have it collected) as much data as possible about its life, while making the best use of it, with the final and indisputable goal of… being better! Better health, better performances, better person, better mood, better potential for the future — everything one cares about, one wants it better! Makes perfect sense to me.

“Quantified-Self is just Data Science… for yourself”

While from one side with quantified-self we have the more common, “traditional” variety of data (heart-rate, sleep, calories, mood), on the other, with life-logging, we have the more controversial entries related to our perception and senses, media like text, audio, video — smell too maybe? — together with other even more peculiar forms of data.

Someone might say that life-logging is really just about collecting data, or that quantified-self is more about numbers, but everything is numbers, and if you collect data is because you want to do something with it (at least that’s the initial good resolution). If you don’t, then the fault it’s of humanity for not giving you good enough tools or motivation, but the initial goal was there: you wanted/hoped to improve your life with data.

Quantified Self (just a brief intro)

I have a Fitbit, and that’s already considered by some as a pretty hardcore device: automatic sleep and heartbeat monitoring using flashing green light? And it shows the time — kind of — when it recognizes that you turned the wrist to check it ?! Astounding!
But then you start reading and chatting around this movement, the quantified-self, start getting interested and fascinated by it, get more and more acquainted with concepts like wearables, continuous monitoring, self-tracking and optimization, cyborgs and transhumanism. You recognize that fancy Fitbit to be just the commercial surface of a deep and variegated ecosystem of choices and products. What else have we there?

Start with basic activities monitoring, for which Fitbit is an example: heartbeat (or the even more important heart-rate-variability), steps, distance covered, activity levels, burned calories, sleep tracking.
Then we can move to more intimate body analysis: genome testing and sequencing, analysis of blood and guts (microbiome), saliva and stool testing. We already approach scenario for which a “wearable” is not enough, tests have to be conducted in a dedicated laboratory, but in a seamless easy process, which is luckily so far away from the tediousness of old fashion hospital routines.

Then there is the brain, still part of our body, but sort of our favorite one. Here the potential is enormous, with a lot of studies and money put into brain computer interfaces (BCI). And even if at consumer level one might see just basic gaming and meditation helpers, different companies provide affordable devices.

And while talking about helpers we can conclude our list on the personal optimization ecosystem, which fades into apps over apps: time trackers, task takers, habits builder, bad-habits wreckers(?), actions loggers and life gamificators, with the omnipresent and always judging Pomodoro technique!

That’s a lot of tracking, and many from one group can see many of another group as crazy or weird just for being into such things. One might ask “is it worth it?”, one should answer “it depends”.
At the same time, many (if not all) are tracked by companies or similar entities in all kind of possible ways, during all their interactions with, practically, anything digital. Seeing the success such companies had using in an intelligent way our data, I expect by analogy that a similar fate should fall upon us once we can do the same with regards to our own goals.

Diversions apart, all this was yes about learning to get and feel better, but still about what I addressed as “traditional” type of data, but my post is about the future of life-logging, so let’s move to some more quirky stuff.

Life Logging

Gordon Bell seems to have been one of the first “playing” around this concept, with a two-sided project of both collection and analysis of life’s data. But then you discover that after all that work he quit, together with another big name like Chris Anderson, and both with practically the same takeaway message: not worth it. Many others point out how dull most of the recorded stuff might end up to be, but that’s such a wrong understanding of the logging concept. Software log everything, not just errors or exceptions, they go as low as possible, down to the finest level, where yes, life is pretty boring, but one can always filter, and who knows, maybe there is so much important insight that can be derived and generated from those lower levels.

Better to bring more than needed, than to find yourself in the need of something you previously decided was superfluous.

As humans, we keep striving for more (for better or worse) and once we will achieve an optimal level of analysis and use of the quantified-self data, do you really want to tell me that richer media such as audio, text, and video would not be the next obvious step? We don’t even need more practical incentives, just more transparent and effortless automation and probably a bit of an increase of apathy toward privacy matters.

Video

Capturing what we see is something most of us do, and something humans have been doing for a while now. Eyebrows rise though when you mention continuous logging: a camera strapped to your chest, some recording glasses or even smart contact lenses. But again, if you want to log, you better log properly, especially if it’s feasible and effortless with nowadays technologies. I know Google glasses — for example — were not really successful as a commercial project, but being this a speculation I can dream of a reality in which products are good just because they are suitable for you and your peculiar needs, and not because there are enough people interested in buying them.
Smart contact lenses could probably deliver unbeatable POV recording fidelity, but it’s also worth pointing out that we might not need external devices to record what we see/perceive; nevertheless, my point would stay quite the same.

With huge databases of images from video life-logs would be relatively easy to implement a search through time, a search through your entirely recorded life. It’s a matter of platform and algorithms scalability, and a lot of research already went into the subject (sometimes nicely addressed as egocentric vision). In correspondence with improvements on the machine learning side, many kinds of natural language queries could be transparently operated on one’s life database: “when I saw person Y?”, “where I was at datetime X?”, “what I did on day Z”, “what did I eat in the year K?”, “how many yellow cars I saw in my life”. Why? Because we can.
There are actually many practical health-related use-cases for such a technology, that cover the entire spectrum of prevention, healing, and enhancement. Food recognition is probably one of the best marketable ones, with automatic caloric and nutritional value calculation (approximate, but the more precise the better technologies get). First world problem, but if there is something stopping me from keeping track of food is definitely how bothersome, impractical and time consuming the recording process is. One can use all this data to infer the influence that some specific food has on him, and discover possible allergies or more subtle effects not diagnosed until then.

This was just about food, but combination and aggregation of different sources of data are the key values for all these “I want to improve” movements: discover how something, how everything affects you. Track luminosity and colors, environments and location, see how habits, mood, performances change. Maybe someone needs technology to tell them who they like or dislike, or whatever other emotions people or situations can influence. GPS tracking can already say a lot about you, but this would be GPS on steroids, with full context awareness.

All this is just about filming what surrounds you, but I realize how much more you could gain if such recording could include also your person and actions. More complex analysis and reporting of hand gesturing, body language, posture monitoring, tics or routines you might not be aware of, with possible reinforcing learning loops via automated feedback.
Plus additional data for all our mood recognition and tracking.
And I feel stupid saying this while staring at the piece of paper taped to my laptop camera, but I suppose that’s the nice thing about speculating, a suspension of disbelief towards some of our credos.

Audio and Text

Many points about video virtually apply to this section too, but I want to reinterpret the audio part, and just focus here on what one says.

Textual messages are already easily available for analysis (Facebook Conversation Analyzer) or OCR if you still happen to scribble around. What we say with our voice is instead most likely and for the most part not recorded. Someone else might be recording it, but they are unlikely to share the data with us. I doubt someone like NSA would share its data about me with me, no matter how much of an enjoyable collaboration that might end up to be.
Recording what we say is the first step to a sea of possibilities in terms of personal discovery and improvement. Once we consider speech to text solutions, we just reduce the analysis tasks to the resources-rich realm of NLP, while leaving the audio data to more specific metrics that are instead lost with pure textual translation.

After my simple analysis of a personal public talk, I thought a lot about this scenario, data, and potential. It’s something simple, but still, until it doesn’t reach your mind is not there, no matter how simple. What if I record everything I say, process it and analyze it. Bear a moment with me and see how much stuff you can get out of it, and this is just was I was able to come up with:

• vocabulary: how rich your vocabulary is (lexical richness), common repetitions of words or idioms, grammatical and syntactical errors
• improvements over time: changes in your “language model” (multiple if you also speak more languages, with analysis of reciprocal interaction and influence)
• pronunciation: accents, quality, weaknesses
• register, timbre, prosody, pace, pitch, volume, mood…

Body language we covered in the previous section, so once you bring good content, here we have the potential to make you the perfect orator.
And I was again restricting myself to what is more normal to us, that is, the recording of public speaking. Instead — in this speculation of ours — everything is public speaking, and so your voice can be analyzed in relation to all sort of situations or people you are interesting in, being it a blind date, a weekend with the family, a mortgage negotiation or a drunk night out.

Even more speculations, and conclusions

Sci-Fi movies and series already said this all, and what I enjoy about them is tinkering for next months about ways to achieve that, what more is there, how technologies I know would behave for such scenarios.

Think how many services you can build around this: more startups for everyone! Given the amount of data, AI becomes the necessary intermediary, and while someone might call it our new — external — brain layer (the successor of the neomammalian one) let’s address it for now in a more friendly way: personal AI coach.
So there you have, a personal AI coach who observes and listens to you 24/7, has a pretty unmatchable universal knowledge, can give you advice on almost everything, know what is good and bad, and can subtly make you better, more like you would like to be… doesn’t this sound magical to you?

Humans don’t seem to excel at rational decision making, especially for daily life activities. A continuous observation of our “doing” has a great potential for improving our stand towards things like cognitive biases, heuristics and logical fallacies; a great potential for better choices overall.
“Better or fewer choices?”, you might ask. As per terms and conditions, our AI coach doesn’t order you what you have to do, is neither responsible nor punishable for what you do based on what it says, cause it’s just an advisor. It gives advice, summarized from such a huge amount of knowledge and computation power. I am not saying it will always be right, but most of the times, definitely most times than you would, so you will seem rather stupid/stubborn not to listen to it, and just do the opposite.

We can even accelerate in our speculation, discarding series and movies and get instead inspiration from books, which are really aiming way higher. Hypothesize about a virtual reality future instead of a physical one, of uploaded brains instead of transhumanism, of citizens instead of fleshers. At that point maybe all that I have said would not make sense anymore, who knows.

Conclusions

I started this entry with measuring my heartbeat with Fitbit and ended up with realities from hard science fiction novels, passing through a lovable/inevitable AI coach and a lifetime of one’s experiences stored somewhere in the cloud, for a possible better life.
I used a lot the word speculation, but for many all that has been here proposed doesn’t even sound so implausible or even too far. We live in interesting times, a steepness for the curve of technological advancement that none of our ancestors had the honor to experience, something to be sure content about if you are a curious mind, no matter what the future will hold.

In a previous short entry, I gave an introduction to chatbots: their current high popularity, some platform options and basic design suggestions.

In this post, I am going instead to illustrate what I believe is a more intriguing scenario: a deep-learning-based solution for the construction of a chatbot off-topic behavior and “personality”. In other words, when confronted with off-topic questions, the bot will try to automatically generate a possibly relevant answer from scratch, based only on a pre-trained RNN model.

What follow are four self-contained sections, so you should be able to jump around and focus on just the one(s) you are interested in without problems.

• Short intro on chatbots tasks and types.
• Details of the RNN model used: high-level model architecture, info on training data sources and pre-processing steps, as well as a link to the code repository. I’m not going into details for RNN, but I’m including what I believe are some of the best-related resources and tutorials.
• Architecture of the final working solution: the working chatbot involves separate and heterogeneous components. I will illustrate them and their interactions, while describing all tools and resources involved.
• Showcase of the chatbot in action: this is technical-details free, and pure entertainment, so just jump here if you not interested in the rest, or need motivation for checking the other sections.

Chatbots

Chatbots (or conversational agents) can be decomposed into two separate but dependent tasks: understand and answer.

Understanding is about interpretation and assignment of a semantic and pragmatic meaning to user input. Answering is about providing the most suited response, based on the information obtained during the understanding phase and based on the chatbot tasks/goals.

This post provides a very good overview of two different models for the answering task, as well as going into great details for the application of deep learning in chatbots.

For retrieval-based models, the answering process consists mostly of some kind of lookup (with various degrees of sophistication) from a predefined set of answers. Chatbots currently used in production environments, presented or handled to clients and customers, will most likely belong to such category.

On the other hand, generative-based models are expected to, well… generate! They are most often based on basic probabilistic models or on machine learning ones. They don’t rely on a fixed set of answers, but they still need to be trained, in order to generate new content. Markov chains have originally been used for the task of text generation, but lately, Recurrent neural networks (RNN) have gained more popularity, after many promising practical examples and showcases (Karpathy’s article)
Generative models for chatbots still belongs to the research sector, or to the playfield of the ones that simply enjoy building and demoing test application of their own models.
I believe that, for most business use cases out there, they are still not suited for a production environment. I cannot picture a client who wouldn’t bring up Tay if proposed with a generative model option.

RNN Model

A Recurrent Neural Network is a deep learning model dedicated to the handling of sequences. Here an internal state is responsible for taking into consideration and properly handle the dependency that exists between successive inputs (crash course on RNN).
Apart from the relative elegance of the model, it's impossible not to get captured and fascinated by it, simply from the many online demos and examples showcasing its generative capabilities. From handwriting to movie script generation.

Given its properties, this model is really well suited for various NLP tasks, and exactly in the text generation context I started exploring it, playing with basic concepts using Theano and Tensorflow for then moving to Keras for the final models training. Keras is a high-level neural networks library, that can run on top of either Theano or Tensorflow, but if you are willing to learn and play with the more basic mechanisms of RNN and machine learning models in general, I suggest to give a try to one of the other libraries mentioned, especially if following again the great tutorials by Denny Britz.

For my task I trained a sequence-to-sequence model at word-level: I feed to the network a list of words and expect the same as output. Instead of using a vanilla RNN, I used a long/short term memory (LSTM) layer, which guarantees better control over the memory mechanism of the network (understanding LSTM). The final architecture includes just two LSTM layers, each followed by dropout.
As for now I still rely on one-hot encoding of each word, often limiting the size of the vocabulary (<10000). A highly advised next step would be to explore the option of using words embedding instead.

I trained the model on different corpora: personal conversations, books, songs, random datasets and movie subtitles. Initially the main goal has been pure text generation: starting from nothing and generating arbitrarily long sequences of words, exactly like in Karpathy’s article. With my modest setup, I still obtained fairly good results, but you can see how this approach doesn’t work on the same assumptions of text generation for chatbots, which is at the end a question-answering scenario.

Chatbot Training

Question-answering is another big NLP research problem, with its own ecosystem of complex and component-heterogeneous pipelines. Even when focusing only on deep learning, different solutions with different levels of complexity exists. What I wanted to do here is to first experiment with my baseline approach and see the results for off-topic questions handling.

I used the Cornell Movie — Dialogs Corpus, and built a training dataset based on the concatenation of two consecutive interactions that resembled a question-answer situation. Each of such q-a pair ends up constituting a sentence of the final training set. During training the model gets as input a sentence truncated from the last element, while the expected output is the same truncated from the first word.

Given such premises, the model is not really learning what is an answer and what is a question, but should build an internal representation that can coherently generate text. This either by generating a sentence from scratch starting with a random element, or simply by proceeding in the completion of a seed sentence (the potential question), one word at a time, until predefined criteria are met (e.g. a punctuation symbol is produced). All the newly generated text is then retained and provided as candidate answer.

You can find additional details and WIP implementation in my Github repository. All critiques and comments are more than welcome.

Architecture

Interfacing with the chatbot is as simple as sending a message on Facebook Messanger, but the complete solution involves different heterogeneous components. Here a minimal view of the current architecture

Data processing and RNN model training have been operated on a Spark instance hosted on IBM Data Science platform. I interfaced with it directly via a Jupyter Notebooks, which simply rocks!
Using Keras callbacks system I automatically kept track of the model performances during training, and backed-up the weights when appropriate. At the end of each training the best snapshot (model weights) was then persistently moved to the Object Storage connected with the Spark instance, together with Keras model architecture and additional data related to the training corpus (e.g. vocabulary indexing).

The second piece is the model-as-a-service component; a basic Flask RESTful API that exposes the trained models for text generation via REST calls. Each model is a different endpoint and accepts different parameters to use for the generation task. Examples of parameters are

• seed: seed text to use for the generation task
• temperature: an index of variance, or how much “liberty” you want to give to the model during prediction
• sentence minimum length: minimum length acceptable for a generated sentence

Internally this application is responsible for retrieving and loading in memory the models from the remote Object Storage, such that they are ready to generate text when corresponding endpoints are called.

The final component is a Java Liberty web application which acts as a broker for the Facebook Messanger Platform. This is responsible for handling Facebook webhooks and subscriptions, storing users chat history and implement the answering-task logic. From one side it relies on the system described in my previous article, using IBM Watson services like Language Recognition and Conversation, on the other, when specific requirements are met or when no valid answer has been provided, can rely on the text generation bit, and call the Flask API at the most convenient endpoint.

Both the Java and the Python app are hosted on Bluemix, and regarding the former one, I’m currently working on the coverage of additional messaging platforms like Slack, Whatsapp and Telegram.

Show Me the Bot!

You can interact with the chatbot simply via Facebook Messanger, but making your bot public (usable by whoever reaches its page) requires some work, demonstration videos and successive official approval from Facebook. As for now, I have to manually add people as testers of my app to allow them to use it, so in case you are interested, just drop me a note.

Nevertheless, I have to admit that already seeing the interactions of the few current testers was a pretty entertaining experience, a good emotional roller-coaster of amusement, shame, creepiness, pride…

Let’s start with some mixed results (on the right the bot replies, on the left some friends’ inputs, nicely anonymized).

Notice that here there is mixed behavior: the “marry me” response is based on Natural Language Classification, while the rest are all generated. There are already grammatically wrong sentences, but at the same time I was for example nicely impressed by the second answer, being fooled into reading a sense of conscious omnipotence in it.

Sometimes replies seem totally random, but still build a nice interaction, a simulation of a shy, confused and ashamed personality, a bit romantic too maybe.

Given the training data, it also learned proper punctuation, so it’s likely to reply with a sentence starting with a punctuation symbol if the previous input was not ending with one itself.
It can also come up with some seemingly deep stuff, for then failing miserably especially given the momentary new high expectations.

Notice that in no way there is context retention between answers, it’s simply not built in the models nor system, is just the interaction flow that gives this illusion, and sometimes can coincidentally give a really good impression:

I am aware that there is no breakthrough in all this, and results might me “mhe”, but after all, so many people get crazily excited about their “baby’s first words”, which from my knowledge are way below the bar I set here…

My decent understanding of the mechanisms behind it, while observing it talking, makes everything even more fascinating. Feels irrationally surprising all it can formulate, with no actual semantic knowledge of what it’s being told and what it’s replying, just statistical signals and patterns… and I wonder, after all, how many people might be actually working in pretty much the same way, just with a more powerful borrowed computation instance in their skull, and a longer and richer running training on their shoulder.

Personality for Your Chatbot with Recurrent Neural Networks was originally published in Towards Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.

This article is supplemented with a Jupyter notebook, which explores the code and methods used for obtaining the results I illustrate here. I relied on common Python libraries (Pandas, Sklearn, NLTK, and Seaborn for visualization) and IBM Watson APIs for the speech-to-text task.

Heart Rate

It’s always a fun exercise to monitor your heart-rate in uncommon, out-of-the-comfort-zone events. We are often aware of our state, we feel the stress, agitation, and palpitation! But we are most likely to lose the focus on such internal state eventually, cause something else requires or shifts out attention.

Tracking your heart rate allows you to partially observe your body reactions to a situation even when the situation is already long gone, gives you the opportunity to approach with a neutral analytical state your past inner behavior.

Here is my heart-rate for the whole day of Tuesday 22nd November 2016

Average value is 81 heart beats per minute (bpm), with lowest and highest measurements respectively of 53 and 134 bpm.

Here a zoomed view around the time-frame during which I gave the talk

I stressed each relevant point with a letter:

• A: arrived at the location of the event (preceded by a 15 minutes fast-walk)
• B: start of the first speaker talk. Here I’m just sitting and listening, also most likely consciously stressing myself by trying to relax myself
• C: on the stage, start of my talk
• D: start of the Q&A session
• E: back to the chair

I already saw a couple of similar graphs, for comparable situations: like many others, I was simply not able to avoid that peak. As you might have noticed, that’s exactly the highest value I reached for the day, evened by a later quick-run after a couple of pints. It is interesting to observe also how HR dropped down consistently as soon as I started presenting. I remember to have been initially highly aware of my words, of what I was saying, for then simply leave space to “auto-pilot mode”.

Theoretically, the more talks I will give, the more I will present in public, the “better” that graph should get. This is my first personal dataset of such kind, but I hope to collect much more data in the following years. Is not only practice/experience that I will have to take into considerations, but also context, aging, and God only knows how many more possible explanatory variables.

Speech Analysis

For the speech analysis, I am going to analyze only the actual transcripts of my talk. A lot of data will be lost cause of this decision, and I’m not talking just about possible inaccuracy of speech-to-text results… here a list of important aspects of public speaking which are lost when considering only a basic textual representation:

• Speech Rate: words spoken in a minute, speed and pauses
• Body Language (posture, gestures, eye contact, facial expressions)
• Voice (register, timbre, prosody, pace, pitch, volume)

That’s surely a lot, but let’s see what we can do and get from the basic text, and let all these aspects to a later stage.

Speech To Text

First thing first: speech-to-text. I didn’t do it on the spot, using tools for real-time generation of text, I instead relied on the video recordings setup for the event.
I got my video file, extracted the audio content, cropped the unnecessary parts (including Q&A session) and went for a speech-to-text solution.

Well known services for speech-to-text (with space for some free usage) are from big names likes Google, Microsoft, and IBM.

Google wanted my credit card number and Microsoft APIs seemed unresponsive to my effort. On the other hand, IBM results are really rich and, together with various optional settings, each recognized word can be accompanied by a confidence level, start and end time (“beginning and ending time in seconds relative to the start of the audio”).

Basic Text Analysis

Considering the pure textual info I already ended up with this basic but neat summary of my talk:

“Here a summary of the conversation. Overall 3315 words have been said, of which 951 unique ones, giving a lexical richness of 28.69%.
With the talk total duration of 21.8 minutes, the speech rate is of 152 Words Per Minute (WPM).”

Basic aspect one might want to explore in this circumstance is words usage: which are the most common words, bigrams, trigrams, used during the talk. Top results will most likely be common and low-informative constituents like articles, adverbs, and pronouns. For my results I found that first significant word is data, ranking 65 in the words that occur the most, which might make sense given the topic of the talk.
To better explore actual relevant words, you might first of all try to simply remove stop words, or rely on more specialized statistics like tf-idf (explained in more practical details in the notebook).

Words Alignment and Speech Rate

Let’s consider again the speech-to-text results from IBM Watson service, here what the first five rows (out of 3315, one for each word) of the cleaned results look like:

The alignment info can be used to overcome some of the limitation of pure text analysis. The following histogram should provide an approximate view of my speech rate trend; the entire conversation has been split in 10 bins of equal size (time) and the x value is the count of words which fall in the corresponding bin.

You can notice for example a slight but constant decrease in my speech rate. Fitting a regression line between such points I obtain a coefficient of -5.23. Considering the 10 bins used, each bin has a size of 2.18 minutes. Very roughly, this is equivalent to say that on average my speech rate decreased by 5.2 words each subsequent 2.1 minutes.

Even more generic, one could simply say that the more time passed, the slower I talked. We should then clarify what one means with “talking slower”. There are two options I can think of:

• using fewer words, which can be caused by two factors: more spacing/pauses between words or usage of longer words, which take more time to be pronounced
• lower word speed (time to pronounce a word, measured as length of word w divided by time to pronounce w)

Based on the results from linear regression fitting and correlation coefficients, an additional summary for the alignment part would be:

Your average speech rate is 152 Words Per Minute (WPM), but an approximately constant and significant decrease can be observed, bringing you from an initial WPM of 166 to a final value of 142. The primary cause of this is the usage of increasingly longer pauses between words, secondarily reinforced by a combination of using longer words, as well as a tendency to slow down the pronunciation of words, while the talk unfolds.

Finally, another interesting addition in the results from IBM APIs, is the presence of a specific keyword: %HESITATION, which unfortunately doesn’t seem to be well documented, but should represent fillers and speech disfluencies such as “Uh”, “Ah”, “Erm”, “Um”, etc.
Here a visualization of the occurrences of such keyword.

I want to stress again that all the text analysis here demonstrated depends merely on the quality of the speech-to-text results, which considering the setup, audio quality, and brief observations are way below optimal. At the same time, the proposed framework is a reusable one, which I’m definitely planning to further expand and put into use on future data of — hopefully — higher quality. As usual, all feedback, critiques, and corrections in particular, are more than welcome.

First of all, great article! Lot a good info and resources to work on.
After my analysis on Fitbit sleep data, I am now working exactly on my heart-beat measurements, and your article provided me with many insight and ideas.

I will also have to dive deeper into the blood related biomarkers; as you pointed out they are harder to get, but definitely valuable!

Again, great post! Keep up with the amazing work!

1. You have to write some code to clean the JSON data returned by the API, and extract the info relevant to you
2. Seems like the API have a limit of 150 calls per hour, so you might need to repeat the operation several times depending on how much and what data you need

Thanks again Esther

I currently work as a Solution Engineer for IBM Watson, Dublin, so I took advantage of the situation for finally building my own chatbot. Needless to say, for the task I relied on IBM Watson Developer Cloud, using services like Natural Language Classifier (NLC), Dialog and Language Detection. I then wrote some code to tie these services together and hook the final solution to Facebook Messenger Platform, hosting the actual broker application on Bluemix. I am not going to explain here in details how I achieved that, there are already many good post about this, I personally followed and suggest this entry. Still, if you interested in a more technical and detailed explanation, feel free to leave a comment or contact me directly.

Let them speak!

Seems like chatbots and conversational agents are a big trend lately (Rise of the Chatbots, Chatbots NBCNews, Available APIs Review). While all these articles sure made me interested in the subject, I have to confess that this one in particular spurred me to actually build something. Because yes, what best use case that a bot acknowledged simply about yourself, its own creator? In all projects I worked on, corpus, knowledge base, ground truth and alike really required a lot of work and experts, depending on the context, subject and coverage, but if it’s about me, well, I should know and be enough! In her article Esther focuses on an overview of “Not-Too-Technical Solutions”, but makes many good points, that’s why I also decided to focus on the recruiting aspect, turning my CV and related info into the knowledge domain/base of the bot. You can ask generic question about me, like how old I am, were I’m from, as well as more specific ones like what is my expertise, my professional interests and if I am interested in a particular job offer.

Chit Chat

Around this corpus of personal data, questions and answers, I played and modeled a bit the so called chit chat, all those utterances that make the conversation much more human, and less database-query like: greetings, compliments, insults, generic questions and off-topic.
Good advice for this part is to add variation and a bit of randomness, but don’t try too hard to fool the user about your bot being a human, for most cases, there is no point in that. Be sincere from the start, put some personality in your bot, but don’t overdo: the user knows she’s facing a bot, doesn’t take much effort to test that, especially if you want to deliver something useful and consistent. Actually, if you know about some really good bot I can easily test for free, please let me know, I am curious to test my personal list of bot-weak-points on products that are out there.

All this part as been managed manly via Dialog service, but I am now willing to give a try to a new experimental service called Conversation, which should unify the dialog and the actual language classification and understanding parts.

UI and Facebook Messenger Platform

While I enjoyed setting up structure, logic and data, I got annoyed pretty quickly playing with the UI part: is not my thing, but at the same I appreciate beautiful design, so I could not settle with a cheap and quick solution of mine, I needed an alternative. I checked again all these articles about chatbots, revisited some of the suggested platform, and the Facebook one seemed the most immediate, intuitive to setup, and most important, didn’t need to bother my friends/testers with “download this app” or “go to this website” just told them “hey, try to chat with this friend of mine”.

Being my bot a simple service, accessible via API, I just had to implement and deploy the broker code, which in fact can be easily improved in order to cover other platforms (e.g. Slack, Telegram, WhatsApp) and services (e.g. Tone Analyzer, Visual Recognition).

Notice that making your bot public requires some work and successive official approval from Facebook, so for now I have to manually add people as testers of the app if I want them to actually chat with it, otherwise they will simply stand there waiting indefinitely for a reply. Again, if you interested, let me know.