The Promise of Personalization: The How and the Why for Retail
She gave birth to a beautiful baby boy. It had been three weeks since and she was well on her way to losing those extra pounds. Spring was also just setting in and she was excited to finally browse through cotton knee length non-maternity dresses. Before her pregnancy, short knee length dresses always used to be her thing- regardless of which season presented itself in New York. Not just dresses that were knee length, but ones with sleeves that covered her elbows. Not only did she not find even a single cotton knee length dress, but only maternity dresses were recommended to her when she logged on to her favorite, most frequented online fashion label — TheUrbanFashion*. She left the site instantly and TheUrbanFashion had lost a loyal customer.
Maria* is an unsatisfied customer.
TheUrbanFashion scrambles for a quick fix and reaches out to their recommendation engine service provider — Curashions *. Is this an algorithmic glitch? No, the personalization algorithms have worked perfectly. Maria was shown maternity dress options based on what she historically browsed through and bought over the last nine months. Less importance was given to the shorter style of dresses she was recently looking at since there was simply not enough user data for the algorithms to pick up on. The data scientists at Curashions are unfazed and their life goes on.
Maria represents a large proportion of disgruntled online shoppers who expect online stores to provide a retail experience that is tailored, dynamic and specific to their shopping needs real time. Maria also represents the precarious conundrum that AI (artificial intelligence) based personalization engine providers such as Curashions face, which is finding the balance between data-driven algorithms and real-time contextual events that occur in a shopper’s immediate past, like Maria’s pregnancy, and long-term past that are not easily captured through conventional technology. Fluctuations in browsing and spending patterns are induced by life events and the recommendations suggested on a page, must be receptive to that. Most importantly, understanding the true intent of a given shopper, ranks extremely high up on the priority list for all stakeholders in the e-commerce ecosystem.
Predictive analytics is sexy. Everyone does it, it’s trendy and it’s creating a buzz. Personalization is specifically designed to enhance the customer experience by showcasing specially curated products to a shopper and also making predictions about what they might be likely to buy in the future. Several studies have found that almost 62% of brands use some form of personalization as a tool to build customer loyalty. However, shoppers are beginning to have unrealistic expectations of these engines, given the Big Data fever and companies like Netflix and Amazon are frontrunners in the race to crack this riddle. Highly optimized personalization engines not only require implicit data that is traceable by technology but explicit user feedback data as well to successfully gauge the intent, context, and background of each online shopper.
Here comes the unavoidable glass ceiling.
The glass ceiling
Recommendation system providers rely largely on large scale data mining algorithms to create personalized user content for e-commerce retail sites. At the back end, personalization engines create a goldmine of user data for companies to continually train and refine their models with. Personalization engine models also use a ranking system that will find the best possible ordering of a set of items for a user, real- time. Scores are given to attributes that are favorable and recommendations are served in that manner. As personalization algorithms keep improving, so does the experience of the users that use these systems, and this cycle repeats. While clickstream user data is paramount, supporting different contexts like sudden changes in user intent, fashion trends and styles, user spending patterns and other types of lifestyle decisions, requires different kinds of data.
The difficulties in cracking shopper intent
The complex task of deciphering the intent of a customer is riddled with several challenges. First, a large proportion of customers access e-commerce sites through the multitude of channels available — social or otherwise. Interestingly, these shoppers tend to be frequent buyers. How does the intent and mindset of an Instagram shopper differ from that of an organic shopper or a traditional brick and mortar store shopper? There is no established method of collecting this type of data. Recommendation engine providers are made to grapple with the possibilities of intent through making educated decisions based on their own internal assumptions and by weaving together all the implicit and explicit data that is available to them. Taking these limitations into account, how can we train our set of codes to identify intent?
Second, a customer who does not trust an e-commerce site, is unwilling to provide qualitative feedback regarding the products that are suggested, further curtailing dynamic personalization. The word “dynamic” here implies the quick reactive and predictive nature of these algorithms to both implicit and explicit feedback and in absence of this, the engines rely heavily on just clickstream data, thereby preventing optimization. How can e-commerce companies and recommendation engine providers get their shoppers to meaningfully engage and confidently provide feedback?
Third, while algorithms continually learn from the ranking systems and variables, the order of listing and the decision to recommend one product over the other is ultimately made by a human being (Raphael, 2016) i.e. the data scientists and product teams at Curashions. Several biases exist in the way the algorithms are trained for personalization and this can potentially expose the lack of innovation and the incapability of breaking the stereotypes. The recommendation system might favor one product over the other due to factors that may not be directly related to the shopper’s preferences. How can recommendation engine providers ensure that these subconscious biases do not interfere with the quality and relevancy of the personalized output?
Finally, algorithms require opportunities for rapid experimentation to iterate intelligently. While a subset of users’ behavior can be modeled, it is unrealistic to expect all users to exhibit typical behavioral patterns. These atypical data points can adversely skew the results and accuracy of these engines. Hesitant shoppers and the lack of established mechanisms to collect product feedback, hinders with the “personal” component of these engines. Adding to this, e-commerce companies are slow in permitting any sort of experimentation on their user base, and understandably so. How can recommendation engine providers and e-commerce companies work together to facilitate continuous algorithmic experimentation and training?
This is not just a data science problem. This is a systemic problem.
Using computer vision and continuous learning systems to decode the psyche of online shoppers
At the outset, a persistent effort to achieve dynamic personalization is absolutely necessary. What does this mean? Algorithms quickly learn with every event click and provide customized fashion choices for each user, real time. These engines have to be continually improved to factor in qualitative feedback from customers in addition to long term and short term click-stream data that already exists. The various flavors of user information that flows in, helps in creating recommendations that stay current and relevant to the specific user at any point in time.
Apart from the data-driven continuous learning systems, visually intelligent computer vision based processes that can extract multidimensional attributes like color, pattern, style, length, sleeve, silhouette among several others, from the product that a customer is engaging with, must be put in place.
Is the product click driven by color or the pattern or the style? Has the customer historically preferred pastels over bright bold prints? Is this a highly engaged style conscious shopper or a convenience seeking discount shopper?
Finding answers to these types of questions will go a long way in understanding the psyche of shoppers. Using Computer Vision based technology, with every click on a product, the specific attribute-wise preferences become apparent and when combined with data-driven historical behavioral patterns, the personalization engine becomes much more powerful and accurate in interpreting intent.
Detecting color similarity as a basis for personalization is a good illustration of this. One would typically assume that when two items from the same category are purchased together, they would tend to be of contrasting colors. A study performed by one of our in-house Data Scientists found that the algorithms that detect similarity in color for co-bought products of a single category, observed a contradictory pattern. As seen below, the color specifications of two shoes that were bought together show that the two items (Shoe 1 and Shoe 2) were of similar color.
This pattern was observed across some other categories as well for the particular client used for the study. Hypothesis-driven studies like these, using other visual attributes of a product, allows for inferences to be drawn with respect to a user’s intent.
Lastly, explicit qualitative feedback from end shoppers is a critical piece of the personalization puzzle — without which intent can never really be pinned down. This can be done in a couple of effective ways. Surveys, written feedback and comment boxes are easy tools to instantly get qualitative feedback. Optimizing user interface design is yet another smart way of mitigating unnecessary navigational distractions and prevents drop offs. Repositioning the graphic elements on a page, creating “like”, “dislike” and “add to wishlist” buttons on product pages can help this process immensely.
Experimentation tools such as A/B tests, that capture the various flavors of personalization, are proactive ways to keep a close eye on the performance and ROI of each version that is released. The process flow illustrated below shows us a bird’s eye view of the elements that go into a personalization engine.
Inputs derived from clickstream data and computer vision based attribute affinities and user preferences are built into personalization models. The models are trained and evaluated continuously through experimentation techniques — like A/B tests, where different versions of the engine can be tested out with the end-customer. The outcome of these tests is fed back into the development stage of these models, facilitating a crisp dynamic learning system.
Is convergence of Big Data and Artificial Intelligence the next step?
Algorithms are where the money is. It is no secret that Big Data giants like Facebook and Google are privy to almost all our personal information. Interestingly, they too are yet to crack the deeper nuances of personalization and, problems like the one Maria faced, continue to exist across the board, in the e-commerce ecosystem. Needless to say, there is a never ending search for better methods of data accumulation and rapid experimentation. The Big Data revolution has undoubtedly pushed the personalization bar high and several companies market themselves as the one stop solution for all things personalization. A proactive engagement with the various stakeholders in the ecosystem is the key to making the next big jump in personalization. The possibilities of optimization have no bounds.
*Note: The names used in the article are fictitious, and purely for illustrative purposes. The incidents, however, might be inspired by reality.
