When building a prediction with Einstein Prediction Builder, sometimes the records you want to use as examples (your historical data) are the same records you want to create predictions for. Think of this as the example set/prediction set overlap problem (or training set/scoring set overlap). Typical predictions that fall into this category include customer attrition, lifetime value, high-value customers, and upsell.

Here are two ways to address the overlap problem: (1) the time horizon approach, and (2) the randomization (two-segment) approach to create your predictions. In the Summer ’20 release, you’ll be able to use filters to explicitly define your prediction set. This feature makes the randomization approach unnecessary because you can choose to use the same records in your example set and prediction set. Read more about this in the release notes!

Read on to understand when to choose the time horizon approach versus the randomization or prediction set filters approach. Use the Prediction Definition Framework described in this blog to think through the upsell and high-potential customer use cases, and see which approach works best in each case and why.

Upsell use case with the time horizon approach

Every company wants to know whether its customers are likely to buy another product or service. This type of prediction is an upsell or cross-sell problem. The goal isn’t to recommend the right product (which is a different class of machine learning problem), but rather to predict how likely a customer is to buy additional products.

Defining your example set can be tricky. Positive examples are easy to identify: any customers who bought two or more products. But what about negative examples? For each customer who bought only one product so far, you have to determine whether this customer has not bought an additional product yet (and thus belongs to the prediction set) or whether they will never buy any more products (and becomes a negative example). In this particular example, set/prediction set overlap problem, you need to differentiate between records to score and negative examples.

To determine the right time horizon to use, create a report, and identify how long it normally takes for customers to buy a second product after the initial purchase. In this case, it’s six months.

The Avocado Framework for this problem might look like this:

1. Dataset: All standard customer accounts.
2. Positive Examples: Accounts with two or more products.
3. Negative Examples: Accounts with only one product and more than six months since the purchase date. For the sake of this prediction, assume they will never buy again. You’ve given up on them because historically, other customers made their second purchase much sooner than six months since the first purchase.
4. Records to Predict/Score (Prediction set): Accounts with only one product and less than six months since the first purchase date. Since they’re still in the green zone (they can potentially make their next purchase soon), create a prediction for these customers.

Here’s how to set it up with filters:

To set up the upsell problem in Prediction Builder, select the No Field option and use the Yes and No example filters to define the logic:

Negative examples:

Tip: One of the conditions involves dates with the comparison: Purchase Date Less than Comparison Now () Minus 180 days. In this context, Less than means Earlier than. Thus, Purchase Date has to be anytime before 6 months ago (for example, 7 months ago or 1 year ago).

In summary, use the time horizon approach if you can confidently define the time horizon to separate between negative examples (“write-offs”) and records to score (“undecided” but with good potential).

Salesforce recommends using the time horizon approach for attrition use cases. Instead of predicting whether the customer will ever attrit, you can predict if the customer is likely to attrit within the first year of becoming a subscriber or buying your product, for example.

What if there is no well-defined period? Then you can try the second approach: create two or more predictions on randomized segments of your data. Or, in the Summer ’20 release, use filters to define your prediction set.

High-value customers use case with randomization or prediction filters.

Let’s say we want to predict high-potential customers: those who are likely to spend more than $X during their lifetime. The actual value of $X depends on your business. You can create a report and see what represents the top-spending 10% of your customers. This is your high customer threshold. In our example, it is $500.

The challenge with this prediction is that it’s very difficult to differentiate between negative examples and records to score. If they haven’t spent $500 yet, does it mean they never will (thus becoming your negative examples)? Or that they haven’t yet reached this $500 threshold, but eventually they will? This is another instance of the negative examples and prediction set overlap problem.

In general, it’s better to use a time horizon approach and identify which customers are likely to spend more than $500 within some specific timeframe, as described above. For example, you can create a report and identify how long it takes for the majority of your customers to reach this $500 threshold. Let’s assume it is 6 months. Then our negative example set will include customers who have not reached this $500 threshold within 6 months. This will probably provide better predictions than the randomized approach described below because your negative examples are much more aligned with the negative behavior — customers not spending $500 within their lifetime.

Alternatively, you can create a numeric prediction, predicting Future Lifetime Value (LTV) for each customer. In this case, you want to use all records as examples (learning from all existing customers’ spending history) and predict for all records (estimate Future LTV for all customers). This is another example set, and prediction set overlap problem.

Two-segment randomized approach

Until there is support, for example, set/prediction set to overlap, use the following approach as a workaround.

Use two randomly created segments to differentiate between the example (training) set and the prediction (scoring) set. Build two predictions: one trained on Segment 1 and predicting for Segment 2, and the other one trained on Segment 2 and predicting for Segment 1.

Prediction Definition Framework — high-potential customers

For this use case, the Avocado Framework looks like this:

1. Dataset: All customer accounts.
2. Positive Examples: Customers who spent more than $500.
3. Negative Examples: Customers who spent less than $500 and are in Segment 1.
4. Records to Predict/Score: Customers who spent less than $500 and are in Segment 2.

Use two randomly created segments to differentiate between example and prediction sets. Using the formula below, we randomly assign customers to Segment 1 or Segment 2 (basically, we are creating an odd or even segment in our data based on some number field):

IF(MOD( INDEX_c , 2) == 0, “Segment1”, “Segment2”)

Now for the final setup using filters:

Then create a second prediction, with customers in Segment 2 as the negative examples, and predicting for customers in Segment 1.

Using Prediction set filters

When the new prediction set feature is available, you can use prediction set filters to explicitly define records to score. This allows the example set and prediction set to overlap.

To predict LTV, select “Score only records that are in the example set,” which creates predictions for all records that you used as examples (the example set and the prediction set fully overlap).

For the customer attrition and high potential customer use cases (negative examples and prediction set overlap), select “Score specific records” and use filter logic to exclude positive examples. In these scenarios, positive examples are easily identifiable (customer attritted or reached a high-value threshold) and don’t need to be scored. Basically, you score everyone, except for the customers who have already attrited or reached a high-value customer threshold. And the same records are used as negative examples and records to score.

How to use your predictions to improve business outcomes

Incorporate these predictions in your business process to streamline resource management and automation:

• Double your efforts on customers who are more likely to buy another product or achieve high LTV. To identify these customers, create a new list view and sort by scores (either the likelihood to buy or achieve high LTV), so sales reps can prioritize customers with the highest potential. You can also review customers in the middle range (those are your borderline opportunities) and identify steps to get them back on track. Perhaps, you can add them to the appropriate marketing campaign.
• Automate creation and allocation of tasks to focus the Sales and Services team on the high potential customers. You can use Process Builder to automate task creation for prioritized customers.
• Show the top predictors for each customer, so your business users can see reasons behind these predictions. Just add the Einstein Predictions lightning component to the Account layout page and select the name of your prediction.
• Use Einstein Next Best Action to provide the right recommendations to the sales reps for each customer based on the predictions and business rules. You can get an idea of what to recommend for each customer based on the top predictive factors in the scorecard.

For more information about the prediction lifecycle, please review this blog.

Summary

If you encounter a situation where you want to score records that you used as examples, first evaluate if you can use the time horizon approach to separate between negative examples (“write-offs”) and records to score (“undecided,” but with good potential). If this is not possible, use the randomization approach or prediction set filters approach (available in the Summer ’20 release).

For more on the latest when planning your prediction, check out the official Salesforce documentation.

Related Blog Posts

1. How to turn your Idea into a Prediction
2. How to use Einstein Prediction Builder for Opportunity scoring.
3. How to Use Einstein Prediction Builder to Predict Opportunity Amounts
4. Which fields should I include or exclude from my model?
5. Understanding your Scorecard Metrics
6. Understanding the Quality of Your Prediction
7. A Model That’s Too Good to be True
8. How do I know if my prediction is working?
9. Custom Logic on Predictions from Einstein Prediction Builder

How to Score Records You Used as Examples: Predicting Upsell and LTV in Einstein Prediction Builder was originally published in Salesforce Einstein Platform on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

The power of Einstein Prediction Builder is how effortlessly it lets you bring your ideas into production, letting it leverage your valuable data to create predictions and new insights. We have written before about the user’s role in coming up with and creating a prediction (https://medium.com/salesforce-einstein-platform/einstein-prediction-builder-how-to-turn-your-idea-into-a-prediction-8393c1319205), but what is the behind-the-scenes magic that makes things happen from there? In this blog post, I want to go into some detail about how the data flow and are transformed through the software backend to help you gain a deeper understanding and appreciation of our product.

Salesforce and Einstein Platform

All data you enter in Salesforce is safely and securely stored in certain formats, ready to be consumed by various Salesforce products, most notably the UI that lets you see your data. Like most applications that do analytics or use machine learning, however, Einstein Prediction Builder can work more efficiently if these data are in a different, specialized format more suitable for large-scale processing. For this reason, the data are Extracted, Transformed, and Loaded (ETL) (https://en.wikipedia.org/wiki/Extract,_transform,_load) into a dedicated software platform called the Einstein Platform. Einstein Prediction Builder is just one of the products running on this powerful platform. Note that the flow described below refers to Einstein Prediction Builder specifically and not to all Einstein functionalities.

The flow of data and user actions

The following diagram gives a bird’s eye view of what happens to your data between Salesforce and the Einstein Platform and how it ties in with the actions you take in the Einstein Prediction Builder UI. It leaves out many important software components in order to clearly explain the gist of things. Let’s walk through the components before we dive into the specifics of the machine learning flow later on.

1. Suppose you create a binary classification (Yes/No) prediction that predicts whether your sales opportunities are likely to be won (following the example from our previous blog post here: https://medium.com/salesforce-einstein-platform/einstein-prediction-builder-which-fields-should-i-include-or-exclude-from-my-model-eae37f231dac).
2. After you create the prediction, a snapshot of all relevant data (e.g., the fields you selected as features and the target field that holds information about whether the opportunity was eventually won or not) is pulled into the Einstein Platform.
3. A machine learning model will be trained on the labeled opportunity data to learn patterns in the data. We will go into more detail on this step below. Also, for an introduction to machine learning that explains these concepts in more detail, see our previous blog post (https://medium.com/salesforce-einstein-platform/an-introduction-to-machine-learning-9f5bfc146942).
4. Several indicators of the quality of the trained model are saved, along with insights into what features the model deems important for predicting whether opportunities could be won or not. These populate the metrics and visualizations of the scorecard, which we discussed in a previous blog post (https://medium.com/salesforce-einstein-platform/einstein-prediction-builder-understanding-your-scorecard-metrics-7e1ef4bba65b).
5. When you approve a model in the Salesforce UI, the Einstein Platform will register this approval and activate the model.
6. Immediately after approval, the previously trained model is fed the unlabeled data, and it then produces a score per record that is the predicted likelihood of the opportunity being won.
7. These scores are then saved in the prediction field that you configured during the creation of the prediction.

After the model is activated in step 5, the training and scoring cycles repeat periodically (currently monthly and hourly, respectively).

• Training loop (steps 2 to 4): the machine learning model is re-trained on the latest data that has received labels, and the scorecard in the UI is again updated with results from this latest re-training.
• Scoring loop (steps 2, 6 and 7): the Einstein Platform will check if there are new or updated records for which predictions need to be made. If so, scoring on those records is performed, and scores are saved.

Inside the machine learning black box

Now that the big picture of how data flows and how user interactions play out is clear, we can come to the heart of things and have a closer look at the most fascinating part: the machine learning pipeline. In this pipeline, data are transformed in various ways in order to distill the insights living in your data into a useful model that can make good predictions on new records.

Data preparation

The data preparation step makes sure that all subsequent steps have clean data to work with. This means removing features (the fields you selected as inputs for the prediction) if they have attributes that are undesirable in a machine learning context. See our blog post introducing machine learning (https://medium.com/salesforce-einstein-platform/an-introduction-to-machine-learning-9f5bfc146942) for details on what is considered undesirable.

Examples of things to look out for include:

• Not enough data: we remove features that have too many missing values in the training or initial scoring data.
• The labeled training data looking markedly different from the unlabeled scoring data. For example, we look at the difference in their distributions and their fill rates. These are indicators that a field started being used differently, for example, because of a migration of old data from another CRM into Salesforce. This can be problematic for machine learning models, and therefore is ground for getting rid of the feature.

The data preparation step also breaks off the flow if, after all this cleaning, there are not enough features or records left to build a useful model. Users get alerted of such situations in the frontend with an appropriate error message.

Feature Engineering

You store data in Salesforce in (custom) fields that suggest a certain type of data: TextArea, Date, Number, Email, etc. One of the things we are most proud of is that we let our software leverage that metadata to do smart, automated feature engineering. Feature engineering is the process of transforming your original data in ways that make the information in them more readily discoverable by a machine learning algorithm. That sounds abstract, so let’s go through a few examples together.

Let’s say you chose the following 4 fields as inputs to your opportunity prediction:

• “Contract size” (Currency field, in $ per year)
• “Subscription type” (Picklist field, with values “Trial” / “Annual” / “Lifetime”)
• “Last interaction” (Date field)
• “Customer sentiment” (TextArea field with some notes from your salespeople)

Now because “Contract size” is a number, our feature engineering step will explore some options for transforming the data to extract more value. For example, it will use machine learning techniques to try to smartly divide these numbers into a number of buckets. Say you have the list of customers shown in the following figure. It is a huge help for most algorithms to have a feature that groups them by their relative contract size. All this is done automatically for any numerical feature!

Both the output of the bucketization process above and the “Subscription type” are categorical, i.e., the data can take one of a limited number of possible values. In order for this data to be consumable by a machine learning algorithm, they are encoded in a different way, as seen here:

Before

After

Note that if the number of possible entries is extremely large, this will result in an extremely large number of columns. This might confuse an algorithm, so sometimes we have to be judicious and only choose the categories that occur most frequently.

“Last interaction” is a date field, so there is a range of possible interesting transformations to be done. For example: we extract the day of the week, the month of the year, the year, the number of days between two dates, and more. We have written before about the complications of date fields, and if you are (considering) using them in your predictions, we strongly recommend you to read that post (https://medium.com/salesforce-einstein-platform/complication-with-dates-v2-2d93bc52edec).

Lastly, and most interestingly, the “Customer sentiment” field. Because this is a freeform text field in which the salespeople can write anything they want, it is (1) a potential treasure trove for predicting the outcome of opportunities, and (2) the most complex field to automatically transform. We use a variety of natural language processing (NLP, https://en.wikipedia.org/wiki/Natural_language_processing) techniques to extract signals from text fields to then feed into our models. Since NLP is a deep and fast-moving field of machine learning, it is an active research subject for us. We frequently experiment with state-of-the-art methods such as pre-trained word embeddings, sentiment analysis, and neural networks for engineering features that are both useful for modeling and human interpretability.

At the core of all the transformations above (and many more!) is a state-of-the-art software library called TransmogrifAI, which we have open-sourced: https://github.com/salesforce/TransmogrifAI.

Feature selection

Our feature selection step is partly about choosing the most promising features with high information content, and partly about dropping features that are “too good to be true.” We have written before about the problem of “label leakage,” where during model training there is information in the input that contains information about the expected output, leading to an unhelpful model that does not perform well without this information when it is time to make predictions (https://medium.com/salesforce-einstein-platform/einstein-prediction-builder-a-model-thats-too-good-to-be-true-f1754e5ca48e, https://medium.com/salesforce-einstein-platform/einstein-prediction-builder-which-fields-should-i-include-or-exclude-from-my-model-eae37f231dac).

Salesforce data scientists spend considerable effort in building ways to automatically detecting these so-called “leakers” using a lot of fancy mathematics. In essence, we remove features that are too strongly related with the label. We use a variety of correlation measures that can suggest such relations for different types of data. Imagine you would have a “Date that contract was signed” feature in the opportunity example. Our software would pick up on the correlation between that and the target “Opportunity was won” label and remove it from the feature set.

One conclusion here is that even if a user accidentally includes leakers in the fields available to the prediction, this step will likely remove them from the training data. On the other hand, users will need to remain vigilant about introducing leakers since this is a very difficult problem for which there is no perfect automated solution.

Finding the best model

Next, we are ready to train some machine learning algorithms to capture the relationships in the data. These mathematical techniques come in various flavors such as linear regression methods (https://en.wikipedia.org/wiki/Linear_regression, https://en.wikipedia.org/wiki/Generalized_linear_model, https://en.wikipedia.org/wiki/Logistic_regression), decision-tree-based methods (https://en.wikipedia.org/wiki/Random_forest, https://en.wikipedia.org/wiki/Gradient_boosting#Gradient_tree_boosting), etc. These algorithms all come with different knobs to tweak their performance in various ways.

We let the various candidate algorithm compete in a tournament, while optimizing their various knobs, and we store the best-performing models for when we need to make predictions.

Calculating metrics

Finally, a wide variety of metrics are calculated and stored after training. These are used to populate the scorecard in the Einstein Prediction Builder UI, where you can explore them. There are separate groups of metrics for:

• Numerical predictions, for which we want to express how close the predicted numbers were to the real numbers. Example: R-squared (https://en.wikipedia.org/wiki/Coefficient_of_determination).
• Yes/No predictions, for which we want to express how often a model correctly predicted the Yes/No situation. Read our blog post about understanding the quality of your prediction for more details about such metrics (https://medium.com/salesforce-einstein-platform/einstein-prediction-builder-understanding-the-quality-of-your-predictions-de5a171ed61e).

Back to you

The results of the above machine learning flow are:

• A trained model, standing by and ready to be used for predictions.
• A set of metrics and insights into the model visualized on the Scorecard.

Now it’s up to you, the user, to determine whether the model seems to achieve the goals you had in mind (https://medium.com/salesforce-einstein-platform/einstein-prediction-builder-understanding-the-quality-of-your-predictions-de5a171ed61e). It’s time to consider whether the expected prediction quality in your Scorecard makes you feel the model will do a good job. Or is the expected performance suspiciously high, and you are worried that you might have included leakers? In any case, check the scorecard for feature importances and see if anything stands out. You might learn something new about your data!

If everything looks good, you can go ahead and approve the model. As shown in “The flow of data and user actions” above, approval of the model means it will start producing predictions on new and updated records. The model will also be periodically updated so that it can learn things from the latest data.

Summary

Even in this brief glimpse behind the scenes of Einstein Prediction Builder, we can already see that there are a lot of moving parts, from the flows of data to the intricacies of machine learning algorithms. Rest assured, there are a lot of great people continuously greasing the wheels and maximizing the value you can get out of your data.

I hope you enjoyed learning something about the Einstein Prediction Builder secret sauce, and have a regained appreciation for what it takes to take machine learning into production.

The Journey of Your Data was originally published in Salesforce Einstein Platform on Medium, where people are continuing the conversation by highlighting and responding to this story.

Welcome to Trailhead University

Imagine a university, say Trailhead University, is using machine learning to optimize its student counselors’ workloads. Rather than have its counselors spend time convincing each and every student to accept their admissions offers, the university wants counselors to focus on the most likely ones. Now, Trailhead University can reinvest these productivity gains into personally reaching out to students who are likely to matriculate.

How would they go about building an accurate predictive model? To predict the likelihood of a student matriculating, Einstein needs to be trained on examples of both students who accepted their offers and students who declined their offers. Let’s assume the training data looks like this:

If the university isn’t careful with its “training data,” it may unintentionally discriminate against some of its prospective students. In other words, the model may be biased. As seen in our previous blog (Einstein Prediction Builder: Thinking Through Predictions with Bias in Mind), we define bias as: to wrongfully impose a relative disadvantage on persons based on their membership in some salient social group, e.g., race or gender. The bottom line is: If your model is trained on “biased” data, its results will also be “biased.”

Before we dive into the four causes of bias, spend some time analyzing the above dataset. Ask yourself questions like:

• “Which groups here are advantaged vs. disadvantaged?”
• “Does the distribution of matriculating vs. non-matriculating students look equitable across the advantaged and disadvantaged groups?”
• “How may utilizing training data like this impact my model’s outputs, and accordingly, my stakeholders?”

Now that you’ve answered some of these questions, you’ve already accomplished the first step (of many) towards building a fair predictive model!

Four Causes of Bias in Data

1. Sample Bias

First off, your training data must be representative of your broader population. E.g., If you’re striving for a diverse class, you need to start reaching out to more groups of students. Since your model has been trained on two types of students, it’s bound to make inaccurate (and perhaps unfair) predictions on types of students it’s never seen before. E.g., Students with disabilities and male students of color aren’t represented in your sample. Hence, you’ve fallen for sample bias.

2. Data Imbalance

Your data has 2 times more men than women, which is likely to skew the model’s outputs towards men. This is a common problem in machine learning because data is rarely balanced. E.g., If you’re making a prediction on the state of California, its most populated cities (like Los Angeles and San Francisco) are bound to dominate the dataset. While techniques like upsampling and downsampling may help balance the datasets, we recommend exploring ways (as in the case of Sample Bias) to improve your marketing, outreach, and messaging towards broader groups.

3. Treatment Disparity

Among men, 75% matriculate. Among women, 16.67% matriculate. Although that’s a staggering difference, Treatment Disparity doesn’t always imply bias. Perhaps one group is more qualified or has more opportunities to succeed than the other. That said, if you notice that men receive a much higher relative % of matriculation than women do, they should also have higher campus visitation rates and demonstrate better cultural fit to support that difference. When in doubt, compare both groups quantitatively and qualitatively. Chances are, huge treatment disparity is hard to explain and therefore implies bias.

Food for thought: Even if men can “explain” their higher relative matriculation rates (through higher campus visitation rates, for example), it’s important to consider that men may also have more opportunities for success than other disadvantaged groups. Thus, Trailhead University may choose to pursue disadvantaged groups at higher rates to achieve equity in the long-run. E.g., Affirmative Action.

4. Outcome Disparity

In the Treatment Disparity example, we see that 75% of men accept their offers, while 16.67% of women accept their offers. If there were an equal number of men and women, then men would matriculate 4.5 times more than women [math check: 0.75/0.1667 ≈ 4.5]. However, our dataset has twice as many men as women, which means that overall, 9 times more men than women matriculate! See how Data Imbalance can multiply the effects of Treatment Disparity to form Outcome Disparity? Outcome Disparity may, yet again, be explained by statistical differences, but it’s up to you to investigate how your business processes lead to these differences in the first place. At the end of the day, your models are training on this label [accepted offers vs. declined offers]. It’s relatively self-explanatory that the model will, by default, favor men over women in its predictions.

Watch out for Proxy Bias

Referring to the diagram below, what are sensitive fields? Coming back to our definition of bias, sensitive fields are personal attributes that define a person’s membership in some salient social group, e.g., ethnicity or gender. Now, what are related fields? Related fields serve as proxies (hence why we call this Proxy Bias) to sensitive fields, i.e., they are correlated to or are predictive of sensitive fields. Let’s see an example of this below:

What related fields can you derive from a person’s name?

1. Gender (common male first names: James, John, Robert; common female first names: Jennifer, Susan, Lisa).
2. Ethnicity (common Vietnamese last name: Nguyen; common Indian last name: Patel).
3. Religion (common Muslim last name: Khan).

Even if you remove sensitive fields like Gender, Ethnicity, and Religion but include the related field Name, bias will be captured subliminally in your models.

The same goes for common related fields like Salutation and Postal Code. However, sensitive and related fields aren’t limited to these basic ones we’ve listed here. They vary based on industries, use cases, and specialties. E.g. In Banking, a loan applicant’s source of income can drastically impact their probability of receiving a loan, but that may not be the case in an industry like Healthcare.

Now what?

So you’ve identified all the types of bias in your data. Now what? Can we flip a switch on our iPhones?

Maybe that’s too easy, how about some code?

import library fairness // use function makeDataFair()
fairData = makeDataFair(biasedData) // problem solved!

Unfortunately, this is a complex issue, and there is no simple, “magical” solution. Bias is a systemic problem that exists because we are imperfect humans living in an imperfect society. The process of bias mitigation is hard work, but it all starts with knowing whether a problem exists. So here’s our advice to you:

1. Start with the 5W’s of Data Quality and Accuracy (see Kathy Baxter’s framework here: Dirty Data or Bias Data?)
2. Follow up on each of these questions with data visualizations that uncover the biases we discussed here.
3. Take your findings with a grain of salt — again, just because you see treatment and outcome disparities doesn’t mean you necessarily have bias. While comparing each of your groups, question your findings (refer to Natalie Casey’s business expert questions: Einstein Prediction Builder: Thinking Through Predictions with Bias in Mind).
4. Once you find potential areas of bias, spend time introspecting on how your business processes impact your data.
5. Lastly, think of ways you can change your business processes to achieve parity across groups. Perhaps, run marketing campaigns to attract more disadvantaged groups to your application funnel. Or build new products that cater to those groups. At the end of the day, your models are only as fair as your data is. And your data is only as fair as the business processes which generate them.

Good luck. And remember, Einstein’s here to help you through these challenges!

Trusted AI: Finding Bias in Your Data was originally published in Salesforce Einstein Platform on Medium, where people are continuing the conversation by highlighting and responding to this story.

One of the most useful predictions you can create for your business is predicting the likelihood of winning an opportunity (opportunity scoring). The higher the score, the more likely this opportunity is to reach the “Closed Won” stage. Next, you will probably want to know what amount this opportunity will close for — which we discuss in this blog. These predictions are very important, since they can affect three important KPIs — revenue, win rate and accuracy of forecasting. This blog describes how to define a use case, gather requirements, think through the problem definition and set up this prediction with Einstein Prediction Builder.

Defining a Use Case

Let’s look at a sample of the Einstein Use Case Worksheet. The worksheet helps walk you through some key concepts:

• What questions does your organization need to answer?
• What’s a good future to aim for?
• What value are we going to drive?

Gathering Requirements

Now that we have identified your use case, let’s gather more requirements to ensure that you:

• Build the right solution
• Identify key stakeholders
• Verify that you’re collecting relevant metrics and KPIs

Planning Your Prediction

To think through the relevant data to support these use cases, you can use the avocado framework (shown below), which aligns with the steps in the Prediction Builder wizard — you start with selecting an object, then decide if you want to focus on a segment and provide examples for Einstein to learn from.

1. Dataset — all records on the Opportunity object. Even though the dataset contain ALL opportunities, you can focus on a specific segment of data and exclude irrelevant opportunities — i.e. those in qualification stage. (Note, you can also use segmentation if you want to focus on a particular type of records — i.e. create a prediction for Enterprise Opportunities only)
2. Positive Examples — Are there any data examples that are showing the behavior you want to find? In the Opportunity Scoring example, you’re looking for won opportunities. They reached Closed Won stage.
3. Negative Examples — Are there any data examples that are showing behavior opposite of what you want to find? In the Opportunity Scoring example, these are lost opportunities. They reached Closed Lost stage.
4. Records to Predict/Score — What records do you not currently know the outcome for, but would like to predict? In the Opportunity Scoring example, these are the opportunities in any other stages, since you do not know the outcome and you want to predict which ones are more likely to be won so you can prioritize these opportunities.

Fun fact: if you are wondering why it is called “Avocado”, here is your answer — the image below looks like a ripe avocado with a pit inside.

While every org may be a little different, enter information according to what the different data buckets would look like in your org.

Tip for Defining Your Prediction Set

You do not have to explicitly specify which records to score since all records remaining in your segment after example filters are applied will automatically become your prediction set:
Segment records — example set = prediction set (or records to score).

Be sure to use the data checker in Einstein Prediction Builder to make sure you have the correct number of records, including positive examples, negative examples and records to score. You can also use reports to verify this, if in doubt.

The diagram below illustrates the final setup:

Setting Up Your Prediction with Filters in Einstein Prediction Builder

With the Spring ’20 release of Prediction Builder, you can set up your prediction without an explicitly defined field to predict. With opportunity scoring, we want to predict the likelihood of an opportunity to reach “Closed Won” stage, but we do not have a checkbox field to represent this outcome. In this case, we can use special filters to specify what outcome is considered positive and what outcome is negative.

This is how we can set this up with Prediction Builder:

1. Select the object you’d like to make a prediction on — Opportunity.

2. Define your segment using the filter under “Want to focus on a particular segment in your dataset?”

3. We are answering the question “Will this Opportunity be won?”, so we need to select “Yes/No” type for prediction.

Note: the prediction will return a number which corresponds to the likelihood of winning an opportunity, but this is still considered a Yes/No prediction.

4. We do not have a custom field created that stores the outcome of opportunities closed won, but we have a picklist instead (such as Stage: Closed Won, Closed Lost, New, Quoted, etc), so we select the “No Field” option.

5. Next, we need to define positive and negative examples using the “Yes” example and “No” example filters.

6. Include relevant fields. We recommend including all fields as you might get some unexpected insights; however, there are a few exceptions discussed in this post.

7. Pick the name of the field where your predictions will be stored. This is the field that will represent the opportunity score or the likelihood of winning an opportunity. It will show you a number from 0 to 99.

8. Review and build your prediction.

Setting Up Your Prediction in Einstein Prediction Builder with a Custom Formula Field

Alternatively, we can set up opportunity scoring using a formula field. The first method (using yes and no filters) is preferred, since it minimizes the probability of an error when creating a formula field and allows defining all prediction elements within the prediction builder UI.

This is how you can define opportunity scoring using a formula field:

1. Create a custom formula field returning text:

Custom Formula Field: Opportunity Outcome
CASE (StageName, “Closed Won”, “TRUE”, “Closed Lost”, “FALSE”, NULL)

• “TRUE” is returned for positive examples — Opportunities in Closed Won Stage,
• “FALSE” for negative examples — Opportunities in Closed Lost Stage,
• NULL is returned for opportunities in any other stages (we will score those).

This is how the setup looks in the avocado framework:

2. In the Prediction Builder wizard, steps 1–3 are the same as above. At step 4, we need to specify that the field to predict already exists.

3. Next, select this new custom formula field “Opportunity Outcome” as the field to predict, and select “Use all records that have a value for Opportunity Outcome”.

4. You can then continue with steps 6–8 listed above.

Next Steps

After you created your prediction, you need to review the scorecard.

If the quality of your prediction is too high, most likely you have hindsight bias and you need to eliminate potential leakers. For example, “Reason Lost” is an obvious leaker, since this field only gets populated once the opportunity is lost.

If the quality is too low, most likely you need to include more relevant data — can you create formula fields to bring data from related objects? Ask your business experts what data they would need to make this prediction — if it is useful for humans, most likely Prediction Builder can learn from it too. For example, you can add fields showing if this is a red account, number of severity 1 cases, % change in number of cases, customer success manager and solution engineer sentiment or assessment score, Account Tier, Account Health Score, average NPS score, lead product, and much more. Read more about prediction quality in this blog — Understanding the Quality of Your Predictions. To create the next iteration of your prediction, select “Clone” from the dropdown menu — it will save all your previous settings and you just need to make some small adjustments.

Do not forget to go to the Details tab of your scorecard. Examine your top predictors and validate that they make sense from a business perspective. Sometimes, you will find some surprising insights there — i.e. positive correlation shows which values of the selected fields correspond to a higher chance of winning an opportunity (positive predictive factors), while negative correlation shows which values of the selected fields are associated with lower win rate (negative predictive factors). Do not be discouraged if the insights are obvious — this only confirms that Prediction Builder is picking up the right patterns in your data.

When you are happy with the quality of your prediction, enable it to get the scores. To see the predicted values, add the Predictions field (opportunity score) to the list views and page layouts and optionally, add the Einstein Predictions Lightning Component to the page layout as well.

After a few weeks or months, you will get real-life data and you will know which opportunities ended up being won or lost. Then you can do a predicted vs actual analysis to understand how your prediction is performing on real data, using Salesforce reports or, if you have access to Einstein Analytics, you can use this Accuracy Template AppExchange package we developed for you.

Using Predictions in Your Business Processes

Here are some of the ways you can use this prediction:

1. Create a list view and sort by opportunity scores, so sales reps can prioritize opportunities with the highest likelihood to close. You can also review opportunities in the middle range (those are your borderline opportunities) and identify steps to get them back on track.
2. To show the top predictive factors — add the Einstein Predictions lightning component to the opportunity layout page, so users can see reasons behind these predictions.
3. Use Process Builder to automate task creation for prioritized opportunities.
4. Add opportunities with low likelihood of closing to the appropriate marketing campaign.
5. Use Einstein Next Best Action to provide the right recommendations to the sales reps for each opportunity based on the predictions and business rules. You can get idea of what to recommend for each opportunity based on the top positive predictive factors in the scorecard. For example, if organizing an executive briefing is associated with a higher win rate, you can recommend executive briefings for Enterprise accounts, while providing a different recommendation for SMB (i.e. industry webinar).

How to Assess the Effectiveness of Your AI Project

How do you know for sure that this AI project was a success? This is where you can go back to your original goal and review your KPIs — Win Rate and Revenue. You can look at YoY changes, but the gold standard for assessing any intervention is to use a control group. For example, you can show opportunity scores only to a small group of sales people, while others continue doing business as usual and will represent your control group (just make sure these groups are quite similar and minimize any other external factors that can influence your outcome). If there is an uplift in the KPIs in the opportunity scoring group compared to the control group, congratulations — your AI project has made the world a better place!

If you’d like to review the full process for building and deploying predictions to end users, see this recent post.

For more on the latest when planning your prediction, check out the official Salesforce documentation.

Related Blog Posts

1. Introduction to Machine Learning
2. How to turn your Idea into a Prediction
3. Einstein Prediction Builder Toolkit
4. How to Use Einstein Prediction Builder to Predict Opportunity Amounts
5. Which fields should I include or exclude from my model?
6. Understanding your Scorecard Metrics
7. Understanding the Quality of Your Prediction
8. A Model That’s Too Good to be True
9. Thinking Through Predictions with Bias in Mind
10. How do I know if my prediction is working?
11. Custom Logic on Predictions from Einstein Prediction Builder

How to Use Einstein Prediction Builder for Opportunity Scoring was originally published in Salesforce Einstein Platform on Medium, where people are continuing the conversation by highlighting and responding to this story.

Opportunity amounts change throughout the opportunity lifecycle, so predicting the final amount when the opportunity is closed becomes extremely important for forecasting. It can also have a positive impact on your revenue since knowing what factors are associated with bigger deals can give you valuable insights, and you can re-design your business processes to drive more revenue.

In this example, we are predicting the final opportunity amount, given that an opportunity is “closed won”. We are excluding opportunities that were lost since those amounts represent estimates by salespeople and are not proven by real-life data.

We highly recommend that your read this post first, so you understand the context behind the frameworks we are using.

Defining a Use Case

The worksheet below discusses the value that the opportunity amount prediction would help drive.

Gathering Requirements

Now that you have identified your use case, let’s gather some requirements.

Planning Your Prediction

For the opportunity amount use case, our avocado framework might be structured like this:

1. Dataset — all records in the Opportunity object. We want to exclude Lost Opportunities because it does not matter for our prediction what the estimated amount was for a lost opportunity. We will use a segment filter to select only records we want for our analysis: Stage Does Not equal to Closed Lost.
2. Examples — all Opportunities in Closed Won Stage and where the final amount was greater than 0.
3. Records to Predict/Score — Opportunities in any other stage.

Tip: You do not need to specify positive and negative examples since it is not a binary classification (Yes or No question). For numeric predictions, you just need to specify which records to use as examples.

While every org may be a little different, enter information according to what the different data buckets would look like in your org.

Setting Up Your Prediction in Einstein Prediction Builder

1. Select the “Opportunity” object and apply a segment filter.

2. We’ll be predicting a number (Opportunity Amount).

3. Add a new condition to specify that we want to only learn from opportunities that were won.

4. Include relevant fields. We recommend including all fields as you might get some unexpected insights; however, there are a few exceptions discussed in this post.

5. Pick the name of the field where your predictions will be stored. This is the field that will represent predicted opportunity amount.

6. Review and build your prediction.

Next Steps

After you created your prediction, you need to review the scorecard, iterate on your prediction, and enable your prediction to get scores. To see the predicted values, add the “Predictions” field (Predicted Amount) to the list views and page layouts.

If the quality of your prediction is too high, most likely, you have a hindsight bias, and you need to eliminate potential leakers. For example, “Sales Commission” is an obvious leaker since we can derive this field from Opportunity Amount.

If the prediction quality is too low, most likely, you need to include more relevant data — can you create formula fields to bring data from related objects? Ask your business experts what data they would need to estimate opportunity amount — if it is useful for humans, most likely Prediction Builder can learn from it too. Read more about prediction quality in this blog — Understanding the Quality of Your Prediction. To create the next iteration of your prediction, select “Clone” from the dropdown menu — it will save all your previous settings, and you just need to make some small adjustments.

Do not forget to go to the Details tab of your scorecard. Examine your top predictors and validate that they make sense from a business perspective. Sometimes, you will find some surprising insights there — i.e., positive correlation shows which values of the selected fields correspond to bigger deals (positive predictive factors), while negative correlation shows which values of the selected fields are associated with smaller deals (negative predictive factors). Do not be discouraged if the insights are obvious — this only confirms that Prediction Builder is picking up the right patterns in your data.

After a few weeks or months, you will get actual values for opportunity amounts, and you will know which opportunities ended up being won or lost. Then you can do a Predicted vs. Actual analysis to understand how your prediction is performing on real data, using salesforce reports or, if you have access to Einstein Analytics, this Accuracy Template AppExchange package we developed for you.

When analyzing predicted vs. actual for numeric values, it is important to look not only at absolute values but at % error as well. For example, if your opportunity amount was predicted to be $100,000, but it ended up as $90 000, an error of $10,000 is substantial in absolute terms (especially if a majority of your opportunities are less than $10K), but represents only a 10% error for this opportunity.

Besides showing the predicted amounts, you can improve your business processes to drive bigger deals — use Process Builder to automate task creation, use Einstein Next Best Action to show different recommendations based on the insights from the scorecard, and deploy personalized marketing campaigns for small, medium and large opportunities.

Finally, to assess the success of your AI project, always look back at your KPIs. Were you able to increase the win rate, revenue, and accuracy of forecasting? You can look at YoY or quarterly changes for comparison. Alternatively, you can conduct a pure scientific experiment with a control group. In essence, your pilot group will follow the improved business process with the prediction while the control group “gets the placebo.” If you see a substantial uplift in KPIs in your pilot group vs. control group, your project is a huge success, and you deserve a promotion. Next, you can predict the likelihood of being promoted, a promotion amount, number of days until promotion… — but with all seriousness, once you start predicting, it is hard to stop. Happy predicting!

Resources

1. Introduction to Machine Learning
2. How to turn your Idea into a Prediction
3. Einstein Prediction Builder Toolkit
4. How to Use Einstein Prediction Builder for Opportunity Scoring
5. Which fields should I include or exclude from my model?
6. Understanding your Scorecard Metrics
7. Understanding the Quality of Your Prediction
8. A Model That’s Too Good to be True
9. Thinking Through Predictions with Bias in Mind
10. How do I know if my prediction is working?
11. Custom Logic on Predictions from Einstein Prediction Builder

For additional help on Einstein Prediction Builder, check out Salesforce documentation and our modules on Trailhead.

How to Use Einstein Prediction Builder to Predict Opportunity Amounts was originally published in Salesforce Einstein Platform on Medium, where people are continuing the conversation by highlighting and responding to this story.

by Thierry Donneau-Golencer, Sr. Director, Einstein Product Management, Salesforce

Einstein Prediction Builder makes it easy to create a custom prediction for your business. You don’t have to worry about ETLing your data, data wrangling or picking which algorithm to use or tuning its parameters. Even better, you don’t have to worry about the infrastructure to run these models in production, model retraining over time, or how to integrate the predictions back into your business processes in Salesforce. Once set up, training and scoring happen automatically behind the scenes and the predictions are available on a custom field on the object where your data is stored, readily available on the records your end users interact with and for automation via Process Builder or Einstein Next Best Action for example.

Keeping all that in mind, setting up a prediction with Einstein Prediction Builder is only one step of the journey. This blog will take you through the six steps of the prediction lifecycle and help you turn your idea into a prediction.

Step 1: Define your use case

It all starts by identifying a business outcome that you want to improve.

For example, your sales team may have a low lead conversion rate, spending hours chasing leads that do not convert. Another example could be that your support team may get low CSAT scores because of high priority cases that got escalated while lower priority cases were being addressed. Maybe your churn rate is higher than you would like, and there might be a way to resolve that by targeting specific groups with special offers before you lose them.

You are probably already thinking about many potential use cases for your business and wondering where to start.

Here are a few questions that may help you select a use case:

1. Is it tied to a high-value business outcome? (if not, nobody will care)
2. How are you going to use the prediction?
3. Is there a business KPI you can measure and determine if the prediction has had an impact?
4. Do you have the data in Salesforce, and can you report on it? (if you can’t report on it, you can’t predict it!)

To help you think that through, there’s nothing like a good-old Mad Libs to make it fun. See this post to learn more.

The example I will use for the remainder of this post comes from a customer in Paris I worked very closely with. This was an up and coming business in the networking services industry and the problem they were facing was that many of their customers were paying their invoices late. In fact, after pulling a quick Salesforce report, they realized that only 35% of their invoices had been paid on time the past year! In turn, that caused them cash flow issues.
To address this issue, they used Einstein Prediction Builder to predict which invoices would be paid late. Here’s how they addressed the questions above:

1. Is it tied to a high-value business outcome? Yes, more money in the bank, less cash flow issues.
2. How are you going to use the prediction? Create a task for account managers, two weeks ahead of the due date, for invoices that were likely to be paid late, so they would remind their customers.
3. Is there a business KPI you can measure and determine if the prediction has had an impact? Yes, percentage increase in invoices paid on time.
4. Do you have the data in Salesforce, and can you report on it? Yes, the data is in the Invoice object in Salesforce.

Step 2: Identify the data that supports your use case

Now that you have picked a solid use case, the next step is to frame your prediction for Einstein Prediction Builder.

Prediction Builder can handle two types of predictions:

• Binary predictions (answering a Yes/No question)
• Numeric predictions (predicting a number)

In our case, we could frame the problem both ways:

• Will an invoice be paid late? (Yes/No question)
• How many days late is an invoice likely to be paid? (Numeric)

To answer these questions, Prediction Builder uses Machine Learning, leveraging historical data to predict the future. In a nutshell, Machine Learning algorithms find patterns in historical data to apply to new data and make predictions. If you want to know more, I recommend reading this Introduction to Machine Learning.

Going with the first question above (“Will an invoice be paid late?”), we will need examples of invoices that were paid late (answer is yes, so we call them positive examples) and examples of invoices that were paid on time (answer is no, so we call them negative examples). These records constitute the Example Set (also called Training Set).

Once Prediction Builder has been trained on those examples, it can then predict on records for which we don’t know the answer yet. These records are referred to as the Prediction Set (also called Scoring Set)

A handy tool to help you frame the problem correctly is the “Avocado Framework” (below).

Here’s some helpful information about the avocado framework:

• The dataset represents all the records in your object. In our case, the Invoice object.
• Within that, you can choose to segment your data if parts of your dataset are inherently different. In our case, we have both B2B and B2C customers. The ways we handle their invoices are pretty different from each other, so we are going to segment our data to focus on B2B customers that have invoices over $10K (as they represent 85% of our business).
• The Example Set is composed of invoices from the past. Some of them having been paid on time (records where invoice balance is 0, and the last payment date was before the due date) and others paid late (payment date is after due date, or it’s past due date, and there is still a balance).
• The Prediction Set is composed of records that have a balance but are not yet due.

If you want to create your own avocado diagram, you can get the template here.

Step 3: Create your prediction

Here is where Prediction Builder really shines. Once you have framed your question using the avocado framework, you can follow the screens in the Einstein Prediction Builder wizard and create your prediction in just a few minutes. Continuing on with our example, I’ve outlined the steps for you here:

1. Select the Invoice object and specify the segment.

2. Select “Yes/No”(the question we are asking is: “Will an invoice be paid late?”).

3. Select “No Field”.

Note: if you already had a checkbox field indicating which invoices have been paid late, you would choose “Field” here and select that field on the next screen.

4. Set up the conditions for “Yes examples” and “No examples” as defined in the avocado framework.

Yes (Positive Examples)

No (Negative Examples)

5. Choose which fields you’d like to include or exclude from the Invoice object.

We recommend keeping most fields selected; however, there are a few exceptions to keep in mind. See this post to learn more.

6. Pick the name of the field where your predictions will be stored, review, and build!

Step 4: Review, Iterate & Enable Your Prediction

Once your prediction is “Ready for Review,” click on the drop-down menu of your prediction and select “View Scorecard”.

The scorecard gives you access to different metrics on your prediction. You can learn more about how to review the metrics of your scorecard in this post.

When reviewing your scorecard, there are a few things to look for:

• On the “Overview Tab”, take a look at the “Prediction Quality.” In general, the higher, the better, but if it’s too high (greater than 95%), it might be too good to be true. This blog post can tell you more about the quality of your predictions.
• If you are in the red zone over 95%, it is probably because your model suffers from a common issue called label leakage. For example, the first time this customer created the late payment prediction, they were in the red because a field called “Late Payment Fee” had been left in. “Late Payment Fee” was a leaker as this information was never available at the time of prediction, but only after the fact, when the outcome was known. Fortunately, we have you covered. This post will help you better understand what leakers are and how to remove them from your models.
• On the “Details Tab”, sort by impact and look at the top 10–20 predictors. Here are some things to keep in mind as you look at this tab:

* Do the top predictors and the sign (positive or negative) for the correlation coefficients make sense based on your business knowledge?

* Are there any potential leakers in your model?

* Are there some fields that should be removed as they could introduce some bias? (this is a tricky one, but we got you covered again in this post)

* There might already be interesting business insights there; predictors that are actionable will be particularly interesting to you as you will be able to integrate them into your business processes right away. For example “Autopay” and “Payment Method” seem to have high impact here, so you could see how you could try to encourage more of your customers to switch payment methods via a campaign with potentially incentives such as a small discount.

Based on this analysis, you are most likely going to need to iterate a bit and tweak your prediction.

Common tweaks include:

• Adding or removing fields (usually leakers)
• Creating a segment to focus on parts of your data
• Updating your example filters
• Adding relevant data from other objects via formula fields or roll-up summary fields (using data from child and parent objects for your prediction is on our roadmap)

What is extra nice is that Prediction Builder makes it very fast and easy to do these iterations, as you can “Clone” or “Edit” your prediction, and everything will already be pre-filled for you.

Once you are happy with your prediction, click on the drop-down menu on your prediction and select “Enable”. This will trigger the initial scoring of your data and all the records in your Prediction Set (right side of the Avocado) will get a prediction.

Moving forward, all new records and updated records in that set will be re-scored on an hourly basis. The model will also be retrained automatically every month, so you don’t have to worry about it becoming stale over time!

Note: We also have a pilot for real-time scoring. Reach to your Salesforce account executive if you’re interested!

Step 5: Monitor your prediction

Enabling your model doesn’t necessarily mean it is ready to integrate into business processes just yet though! In fact, it is recommended to let it run for a period of time behind the scenes (or only surfacing scores to a small number of users) until you can assess its performance on new data.

After your model has run for a while on new data, you will indeed be able to know if it is really working by comparing your predictions with what actually happened! The timeframe for this analysis will vary based on your data throughput and the length of your business cycles, but a good rule of thumb is a couple of months.

An easy way to do this analysis is by using reports. This post will give you step by step instructions on how to set those up.

Below, you can see that for higher scores, most invoices ended up being paid late, while for lower scores, most were paid on time. It seems that our model is performing pretty well!

For those of you who have access to Einstein Analytics, we have also created this nifty little package that you can get on the AppExchange for free: Einstein Prediction Builder Model Accuracy Template

If you are happy with what you see, you are ready to move on to the next and final step - using your prediction!

Step 6: Deploy and use your prediction

There are multiple ways to use a Prediction Builder prediction in Salesforce:

• Add the prediction field to a list view and sort by score. If we used this method in our example, invoices that are more likely to be paid late will be listed at the top and brought to everyone’s attention.
• Add the Einstein Predictions Lightning Component to the Invoice page, so you can see the top predictors that influence each particular prediction. That will give you helpful information when you reach out to each customer.
• Run automated flows based on the prediction, using Process Builder. For example, sending a task to account managers two weeks ahead for invoices that are predicted to be paid late. This post will tell you how to set that up.
• Use your prediction along with business rules in an Einstein Next Best Action strategy. For example, you may decide to send a reminder to customers that are likely to pay late but only if they have not had a meeting with you in the last month and they are not part of an open up-sell opportunity.

However you decide to use your prediction, it is important to:

• Track your KPI from Step 1 of the prediction lifecycle. You can create reports to track those and review them regularly. This post will give you some ideas on the type of dashboards you can set up
• Consider a phased roll-out to collect qualitative and quantitative feedback from your users and improve as needed.
• Review your assumptions and the reports from Step 5 regularly. Even though the model gets retrained automatically, your business will change over time; new processes will be put in place, new fields added which may require modifications to the model.
• Manage the change in your organization. Transforming your business with AI will take time, and some business processes will have to evolve to reap the maximum benefits. Folks often ask me how AI can become a pain-killer instead of a vitamin for business, and for me, it has to do with the depth of integration into business processes. You will need to get buy-in from everyone involved (management and end-users) and show the benefits (hence, the importance of hard KPIs you can measure over time).

Now that you have set up successfully your first prediction, you will likely uncover many more predictions that would help your business and that you can set up with Einstein Prediction Builder. Fortunately, Einstein never sleeps!

—
Here are the different blog posts I mentioned that could guide you through the various steps of your journey from idea to prediction:

1. Introduction to Machine Learning
2. Einstein Prediction Builder Toolkit
3. Which fields should I include or exclude from my model?
4. Understanding your Scorecard Metrics
5. Understanding the Quality of Your Prediction
6. A Model That’s Too Good to be True
7. Thinking Through Predictions with Bias in Mind
8. How do I know if my prediction is working?
9. Custom Logic on Predictions from Einstein Prediction Builder

For additional help on Einstein Prediction Builder, check out Salesforce documentation and our modules on Trailhead.

Einstein Prediction Builder: How to turn your idea into a prediction was originally published in Salesforce Einstein Platform on Medium, where people are continuing the conversation by highlighting and responding to this story.

by Kevin Moore, Lead Data Scientist, Salesforce

Machine learning algorithms will learn patterns that are present in the data you show them, so be careful what you show them.

When you train a machine learning model, you’re implicitly telling the algorithm that the data you’re feeding it is trustworthy. You’re telling it, “Here are some examples of successes, here are some examples of failures. Extrapolate patterns from these so that we can predict the outcome of new records.” This can sometimes lead to surprising and unhelpful results, where the algorithm picks up on data that is filled in after the outcome is known.

As an example, imagine you’re a realtor and want to have predictions on whether a house will sell in a given timeframe. You’ve diligently collected the relevant data in a custom Salesforce object House__c, and have many past examples of houses that did and did not sell within the timeframe of interest — let’s say 3 months. Based on what you’ve read in Trailhead, this sounds like a great candidate to apply machine learning. A simplified version of the House__c object may look like this:

The houses you want to make predictions for would be houses that are on the market (preferably close to when they are posted), but not yet sold. This means certain pieces of information in the object will not be available, such as the final sale price, the closing date, closing costs, etc. You would instead expect your prediction to be based on information available before the house is sold, such as size/location data of the house in question, the asking price, and other similar quantities.

A human manually building a model would look at this data and manually exclude fields that aren’t available before the house is sold so that that model will only depend on the relevant fields that exist when predictions are required. However, the machine learning algorithms do not have wisdom on their own; they will simply do what you tell them (for more on how machine learning works, check this post out). If you ask a model to use all the fields in your object, then the algorithms will happily crunch all the data and find strong predictors of the outcome regardless of whether it makes business sense to use those fields. For example, a model may consist of the single rule that whenever the difference between” initial posting “and” close date “is less than 3 months, then the label is True! Moreover, the model will think it has done a great job since all its predictions on the existing labeled data turned out to be correct — even on the unseen holdout set it didn’t train on!

Unfortunately, this model would not be useful in making predictions in the context the realtor cares about. All its predictions on unsold would be negative because the close date is never filled out on unsold houses! We call this problem “Hindsight Bias or “Label Leakage,” see this Trailhead for more examples.

The machine learning pipelines powering Einstein Prediction Builder will do their best to remove fields that look like leakers, but these methods are not perfect. Hence, you need to be vigilant in inspecting your models for potential label leakage.

The main question you should ask yourself to determine if a field should be included in your prediction is:

• Do the values of this field look similar to the values on the records for which I want to make predictions?

This should filter out common leakage sources where a field is modified after the label is known. An example of this would be leaving in a “closed reason” field that can only be filled out when the prediction is negative. All records you’d want a real prediction on would not have this filled in, so it doesn’t make sense to include in your model.

It can also help filter out fields whose usages have drifted over time. Perhaps you used to have a process where you would try and predict the outcome by hand or use a prediction from some other source. Unless you are very careful about leaving these fields unchanged once the label is determined, then including them can cause label leakage. Or perhaps you just have a field (IdInUnusedExternalSystem) that was used in the past but isn’t used anymore. It’s better to leave that field out since it won’t be filled in on any of the new records that you want to make predictions on.

How to Diagnose Label Leakage in Your Model

The first thing to check is the model scorecard (see this post for more information on the Einstein Prediction Builder Scorecard). If the model quality is listed as “Too High”, then that may mean there was label leakage. It could be that the model was just able to do an excellent job at predicting what you asked, but such models are often too good to be true, so you should be especially wary of high model quality. Even if your model quality is modest, it is still essential to check your top predictors and see if they make sense for your use case.

In an extreme example, where there is a leaky field that didn’t get removed automatically, you could see it contributing much more than other fields of your object.

If you see a single feature jump out as much more impactful than everything else in the model, then you should inspect your data and see if that makes sense for your use case. This can sometimes be a sign of label leakage but is not always a bad sign. Your data and use case may have a single field that’s legitimately much more important than everything else, but it’s advisable to do a double-check.

Another way to diagnose things after a model is built is by looking at the predictors’ detail page on the ScoreCard, which will show a table containing information about the top features (by the impact), in particular the feature name, its impact, correlation, and weight.

The main things to pay attention to are the impact (this is the weight scaled to be between 0 and 1), and correlation. Are there any features with large correlations or large impacts that should not be there?

Einstein Prediction Builder will automatically remove features above a certain correlation threshold because they are typically proxies for the label. However, it can still leave in features that you don’t want in your model. Check the correlations and see if any of the high correlation features belong in your model. Are any of these features known before the label is known? Are they modified at all after the label is determined? If they’re modified after the label is known, you should make a new prediction where you remove them, since the model may have learned from information that is unavailable when you want to make predictions.

There are also automatic tests that check whether selected fields look similar between the training data (labeled data that passes the custom training filter) and the scoring data (everything else). If a field looks radically different between the training and scoring data sets, then that indicates the field is not useful in the prediction because the model will learn patterns from the training data that are not present in the data on which it will make predictions. For example, this would catch and remove a field that is always filled in for training data (like Close_Date__c in the house price example from the beginning) but is never filled in on the unlabeled records on which you want to make predictions.

Putting This Into Practice

Here are a few examples that are similar to real-world cases we have run into when diagnosing models.

In this case, the label is the “Converted” field, making it a binary classification problem. The status field encodes more detailed information on why a record converted or why it didn’t. If you were to train a model that predicted the “Converted” field and included “Status” in your model, then you would be introducing label leakage in your model. This is a more subtle case than the ones shown before because some choices do not leak information and could exist on the unlabeled records you want to make predictions on (e.g., “Waiting”). Still, other decisions clearly indicate what the label is. The “Too expensive” choice always goes with a negative outcome, while the “Converted — 12mo subscription” choice always goes with a positive outcome. The field as a whole is a leaky field, even though not every choice is. You can again apply the test of comparing what the values look like on labeled vs. unlabeled records, here a very different set of choices, and determine that this field is not a good field to include in your model.

To give one more example, let’s say you want to predict whether a customer will make a late payment on an invoice — here the “Late” binary field. Assume that this field is filled out either after the due date has passed without payment, in which case it’s late, or when payment is received. There’s also a “Days late” field that defaults to 0 and corresponds to how many days late the invoice payment is. A value of 0 in this field means either the due date hasn’t arrived, and there’s been no payment yet (so not late yet), or the payment was received on the due date. Negative values correspond to early payments, and positive values correspond to late payments. Including a field like “Days late” in your model will also contribute label leakage into your model because the value often depends on the label itself. Applying the test of comparing what the values look like on labeled vs. unlabeled records, you can see that this field typically looks different between the two, which means it is not a good field to include in the model.

Summary

Before training a model, be careful to inspect your data for fields that leak information about the label that is unavailable at prediction time. For each field you include, ask yourself, “Do the values of this field look similar to the values on records I want to make predictions on?” If the answer is no, then you should not include that field.

Einstein Prediction Builder: A Model That’s Too Good to be True was originally published in Salesforce Einstein Platform on Medium, where people are continuing the conversation by highlighting and responding to this story.

by Sara Asher, Senior Director of Product Management at Salesforce

Hi everyone! I’m very excited about one of our new features in the Spring ’20 release:“Filter-based predictions.” The purpose of this feature is to make it easy to construct your prediction within the Einstein Prediction Builder (EPB) wizard without needing to have a field to predict prepared in advance.

Before we go into details about filter-based predictions, let’s talk a little bit about how predictions work in general. Einstein Prediction Builder (and other machine learning systems) learn from examples in the past to make predictions on the future. One of the types of problems Einstein Prediction Builder concentrates on are yes/no questions (binary classifications). Examples of yes/no questions are: Is this opportunity going to convert? Is this invoice going to be on time? Is this student going to graduate high school? For new records, Prediction Builder will return a number between 0 and 100 that represents the likelihood that the answer to the question is yes.

In order for Einstein Prediction Builder to try to answer a yes/no question, it needs to have example records where the answer is definitely yes (positive examples), and example records where the answer is definitely no (negative examples). Once Einstein Prediction Builder has those examples, it can then predict on new records or on records where we don’t know the answer to the question yet.

So how do you tell EPB which records are your positive examples, and which records are your negative examples? In previous releases of Einstein Prediction Builder, you could select a checkbox and define an example set. Then, all records in the example set that were checked are the positive examples, and all records in the example set that were unchecked were the negative examples. (And the everything not in the example set would get predictions.)

That system worked pretty well for many use cases, but what should you do if you don’t have a checkbox like that? There is the option of creating a formula field, but that tends to be complicated and perhaps a bit error-prone. Instead, there should be an easy way for you to simply tell us which are the positive examples and which are the negative examples.

Introducing filter-based predictions!

Now, instead of needing an already existing field that represents what you want to predict, you can directly define your positive and negative examples within the wizard. Let’s run through an example to see how it works.

Let’s jump back to the question of whether a bill is going to be paid on-time. The positive examples are invoices where that balance is zero and was paid before the due date, and the negative examples (the late invoices) are ones where they paid, but the invoice was only paid off after the due date OR they still have a balance due, and the due date has already passed.

It is pretty easy to think of these as filters on your data.

Positive examples: Records where invoice balance is 0, and the last payment date was before the due date

Negative examples: Payment date is after due date OR it’s past due date, and there is still a balance

One note to consider when setting up filters like this: your “yes” examples and your “no” examples should be distinct from each other. Einstein can’t differentiate between your positive and negative examples if your records are both at the same time!

Anyway, now that we have a feel for how we could create filters to represent our positive and negative examples, let’s see how we would create this prediction within Einstein Prediction Builder.

Building a filter-based prediction

Just like any prediction, you first need to tell us what object the prediction is for and then let us know what kind of prediction you intend to create. In this case, the prediction is on the Invoice object, and we wish to answer a Yes/No question.

At this point, you will notice a new question from Einstein Prediction Builder: Is there a field that can answer your prediction question? If you already have a checkbox or formula field set up, choose the “Field” option. But in this case, we don’t have a field, so we choose the “No Field” option.

After selecting that option, you will see that you now have the opportunity to fill in “Yes” examples and “No” examples, which we can do just as we planned out above:

Once you’ve entered in your two filters, you can validate your logic by using the data checker to the right of the screen. The data checker will tell you how many positive and negative examples you have and how many records will receive predictions. The rest of the wizard remains the same. And that’s it! You can now create predictions without needing to have a field to predict prepared in advance.

We hope these new filter-based predictions are helpful when setting up your new predictions! Please try it out, let us know if you like it, and have fun predicting!

New in the Spring ’20 Release: Filter-Based Predictions was originally published in Salesforce Einstein Platform on Medium, where people are continuing the conversation by highlighting and responding to this story.

by Michael Weil, Senior Data Scientist at Salesforce

Customer data in Salesforce has a rich variety of field types ranging from string, double, picklist to date, datetime, and time. Einstein Prediction Builder takes into account these different types to prepare the data for modeling (for more information on modeling and some of the other terms used in this post, check out this blog).

In the case of date fields, it will leverage information about the day, day of the week, month, and year, the number of days between two dates, and more.

With the usage of date fields, here are some common misconceptions users should be aware of:

1. Leakage with Date

Dates fields are not immune to problems of leakage. A more in-depth post on data leakage is coming soon. The idea is that machine learning models will learn from information present in the data during training, but that will be missing on new records one is trying to predict until the true label is revealed.

Consider an example of an Opportunity object where the goal is to predict whether the opportunity will be won or not.

In the example above, Customer B and Customer C do not have the ClosedDate field filled in yet, the reason being that those are still in an intermediate stage- Negotiation/Review, Prospecting. By default, the label IsWon is No for these two customers. We can think that the label should be empty during intermediate stages, but we might have an example of records in which customers are “lost” early during sales development, and admins do not update the Stage field.

If we are training a model on this data and by keeping this ClosedDate field, it will “remember” the association between missing CloseDate and IsWon being No. On new data, the opportunities are not closed yet; therefore, the date field is missing. As a consequence, the model would predict No.

The problem boils down to dates that are posterior to the label. Other fields having similar issues as CloseDate include DaysToDate and DaysInStageX.

Other sources of leakage are automated processes done with the dates. Let’s take the example of Lead data where we are missing information on some lost leads: leads lost during a month have by default the first day of the month as the open date. If a lost lead had been opened in December 2019, the open date would be by default 12/1/2019. In that case, the model will learn an association between the open day being the 1st and the record being lost. This model will be heavily biased towards new records that are open during the first of the month.

Admins should also be cautious when trying to predict a formula field. For example, if the label is a formula field in relationship with a date field : LABEL = If(DATE >= 08/03/2019) TRUE Else FALSE
The field DATE determines LABEL, therefore it should be hidden from the model.

The same goes when including a date that is a formula field using the label: DATE = If(LABEL == TRUE) 12/31/2019 Else Null
This DATE will be useless in training since LABEL will be missing for new records to predict.

As you can see, dates are a significant source of leakage, and most of the time, it makes sense to exclude them when selecting which fields from your data to include in your model. For more guidance on which fields to include and exclude for your prediction, check out this blog.

2. Dates disguised as Strings

Admins can create many custom fields of different types. But sometimes the salesforce field types are misused. For example, admins might be tempted to create a custom field of type string despite containing dates.

Einstein can leverage interesting information from date fields such as the day of the week, the month of the year, etc. But in that case, as this CustomField is of type string, it can’t be inferred as a Date; therefore, we are losing this information. Be aware of this when choosing your field types!

The use of the typed string usually comes from the fact that dates are not in the same format. In the example below, some dates are not in the MM/DD/YYYY format. Besides making Einstein Prediction Builder’s life easier, using a Date type will bring consistency to your data as an added benefit!

3. The Case of System Fields

In addition to custom fields, Salesforce contains generic fields called System Fields. Those are fields that are updated during API operations such as record creation, record updates, etc. Some of these System Fields are dates: CreatedDate, LastModifiedDate, SystemModstamp. In general, when training the model, these fields are automatically filtered out as those dates are irrelevant for building a prediction. But there might still be a risk.

Let’s take the example of an admin trying to predict a Sale Cycle Length using this formula :
Sales_Cycle_Length__c = CloseDate__c - CreatedDate

This formula is probably not what the admin wanted, as the system field CreatedDate indicates when the API created the record, not necessarily when the user did. For instance, if the data has been uploaded once in bulk, the value of CreatedDate corresponds to the date of this bulk upload.

You should consider removing fields that are (or related to) System Fields. Also, you should specify your own created date (as a custom field) as a best practice: CreatedDate__c

Another word of caution regarding system fields: fields are not being reevaluated in real-time.

For instance, let’s say you have a formula field with Now + X # of days, , for example, you define your training set for a membership renewal scenario as: CreatedDate > Now + 90 days. “Now" will not be updated automatically daily but only once a month, at the time of training, when it will be substituted with the actual date and records that meet Training filter requirement at that time will be used for training

4. Mixing historical data

For some use cases, a wide range of historical data might be available throughout the years, and it might be better to segment data accordingly to avoid some mix-up. Especially if the business processes what a specific file is used for, or the way to collect data has changed over time.

There is also the odd case where the same instance is evolving over time. For example, if an admin wants to predict who is likely to become part of a frequent flyer program, it could be that some customers have fallen in and out of status over time, so there is a chance to encounter multiple instances of the same customer :

In this case, there are records of Customer A in both 2020 and 2018. In 2018, this customer was a frequent flyer; in 2020, she is not anymore. This indicates that this data has a time component in which records change over time. It is not necessarily a yearly cadence; The period can be in months, days, seconds.

In that sort of problem, it would be desirable to select the data accordingly. Potential ways to address this scenario include training on 2019 data in order to predict 2020, picking the most recent record for a given customer, or setting it up in such a way that a customer is considered a Frequent Flyer (“Yes Label”) if she/he has ever been a Frequent Flyer.

5. Time Series

As seen above, admins sometimes want to solve specific problems where dates/time play a huge part. In the case of records that are ordered by time, the use of models to predict future values is then called time series forecasting. A date field indexes data and usually equally spaced by time (minutes, days, months,…).

Examples of such predictions include predicting sales price, weather temperature, number of bookings, and case volume.

Time Series is generally composed of a systematic pattern and some random noise. In addition, you can decompose the pattern into:

• Trend — a component that changes over time and does not repeat.
• Seasonality — a component that repeats periodically.

Time Series forecasting has its variety of techniques like (seasonal) ARIMA models or Deep Learning.

Einstein Prediction Builder does not currently support those methods.

If you think your prediction might be a time-series, please consider another tool for predicting the forecast, such as Einstein Analytics Time Series.

Complication with Dates V2 was originally published in Salesforce Einstein Platform on Medium, where people are continuing the conversation by highlighting and responding to this story.

When configuring a new prediction, one of the steps involved is choosing which fields from your data you would like to include when building a predictive model. Since all of the predictive power of a model comes from what data we choose to show it, selecting the right fields is important for getting good predictions.

Consider an example of predicting which of your sales Opportunities are most likely to be won. An example of how the set of Opportunities could look is below:

The field we want to predict in this example is IsWon and the other fields are possible candidates to include as inputs into the predictive model.

What to Include

In short, include as much as you can. You may have some ideas about certain fields that would be useful for making predictions already. For our Opportunities example, maybe you know that they are more likely to be won when the Amount is not too high or when they come from a certain LeadSource or when the LastActivityDate is not so long ago that it has become stale. You should certainly include those fields. However, there could also be predictive power in fields that you might not expect. The opportunities from certain ContactId’s might convert better and lots of information could potentially be gained from the Description field even though it is just free text.

The point is, there can be many tiny signals in your data that can help indicate what the final outcome may be. You may not always notice them yourself or even be aware of them, but a predictive model can leverage them to make your predictions as good as possible. Generally, the more data you give to it, the better it can be.

What to Exclude

With that said, there are still certain kinds of fields that you should probably not include in your model. While generally more data is better, there are some exceptions for ethical, legal, and prediction quality reasons.

Ethical Concerns

If you are using the predictions of a model to make any kind of business decision, you are also indirectly using the information you used to produce that model in your decision. There can be a variety of reasons that using certain types of data for decision-making can cause ethical concerns, and it will depend both on what’s in the data and the problem you are applying it to. For example, it would make a lot of sense to include a customer’s gender when you are trying to decide items of clothing to recommend, but you would probably not want to use it if trying to predict what salary you should suggest when making a job offer. A quick check is to fill in the data field you are using and the problem you are solving into the following statement:

I am using <field x> to help me with <problem y>

If we apply this test to the examples above it would become,

I am using a customer’s gender to help me make the best clothing recommendation possible. ✅

I am using a customer’s gender to help me decide what starting salary to offer. ❌

If you are not comfortable making that statement, the field should not be included in your model. For more details on ethical use of data and bias, see this post.

Legal Concerns

There can also be situations where it is prohibited by law to use certain information when making decisions. If a field contains information on a person’s race, religion, gender, or nationality, you wouldn’t want to use it as input on something like making hiring decisions in places, such as the United States, where such a thing is not allowed.

You can apply the same test as when evaluating potential Ethical Concerns here as well and ask yourself if there could be any legal restrictions on including certain fields in your decision-making process. If your business involves things like decisions on employment, lending, healthcare, or any other similarly regulated areas, it is worth reviewing the list of fields you are using.

Fields with “Hindsight Bias”

There are certain situations where including a field in your predictions can actually make them worse. We can say that these fields show a “hindsight bias”. This is a field where the contents are filled in or updated on a record some time after the final value of the prediction field is determined. An example of this would be filling in the sale “Value” of an Opportunity at the time when it is won. The Value field would appear to be a very good predictor of winning an Opportunity since whenever it is present, the Opportunity is won every time. However, we cannot actually use Value as a predictor in practice since it is never available before the Opportunity is won (that is, it only looks like a good predictor “in hindsight”). Some other general examples of this type of issue include:

• Fields that are only filled at time of “conversion” or after, such as in the “Value” example above.
• Formula fields that depend on the thing you are trying to predict should be excluded. For example, you may have a field that you use to identify a follow-up after an opportunity is won whose formula starts with IF IsWon AND .... This field should not be included.
• If the field you are trying to predict is a formula field, any fields that appear in that formula should not be used. Suppose that instead of predicting IsWon, you are predicting another field, ExpectedValue which is equal to the formula (Value * LikelihoodToWin). In this case, you should exclude both the Value and LikelihoodToWin fields.

If you have any fields that fit these criteria, they should probably not be included in making your predictions.

Using Feedback from The Scorecard

You may also find additional fields to exclude from you model by looking at your model scorecard after the first time you produce a prediction. We can look at the Predictors and Details tabs to see how fields have influenced the predictions and look for a few different indicators.

If you see things like a field that has a correlation that is much higher than you would expect, especially if that one field has a much much higher Impact than everything else, you may want to consider excluding it from your predictions.

The example scorecard above shows a good example of a field that should be removed. The combination of a Prediction Quality that is “Too High” (99) and a single Top Predictor that is much higher than the others (Value) is a good indicator that that field should be considered for removal.

You can learn more about the Einstein Prediction Builder scorecard here.

Einstein Prediction Builder: Which fields should I include or exclude from my model? was originally published in Salesforce Einstein Platform on Medium, where people are continuing the conversation by highlighting and responding to this story.