Lessons Learned as an AI Product Manager at Tesla Energy
- Real-world datasets are messy. Identify early on which aspects of the data are critical and which parts are deficient.
- Focus on the customer and the value proposition. Ruthlessly simulate, educate and simplify.
- AI is not a silver bullet. Many non-AI factors are integral to building a successful AI product, or deciding whether AI needs to be used in the first place.
For the past three years, I worked on Opticaster, Tesla’s artificial intelligence (AI) software for commercial energy storage products. It has been truly humbling to witness this software being shipped to thousands of homes and businesses across the globe, minimizing customers’ utility bill expenses, increasing renewable consumption, and keeping the lights on when off-grid. I would love to share four lessons learned through my experience as an AI product manager in clean energy. (Many of these lessons will apply to enterprise machine-learning-as-a-service software product beyond energy.)
Lesson 1: Energy Data Is Messy
Data is messy by default. No one ever believes that they will have a perfect dataset. However, I was still caught off guard when I started dealing with real-world energy data at Tesla.
Energy data is ubiquitous, dynamic, and incredibly disorganized. Smart meters can now track near real-time energy consumption profiles at the sub-minute level; however, the quality and format of such data vary drastically. Little is known about the day-to-day usage patterns of the majority of ratepayers without smart meters, which limits the information with which models can be trained. Energy transactions and operation data are also market-specific, with arbitrary protocols, interfaces, and latencies. Every utility and government agency has its own way of consuming, publishing, and maintaining utility rate plans, incentives programs, and regulatory requirements, which become buried in mountains of PDF documents.
The problem is that there is no universal data sharing or standardization of energy data. While we see many initiatives from both the public and private sectors to consolidate how we use, communicate, and interpret energy data (e.g., OpenADR, OCPP, MESA, etc.), databases and standards are poorly maintained. Often, they are too loose and open to interpretation, or too restrictive to be widely adopted by industry.
So, how do product managers deal with this mess? First, we must see beyond the chaos by focusing on the part of the dataset that truly matters. It is easy to become distracted and overwhelmed by all the edge cases and lose sight of the customer’s perspective. We must ask ourselves: do we really need to ingest this data? How much will it impact the customer experience? Would the customer even notice the <1% improvement on bill savings? For example, we don’t need complicated utility discounts and cost adders in the inputs when the value of the battery only comes from time-of-use arbitrage. Adding local weather into demand forecasts is unnecessary if customer operations are not affected by temperature, irradiance, or cloud cover.
Secondly, product managers need to help identify data deficiencies proactively. Never view data cleaning as just an engineer’s job. We must be fully aware of data quality because a problematic training dataset not only hurts performance when building models — it can also result in numerous production issues. Time series data is intimidating, but data-centric product managers need to be comfortable inspecting datasets in order to provide context and guidance on data sources to the engineering team early in the data processing stage.
Lesson 2: Understand the Customer and All Stakeholders
The B2B energy storage world contains many types of stakeholders, each with unique interests and responsibilities. Channel partners may secure deals with end-customers who cannot be reached directly. The corporate leadership of the end-customer is often keen to understand impacts to sustainability. Meanwhile, a bank may finance and own a project and only cares about returns on assets. Authorities Having Jurisdiction (AHJs), such as the local utility company, must also inspect, test, and approve the project in order to operate. Separately, incentive administrators who subsidize renewables projects often require performance and compliance reporting, and the facility manager always wants to know if the system is operating appropriately.
A B2B Energy AI solution, like any enterprise software product, will never make all stakeholders happy. In fact, I spent the majority of my time balancing stakeholders’ needs throughout project life cycles rather than deploying the most cutting-edge algorithms. As a B2B product manager, it is crucial to ruthlessly search for the highest-value common denominator to prioritize while juggling complicated requests and tradeoffs among stakeholders. People with different backgrounds, technical expertise, and business interests will perceive the software very differently. It is important to carefully craft the product message accordingly for each stakeholder.
Lesson 3: Focus on the Value Proposition
AI in the internet industry is intuitive to end-users. Google returns a list of answers to our searches. Netflix suggests hundreds of movies for us to watch every week. Cars change lanes and stop at traffic lights based on surrounding traffic. Customer perceptions of these AI-powered recommendations or actions are straightforward.
However, this is not the case for energy.
For energy storage and other dispatchable energy assets, a core part of the value of AI and machine learning (ML) is enhanced model predictive control. The system forecasts various aspects of the storage system’s environment, such as renewable generation, power consumption, greenhouse gas emission, market prices, etc. With these forecasts and a set of constraints, a Battery Management System (BMS) executes an optimized set of battery charge/discharge commands to maximize the value delivered by the system. The value can be a demand charge reduction, time-of-use arbitrage, incentives, and grid service revenue, or a combination of multiple value streams.
These controls are complex and the outputs are non-intuitive. Can you make sense of the time-series graph below?
When a customer cannot immediately comprehend the model outputs focusing on the value proposition of the AI solution becomes critical.
How do we do that? Three keywords: simulate, educate, and simplify.
Simulate
A well-rehearsed product demo or a fluffy case study is not sufficient to entice customers to sign up for an AI solution. Logical, value-driven B2B customers always crave customized insights that resonate with their current conditions. “Your product looks great,” they’ll say, “but what can it do for me?”
Simulations with customer-specific inputs in the sales stage are a powerful way to build trust with customers. They will immediately see how the AI solution would specifically benefit them well before signing a contract.
Recently, I asked an existing customer to participate in our virtual power plant program. They were very hesitant at first because they didn’t even know what a virtual power plant was. So, we took their historic data, modeled the potential outcomes with the exact ML algorithms deployed in the field, and laid out the risks and benefits. That customer immediately signed up after the meeting.
Of course, simulations can be time-consuming, require technical knowledge, and create additional sales overhead. Thus, AI product managers must build easy-to-use tools that scale and democratize simulation functionality. Instead of relying on a handful of engineers to run simulations, we should strive to empower everyone (e.g., sales, product, fleet service, engineers, and partners) to run simulations themselves and pitch the benefits of AI to customers.
Educate
AI-powered energy management solutions are rarely self-explanatory. Customers will inevitably detect unexpected control behaviors that cannot be addressed by a quick inspection. I often ran into situations where customers wrote lengthy angry emails about why the battery should have done X, Y, and Z. They would come in with very biased opinions on how exactly our product should behave, and completely ignore forecast uncertainties, changing site conditions, and hardware limitations.
However, the customer is not always right. Customers need constant education at every stage of the product life cycle. This can be accomplished by creating easily accessible documentation such as product overviews, FAQs, and tooltips. For more sophisticated customers with advanced technical appetites, it can be helpful to display forecasts and real-time control reasons. The more educated customers are about the control decisions made by the technology and the value they provide, the happier they will be using the product.
Simplify
Customer education and simulations are effective, but they are also resource-intensive. We can’t ask every salesperson to apply simulation in every deal, or force customers to read user manuals back-to-back. In my view, the most powerful approach is to radically simplify the customer’s experience when using the product.
Instead of having customers waste hours scratching their heads over an abstruse battery dispatch profile and desperately calling technical support for help, simplify their experience by giving them what they actually need. Provide cost savings reports regularly. Send paychecks for their participation in virtual power plants. Aggregate the key metrics so that they can present a dashboard in their executive meetings. Sell them performance guarantees that promise revenues regardless of what happens.
One elegant, simple solution Tesla offers is the Tesla Energy Plan. By enrolling in this program, Tesla customers are immediately qualified for a discounted rate by utility companies. The Tesla Energy Plan has already launched in Australia and the UK, saving over 30% on customers’ electricity bills and preventing blackouts for millions of residents.
Simulate, educate, and simplify. AI product managers must continuously quantify and articulate the value propositions of AI products to customers.
Lesson 4. AI Is Not Everything, and in Some Cases, It’s Hardly Anything.
Okay, let’s take a deep breath and face the hard truth.
Sometimes customers just don’t need AI. Either they can do better themselves, or they just don’t care. For customers with intimate knowledge about their facility operations, humans and rule-based logic can do a decent job.
I once had a customer who operated a heavy-duty machine based on appointments. They knew exactly when and how much power it would generate at any given time. As a result of this careful control, they knew exactly what to do with their batteries, and AI in their specific use case only serves as an optional secondary control layer in case rule-based control fails. Even if an autonomous solution is better than a manual, human-driven process, the marginal benefits sometimes hardly justify the price premium.
Second, emotional triggers can heavily influence people’s perceptions of AI-driven energy products. This is difficult to capture in the algorithms. For example, when the utility notifies you of a potential power outage coming up in two days, do you want to have your battery optimizing the utility bill or reserving its energy for the upcoming outage? What if your power will be cut off in just two hours? Do you want Google Nest (i.e., a smart thermostat), to mess with your temperature settings in the middle of a major heatwave?
There is also too much at stake for the energy sector to fully hand over the reins to AI. Unlike an easily discarded, ML-selected Spotify song recommendation, the wrong power system decision could cause a blackout across the system with huge economic impacts. Large utility customers will never let a third-party software run on Autopilot and take complete control of their assets; they want to be informed by AI in decision making, but they want to make the final call. Until AI can prove a step-change in reliability, safety, and financial upsides, energy customers will continue to rely mostly on rule-based, empirical solutions. AI can help augment operator experience, but it will take time for it to fully automate the energy industry.
Lastly, AI is only a solution, and it’s just part of the solution.
Energy system operations are crucially dependent on the physical world. The success of an AI-powered intelligent control requires a highly reliable infrastructure and properly-functioning hardware. People seem to forget that IoT stands for the Internet of Things. Well, these “things” are devices that must work in the first place in order to generate value. AI means nothing if a device constantly loses communication or has a faulty piece of hardware. Building high-performing AI/ML algorithms are only one part of an AI-powered energy product. Hardware and software must be fully tested and seamlessly integrated to deliver an impactful customer experience.
Many clean-tech companies tout their cutting-edge AI technology. This may be an effective marketing tool, but the exact solution is often poorly defined. Product managers must dig deeper into customers’ needs and pains before jumping the gun and whipping up some AI elixir.
This phenomenon is nicely captured in this quote from an energy software startup CEO:
“AI and machine learning shops are too often looking for problems to solve rather than addressing the very specific problems that utilities face.”
— Joshua Wong, CEO, Opus One Solutions
Conclusion
My four lessons as an AI PM at Tesla Energy are as follows: (1) master your messy energy datasets, (2) prioritize among various stakeholders, (3) stay laser-focused on the value propositions, and (4) recognize AI as only a small part of a solution. It has been a transformative, humbling experience to tackle some of the most difficult challenges in the energy industry. Together, let’s harness the power of AI to save our planet.
This article was adapted from my talk for the Undistancing Project (now GoBite) on Jan 20, 2021. Opinions are my own and not the views of Tesla.
Acknowledgments
Many thanks to Jason Koeller and Katherine He for reviewing the draft of this article and providing helpful feedback.
Siyi is on a mission to make affordable, reliable renewable energy and EV charging infrastructure accessible to anyone, anytime anywhere. She is currently a senior product manager at EV Connect. Previously she led the B2B product lifecycle of machine learning algorithms and analytic tools for Tesla Megapack, Powerpack, and Supercharger.
