Power Take-Off: Processing AUX Signals to Account for Productive Idling

Power takeoff (PTO) refers to taking power from an engine and transmitting it to another machine, such as a cement mixer. Motive has recently released a PTO feature, which detects idle time and provides fleets with PTO usage data, enabling them to distinguish their vehicles’ idle time (engine on, no motion) from productive idle time (engine on to power equipment). In this blog we describe the product and scaling challenges our engineering team overcame in building and implementing the PTO feature.

Background

Many of our customers in the construction, agriculture, oil, and gas industries use fuel to run various equipment attached to their vehicles, such as cement mixers, sweeper trucks, etc. In these industries, the term for this utilization of power is power takeoff (PTO), used when power from a vehicle engine is used to run another machine. When a vehicle engine is running and not in motion, it is either actually idling or productively idling to power attached equipment.

To measure efficiency, it is crucial to distinguish between actual idle time and productive idle time. Fleets with such equipment need accurate PTO usage data to track driver activity, fuel use, and safety. Until recently, Motive counted PTO time toward idle time, but we wanted to account for cases where a vehicle is idling with its engine on so that a cement mixer could do productive work. We recently introduced the PTO feature to enhance our product offerings.

Architectural Context

A few words about the architectural context for building this feature: Information about each vehicle is collected through Motive’s Vehicle Gateway (VG) — a hardware device connected to the diagnostic port (ECM) of a vehicle — and sent to our backend. The role of Motive’s Vehicle Gateway is to read the CAN bus data, engine data, and the odometer and vehicle speed data to detect motion events and idle events. Some of these data are posted as “breadcrumbs.” We build several applications on top of the high frequency telematics data we receive from the VG.

Objectives and Challenges

Our primary objectives for this new feature were:

• Include information on location and duration of PTO use (important for fleets to bill customers accurately and to identify driver misuse)
• Distinguish true idling from idling due to PTO (fleets can find opportunities to save money on fuel through driver coaching)
• Receive alerts during unsafe scenarios where PTO is engaged on a moving vehicle (fleets can improve safety)

Meeting these objectives required overcoming a number of challenges:

• Designing for scalability to record the hundreds of millions of signals sent from all of our Vehicle Gateways per day
• Supporting backdated user configuration updates, requiring the activity of auxiliary equipment (such as concrete mixers) that we report to the user to be re-computed
• Handling non-contiguous signals (out of order messages) that arrive after those with later timestamps have already been grouped into events
• State management of the auxiliary equipment cable’s connection object, which represents the unique combination of a vehicle, its Vehicle Gateway, and the VG’s AUX input slot

Below, we explain these challenges in more detail and present our solutions.

Basic Building Blocks

Motion and Idle Events

An event, motion or idle, is a time period (with a started_at and an ended_at timestamp), during which a vehicle was either in motion or idling with the engine on. As the name suggests, a motion event represents the time the vehicle was in motion, and an idle event represents the time the vehicle was stationary, but with the engine running. Now, each of these motion and idle events contains information about the fuel readings at the start and end of the event, start and end locations, odometer values, and other telematics data that we collect from the vehicle.

Note that the idle event and motion event are mutually exclusive events.

Motion and idle events are generated inside the Vehicle Gateway and then pushed to our backend. Our backend processes these events to generate further data, insights for reports, and web dashboards.

Breadcrumbs

Breadcrumbs are messages created both at regular intervals and when triggered by a change in the important parameters of a vehicle, such as speed, state of the vehicle (motion, idling, auxiliary input status on and off, engine on and off). A breadcrumb carries a lot of information, such as the vehicle’s fuel reading, odometer reading, engine hours, start and end times, etc. Think of a breadcrumb as a snapshot of the vehicle at a point in time.

We have 700K+ vehicles that send breadcrumbs almost every 30–60 seconds, which adds up to around 400M messages per day. These messages are then processed in the backend for various use cases, such as alerting and real-time tracking.

Detecting PTO Activity

Motive’s Vehicle Gateway is responsible for monitoring electrical signals and reporting the status of the monitored equipment. The first solution we needed was a way for the Vehicle Gateway to interface with external equipment. For this, our hardware team created a new cable designed to monitor discrete electrical inputs, which can come from equipment such as a seat belt, panic button (provided in vehicles to notify authorities in case of emergency; in addition, we can detect its use and inform the fleet managers specified during configuration), PTO equipment, and more. This auxiliary (AUX) universal cable works for any equipment that provides an electrical differential between states (that is, seat belt connected vs. seat belt not connected, or PTO engaged vs. PTO disabled).

The Vehicle Gateway monitors this binary input and periodically reports the status of the signal to our servers. Because the external equipment and the precise signal that indicates an “ON” status for each type of equipment can vary, configuration is required on the server side to properly report signal status. For example, when some types of equipment send a low electrical signal, it means the status is ON, while for other types of equipment, a low electrical signal means the status is OFF (we refer to this as reverse polarity).

To take reverse polarity into account when reporting AUX activities, we asked users to set up the AUX connections on our dashboard and configure them appropriately as a one-time setup before using this feature.

Figure 1. User configuration in the dashboard to assign ‘ON’ or ‘OFF’ meaning to a ‘HIGH’ electrical signal

To mandate users to update the configuration, we designed a state machine for each AUX connection established, which tracks various factors, including whether the user has completed the configuration we need to infer signal meanings. We do not generate AUX activities from the AUX events for reporting until the AUX connections for the user are in a desirable state (meaning that the user has configured signal meanings on our dashboard, and the connection is sending signals; we talk more about AUX connection states further on).

Transmitting PTO Data to Server

Now, let’s explore how we encoded the data in the breadcrumb and sent it to our server. Our Vehicle Gateway supports two AUX connections at any given time. A customer can connect up to two universal AUX cables from the VG to any two pieces of equipment they want to monitor (we plan to support more connections in the future). For efficient transport of information, we use a single 64-bit integer to transport the state of all the connected AUX connections in the breadcrumb. This addresses scalability: While we support two AUX inputs today, in the future we will be able to support up to 32.

As illustrated below, each bit represents an AUX connection status. Later, that signal is interpreted as either ON or OFF, depending on the configuration set up by the user.

Figure 2. Inputs from the AUX cable

Persisting PTO Data

Just as we generate idle and motion events for the vehicles themselves, so we needed to generate events for each AUX activity. We build events from groups of time-contiguous breadcrumbs that all report the same status. For instance, a contiguous stream of breadcrumbs sending a LOW signal is captured as one event, which ends when a breadcrumb reporting a HIGH signal arrives.

As mentioned earlier, breadcrumbs provide us with snapshot information about the vehicle’s status (fuel, odometer, engine, start and end times, etc.). We use the snapshot information of the first LOW-signal breadcrumb to describe the vehicle’s state at the start of the event, and the information from the first HIGH-signal breadcrumb to describe the vehicle’s state at the end of the event, as well as its state at the beginning of the next event.

Not only does an AUX event have information describing the vehicle at the start and end of the event, but each AUX event also has a state of either HIGH or LOW, depending on the signal type received from the Vehicle Gateway. As we mentioned earlier, a HIGH signal doesn’t necessarily denote an ON activity, so we consult the AUX configuration set by the user in order to infer the ON or OFF state using this signal. Therefore, AUX events are just a raw representation of what we receive from the vehicle; they are not what the customer sees. We process raw events into activities using additional information from the Vehicle Gateway and the customer’s configuration.

In addition to a state, an event also has a duration. Event duration is defined by changes in state reported by the breadcrumbs. In Figure 3 (below), we use a dotted line to represent multiple breadcrumbs received over a period of time. Let’s say that each dot is a breadcrumb. Because breadcrumbs are sent frequently, most repeat the same signal, such as HIGH, HIGH, HIGH, and HIGH. We group all these into one event, whose state is HIGH.

Eventually, we receive a change of signal and we use the data in this breadcrumb to finish the current event and start a new event. In Figure 3 (below), the preliminary AUX signal is HIGH and the breadcrumbs continue to send HIGH signals until we eventually receive a LOW signal. These two breadcrumbs serve as bookends to AUX Event 1, whose state is HIGH.

Figure 3. How breadcrumbs become events

To distinguish actual idling from productive idling, we had two options:

1. Change how idle events for the vehicle currently work in our system; that is, end an ongoing idle event when PTO is turned ON and start a new vehicle idling event when PTO is turned OFF.
2. Generate the AUX events in the backend, counting the productive portions of idle events in a separate bucket, while continuing to generate actual idle events for the vehicle as they are currently defined.

We selected the second option: generating the AUX events in the backend. With this option, the processing is performed in the backend and we can provide faster turnaround in case of bugs.

You may notice that in Figure 3 (above), there are two consecutive HIGH events: (1) AUX Event 3 and (2) Aux Event 4. This is not uncommon. The same electrical signal can be sent in multiple scenarios. For example, for a user with reverse polarity, we receive a LOW signal when the auxiliary power source is ‘ON’, but we also receive a LOW electrical signal from the auxiliary power source when the main engine is turned off. For these two breadcrumbs the actual events are very different. If the two breadcrumbs arrive consecutively, it would be a mistake to group them into one event. We use other messages from the Vehicle Gateway to detect whether some additional state other than the AUX has changed (such as the main engine being turned off and on). If we detect other pertinent changes, we create two separate events. They may have the same electrical state, but using additional VG data may end up creating separate activities from them downstream.

For this reason, we process breadcrumb data to help us define these raw events and then we process events, together with the customer’s configuration setup, to define activities–what the customer sees.

Challenges and Design Decisions

Having discussed the moving parts, let’s talk through the challenges we faced and how we implemented this feature.

1. The Scalability Challenge

As stated earlier, we use the breadcrumbs to generate AUX events. Because we have hundreds of thousands of vehicles actively using our Vehicle Gateways, we needed an efficient way to record the signals sent from all of these VGs.

We used two attributes in the breadcrumb to scale this new feature. Consuming all the breadcrumbs sent by all the vehicles was not a scalable option, so we decided to filter them based on AUX signal changes. To avoid taxing our system, we designed a Kafka relayer service that filters out the repeating breadcrumbs in the Kafka stream to consume only breadcrumbs with a change to report. This design choice significantly reduced the number of breadcrumbs we need to process.

As a result, we can generate a new event for every breadcrumb received, setting the breadcrumb data as the (current) ongoing event’s end parameters and the new event’s start parameters.

2. Backdated Configuration Updates

We ask our users to configure their AUX setup on our dashboard to help us interpret the signals received from their vehicle’s Vehicle Gateway. We also wanted to support backdated configuration changes for our customers. In the case of a backdated change, the AUX activities we report to the user must be re-computed. This means we must store the raw events sent by the vehicle and then separately derive the events for reporting. Only when the AUX connection is established (more on this later), the configuration on our web dashboard is completed and signals are being received should we start generating the AUX activities that correspond to the AUX events.

Figure 4. How events become activities

Introducing an AUX activity entity helped us in two ways:

1. On a backdated configuration update, we only need to regenerate the AUX activities, while the events remain untouched.
2. Because AUX activities serve as a presentation layer on top of the raw AUX events, we can merge consecutive AUX events of the same state (two HIGHs or two LOWs) into a single AUX activity (illustrated in Figure 4 above).

However, introducing AUX activities created another major challenge for us: out-of-order breadcrumbs.

3. Out of Order (OOO) Breadcrumbs

What if a breadcrumb’s delivery is delayed and it arrives after breadcrumbs with later timestamps have already been grouped into events? The way we treat the events and activities associated with an out-of-order breadcrumb depends entirely on how its electrical state (0-LOW or 1-HIGH) compares to the states of the breadcrumbs that flank it.

We started by charting out all the different combinations of signals that could occur when an out-of-order breadcrumb is inserted between two existing breadcrumbs. We then grouped the combinations into categories that we could implement with the least effort, while keeping the design as simple and extensible as possible.

Of the many ways that an out-of-order breadcrumb can relate to its neighbors, there are two major categories (described below) of combinations that require updates to existing AUX events and activities.

A. The OOO Breadcrumb Is Sandwiched Between Two Breadcrumbs of the Opposite State (0-*1*-0 or 1-*0*-1)

Figure 5. An out-of-order breadcrumb with state 1 falls between two breadcrumbs, both with states 0

This scenario occurs when an out-of-order breadcrumb chronologically sits between two breadcrumbs with signals of opposite state. In Figure 5 (above), the out-of-order breadcrumb’s signal is 1 (HIGH) and it is injected between two 0 (LOW) breadcrumbs. The existing breadcrumbs (both LOW) had previously bookended a LOW event (AUX Event 2), but this event is now affected by a new breadcrumb. The bisected AUX Event 2 must now be broken into two events because of the injected breadcrumb, and the affected AUX Activity 1, which we previously reported as an OFF activity, must be split into three activities.

B. The OOO Breadcrumb Is Sandwiched Between Two Breadcrumbs of Differing States (1-*0*-0 or 0-*1*-1, etc.)

Figure 6. An out-of-order breadcrumb with state 0 arrives between two breadcrumbs whose states are 1 and 0, respectively

In Figure 6 (above), the out-of-order breadcrumb chronologically fits between a HIGH and a LOW signal. The two existing signals previously bookended AUX Event 2 (a HIGH event) which fell into AUX Activity 1. Now, AUX Event 2 is split into two events. As for the activities, instead of being split, the AUX activities only need to be resized, as shown in the diagram.

4. AUX Connection State Management

An AUX connection is an object that represents the unique combination of a vehicle, its Vehicle Gateway, and the VG’s AUX input slot. (Recall that we have two available slots in the VG so that a given vehicle can connect a maximum of two AUX objects.)

Every AUX connection has its own state of operation. The possible states include:

• The AUX connection state in CONFIRMATION_REQUIRED, the AUX signal polarity has not been configured on the dashboard, and we don’t know how to interpret the raw signals. For an AUX connection in that state, we can only create AUX events, but not any AUX activities.
• If the AUX connection is established, signals are both being received and have been configured on our web dashboard, it is in a CONNECTED state. In this state, AUX activities are generated and reported to the user.
• When the AUX connection is DISABLED, it is not in use, and we do not generate AUX activities. Once a HIGH signal is received, the AUX connection state is changed to CONFIRMATION_REQUIRED, automatically. (If a LOW signal is received, it is indistinguishable from an unavailable signal/unconnected cable and is ignored.)
• SIGNAL_UNDETECTED means that the AUX connection is configured on the web dashboard, but no signals are being sent from the VG. Once signals are received, the polarity needs to be confirmed.
• If the VG is unable to acquire data from the configuration after the hardware is set up, the cable sends us a unique signal to relate this, and we assign the UNSUPPORTED_HARDWARE state to it.
• INITIALIZED is an intermediary state, which doesn’t persist to the DB. The AUX connection object acquires that state at the time of object creation (new AUX connection) and transitions into CONFIRMATION_REQUIRED, SIGNAL_UNDETECTED, or UNSUPPORTED_HARDWARE, depending on how it is created.

We listed all the possibilities to come up with the state machine shown below and implemented it using looplab’s FSM package.

Figure 9. AUX connection states

Impact and Applications

The diagram in Figure 10 (below) illustrates a vehicle with two AUX connections configured for PTO and engaged simultaneously. It also depicts how the productive idling is distinguished from unproductive idling by accounting for the overlap between the idle event and a PTO AUX event activity (generated by merging AUX activities from input 1 and 2 of the same vehicle).

Figure 10. Distinguishing productive idling from actual idling in a vehicle with two AUX connections

Accounting for Productive Idling

We started storing the productive idling part of every idle event and updated our dashboard with this information; wherever we were displaying idle time, we now subtract productive idling from it. Now, the new formula for idling duration is:
idling end time - idling start time - productive idling time

Motion While PTO Engaged Alert

If the PTO is engaged and the vehicle starts moving, it is a potential safety hazard, and important to detect. To avoid such risks, we enabled our customers to configure alerts once the AUX cable is set up and the signals from the Vehicle Gateway are processed to trigger these safety alerts to the driver (in near real time).

Prolonged PTO Alert

Another alert use case that we support is to detect and notify when the PTO is engaged for a longer period of time. This is of potential use to customers who want to learn if a PTO activity has been ongoing continuously for a long period of time.

Panic Button Alert

As we mentioned briefly above, some vehicles offer a button to notify authorities in case of an emergency. Vehicle owners can use our universal AUX cable to notify Motive as well, so that our system can inform the specified fleet managers.

AUX Usage Reporting

We also wanted our customers to be able to download reports informing them when the AUX equipment was engaged, along with associated information such as location, odometer readings, start and end times, etc. This enables our customers to bill their clients based on the usage we report.

Conclusion and Future Work

When designing systems with many moving parts, it is important to keep the design simple. Ensuring that client-side elements (Vehicle Gateway) require a bare minimum of intelligence while the server does the heavy lifting allows generous scope for experiments and adaptation.

Design decisions that take future scope and edge cases into consideration (such as backdated AUX configuration) make systems robust and easy to maintain when new requirements arise. For example, in a stream of data being consumed from a queue, consider the possibility that the data may be unordered, and prepare your systems for every possible scenario.

Currently, we don’t know how to account for fuel consumption during a PTO activity, especially when multiple PTO equipment pieces are engaged at the same time in the same vehicle. We are going to tackle this problem in the future.

Acknowledgements

This project was a joint team effort and would not have been possible without help and guidance from Prahadeesh Giridharan, Gautam Das, Himanshu Dhillon, Soumya Das, and Matt Brown. Acknowledgments would be incomplete without a huge shout-out to the QA team, comprising Duaa Zahra Khan and Avinash Golla. The same goes for our Technical Program Manager, Sean Marshall. Finally, we thank the server team, comprising Vivek Rai, Akash Prasanna Basabhat, Anirudh Pundhir, and myself: N. Vijay Kumar.

Come Join Us!

Check out the latest Motive opportunities on our Careers page to join our rad engineering team!