In this article, we will discuss car infotainment ecosystems from the perspective of an application developer, covering their past, present, and future.

We will also explore how to interface with a car and retrieve information, such as vehicle speed, from its network.

Do cars need software applications?

When cars first went into mass production, they did not have radios. The driving experience was exciting enough on its own.

The very first commercial radio appeared in 1930, and it took several years for it to become an established standard equipment. People clearly saw the value of the radio, and radio stations were happy about it too.

Today, large, colorful displays with touch and speech control are all the rage. Much as we love analog technology and physical buttons, we cannot deny that such user interfaces offer drivers far greater value.

These days, people expect cars to offer more than just a radio.

Why now?

Displays are now just as much a part of any new car as a radio. But why are we talking about it now? What changed?

Until recently, in order to develop an application for a car’s built-in system, you had to talk to OEMs directly (e.g., to BMW, GM, Ford, Volvo, etc.), and use their proprietary framework. This resulted in virtually no market for car-specific applications. This is still the case for most OEMs today.

However, this is changing.

Android Auto & Apple CarPlay

Today, there is an established market of Android Auto and Apple CarPlay. These are so-called projection modes. Simply connect your phone to your car and enjoy your favorite apps on the go.

When it comes to developing car applications, these are clearly the best platforms to target today. That is where the users are. Consequently, it receives the most attention from both Google and Apple as a platform for automotive apps.

However, a new platform has emerged: Android OS installed directly in a car. This is the future.

Native Android in cars

In the recent years there has been a dramatic shift in the technology being used for car infotainment systems. This change was enabled by the release of Android Automotive OS (AAOS), which included many useful car-specific features in the Android Open Source Project (AOSP).

Previously, most OEMs used their own customized Linux solutions; now, most support Android APIs. We will not go into motives of the change. Instead, let us focus on the opportunity to develop for it.

What do the cars support?

A developer who wants to build an Android application specifically for cars, or add certain in-car use cases to an existing Android application, is faced with the following choice of the target today:

• Cars with support for Android Auto,
• Cars running Android Automotive OS (AAOS) with Google Automotive Services (GAS) — these are now marketed as “Google built‑in”, and
• Cars which run an OS based on Android Open Source Project (AOSP) or AAOS without GAS.

For a more in-depth overview of these platforms, take a look at these great articles by Antonio Roa-Valverde:

• Navigating the Android Automotive OS Third Party App Ecosystem: a Developer’s Perspective,
• Android Automotive OS app stores: understanding the ecosystem.

Car API overview

At the highest level, Google provides an abstraction which is called Android for Cars App Library, also known as Car App Library (CAL). It works for both Android Auto and AAOS.

Ideally, all car APIs would be exposed exclusively via CAL.

Unfortunately, there is still some fragmentation, even among Google’s own APIs. AAOS has certain APIs that can (or have to) be used instead of CAL.

Let’s categorize all of the Android car APIs into 3 groups:

1. Car APIs that mirror standard Android APIs
2. Car-specific APIs for user interaction
3. Car-specific APIs for vehicle data

1. Car APIs which mirror standard Android APIs

CarSensors (CAL)

Suppose you want to access an accelerometer.

Since Android Auto runs on a phone, developers have access to two accelerometers: one on the phone, and one in the car. CarSensors gives you access to the one in the car.

AAOS does not have this ambiguity. Therefore, we must use SensorManager and LocationManager there instead.

The choice is straightforward:

To target both Android Auto and AAOS, you must check which platform your application is running on at runtime, or build separate versions of the app.

The reason CarSensors returns an unimplemented status on AAOS instead of wrapping calls to SensorManager and LocationManager remains unclear.

CarAudioRecord (CAL)

This is a car-specific subset of the AudioRecord API. It is minimal and integrates well with other CAL APIs.

There is also generic MediaRecorder API, but its scope is different.

2. Car-specific APIs for user interaction

This cluster of car APIs is new to Android. Cars are not phones, and must have different standards for distraction optimization and general user interaction.

These APIs include App Types, Templates, Media Apps, etc.

Depending on your app’s category, target platform and whether it must be usable while driving, you will see which of these APIs are available and possibly even enforced.

For a great introduction, see Section 4, “Building your App,” in the Appning Automotive Apps Market developer documentation.

Google has documented these APIs quite well: https://developers.google.com/cars/design/create-apps

The libraries and documentation are both in active development, so it is best to regularly check these resources. This is also why we omit the arguably largest section of this article and focus on data APIs instead.

3. Car-specific APIs for vehicle data

The usage of vehicle data is an important feature that sets a car-specific Android application apart. Examples of such vehicle-data-centric use cases include:

• Fleet management: fuel and charge levels, pending maintenance, location
• Insurance: driving performance evaluation, tire pressure, speeding and braking events, indicator usage, timely maintenance
• Entertainment: driving feedback, control of peripherals such as ambient lights

One specific example is the Carchestra app. It is a media application that reacts to your driving style that was created by us at Paradox Cat. It won the Appning App Challenge, so give it a try!

Currently, there are the following ways of accessing car data:

1. CarInfo as part of CAL. It works for both Android Auto and AAOS.
2. CarPropertyManager, which only works for AAOS-based cars, because it uses Car Service and Vehicle HAL.

Here is how to choose:

From a software architecture perspective, it is questionable whether we even need a CarPropertyManager. Again: ideally, all Car APIs would be exposed exclusively via CAL. Perhaps Google will address this issue eventually, but for now, this is the current state.

Now, let’s take a deeper look at both CarInfo and CarPropertyManager.

CarInfo

Starting with Car App API level 3, the Car App Library has APIs that you can use to access vehicle properties and sensors.

This quote is about CarInfo, which does not expose as much as most use cases would require. Here’s the full list of APIs as of today:

It is definitely not all the data which could be made available to Android applications. Unfortunately, even automotive OEMs cannot expose additional data via this library, because the interface is specified by Google for all cars.

From our experience of using CarInfo with Android Auto on non-Android cars, we have found that most OEMs only deliver the car model and speed. The rest of the CarInfo methods usually do not return actual data.

On AAOS, CarInfo is implemented using the CarPropertyManager API. CarInfo essentially exposes a subset of these APIs in a slightly different form.

Therefore, more data is available on AAOS cars than on non-Android cars using CarInfo.

However, AAOS directly exposes the CarPropertyManager API, so let’s explore the benefits of using it.

CarPropertyManager

Note: CarPropertyManager is an Android Automotive OS (AAOS) feature, it does not exist on Android Auto.

Have you ever wondered how OEMs who use the Android OS in their cars are supposed to access car data points themselves in all of their first-party applications such as Climate, Telephony, or Navigation?

The answer is: the CarPropertyManager API.

What is a Car Property?

Google also refers to them as vehicle properties. A vehicle property is a data point that represents the state of a car.

This data is sent from the automotive network to the Vehicle Hardware Abstraction Layer (VHAL), where it is mapped to properties. CarPropertyManager allows application developers to access these properties.

Properties can be read-only, write-only, or read-write.

Each property has an area (such as VEHICLE_AREA_TYPE_SEAT), a strong data type (such as Integer), and a change mode (such as VEHICLE_PROPERTY_CHANGE_MODE_CONTINUOUS) associated with it. Additionally, each property declares the necessary permissions to access it.

Below are some examples of vehicle properties:

The full list of property IDs can be found here. As of today, there are over 250 different vehicle properties, which by far surpasses the number provided by CarInfo.

For more information on how it works under the hood, please refer to this article by Rashik.

System Properties vs Vendor Properties

By design, Google is expecting OEMs to provide data via a predefined set of so-called system properties. This set contains over 250 properties, as shown above.

At the same time, Google recognizes that this set may not be ideal for all OEMs and provides a mechanism to extend it with vendor properties. It is important to note that OEMs that want to be officially compatible with Android must adhere to these rules:

Always try to use system properties first, vendor properties should be used as a last resort when none of the system properties feeds your requirement.

To prevent ecosystem fragmentation, vendor properties must not be used to replicate vehicle properties that already exist in the SDK VehiclePropertyIds. To learn more, see Section 2.5, Automotive Requirements in the CDD.

It is unclear whether all car manufacturers who use AOSP comply with this requirement.

So, there are even more vehicle data points. But who has access to them?

Permissions: a word of warning

If the property requires a “signature” or “signature | privileged” permission level, it means that you cannot obtain permission without the platform being updated. Each OEM must update their AOSP to whitelist your application package. You will only receive the permission when the OS image is updated in each car. Not to mention, a trusted relationship with an OEM is required.

This is a topic that we are currently addressing with many different automotive OEMs at the Connected Vehicle Systems Alliance (COVESA).

One of the goals there is to avoid fragmentation by standardizing all vehicle APIs for Android. No need to talk to each OEM separately, flexible permission model, the ability to reuse Vehicle Signal Specification (VSS), and so on. The group is open, and software is open-source.

Given the current permissions situation, three categories of developers could use the CarPropertyManager API:

• OEMs developing first-party applications;
• third-party application developers who are in agreement with OEMs to have their applications whitelisted to access specific properties;
• third-party application developers who are only interested in properties with “normal” or “dangerous” (runtime) permissions.

Disclaimer

The following tables are merely a snapshot of the current API. Please, always refer directly to https://developer.android.com/reference/android/car/VehiclePropertyIds

The following is a list of the 37 properties with “normal” permissions today:

Below is a list of the 33 properties that require “dangerous” permissions to be granted at runtime:

Code example

Now that we have the permissions out of the way, we can move on to the actual code example.

At Paradox Cat, we have recently released an open-source (MIT-licensed) wrapper library for the CarPropertyManager API. This library greatly simplifies integration with the Kotlin codebase and handles all connection logic out of the box.

This is how easy it makes to subscribe for car speed (omitting permission request):

val kpm = KarPropertyManager(context, scope)
val speedFlow = kpm.getProperty<Float>(
        VehiclePropertyIds.PERF_VEHICLE_SPEED, 0, 60F).valueFlow

Here is the GitHub repository: https://github.com/Paradox-Cat-GmbH/KarPropertyManager

We hope you’ll find our library useful! Feel free to report any issues or suggest features you’d like to see, we would appreciate your feedback.

Challenges

From an application developer’s perspective, the situation is pretty dire.

Confusing API surface

It’s simply too much for a typical Android application developer to grasp. Just look at the size of this article, and we mostly focused on the Google experience.

Lack of Testing Tools

The Google landscape is more or less covered by the official Android Automotive emulator (with GAS). Additionally, one could rent cars of the following brands and expect similar experience: https://built-in.google/cars/#explore-cars

It would also be helpful to feed the emulator with real car data. There is a free demo recording provided by remotiveLabs, but Google has not yet provided anything like that. A comprehensive car simulator that includes all displays and interaction capabilities would also be helpful.

As for non-Google OEMs using AAOS, it’s certainly difficult to find a representative platform. COVESA recognized it and we supported this by setting up a vanilla AAOS emulator CI pipeline. See https://github.com/COVESA/aosp_device_covesa_emulator.

Next steps: https://github.com/COVESA/aosp_device_covesa_emulator/issues

Discussions: https://github.com/COVESA/aosp_device_covesa_emulator/discussions

As of this writing, further development and maintenance of this emulator has been paused due to lack of interest from automotive OEMs. Each OEM has its own emulator, but they differ, and most are not public. This makes it difficult for developers to verify whether their apps would work on non-GAS cars.

Once the issue of a standard reference emulator is resolved, non-GAS OEMs will face further challenges in standardizing their interfaces if they cannot be identical to Google’s.

With regard to the rental of non-GAS cars, we have also recognized that there is no official channel for sideloading the application for testing purposes. ADB is usually disabled.

Google has an internal test track in the Play Store, but it is only usable with GAS. Perhaps other app stores should make it publicly available too.

OEM support of Template Host

Whether CAL’s Template-UIs are supported depends on whether the OEM has integrated so-called Automotive App Host. Cars with Google built-in (GAS) come with the Google Automotive App Host, which Google keeps up to date via Play Store updates. Without GAS, there is no guarantee that the OEM has integrated their own version of the Automotive App Host that supports the latest Car App Library version.

Fragmentation occurred when Google updated the GAS version of the Automotive App Host, but not the open-source version.

1. OEMs running AAOS without Google Automotive Services (GAS) are stuck with Car App Library API level 4.
2. Some OEMs and App Stores started to extend it. Unfortunately we are not aware of any implementation of Template Host that supports the latest Car App Library APIs except for Google’s proprietary one.

Hopefully, Google will continue to update the open-source Template Host to support the latest Car App Library APIs.

To ensure compatibility today:

• Wait for OEM support: Avoid using features that require API level 5 or higher until all manufacturers have updated their systems.
• Build with fallbacks: If you need newer features now, check the API level at runtime and provide alternative implementations for cars running older API versions.

OEM support of Media Apps

The MediaBrowserService and MediaSession APIs are already supported in AAOS. Therefore it is unlikely that OEMs would use something else.

However, OEMs typically implement their own media player, which may or may not function as expected.

Access to Vehicle Data

Issue #1

Car manufacturers are not required to support all vehicle properties.

Currently, even cars with “Google built-in” are only required to implement 4 (!) vehicle properties out of the 70 publicly available (with “normal” and “dangerous” permissions):

[7.3/A-0–1] MUST implement and report GEAR_SELECTION, NIGHT_MODE,PERF_VEHICLE_SPEED and PARKING_BRAKE_ON.

Therefore, application developers cannot rely on CarInfo or CarPropertyManager to deliver more than these four properties.

Another issue is that some OEMs do not even use AAOS; they use the tablet version of AOSP, which lacks CarPropertyManager.

The solution will take time. The more apps that start using these APIs, the greater the incentive for OEMs to support them.

Issue #2

The permission model is not flexible enough for OEMs to adopt.

Currently, 180 out of 250 standard vehicle properties are only available with OEM signature.

There’s an initiative at COVESA to enable a better permission model with minimal AOSP changes: https://covesa.atlassian.net/wiki/spaces/WIK4/pages/262766594/COVESA+AOSP+Vehicle+Data

Ideally, this will be merged into AOSP, which will avoid fragmentation.

Issue #3

Most OEMs require more than Google’s set of properties.

These end up as vendor properties, as indicated above, which third-party applications have no access to.

If all OEMs agreed to reuse the Vehicle Signal Specification (VSS) or any other standard, developers would have access to more car data points without needing OEM-specific software development kits (SDKs).

Summary

The world needs more car-specific applications, and now is the time to develop them because almost all OEMs are using Android as the main OS for infotainment.

Begin by consulting Appning’s documentation, followed by Google’s.

Refer to this article for details on how to work with vehicle data and what to be aware of.

If you are using Kotlin, check out our KarPropertyManager wrapper.

As a third-party developer you can only access the properties with “normal” or “dangerous” (runtime) permissions. Do not assume that you have access to all the data.

Both developers and car manufacturers face many challenges, but most of these will be solved once the market saturates and stabilizes.

So go ahead and develop some apps for cars! Contact us at Paradox Cat if you need any help.

Building Apps for Cars in 2026: Android Auto, AAOS, and Vehicle Data APIs was originally published in Paradox Cat Tech Hub on Medium, where people are continuing the conversation by highlighting and responding to this story.

• Inside a car, even in a day with a moderate outside temperature, the vehicle may reach dangerous higher temperatures within minutes. What may look like a partial solution, such as letting the window slightly open, parking in shade, or using air conditioning, do not prevent the vehicle from accumulating heat.
• Children’s physiology makes them more susceptible to higher temperatures, because their thermoregulatory system, such as their ability to sweat and to regulate their body temperature, is different from adults. Therefore, children cannot dissipate heat effectively and their body temperature rises three to five times faster. The case is more serious for newborns and toddlers.
• Since the temperature inside a car can reach dramatic levels in a short time, combined with the fact that a child absorbs heat faster without having the capability of regulation, heatstroke conditions develop quickly, leading to a life-threatening scenario.
• The child’s inability to exit the vehicle on their own.

At the same time, while parents have a primary responsibility to take care of their children and their health, society also has a responsibility to develop systems and mechanisms to prevent such situations from happening.

To address the risk of children being left alone in cars, the European New Car Assessment Programme (Euro NCAP) and China NCAP (C-NCAP) have recognized Child Presence Detection (CPD) systems inside the vehicles as a feature in their safety rating systems since 2023. CPD may become mandatory in the future, depending on the outcome of current EU and UNECE regulatory processes.

A CPD system should be capable of detecting unattended children anywhere inside the vehicle, including all seating positions (optional and removable seats) and unusual locations such as footwells. It should operate reliably under diverse lighting conditions, including darkness, shadows, or glare, and handle occlusions, varying postures, and other challenging scenarios such as children covered by blankets, objects on seats, or atypical seating arrangements.

According to Euro NCAP’s 2026 protocol, a CPD system must rely on direct sensing methods, such as detecting movement, breathing, or even heartbeat. In addition, the system must also issue a visible and audible warning within 15 seconds if a child is detected after the vehicle has been locked.

To meet these requirements, radar-based technologies, such as millimeter-wave and Ultra-Wideband (UWB) radars, are among the most suitable options. Our company offers solutions for both, including a CPD system based on 60 GHz mmWave radars. However, because UWB is already widely used in vehicles, particularly for applications like keyless entry, integrating it into existing automotive platforms is often simpler, more cost-effective, and faster. Beyond ease of integration, UWB also offers several technical advantages, such as lower power consumption, better penetration, and occlusion handling. Still, its use in current automotive systems remains one of the strongest reasons for choosing UWB for CPD.

How do UWB radars work, and how can they detect if a child or newborn is left alone in a car?

To start, let us recall the physical behavior of electromagnetic waves, such radar waves, in simple terms. Imagine this scenario: If you go to a lake and throw a rock, you can see patterns of waves in the water. In this analogy, the rock represents the part of the radar called transmitter, which generates invisible waves in the air. The pulse repetition frequency of the transmitter is determined by “how many times you throw a rock in just one second”. The distance between the ripples created by each stone corresponds to the radar wavelength, which is determined by its carrier frequency (6.5 to 8 GHz for UWB radars).

Just like an echo when you scream in a cave, radar waves are also reflected when they hit an object: if you scream, your ears recognize your own voice bouncing back. With radars, the receiver antenna recognizes the reflected wave pulses, but the radar “shout” is a billion times faster than sound and the “ears” measure nanoseconds. The time between transmitting a pulse and receiving its reflection is called Time-of-Flight (ToF). The ToF is proportional to the distance between the radar and the object, and it is used to estimate the distance and speed of objects. In this way, the reflections read by the radar carry information about the location and movement of surrounding objects.

The UWB radar periodically emits a sequence of extremely short pulses. The reflection pattern acts like a map, capturing how reflections appear at each moment. If an object moves, this pattern changes. UWB radar can “see” tiny movements, detecting small periodic changes in the reflected signal. This makes it ideal for recognizing a child or adult breathing inside the car, as each breath causes the chest to move around 4 to 8 mm and follows a periodic pattern with a frequency of approximately 0.2 to 1 Hz. The breathing rate of an adult is different from that of a child or a newborn.

When two antennas are used to receive the signal, it is possible to get information about the direction of arrival by comparing the time difference of the received echoes. The technical term is angle-of-arrival estimation and it allows us also to check the location of an individual inside the car.

The capability of reflection depends on the object material and thickness. Metal, water, and the human body have strong UWB reflection, while soft fabrics such as blankets and clothes, foam, or thin plastic partially absorb or scatter the received pulses, reflecting very little. As a result, much of the signal penetrates through the blanket, allowing the radar to ignore the fabric and to receive the echo from the body. This property of the frequency band of UWB radars enable us to measure the reflections of people and children, focusing on our goal: to detect if there is a kid alone inside the car.

Once we receive the UWB radar signal for a maximum duration of 15 seconds (required by Euro-NCAP), we then extract relevant information received for each moment of time. First, we need to filter the received data to remove unwanted components and enhance the parts of the signal that correspond to life presence detection. For instance, we want to remove all reflections that are related to static objects, such as car seats. We also do not want reflections that are associated with locations outside the car. This step is called pre-processing.

The received UWB radar signal contains multiple forms of information, commonly referred to as features. The subsequent processing stage focuses on extracting those features that are relevant and discriminative for CPD. There are amplitude-based features, which describe how strong the reflections are by measuring the intensity of the returned signal. If the radar uses more than one antenna, we can also calculate the angle or direction of arrival of those reflections, which is crucial to distinguish between signals from inside or outside the car. Additionally, by extracting the repeating patterns in the signal over time, we compute frequency-domain features, which help us detect periodic motions, such as breathing or other rhythmic activity, from otherwise still objects in the car.

After extracting the relevant features, we want to group all of them as an input for a machine learning (ML) model, which classifies whether the radar data represents reflections of a child or of an adult, or of an empty car. The ML model learns from the subtleties of the data that may not be obvious when we analyze it with a simple mathematical model. It acts like a decision-maker step, built on top of all features extracted. Once the model is trained, it can then perform real-time inference, pointing out to the CPD system, whether there is a child left alone inside a car or not.

Therefore, UWB radars are a good solution for a CPD system, having low-power consumption. The combination of UWB radar sensing and artificial intelligence allows us to implement a reliable and cost-efficient solution.

Radar-based sensing opens many possibilities in vehicles: what applications do you think it could unlock? Let us know in the comments!

For companies looking to implement vehicle safety with Child Presence Detection (CPD) solutions, get in touch with us today to explore how our cutting-edge radar technology can help you bring life-saving features to market. We are flexible in sensor choice and positioning, and our algorithms can be adapted to work with different radar models and configurations, ensuring seamless integration into your vehicle systems.

How Can UWB Radar Protect Children Left Alone in Cars? was originally published in Paradox Cat Tech Hub on Medium, where people are continuing the conversation by highlighting and responding to this story.

We are excited to announce that our new app, Carchestra, is now available for Android Automotive OS (AAOS)! This release is the first preview version with more features to follow soon. Carchestra is an audio app where the music adapts to your driving style in real time, reacting to speed, acceleration, braking, turning, and more! You can read more about it on the project website and in our video:

In this blogpost, we, the development team of Carchestra, will give you an insight into how this project came about, how it works, and what technical challenges we faced while creating the app.

How to use it

To download Carchestra, click on the link, or alternatively find it in the Play Store in your car:

Then, find the Carchestra icon in the app drawer or select it as a source in your media player.

You will now have to accept our license agreement, and grant the location permission, so that Carchestra can access vehicle sensors such as speed, accelerometer, and gyroscope.

Finally, you can hit play and start driving!

Automotive app stores

In the past, most “apps” on automotive infotainment systems were provided by the OEMs themselves. More recently, CarPlay and Android Auto, which rely on a connection to the driver’s phone, brought a wider choice of apps to cars. Now, with more and more OEMs choosing AAOS, third-party apps can be downloaded from app stores to run natively on the vehicle, creating a more seamless experience.

Depending on the OEM, there are different major app stores available on AAOS vehicles. In a future blog post, we will go into more details about the automotive app store ecosystem.

• Google Play Store for Google’s flavor of AAOS (Google Automotive Services / “Google built-in") such as Volvo, Polestar, Renault, GM, Ford
• Appning by Forvia (previously Faurecia Aptoide) – white-label app stores for brands such as BMW, Mercedes-Benz, Dacia, Lynk & Co and many others
• Harman Ready Link Marketplace (previously Harman Ignite) – white-label app stores for Audi, Porsche and others

Carchestra is now available in the Google Play Store, it is currently in review at Appning, and we are planning to submit it to Harman as well.

How it started

Every project starts with an idea: As a software development service provider specializing in HMI, we at PARADOX CAT are already working on AAOS systems and apps in customer projects, such as on BMW’s latest Android-based infotainment system. At one of our regular Ideathons, we were thinking of a new side project for our company: Building our own AAOS apps and distributing them through the automotive app stores. Soon, the idea of a music app that adapts to your driving style was born, and became popular among developers, data scientists and music enthusiasts in our team. Shortly after, we found a great name for the app as well: Carchestra!

The Appning App Challenge was an additional motivator for us to start this fun project. With a project start on January 7, 2025, and a submission deadline for the App Challenge on February 23, this gave us just shy of 7 weeks to complete the first MVP. The development team was formed, and the fun began!

Challenges

And fun we had! Of course, not everything went smoothly:

Testing on cars:

An Android developer’s testbench usually looks like this:

Unfortunately, we can’t afford this as Android Automotive developers:

Still, we got access to various cars and tried to make the most of the time we had with them:

• Carchestra needs real-time vehicle data to work, so it is hard to test it on an emulator. To address this, we built a separate data recording app and built a collection of test drive data and videos from different cars. Then, we added functionality to replay recorded data synchronized with the videos on the emulator.
• During the test drives, we also noticed quite a few differences between the AAOS implementations on the cars that we tested, such as certain sensors not being available or returning incorrect data, or different quirks in the behavior of the media app. Our collection of replayable data recordings made it possible to identify and address these differences.

Being a 3rd party

As a third-party app developer, you usually cannot get a direct Android Debug Bridge (ADB) connection to a production car’s head unit to install and debug apps, as this is restricted by the OEM. Our workaround for this was to use Google Play’s internal test track, which allows us to upload app updates and make them available in the car within a couple of minutes. For debugging, we then also implemented a solution to upload logs to the cloud. Still, this is not ideal, and we would love it if OEMs provided ways to unlock ADB for approved developers.

Cars are not phones

Building user interfaces for third-party AAOS apps also differs from phone apps: Depending on the app category, the system provides distraction-optimized UI templates that are safe to use while driving. In our case, as a media app, we use the MediaBrowserService and MediaSession APIs to expose our capabilities to the OEM’s own media app. Our settings, built with standard Activities and Jetpack Compose, are only available in standstill.

Real time audio is hard

Finally, we also encountered a challenge that was not specific to automotive: Implementing real-time audio mixing based on Android standards for low-latency audio using Oboe and the NDK. Which is like laying tracks without using power tools while the train is driving full speed.

Carchestra Preview Release

Despite these challenges, we managed to build a working MVP and submit it to the contest. This first version of Carchestra includes one song, “Without You” by Michael Badal, with different intensity levels triggered by speeds up to 60 km/h (we call it the “urban mode”), and effects like turning or coming to a standstill.

Speaking of challenges, the final challenge was of course the Appning App Challenge. And when we got an email in mid-April, we were excited to hear that we had won the challenge! Thanks a lot to Appning and the Jury for recognizing our work. We are looking forward to working with the Appning team to bring Carchestra to BMW and other OEMs.

𝗜𝘁'𝘀 𝗼𝗳𝗳𝗶𝗰𝗶𝗮𝗹 - we have our winner! | Appning by FORVIA

If you read the blogpost until this point, you are probably interested in some more technical details. We are planning to release more blog posts in the future about technical aspects of development for cars in general and Carchestra specifically.

Future plans

Speaking about future plans, we of course have many ideas for Carchestra already. Of course, we are planning to add more music and more sophisticated effects. As already mentioned, we also hope to make it available to more cars in the near future through all the major AAOS app stores.

First, we would like to hear your feedback: Which features and what type of music would you like to see in Carchestra? Is everything working smoothly on your car? You can reach us at carchestra@paradoxcat.com or through the contact form on the Carchestra website.

Is your company interested in Android apps for cars? Then please get in touch with us – we can support you with the development of new or porting of existing apps.

Authors: Johan von Forstner, Adrian Tappe, Lucía León Milán

Introducing our new app: Carchestra — winner of the Appning App Challenge was originally published in Paradox Cat Tech Hub on Medium, where people are continuing the conversation by highlighting and responding to this story.

After having a great time at last year’s ICCV in Paris learning about the latest scientific advances in Computer Vision, we at Paradox Cat’s AI team were delighted to see that another major computer vision event, the European Conference on Computer Vision (ECCV) in October 2024, took place even closer to Munich, in the beautiful city of Milan in Northern Italy 🇮🇹. This time, our working student Priya and I took the trip together using our Paradox Cat training budget and it was a lot of fun!

After a scenic train ride through the Alps we arrived in Milan on a Saturday afternoon with beautiful sunny weather and 26 °C 🌞. The weather wouldn’t stay quite as nice over the rest of the week, but that didn’t matter too much to us as the conference was jam-packed with interesting content. 😸

Sunday and Monday were dedicated to workshops and tutorials, which could be regarded as mini-conferences focusing on specific subjects within the field of Computer Vision, organized independently from the main conference and with a few varying formats (invited talks, challenges, poster and oral sessions). We enjoyed many different engaging workshops and tutorials such as the annual workshop on observing and understanding hands in action, Meta’s Project Aria tutorial, or the workshops on human analysis, spatial AI, and map-free visual localization.

This year’s ECCV was the largest to date with about 7,000 participants — and the scale of this event only really became clear on Tuesday, the first day of the main conference, where the opening ceremony took place and was streamed into all three large lecture halls. But the organizers did a great job in preparing for the crowds, with ample seating available in all the oral sessions and also a generous spacing of posters in the exhibition hall.

The opening ceremony also included the Best Paper Award, which was presented to Minimalist Vision with Freeform Pixels, an intriguing approach to solve vision problems with a minimalist camera with as few as 8 pixels of irregular shapes, providing benefits for privacy and power efficiency.

The main conference was again filled with so many great presentations that it’s impossible to list all of them. Among the most popular topics were generative AI (image and video), multimodal models (vision & language, vision & audio), applications of foundation models, 3D and multi-view vision techniques, and many more. I’ll mention two papers that I was particularly impressed by:

Sapiens: Foundation for Human Vision Models
Self-supervised pretraining of large foundation models has been a big trend fueling the recent breakthroughs in text and image generation models. This paper from Meta marks the first success in bringing this concept to the realm of human-centric vision tasks: They pretrain a Vision Transformer model on a large dataset of high-resolution pictures of humans using the masked autoencoder (MAE) technique. This pretrained model can then quickly be fine-tuned for downstream tasks like body pose estimation, body part segmentation, or depth and surface normal estimation, surpassing previous state-of-the-art results.

Grounding Image Matching in 3D with MASt3R​
This paper won the map-free visual localization challenge, surpassing baseline methods by such a large margin that the organizers initially suspected the authors had cheated. Given just two images of the same scene under challenging conditions (outdoor scenes captured on different days with different lighting and extreme viewpoint changes), the task of the challenge is to accurately reconstruct the relative camera pose between the two images. MASt3R solves the problem in 3D, taking into account not just local feature descriptors but also the 3D geometry via a depth map estimated from each image.

Of course, there were also a lot of popular sessions about new trends in generative AI and multimodal foundation models. But besides these highlights, we also found many more niche presentations in the poster sessions that were still quite relevant for our work in automotive in-cabin sensing.

In addition to the scientific program, ECCV included three keynote talks on Tuesday, Wednesday and Thursday. The first one was from Lourdes Agapito and Vittorio Ferrari, presenting the journey of their startup Synthesia and their newest advances in AI avatar video generation, with some impressive and funny demonstrations at the end. In the second keynote, law and AI ethics expert Sandra Wachter gave insights into fairness and transparency in AI systems and how current anti-discrimination legislation in the EU is not quite prepared for dealing with these issues. Finally, on Thursday, Stanford professor Sanmi Koyejo talked about distribution shifts in AI and described how, despite the advances in foundation models with improved generalization capabilities, distribution shifts are still an issue in many Machine Learning applications and domain adaptation/domain generalization techniques are still a crucial field of research.

After a fun and insightful week in Milan, we headed back to Munich on Friday afternoon. Thanks a lot to the ECCV 2024 team for organizing such a great conference and to all the authors who presented their awesome work. And of course thank you to Paradox Cat for allowing us to participate! 🐱

Gatti pazzi a Milano —  PARADOX CAT at #ECCV2024 🍕 was originally published in Paradox Cat Tech Hub on Medium, where people are continuing the conversation by highlighting and responding to this story.

In this article we are going to unveil:

• 4 Reasons you should use Android on Raspberry Pi
• Where should you get Android for Raspberry Pi?
• Which one to choose?
• Future prospects.

4 Reasons you should use Android on Raspberry Pi

Raspberry Pi is arguably one of the world’s most popular single-board computer. You probably already have one, don’t you?

Android is the most popular operating system in the world. There is a huge variety of applications to choose from.

This popularity, combined with the increasing power of Raspberry Pi, opens up many possibilities for using Android on Raspberry Pi. Let us explore these briefly.

Reason #1: Gaming

Raspberry Pi with Android is one of the most affordable gaming platforms on the market, whether using a keyboard and mouse with your desktop monitor, or using controllers with your TV. Simply install an Android game as an APK or use another app to emulate one of the retro consoles.

Reason #2: Home Entertainment

Once your built-in TV functionality gets old, there are plenty of affordable ways to breathe some life back into it. You could obviously get something like a FireTV stick, Apple TV, the discontinued Google Chromecast, or the all-new Google TV Streamer. If you feel adventurous, you could also take a look at the myriad of chineese streamers which retail on Amazon and Aliexpress. However, none of these would give you the flexibility of a Raspberry Pi.

The Raspberry Pi opens up so many more possibilities. It is a great centerpiece for home entertainment.

You can choose an Android TV Lineage OS build or use any Android flavor and simply install an app like Plex to have a centralized streaming console.

Raspberry Pi officially supports Widewine, which means that Netflix, Disney+ and all the major streaming services work.

With a help of HiFiBerry, you could do multi-room audio, turn passive speakers into active ones, correct room acoustics with DSP, build a high-quality streamer, and dive deep into DYI audio.

Projects like Hyperion enable DYI lighting solutions for bias lighting.

Simply plug a Webcam into a USB port of a Raspberry Pi and your TV turns into a video-call enabled device. Due to the fact that all major videoconferencing software runs on Android phones, the chances are high that it would work really well.

Reason #3: Low-Volume Product

Let’s say you need to build a point-of-sale terminal and you only need 10 of them. Maybe you already have an Android application that you could at least partially reuse. Or you could quickly deploy a solution using apps like Square or Vend.

Not every company needs a professional and scalable hardware solution from NXP, Qualcomm, or other vendors right away. Sometimes you need a quick and inexpensive way to test your proof of concept.

Android is a great platform for an embedded solution with a touch display, that has a lot of features you would need to implement yourself in a more standard Yocto distribution.

Thus, a combination of Raspberry Pi and Android is a very lucrative option for such prototyping.

Reason #4: Car Navigation System

Once your car’s embedded system can not keep up with technology, the typical solution is to use your phone either directly or via a so-called projection mode such as CarPlay (for iPhones) or Android Auto, effectively projecting your phone’s screen onto the car’s display(s).

Not all cars support projection modes, so there are things you can do with Raspberry Pi and an external display to get around this.

There are many off-the-shelf solutions on the market, mostly from the Far East. However, if you want an up-to-date Android system without built-in backdoors and with some DIY capabilities, Raspberry Pi with an external touch display is a great place to start.

The main goal of the project would undoubtedly be navigation with Google Maps. But again, just like in home entertainment, you could pimp your sound system, install some ambient lighting, and configure it all to play well with your Android system.

Bonus: Other Reasons

There are plenty of projects which may be realized with a combination of Raspberry Pi and Android.

However, this may not be the best solution for organizations due to licensing issues or lack of hardware performance.

Feel free to contact us at Paradox Cat if you have a project in mind. We can help you understand the costs and benefits involved, and we can build a custom AOSP-based solution that fits your needs precisely.

Where should you get Android for Raspberry Pi?

To run Android, Raspberry Pi needs an Android Board Support Package (Android BSP). This BSP is a collection of tools and drivers that allow specific hardware to boot into vanilla AOSP. Normally, it is the hardware vendor’s responsibility to create such a BSP.

However, the Raspberry Pi vendor (Raspberry Pi Foundation) only officially supports the Raspberry Pi OS.

There is no official Android support for Raspberry Pi.

Unofficial Android Support

Fortunately, Google provides the source code for the main part of the Android OS via the Android Open Source Project (AOSP). This, combined with the open source nature of most of the Raspberry Pi drivers for Linux, enabled people to build their own Android BSPs for Raspberry Pi.

The development of Android BSP for the original Raspberry Pi dates back to 2012. After a while the developer community has been centered around the android-rpi Google group, which is still active today.

Let us take a look at the landscape of Android BSPs for Raspberry Pi today.

Disclaimer

What follows is our humble research of available Android BSPs. We are not affiliated with any of the authors, and would be happy to be corrected. It is also difficult to determine the original authorship of the code, as some of it has often been copied without proper attribution. It seems that all the BSPs developers have been working in parallel, looking at each other’s changes and cherry-picking some, but not all.

android-rpi

android-rpi is a developer community found on Github (https://github.com/android-rpi), as well as Google Groups (https://groups.google.com/d/forum/android-rpi).

The history of this community goes back to 2015 with the release of Android 5.0 on Raspberry Pi 2. The community is still active and continues to release source code of Android BSPs for new Raspberry Pi versions.

There are no binary images available for download.

The entry point for building your own image for Raspberry Pi 5, for example, can be found at https://github.com/android-rpi/device_arpi_rpi5.

This build is based on Android TV and comes with its own open source launcher called RpLaucher and TV settings app called LbSettings. If you want to know more about Android launchers, we have written another article about them.

raspberry-vanilla and lineage-rpi by KonstaKang

KonstaKang has been consistently providing ready-to-use Raspberry Pi Android images for years. Looking at the website https://konstakang.com/ along with other devices, one would find a lot of different binary Raspberry Pi Android images for free download:

• Raspberry Pi 4: https://konstakang.com/devices/rpi4/
• Raspberry Pi 5: https://konstakang.com/devices/rpi5/

Most of the source code for these images along with build instructions, can be found in two different GitHub “organizations”:

1. The raspberry-vanilla collection of repositories has everything you need to build a vanilla AOSP for Raspberry Pi 4 or 5. You can build the kernel and the Android OS all by yourself.
2. The lineage-rpi collection of repositories is used to build an Android OS distribution based on LineageOS, which provides a little more than vanilla AOSP, including its own TV launcher.
   NOTE: these repositories became private after Lineage OS 17.1, so you can only build the kernel, but not the rest.

GloDroid

The project was initiated by Roman Stratiienko at “Global Logic” with the main goal of creating a platform for training Android BSP developers [#44 Proof My Concept : GloDroid]. Github profile: https://github.com/rsglobal

It was originally started for Orange Pi. Raspberry Pi support was recently added in 2023: https://github.com/GloDroidCommunity/raspberry-pi/commit/7c0e3b199f9c7f5a43ad76cf9688811300ce0490

Source code: https://github.com/GloDroidCommunity/raspberry-pi

Prebuilt images: https://github.com/GloDroidCommunity/raspberry-pi/releases

Project status: https://github.com/GloDroidCommunity/raspberry-pi/issues/1

Discord channel: https://discord.gg/5H8cW5xA

GloDroid also provides builds based on LineageOS, but you have to build it yourself using this script: unfold_lineageos.sh. See README.md for details.

OmniROM

OmniROM is a custom ROM distribution founded in 2013. The OmniROM distribution also founded TWRP (Team Win Recovery Project), which is included in the raspberry-vanilla and lineage-rpi builds as the default recovery partition. Android 13 builds for the Raspberry Pi 4, when tested by us resulted in an unstable experience, with frequent OS crashes and visual stuttering. Due to these unresolved issues, we would not recommend using this distribution for this particular version of Android. The latest stable release for Raspberry Pi 4 appears to be Android 12.1.

Emteria Android OS

Emteria customizes and maintains the Android OS for popular off-the-shelf hardware and industrial platforms, including Raspberry Pi. This BSP is closed source and commercial.

More info: https://emteria.com/

Which one to choose?

It depends on your requirements and your use case. Here is a checklist:

• What version of Raspberry Pi do you target? Different versions have different hardware, and require different Android BSPs.
• Can you build it yourself? Current AOSP builds require a Ubuntu machine with at least 64GB ofRAM.
• Do you need support for all the hardware that Raspberry Pi offers? Is anything in particular more important than the rest?
• Do you want to use it commercially? Check the license.
• Do you want the vanilla AOSP experience, LineageOS, or something else?

As you can see, the KonstaKang and GloDroid BSPs are the most prominent AOSP distributions for the Raspberry Pi 4 and 5. However, while similar, these images are built differently and offer different experiences and are intended for different applications. To help you make your choice, we have compiled the following table with the functionality of various components:

The flashing procedures also differ between the two. When flashing to an SD card, the scripts and instructions are different.

Besides using an SD card, a nice addition in GloDroid is fastboot over USB. Simply call:

adb reboot bootloader

and off you go, fastboot is enabled.

KonstaKang builds are using TWRP recovery partition for easy OTA updates. To enter this recovery mode, simply flip a switch in the Raspberry Pi submenu of the modified Settings app and reboot. This submenu also offers some nice features, such as the ability to view and change the screen resolution.

Another difference is that GloDroid uses U-boot, the “Universal Boot Loader”, while the KostaKang builds boot directly from the standard Raspberry Pi EEPROM bootloader.

Future prospects

It seems that the fragmentation of Raspberry Pi Android BSPs is mostly caused by the different goals of the projects. We do not see these projects converging in the near future.

It is also highly unlikely that the Raspberry Pi Foundation will start officially supporting Android.

Fortunately, we have the community where we can find support and play around with Android on Raspberry Pi.

Special thanks to James Gatt for co-authoring this story.

Everything you need to know about Android on Raspberry Pi was originally published in Paradox Cat Tech Hub on Medium, where people are continuing the conversation by highlighting and responding to this story.

Spatial computing, an emerging technology that encompasses the digital integration of physical and virtual environments, has transformed numerous industries by enhancing user engagement, increasing operational efficiency and enabling innovative applications. With the rise of new headsets like Apple Vison Pro and Meta Quest 3 in the automotive sector, spatial computing has a significant potential. The car showroom experience and configurator can become highly immersive opening doors to a new experience, particularly through augmented reality (AR).
This Insight explores the use of spatial computing by means of augmented reality applied to interactive car configurators, highlighting its benefits in enhancing customer experience, increasing sales, and strengthening brand loyalty.

What is spatial computing?

Spatial computing is a paradigm that involves the interaction of digital and physical worlds through the integration of spatial and contextual data. It includes technologies such as augmented reality (AR), virtual reality (VR) and mixed reality (MR), which enable users to interact with digital content in a spatially aware environment.

Importance in the automotive industry

The automotive industry is increasingly adopting digital technologies to meet evolving consumer expectations. AR as a component of spatial computing, offers unique opportunities to engage customers more deeply, providing interactive and immersive experiences that traditional digital interfaces cannot match.

Augmented reality online car configurator

An interactive car configurator allows customers to personalize and visualize their vehicle choices in real-time. Integrating AR into these configurators elevates the user experience by enabling customers to see a life-size, interactive 3D model of their customized vehicle in their own environment.

Benefits of immersive
car configurator experience

1. Best use of retail space: More cars in limited retail space

• Digital exhibition space for city car showrooms: The recent city car showroom displays a lower amount of retail space and is usually focused on one or two example vehicles.
  AR eliminates the need for physical display models, allowing dealerships to display an unlimited number of car models and configurations digitally. This reduces the physical space required for car exhibitions, making room for a more engaging digital showroom.
• Cost efficiency: Reducing the number of physical cars on display lowers storage and maintenance costs while maximizing the use of available space.
  This reduction in physical footprint can lead to significant cost savings in real estate and facility management.

2. Ready to “WOW” your customers: Immersive experience

• Immersive visual and interactive appeal: AR creates an immersive experience that captures customers’ attention and imagination.
  The ability to visualize their customized vehicle in a realistic 3D space, walk around the car, view it from different angles, with different environments and interacting with it in 3D seeing how different customization options look in real life, significantly enhances the “wow” factor.
• Interactive experience: Customers can engage with the car’s features in real-time, exploring different configurations and options in a highly interactive manner, such as changing colors and rims and adjusting interior settings, leading to a more satisfying, enjoyable and memorable shopping experience.

3. Innovative Features: AI + Personalization

• Customization and personalization: AR technology allows for extensive customization options, enabling customers to tailor every aspect of their vehicle to their preferences.
• New AI features: The possibilities with AR are vast, including integrating AI for personalized recommendations, overlaying information about car features, and even simulating driving experiences in a virtual environment.

4. Increase brand experience at the showroom

• Emotional experience: Due to the high immersive character of the experience, customers can explore different realities and at the point of sales in a multisensorial environment and at a 1:1 scale ( real size of the car).
  This can have a high impact on the customer choice and on the emotional experience with the car.
• Brand Differentiation: Adopting cutting-edge AR technology distinguishes a brand as innovative and customer-centric, enhancing its competitive edge in the market as well as its reputation and appeal.

5. Increased sales and conversion rates

• Informed decision-making: By offering a detailed and realistic view of the vehicle, AR helps customers make more informed decisions, reducing uncertainty and increasing confidence in their purchase. In an immersive experience the customer can see the actual personal variant of the product in actual size, rather than using samples.
• Upselling opportunities: AR can visualize the appeal of additional features and options, encouraging customers to consider higher-end configurations and accessories via the true size scale representation of the vehicle.

6. Easy to use and roll-out in the car showroom

• Plug & Play: Modern AR solutions are ready to use with minimal setup, allowing dealerships to quickly integrate AR configurators into their sales processes. Due to technology advancements, the new spatial computing technologies became more user friendly. Out of the box solutions that are ready to use and don’t need controllers nor advanced rigging systems are increasing the user experience and decrease the complexity of past VR systems.
• Intuitive interfaces: AR platforms are designed to be intuitive, requiring little to no training for customers and sales staff to use effectively.

7. Data & customer relationships

• Data-driven insights: AR configurators can collect data on customer preferences and behaviors, providing valuable insights for personalized marketing, improving customer relationships and tracking customer preferences.
• Market expansion: The flexibility of AR allows brands to reach a wider audience without the limitations of physical showrooms to display real scale digital car models. This makes it easier to expand to different markets due to its “out of the box” roll out specifications.

Technical implementation challenge

AR technology stack

Implementing an AR car configurator requires a robust technology stack that includes:

• 3D modeling and rendering: High-quality 3D models of vehicles and customizable parts.
• AR SDKs and frameworks: Tools such as ARKit (iOS), AR Core (Android), or Web-based AR solutions.
• Backend infrastructure: Efficient data management systems to handle user configurations, preferences, and interactions.
• User Interface (UI) Design: Intuitive and user-friendly interface to facilitate seamless interaction with the AR environment.

Integration with existing systems

To ensure a smooth and effective implementation, the AR car configurator must be integrated with the brand’s existing digital infrastructure:

• E-commerce platforms: Synchronization with online stores for real-time pricing, availability, and purchasing options.
• Customer Relationship Management (CRM) systems: Integration to track user preferences, interactions and feedback for personalized marketing and support.
• Analytics tools: Deployment of analytics tools to monitor user behavior, engagement metrics, and configurator performance.

Future prospects

The future of AR in car configurators is promising, with advancements in spatial computing poised to offer even more sophisticated and immersive experiences. Future developments may include:

• Enhanced realism: Improved rendering technologies will enable even more lifelike and detailed visualizations.
• Expanded interactivity: Increased functionality, such as test-driving virtual vehicles in AR environments.
• Integration with AI: Artificial intelligence can provide personalized recommendations and adjustments based on user preferences and behaviors.

Conclusion

Augmented reality represents a transformative opportunity for the automotive industry, particularly in the realm of online car configurators. Due to the intuitive user interface and easy to roll-out solution, this technology has the potential to take car personalization at the car dealership to a new dimension. By providing an immersive and personalized experience, dealers can enhance customer satisfaction, drive sales and increase customer experience at the showroom.

Automotive brands that will use AR technology in the future have a high potential to redefine the usual showroom experience for the customer and make a better use of retail space.

More than this, by adopting AR in online car configurators, automotive companies can significantly improve the customer journey, from initial at home immersive exploration to final showroom purchase, establishing themselves as first moves in the digital transformation of the automotive retail experience.

Authors

Vladimir Moldovanu
Creative Director rpc

Alvaro Alonso
Sales Director ParadoxCat

Can we help you with your project? Contact us at sales@paradoxcat.com

Immersive Showroom Experiences: The Future of Car Buying with spatial computing and augmented… was originally published in Paradox Cat Tech Hub on Medium, where people are continuing the conversation by highlighting and responding to this story.

After getting off the train on which I had been speeding through the French countryside at 320 km/h, I headed directly to the conference venue at Paris Expo Porte de Versailles, where the workshops and tutorial sessions were already ongoing. I joined the tutorial on Understanding the In-Camera Rendering Pipeline and the role of AI/Deep Learning, which gave an extensive background on how modern cameras perceive the world and which algorithms are needed to turn the raw sensor output into beautiful images.

The following three days constituted the main conference, with two tracks of oral sessions, as well as poster sessions in between. Certainly by now, it became clear that the field of Computer Vision and AI is really exploding — with many poster sessions rivaling the Paris Métro in terms of crowding, it was not always easy to push one’s way through towards the poster one wanted to see. Especially presentations from the well-known Big Tech companies and the award-winning papers like Tracking Everything Everywhere All at Once, Segment Anything and ControlNet received their well-deserved attention.

It was also interesting to see that Computer Vision is becoming very interdisciplinary, in the sense that many methods are not just bound to specific problems, but are often applied all across the field and combined in new creative ways. This also meant that it probably became pretty hard for the conference organizers to group the papers into distinct sessions, and there really was something interesting in almost every session.

Besides the oral and poster sessions, the final two days also incorporated two invited keynote sessions. Especially the talk from Pushmeet Kohli (Google DeepMind) was very inspiring, giving an insight to how DeepMind is using AI to help solve hard problems in science, such as their recent success predicting protein structures with AlphaFold.

Au revoir, et merci beaucoup, Paris 🥐🇫🇷!
Thanks a lot to the ICCV 2023 organizers for this great event, and thank you to Paradox Cat for letting me participate! I hope that we can also present some of our own work at a future conference.

PARADOX CAT at #ICCV2023 🥐 was originally published in Paradox Cat Tech Hub on Medium, where people are continuing the conversation by highlighting and responding to this story.

Being part of Paradox Cat means that from time to time you get to do some specialisation trainings on some technologies that not so many people get access to. In our case we had the opportunity to participate in an exclusive online training course about the topics most of us need for working with our customers. This time we did a 40-hour training on Android Open-Source Project (AOSP) with a dedicated day on Android Automotive given by an expert on embedded systems that has been in the industry for so many years already.

During this course we were able to listen to the lessons and then get our hands dirty trying the concepts ourselves on a preconfigured remote machine in the cloud (that made the setup for the course a breeze). It is proven that when you want to study something new, listening to a few lessons is not enough. We developers learn by doing and tinkering, and to do so for this course, we had a safe sandboxed environment to try out the commands we just learned, change some parameters, experiment, etc. and see the results ourselves without the fear of breaking our system.

We touched on many topics involved in the building and customization of such a complex operating system like Android. We started by creating a new “product”, this means we created a new device configuration that allowed us to customize it by adding new packages, services and features.

Once we had this new “product” we applied all the things we saw during the lessons. From replacing the kernel with a new one we built from scratch, looking at the startup scripts and adding a new booting animation, to adding new modules and system services, modifying SELinux policies, adding a new HAL and calling JNI code. Then, for the Automotive part, we looked at the Vehicle HAL, vendor properties and the Car Service.

By getting this knowledge we could be more effective for our customer project and make more informed decisions about the development of some parts of the system where we are involved. Since AOSP doesn’t really provide a lot of public documentation, it is comforting to have someone who understands it to explain to you the most difficult parts and be able to ask questions. And even better, that you get to do it yourself.

All in all, it was a lot of content condensed in just 40 hours, which means we had to review and play ourselves afterwards, just to get the most of it. However, I think it was very useful and allowed us to understand at a lower level how AOSP works and how the overall architecture is designed to work on mobile devices. As an app developer, it is completely another Android world that you don’t get to see when you create mobile applications. It means really getting into the nitty gritty details of an operating system, and if you have never done it before, it is both exciting and daunting at the same time, but if you are eager to learn new things outside of your comfort zone, this is your place. I am very happy that Paradox Cat gave me this opportunity to take this training and expand my knowledge, enabling me to grow as a Software Engineer.

From apps to Operating Systems, how an AOSP training changed our perspective. was originally published in Paradox Cat Tech Hub on Medium, where people are continuing the conversation by highlighting and responding to this story.

Recently, PARADOX CAT took part in Auto.AI Europe 2022 in Berlin. Auto.AI is one of the leading conferences on the applications of Deep Learning in the automotive industry, including but not limited to Level 4 and 5 of autonomous driving. As a company that specializes in in-cabin AI applications, we hosted a World Café session on the topic of architecting robust machine learning systems for HMI interaction within autonomous vehicles, moderated by Johan von Forstner (Principal Machine Learning Engineer) and Alexander Schaub (Director Development).

Our World Café session focused on several important challenges in the development of AI systems for natural interaction in the vehicle, and many attendees participated in the interesting discussions around these topics.

Auto.AI takeaways

Leveraging from new and existing sensors

Safety-critical in-cabin sensing applications such as driver monitoring or child presence detection, but also convenience features, like gesture control, greatly benefit from dedicated sensors such as interior cameras with RGB and near-infrared capabilities or depth sensors like time-of-flight cameras. Modern millimeter-wave radars, which are traditionally used as exterior sensors for autonomous driving, are also quickly gaining popularity in the interior space due to their low cost and power consumption. They are even accurate enough to detect vital signals, including breathing and heart rate. On the other hand, participants stressed, that it should be possible to leverage the existing sensors in the cabin for new use cases with machine learning and sensor fusion, including microphones, steering wheel and seat occupancy sensors.

Avoiding false positives

In gesture or voice interaction systems, false positives can easily lead to frustration and increased driver distraction if they trigger unwanted actions in the HMI. There is always a tradeoff between making the system as natural and easy to use as possible while also keeping the gestures or voice commands distinct enough to avoid triggering them accidentally. In other words: Should the system be trained to work well for every user, or should the user need to be trained to use the system correctly? This problem is hard to solve in general, so it requires tailored solutions depending on the application. Possible approaches include personalization of the system to the user through active learning, as well as combination with other modalities like gaze and pose detection to provide additional context.

Deploying active and federated learning systems

Just like autonomous driving systems, in-cabin sensing applications usually benefit from large and diverse datasets collected from the fleet. However, sending data back from customer vehicles may raise privacy concerns, especially in the case of interior cameras. Federated learning, where only gradients of the deep learning model are transmitted, provides a privacy-friendly approach to this, however it is not directly applicable in cases where we cannot benefit from self-supervised techniques to label the data. In these cases, it could be combined with active learning, where the user can directly help to improve the AI system by providing labels for a couple of data samples. In the car, special care needs to be taken to choose a suitable UX design for such active learning sessions.

PARADOX CAT and in-cabin AI

PARADOX CAT has created a new division (PARADOX AI) with focus on artificial intelligence and machine learning, which has a startup character within our already established software development business. We strongly believe that this technology will have the greatest impact on everyone’s lives in the future. Focusing on new in-cabin applications of AI, we enable our automotive customers to develop prototypes in this field based on cutting-edge research. In addition, with our experience in series development of automotive HMI systems, we can support customers with the integration of AI applications into these platforms to bring them into production. Do you want to learn more about our PARADOX AI team and services? We are looking forward to get in touch.

Originally published at paradoxcat.com on September 29, 2022.

Machine Learning systems for HMI interaction — insights from our World Café session at Auto.AI was originally published in Paradox Cat Tech Hub on Medium, where people are continuing the conversation by highlighting and responding to this story.

Disclaimer. I have presented most of this material at DroidCon 2023 in Berlin. If you prefer a video, you can find it here: https://www.droidcon.com/2023/08/01/custom-launcher-why-and-how-do-i-build-one/

What is a launcher?

According to the source code of Android OS, a Launcher or a Home App is an application which “replaces the home screen and gives access to the contents and features of your device”. It is usually responsible for listing and starting other applications and for hosting widgets.

Here is an example of a launcher:

Guess what your phone screen would look like without a launcher? You would only see what is called System UI:

A home app is essential for any Android system. Try to start your applications without it!

Every Android device comes with a pre-installed launcher. Device manufacturers may create their own home app, or simply reuse the vanilla launcher that comes with AOSP (Android Open Source Project). Since it is Open Source, you can actually inspect the original launcher source code. For instance, Launcher 3 for Android phones can be found here: https://cs.android.com/android/platform/superproject/main/+/main:packages/apps/Launcher3/src/com/android/launcher3/Launcher.java

User can choose default home app: Settings — Apps — Default Apps — Home app:

A default launcher will be the default “HOME” action, which can be triggered by a hardware button or a swipe-up-from-bottom gesture. The selected default home application will always be at the bottom of the activity back stack. In simple words, if you keep going “BACK”, you will always arrive at the default launcher.

Why create new launcher?

There are multiple possible reasons:

• Performance
• Features
• Usability
• Design
• Customization

These are the reasons behind every custom launcher you see in the Google Play Store. In fact, most of today’s stock launchers features today were once only available in custom launchers.

If you check Wikipedia, you will find a table with more than 60 launchers for phones/tablets only. Obviously, not all of them can keep up with Android development and some became obsolete when their features were integrated into stock launchers.

Another reason to create a new home app is the plethora of Android devices out there: phones, foldables, tablets, TVs, cars, wearables. They all need a launcher.

There are many possible applications for Android OS. Here are just a few of them.

• Vending machines, coffee machines, beer machines
• Public transport: entertainment, info screens, ticket machines
• Shared & private transportation: cars, bikes, scooters
• Smart home, IoT devices

Of course, not all of them run Android (yet), but if they do, they all need a launcher.

How to develop a launcher?

First of all, make your application a launcher. Add this category to your intent filter in AndroidManifest.xml:

<category android:name="android.intent.category.HOME" />

Actual meaning of this category and other related ones can be found in Intents.java, here are the example definitions extracted from this file:

android.intent.category.HOME
This is the home activity, that is the first activity that is displayed when the device boots.

android.intent.category.HOME_MAIN (hidden API, only for platform)
This is the home activity that is displayed when the device is finished setting up and ready for use.

android.intent.category.SECONDARY_HOME
The home activity shown on secondary displays that support showing home activities.

android.intent.category.SETUP_WIZARD (hidden API, only for platform)
This is the setup wizard activity, that is the first activity that is displayed when the user sets up the device for the first time.

android.intent.category.LAUNCHER_APP (hidden API, only for platform)
This is the home activity, that is the activity that serves as the launcher app from there the user can start other apps. Often components with lower/higher priority intent filters handle the home intent, for example SetupWizard, to setup the device and we need to be able to distinguish the home app from these setup helpers.

Next step in becoming a launcher is to list the apps. To get a list of apps, use Package Manager’s queryIntentActivities() API:

val intent = Intent(Intent.ACTION_MAIN)
intent.addCategory(Intent.CATEGORY_LAUNCHER)
val flags = PackageManager.ResolveInfoFlags.of(
    PackageManager.MATCH_ALL.toLong())
val activities: List<ResolveInfo> =
    context.packageManager.queryIntentActivities(intent, flags)

For ease of use, map this list to your app objects:

data class App(
    val name: String,
    val packageName: String,
    val icon: Drawable?
)

val installedApps = activities.map { resolveInfo ->
    App(
        name = resolveInfo.loadLabel(packageManager).toString(),
        packageName = resolveInfo.activityInfo.packageName,
        icon = resolveInfo.loadIcon(packageManager)
    )
}

Start an app via such object:

fun App.launch(context: Context) {
    val intent = context.packageManager.getLaunchIntentForPackage(packageName) ?: return
    context.startActivity(intent)
}

How to display applications?

Let us start by looking at the existing user interface modalities. It seems that the world has settled on multi-screen navigation with a configurable grid of icons grouped with the app names.

However, grid is not the only option. Experiment with simple flat lists, embedded search or filtering using text or speech, fast scrolling using first letters, etc.

Take a look at Niagara Launcher’s approach:

Or it could be as simple as OLauncher:

Also, remember to give users a way to see all of their installed apps in one place. For example using an App Drawer which one could “draw” from the bottom of the screen with a swipe-up gesture:

That said, do not limit yourself to application icons and app widgets only. Get creative and build the user interface around what your users actually need. For example, a SpaceX rocket launcher (pun intended) might look something like this:

App Widgets

A story about launchers would be incomplete without talking about App Widgets.

This is an example of a standard clock widget:

I will not bother you with details on implementation or widget types, you can find great official documentation here. If you use Jetpack Compose, check out the Glance library, it is truly great!

Instead, let us discuss the uses and limitations of App Widgets.

It is clear that widgets are quite useful for applications like calendar, clock, and weather. However, if you want to have something like a fully interactive map on your home screen all the time, App Widgets unfortunately are not going to cut it.

And why is that?

• Interaction. Gestures are limited to only touch and vertical swipe, so as not to conflict with Home App gestures.
• UI. Not all UI elements are available, so you cannot create a full-featured home screen app as an App Widget.
• Battery. By default, App Widgets refresh once every 30 minutes (!). That says a lot about the design decisions.

But wait a minute, isn’t that what the default launcher of Android Automotive OS does? Showing a giant interactive map in a launcher?

Yes, it does:

If it is not an App Widget, what is it?

In the source code, Google calls these widgets “Cards”. Cards are implemented in terms of lower-level window management facilities. This facility used to be an ActivityView, which was deprecated and eventually removed in Android 12. Now it is a TaskView, which is rightfully a hidden platform API.

What does “hidden platform API” mean for application developers? Simply put:

• it should not be used by non-system applications,
• it will likely change in future Android version.

Yes, there are ways to expose this API to applications, but if you want your application to survive Android updates, I urge you to consider a better design alternative.

Implement the element you want to “embed” directly in the launcher. As long as you have control over the providing app, you can extract the relevant code parts into common libraries and design proper communication between the “Widget” and the “App” yourself.

That said, your launcher should probably still support embedding App Widgets as well, some users depend on it.

We have covered a lot of ground in this story. To summarize, we have learned the following.

• Launcher is a home screen application.
• The world needs new launchers that are fast and stable.
• Now you know how to make an application a launcher, how to list other applications.
• You should support App Widgets, but do not let them limit your design.

Be creative and focus on what your users need.

Custom Android Launcher — Why and How do I build one? was originally published in Paradox Cat Tech Hub on Medium, where people are continuing the conversation by highlighting and responding to this story.