Now Playing: Ambient Music Mod v2

Note: If you’re just here for the update, check out the screenshots, tl;dr and download links at the bottom of the page.

Ambient Music Mod was first released just over a year ago now, it was a lockdown project — getting Now Playing from the Pixels to work on another device. It was something like the fourth time I’d attempted it, but the right combination of device (a OnePlus 7T Pro) and software allowed it to actually work: 1,270 days after the Pixel 2 was released. Since then something has become quite clear: The method used in the original Ambient Music Mod is incredibly finicky, and a new route was needed to allow it to work on newer Android versions and different devices.

How does Now Playing work anyway?

If you’ve ever used a Pixel with Now Playing enabled, you may have been amazed at its ability to recognise music. I certainly have — there have been numerous times where it’s picked up a track I can barely hear myself, or I’ve woken my phone up in a shop and been greeted with the name of the song that’s being played over the world’s worst PA system. It’s certainly clever, but how does it actually work, and how does it manage to recognise music so efficiently and without an internet connection?

The basis of recognition on Pixels lies in the “hotword” system. This is the same code and hardware responsible for reacting to the “Hey Google” wake-word, which is used by the Google Assistant — a constantly model that processes ambient audio to recognise the phrase. On almost every modern Android device, there’s a dedicated signal processor (DSP) that takes responsibility for this recognition, minimising battery life. On Pixels since the Pixel 2, this also serves another use, music detection. Contrary to what many believe, the actual track detection is not actually done at this level, the model that runs simply recognises that some music is probably playing, and tells an app — Android System Intelligence. If some other requirements are met, for example the user isn’t on a call or playing the music themselves from the phone, it will start an 8 second recording and run it through the actual track recognition engine.

A simplified look at how Now Playing works on a Pixel

The music recogniser itself creates a fingerprint of the recorded audio and compares it to a known database of tracks (if you want to know more about how this works, you can read a research paper on it). This database is stored offline, and totals around 250MB, containing in the region of 69k tracks (USA database, June 2022). The tracks vary by country, to give Now Playing the best chance of recognising that barely audible song in the back of a convenience store.

Breaking Down Barriers

Due to how hardware-specific the DSP model is, getting it to even load on non-Pixels is very difficult. If you ever tried the original Ambient Music Mod, you probably encountered some of the errors, and the response was always the same: sorry, if the model doesn’t load, you’re out of luck. It seems the OnePlus 7T Pro running Android 11 (the device I was dailying at the time) coincidentally is one of the few devices that was able to load the model, giving a false sense that it was more widely possible. To make matters worse, loading the model was only a requirement to get the method working, the model didn’t actually detect music was playing, instead Ambient Music Mod bypassed it and injected recognitions using time-based triggers (every x minutes).

When it became clear that the actual number of users that Ambient Music Mod was usable for was very small, I decided to take another look at the project and the state of the options available. Since its release the generally used Xposed implementation has shifted away from EdXposed to LSPosed, and while both are Magisk modules and may seem similar enough from the outside, LSPosed development involved creating LSPlant, an efficient method hooking framework. LSPlant itself was used by the developers of Aliucord (a Discord mod) to create the aptly named hook, which allows for very easy Xposed-like modifications to apps that are decompiled and re-compiled with APKtool.

With Hook, it’s now easily possible to inject code into and modify an app that is installed as a regular APK, so Android System Intelligence can be modified to disable all functionality except Now Playing, and with full control of the app we’re able to bypass the whole DSP model requirement, replacing it with the same time-based trigger that was previously used.

Even better, the permission required for accessing the hotword microphone source (CAPTURE_AUDIO_HOTWORD ) has been granted to shell since Android 12 (for Compatibility Test Suite device certification purposes), so by using Shizuku we can actually remove the need for root entirely, at least for the core music recognition features of the mod.

A simplified look at the implementation of Now Playing with Ambient Music Mod

Now Playing: Ambient Music Mod’s Implementation

Ambient Music Mod now has three components, including two APKs to install:

• Now Playing, a modified version of Android System Intelligence with all the features except music recognition disabled
• The Ambient Music Mod app, which provides a UI for changing settings, running manual recognitions and updating the music database
• The Shizuku service, run from the Ambient Music Mod app. This provides a service for Now Playing to connect to, and run the audio recording using the “hotword” audio source. For technical reasons, this service is “proxied” by the Ambient Music Mod app as a regular bound service, to allow Now Playing to access it without itself having to have a Shizuku service.

A more complex look at how data flows between Ambient Music Mod, Now Playing and the Shizuku service

Ok, what’s the catch?

Time.

Because Now Playing on the Pixels has the DSP music detection, it knows when music has stopped, and most importantly when it has started. It knows when you’ve walked into a room with music playing, and so it should start a recognition. Without this, Ambient Music Mod has to use some arbitrary time-based triggers. There are a couple of things we can do to improve this though:

• The Now Playing engine is able to estimate the remaining time of a song it’s detected, so for songs that are being played back-to-back we can dramatically improve the likelihood of successive recognitions being successful by simply delaying the next recognition until the current song is due to finish, plus a little buffer time to allow the next song to get started. This also has the benefit of improving battery life since it means less recognitions have to run (so long as the remaining song time is less than the regular delay time)
• We can use user triggers, such as running a recognition when the screen is switched on, or the pressing of a button on the widget, to add additional recognitions, improving the performance of song detection when it’s needed most

There are however some downsides to this approach that can’t be fixed:

• Songs that are played for a short time outside of the trigger time won’t be recognised, for example during TV or radio advertisements (some may see this as a benefit)
• Songs that finish early, for example if you leave a room, will still continue to delay the next recognition until the song would have finished, if it were still playing. On a Pixel, the device is able to recognise this stop, and if another song starts playing will run a new recognition.

Performance Testing

To test how well Ambient Music Mod works against the real Now Playing, I ran a “very scientific” test, putting my Pixel 6 Pro (with the real thing) and OnePlus 7T Pro (running Ambient Music Mod) next to each other, within audible distance of a radio. Both were using the same music database: GB, June 2022. The radio was tuned to Absolute Radio*, who publish their playlist online. They ran for 5 hours, during which time 56 songs were played.

As you can see, the Pixel got all 56 songs. It actually also got two more — both appeared to be recognised during ad breaks (not included as they are not on the station’s playlist). The OnePlus got 55 songs, missing only one song. Therefore the mod has almost as good performance as the real thing, at least for when listening to a radio station. This is especially good, considering the music played is not all recent either, it includes tracks from the 60s all the way to some released this year.

Throughout this time the OnePlus showed that Ambient Music Mod and Now Playing combined to use a total of 5% (of the 18% idle) battery. AccuBattery showed a usage of 2.4mAh. Unfortunately, since Android System Intelligence is excluded from battery metrics on the Pixels, we can’t get a measure of how much battery that uses (nor would it be fair given ASI also has other features), but the Google research paper linked above reports a daily usage of “less than 1%”.

*Absolute Radio is broadcast in 80Kbps mono, over DAB. It being in mono may have slightly improved the recognition (although the sound quality is pretty awful).

Additional New Features

The original Ambient Music Mod used an old build of Pixel Ambient Services (the app that previously handled Now Playing on Pixels before it was moved to ASI), so it was locked to that version’s features. With the new mod’s ability to use the latest and greatest build of Android System Intelligence, it can take advantage of some new features:

• Now Playing History has favourites enabled, so you can mark a song as a favourite to come back to later
• On Demand is enabled where possible, which allows you to use the Assistant-backed internet music recognition to recognise a song when asked (you must trigger it manually) that is not in the local database. In addition, this track’s fingerprint will then be stored locally so if the device hears it again, it will be recognised without internet.
• The recognition model in the newer build is significantly better, meaning tracks can be recognised with lower quality and quieter audio, and without needing huge gain adjustments that Ambient Music Mod previously relied on.

The aforementioned On Demand feature is actually a system service, introduced in Android 12. OEMs can now define a “MusicRecognitionService”, implemented in an app. On Pixels, this is handled by the Google app. For it to work with Ambient Music Mod, it requires both a specific variant of the Google App to be installed, and a system config value to be set to point to the Google App’s service. Unfortunately, as most devices don’t include this variant of the Google App, the system value is not set and a Magisk module is required to set it correctly. The app guides you through this process, if you wish to install it.

On top of the new features in Now Playing, a number of features return to Ambient Music Mod that were originally in the mod:

• Album Art can be shown in the Now Playing History (sourced from YouTube Music)
• You can specify the recognition period (delay between recognitions when music is not recognised), and trigger manual recognitions. You can also disable automatic recognitions entirely, if you want it to only be triggered when requested.
• You can show recognitions on the lock screen, using an overlay
• You can have recognitions automatically run when the screen is switched on
• You can view the entire track list of your selected country’s database, to check which songs Now Playing can recognise

And some new mod features too:

• Two widgets to show the currently recognised track, and trigger normal or on-demand (online) recognitions if available
• Bedtime Mode, to disable automatic recognitions overnight when music is unlikely to be playing
• A Quick Settings tile to instantly enable or disable recognitions
• An app shortcut, to give quick access to recognitions from your homescreen
• Backup and Restore, which not only backs up your settings but also your full Now Playing History, Favourites, and locally stored on-demand fingerprints too.
• The lock screen overlay will show a “Music Search” button when On Demand is enabled, and a track is not recognised, to instantly search for a track using the internet-backed method, without even needing to unlock
• You can now show the currently recognised song in the “Owner Info” on the lock screen, for seamless integration with OEM skins (root required, experimental)

The new widgets in Ambient Music Mod

Protecting Your Privacy

You may have noticed on a Pixel that Now Playing’s recognition does not show a microphone icon in the privacy indicator, nor does it show in the Privacy Dashboard. This is because the “hotword” microphone source (which, again, is protected by a system-only permission) is excluded from being shown to the user. Obviously this is a concern for users of mods like this, since you’re giving an app access to potentially record audio whenever it pleases.

In the Android CDD, the system will make the microphone indicator appear when an app is using the microphone, unless (amongst other things), SOURCE_HOTWORD is being used by the app.

Ambient Music Mod has always been open source. It will continue to be. In addition, the modifications to Android System Intelligence are open source, including build instructions to go from a clean ASI APK to a fully modified version.

Android System Intelligence is also unique in terms of system apps, in that it is required to be extremely limited in terms of interactions with other apps and the internet. This is also defined in the Android CDD:

Now Playing follows these requirements, with Ambient Music Mod providing the equivalent of the Private Compute Services app, giving well-defined links between Now Playing and the outside world. These are used for internet (for database downloads only) and communication with the Shizuku service only, and can all be viewed in the source code for Ambient Music Mod. Private Compute Services’ open source internet component is used in Ambient Music Mod to provide internet access to Now Playing to download database updates, and is also defined in the source code, in the astrea module. On Demand in Ambient Music Mod runs via the system Music Recognition Service mentioned earlier, and does not interact with the internet at all on its own — it’s all done by the Google App.

Screenshots

Ambient Music Mod v2 uses the same Material You style as my other recent apps

tl;dr

Ambient Music Mod, previously one of the most restricted and difficult Xposed and Magisk mods, now runs on far more devices. Root is no longer required (on Android 12+), and Xposed is not required at all. There’s a ton of new features, and a brand new design too.

Download & Install

You can get Ambient Music Mod from the GitHub releases, it will install Now Playing and the latest database as part of the setup process. Either Shizuku (non-root or root) or Sui (Magisk/root only) is required.

Updated 27/06:

A previous version of this story stated that the United States database contained 53k songs. Due to a bug in the database reading code, this is incorrect, it actually contains 69k songs.

With thanks to Mishaal Rahman for some of the research throughout the development of this update.