Working with Motion Photos
Motion Photos is a feature which launched with the Google Pixel 2 and what it does is, along with the photo, it records a video of up to 3 seconds (1.5 seconds before and 1.5 seconds after you took the photo).
For an Android app, Motion Photos behave just like normal JPEGs, but if you are a photo editing app and want to overwrite the original photo with the edited one, you need to know that if you don’t take care of it, the new edited photo will lose the embedded video.
Diving into the format
A motion photo consists of a JPEG file whose name starts with MVIMG: MVIMG_xxxxxxxx_xxxxxx.jpg (this seems to be what Google Photos checks to see if it’s a Motion Photo or not, because removing MV from the start of the filename causes Google Photos to stop showing the video).
If we open the file with a hex editor
xxd MVIMG_xxxxxxxx_xxxxxx.jpgThen we discover that at the end of the JPEG photo there is a MP4 video (on the image below we can find ftypmp42, which indicates the start of a MP4 video)
So, with this step we learned that the structure of a motion photo file is as follows:
Because the file starts with the JPEG, apps can open it as a regular photo. But if we want to get the video, then we need to know the offset at which it’s located (which is equal to the image size, which is obviously different from the file size, since it includes both the photo and the video).
Luckily for us, inspecting the EXIF data reveals a “Micro Video Offset” key which offers exactly the value we wanted (the offset is from the end of the file).
Extracting the video
On a UNIX system we can use dd to extract the video from the Motion Photos. Since the offset is from the end of the file, we need to calculate (file-size — offset from EXIF) and pass it as bs argument to dd
dd if=MVIMG_xxxxxxxx_xxxxxx.jpg of=video.mp4 bs=3239850 skip=1If we view the video details with mediainfo, we discover some interesting things, like the video is recorded at 25 fps.
Working with Motion Photos on an Android app
In order to get the JPEG we operate like with a standard photo, but in order to get the video we need to seek to the offset specified on the EXIF data.
Opening the image again with exiftool but dumping the result to html reveals that this information is located inside of the APP1 XMP segment, which is stored as RDF
exiftool -htmldump MVIMG_xxxxxxxx_xxxxxx.jpg > exifhtml.htm
Extracting the video offset
The problem with this is that Android’s ExifInterface doesn’t allow us to read that value, so we need to use an external library, like metadata-extractor.
We can set up the library with the following Gradle dependency:
implementation 'com.drewnoakes:metadata-extractor:2.11.0'And we can retrieve the video offset simply by reading the “GCamera:MicroVideoOffset” XMP key:
Displaying the image
You can display a Motion Photo just like a regular photo, for example passing its URI to an ImageView:
imageView.setImageURI(uri)Playing the video
In order to play the video, we need to skip to the offset where the MP4 video starts on the file
Playing with MediaPlayer
You can play the video by simply passing the offset to MediaPlayer setDataSource method, passing a FileDescriptor, the offset and the length of the video (which is equal to the offset since the offset is from the end of the file). You can then set the MediaPlayer to looping to get the same behavior like on Google Photos.
Playing with ExoPlayer
The first step is to create a new DataSource, which is based on the standard FileDataSource, but when open is called it delegates to FileDataSource passing the original DataSpec. This is necessary because if we use the regular FileDataSource then when playing ExoPlayer will call open with a DataSpec without the offset, causing playback to fail.
And then we can play the file by simply creating a DataSpec object and passing the required offset to it, and then using MotionPhotoDataSource.
The rest of the implementation is just regular ExoPlayer code, with the addition of using a LoopingMediaSource to mimic the looping behavior of Google Photos (we are also using a PlayerView provided by ExoPlayer in order to display it).
Extracting the embedded video
Now that we have the offset available, we can extract the video from the photo simply by copying the bytes from that offset to the end of the file (note that the offset is from the end of the file, so we need to compute file size — offset as start point):
Creating a Motion Photo
But what about creating a motion photo? (Maybe the user is editing some photo and we want to preserve the video attached to it, or even also edit it using ffmpeg or a similar tool).
For this first we need to output the image to a file like with a standard JPEG and then append the video at the end like we did above
But if we try to open this image with Google Photos, it’s handling the image like a normal photo (no option to play the video). For this to work we need to attach the XMP metadata to the photo, and for this we need to use a library which allows us to write XMP metadata (metadata-extractor, used above, only allows us to read it), like Apache Commons-Imaging.
We need to add the apache repository
repositories {
maven { url 'https://repository.apache.org/content/repositories/snapshots/' }
}And then set up the dependency
implementation 'org.apache.commons:commons-imaging:1.0-SNAPSHOT'Now after writing the image to the file we add the required metadata to it:
The trick on the code above is to create a copy of the image (_xmp as shown above) and then rename it to the original image afterwards, because if we try to pass the same file to JpegXmpRewriter both as input and output we’ll encounter errors.
The updateXmpXml function takes as arguments the saved photo, an OutputStream where to save the updated photo with the XMP metadata and the XMP XML string. You can copy this string mostly unchanged, because it’s mostly defining XML namespaces and a RDF document. What you might need to change is the offset: here I’m passing the same value retrieved from the image since I’m saving the video unmodified with the same dimensions and size, but if you are modifying the video you might need to adjust the offset to be equal to the new video size (since the offset is from the end of the file, the image size does not affect).
And that’s it! Hopefully now you have learned how to manage Motion Photos and found interesting use cases for your apps
