How I edited 5100 photos for my last timelapse

So, I made a timelapse. Of a sunset. Yes, I think I’m the first in the world to have this idea…

If you want to see it now, before the details about the image manipulations, here it is:

I’m new in the timelapse world, and I made mistakes during the shooting.

But still, mistakes or not, there is a lot of work to do when you take 5100 photos (one per second)

In this post I’ll describe how I:

  • found the images where the camera moved
  • corrected them
  • cropped all images to the area I wanted to keep
  • corrected the fact that I shot, in JPEG, with a too high ISO
  • removed the flickering
  • removed birds and planes from the sky

Preparation

Tools: I work on Linux (Ubuntu 14.04) and to do this I used ImageMagick and Gimp (in batch mode for the repetitive tasks)

Directory: I created a timelapse directory, in which I created a source sub-directory to put all the images from the camera (~32GB, in JPEG)


Find the moved images

I put the camera on a tripod, but still, it moved. There was some wind, and I don’t remember touching the tripod during the shooting but…

At first, I couldn’t see any problem.

But a thing I love to do when shooting timelapse is to do a “mean” of all images, and, sometimes, also a “min” or “max”

The “mean” picture is a picture where each pixel is the average color of all the pixels at the same position on all images. “min” take the minimal color (black is the lowest), and “max”, the maximal (white is the highest).

To create theses pictures, the easiest way is to use the convert program provided by ImageMagick.

You usually use it this way:

But…. you don’t want to do it this way when you have 5100 images :) You won’t have enough memory, CPU, and time :)

So, you want to split the work into many part. And it will work, because to compute a mean, you can do it in many steps.

And why not doing these steps in parallel? With the parallel command for example.

YOU DON’T WANT TO (believe me, you don’t)… because convert will use all the available CPU to do this kind of work. But we’ll use it many times later.

So, what we want is:

  • split the images in batches
  • run convert on each batch
  • then compute the mean of the generated images

So we go to the source directory, and with the help of xargs, we can do this easily:

You can see in this command the three steps: split with ls + xargs, then convert each batch (of 50 images) via xargs, then the final convert

It will take some time, using all your CPU!

You can do the same for min and max (don’t forget to replace all the occurrences)

And then I wanted to see these images. It’s at this moment that I saw I had a problem. It is obvious something was wrong.

For example, here is P110600.JPG, which tell me there is a problem between P110600.JPG and P1106049.JPG. You can clearly see the movement.


Correct the moved images

I knew I would crop the images for two reasons:

  • focus on the interesting part of the sunset
  • fastest image processing: the sources images are 4592*3448 and I will create the timelapse video in HD so all I need is 1920*1080. It’s 7.6 times less!

Knowing that, I can easily correct the moved images by “moving” them (and/or rotating, if needed), because the borders will be ignored in the crop step.

Here is how I manually updated the 7 moved images.

I take the image preceding a moved one and open it in Gimp:

I add the image to update in a new layer. Menu File > Open as layer

And I set the new layer mode to Difference:

This way, I’ll see in dark the part of the images that are the same (or nearly, because clouds move), and in bright, the difference. Here is a zoom on the image in Gimp:

The only thing I have to do is to move the layer until I cannot see any bright part. Here I moved 16 pixels to the left and 4 to the bottom:

Then I can set back the layer mode to Normal and export the image on top of the modified one (at this time I still had the original on the SD card)

Rinse and repeat, for each moved image.


Crop the images

So we want our images to be 1920*1080 to be ready to be used in a HD video, and to process them a lot faster in the following steps.

Open an image in Gimp and open the Image > Canvas Size menu.

With the anchor NOT “activated”, enter 1920 as width and 1080 as height, using px as unit, and THEN, activate the anchor:

Then you can freely move the image in the preview, and you can raise the height or width (with the anchor activated: if you change one, the other will change too, to preserve the ratio).

Here is my selection:

Do NOT resize the image, but take note of the different values.

Here, we have, ignoring the minus signs:

- height: 3280
- width: 1845
- offset X: 716
- offset Y: 1248

You can close Gimp now.

Now, on the command line, we’ll use convert from ImageMagick again to crop and resize the image. The base command looks like:

As we have 5100 images, we can take advantage of the multiple cores we have on our machines, so we’ll use parallel.

Here is the final command line for me:

Here I ask parallel to launch the command for all .JPG file, using 8 processors, and save the result images in the sized directory.

It will take some time, using all your CPU!

At the end, I have 5100 images in the sized folder, resized and cropped, now with a weight of ~1MB instead of 6.5MB for the original ones:


Correct bad ISO choices

I’m new to the photo world, and I easily make mistakes. For my previous timelapse, I set the ISO to 1600, but at the end, the sky was really dark and I had to change the exposure but also lost a lot of “data” not taken by the camera.

This time, I wanted to avoid this, so I raised the ISO to 6400. I forgot that the last time, there were dark clouds hiding the sun, and not this time. So the sky stay bright all time…

So, now, I have 5100 images with a lot of noise.

Here is a zoom to see it:

There are many ways to correct this. One is faster, but not as good as the others, but as we have 5100 images to update, I chose it: selective gaussian blur.

I let you search on the internet to see how it works but the main idea is to only blur areas with contrast below a certain threshold. So the line separating two very contrasted areas won’t be blurred.

It exists in ImageMagick but it takes forever, so we’ll use Gimp.

First, open an image in Gimp, then go to Filters > Blur > Selective Gaussian Blur:

Then, try to choose the value you find appropriate.

For me it’s 5 for the blur radius, and 20 for the max delta. A biggest delta was better for the sky, but not for the sunset colors. So you have to try.

Here is the same area as before but with the blur applied:

But how to do it for the 5100 images? We can use the powerful batch feature of Gimp. As it was my first time, I searched on the internet and found exactly what I wanted on this page:

http://www.miguev.net/blog/en/2007/12/25/batch-noise-reduction-with-the-gimp/

It’s quite old (NINE YEARS!!!) but works perfectly. I didn’t change anything, just saved the script in the Gimpscripts” folder (~/.gimp-2.8/scripts/batch-noise-reduction.scm for Gimp 2.8)

Just in case the link above doesn’t work when you’ll read this page, here is a gist with the script: https://gist.github.com/twidi/646f1e863a43f1e1d5db14d2aa768002

As the script run in a whole directory, updating images in place, and because I didn’t want to loose the cropped images (in case of problem), I copied the sized directory into a blur one.

And to be able to run this in parallel, I then split this directory in chunks, using the dirsplit command. This command expect a “max size” for the new directories: we cannot pass it a number of files. So, instead of dividing the number of files by the number of process we want, we divide the size of the directory:

What we do here:

  • We get the full size of the directory in bytes, with du (keeping only the number with cut),
  • We divide this number in 8 (the number of process we’ll tell parallel to use, or a factor of this number: for example if we use 4 cores, 8 directories is ok) using bc, but multiplying by 1.01 to prevent dirsplit to create a 9th directory for adjustment. And then removing the floating part with cut,
  • We use dirsplit to move (-m) the files in a “simple” (alphabetic) way (-S), specifying the max size of each directory with -s, the prefix of the directories to create with -p, and finally the directory to split.

The result:

Now that I have all my directories, I can run the noise reduction script in parallel.

Due to a problem with the ”*.JPG” being badly interpreted when invoking Gimp in batch mode in parallel, I created a simple script, noise-reduction.sh, to be saved in your timelapse root folder:

Then I could run this:

As you can see I told parallel to run 4 process in parallel, not 8. Because in my tests, 8 was really heavy for the computer, and not faster than 4. Both are ~3 images per second (total for all processes). Instead of 0.8 per second without using parallel!

Note: how to know the number of images processed per second and see the progress? Before running the command, in another terminal, I created a “timer” file at the timelapse root, then used watch to count the files updated after the “timer”:

It shows the number of files updated after the creation of the “timer” file, every 10 seconds. So it’s easy to compute the difference mentally and divide by 10 to get the number per second (it would have been a lot slower if we did it on the source pictures, without resizing!)

Now we can move the files back to the blur directory:


Removing the flickering

There are many reasons we can have flickering on this kind of photo series. You may not see it at first, but go through a suite of photos at high speed and you’ll see it: inconsistent lighting or luminance variation between consecutive pictures.

But of course we’re not the first to encounter this problem.

During my researches on the subject, I found a perl script that uses ImageMagick: timelapse-deflicker.pl, available on Github: https://github.com/cyberang3l/timelapse-deflicker/blob/master/timelapse-deflicker.pl.

I created a gist, just in case: https://gist.github.com/twidi/6d2f0ea3769f011a07a67dfb6c94eeae

You’ll need to install some perl modules to use it. On Ubuntu:

This script will read the luminance of all pictures, save them (to avoid doing it again in case of problem, because it’s a slow operation, especially on 5100 images), then adjust it for each image based on the previous and next ones, saving the new images in a Deflickered sub-directory.

It can do it in two passes for a better result (the -p 2 argument below).

Use it this way:

It’s a long process too. Imagine if we executed it on the source pictures!

Here is a little video (4 seconds) with the screen split in three to compare the source image, the blur, and the deflickering

http://p.twidi.com/demo-blur-and-flickering.mp4

It’s a direct link to the video because when uploaded to Youtube, the noise was automatically removed, so there was no point to keep it (but I kept it to let you see the difference with the original: https://www.youtube.com/watch?v=tFjYRN5cwPs)


Clear the sky

During 90 minutes, on a large view, the number of birds and planes (mostly birds) that cross the sky can be very high. In my previous, and short, timelapse, I removed each bird manually, but there were only a few.

And it is worth noting that the same birds may be at different places of the sky in consecutive pictures!

Why removing them? Because if you don’t, very fast black flash dots will appear when looking at the final video!

How to see if you have a lot of birds? It’s simple, we’re going to create again the “min” image, but on the cropped, blurred and then deflickered images:

You can see a lot of birds:

And there are not all there, because at the end the sky is darker, so small birds in the first parts are not shown, but with the intermediate min pictures, we can see them:

Of course, seeing all these birds, I wanted to automate the removal. I am a beginner in image treatment, and knew nothing about programming scripts for Gimp. But it was my goal for this step.

First, I had to try on Gimp, manually.

Here was my plan:

  • find an image with a bird
  • open the previous one with Gimp
  • open the image with the bird as a new layer in Difference mode (as seen earlier)
  • add a mask to ignore the static part of the images (the ground, trees, buildings…)
  • flatten the layers in one
  • convert to black and white to ignore small differences (the lighter the color in the difference view, the more important the difference is: a black bird on a bright sky)
  • select the black
  • invert the selection to get the white/gray
  • grow the selection
  • add again a layer with the image with the bird (and ensure it’s the active one)
  • use the “Resynthetizer” Gimp plugin to magically remove the bird
  • save the image

Some screenshots of the whole process:

A part of the sky on the image just before the one with a bird:

The same area on the next image, with the bird:

The difference on the bird:

We can see we that have some difference on the monument and the trees too, because of the light changing, the wind moving the leaves…:

We know we can ignore theses little differences, and all the dark part of the image, which is nearly the same on all the pictures. So, we’ll create a mask of this part.

To do this, we’ll start by creating the “mean” of all images:

Then, open the mean/mean.JPG in Gimp (as a new image).

We have this (which is an image I always create when doing timelapse of the sky, and that I love looking at ):

We’ll convert it to black and white to have the static parts in black and the rest in white.

Go to the menu Colors > Threshold

Then we’ll update only the “black” part by moving the black arrow. By default the black should be at 127 and the white at 255.

At 127, the static part is black, but also the sky, that is dark:

So we’ll reduce the black value until we find something correct. For me it’s 110:

Now we can select the black part of the image, using the Select by Color tool (you can see at the bottom the options I used for this tool. I think they are the default ones):

Then click on any black part to select the black area.

And we’re going to grow the selection we just made a little to compensate the threshold:

We now fill the whole selection in black, by selecting the Bucket Fill tool. Set it to fill the whole selection using the foreground color (in the options at the bottom):

Then click on any part of the selected area to fill it.

Now you can cancel the selection:

We have something like this:

To skip some manipulations later, we’ll set the white part as transparent.

First, we add the Alpha Channel to enable transparency:

Then, using the Select by Color tool, as seen before, click on any part of the white area and press the del touch on your keyboard (or menu Edit > Clear)

We cancel again the selection and we have:

Don’t hesitate to add black on the parts where you are sure changes must be ignored.

Our mask is now ready, we can export it:

Save it as mask.png in your timelapse root folder, using the default provided options in the Export image as PNG dialog box.

Now we can go back to the Gimp image with the bird. We were in Difference mode. We’ll add the mask as a new layer, and flatten the images:

Now re’re going to convert to black and white. And to choose the right threshold that will be OK for the whole images series, we want to repeat the process (except creating the mask: we’ve done it once for all) with many birds.

For example one with a darker sky and a lot of small birds.

Here is the part with the birds, and after doing the difference, adding the mask, and flattening the image:

And another with a plane:

So let’s apply a threshold, as seen before (menu Colors > Threshold).

The value of 127 (for the black arrow, because we don’t change the white one) is too high. We have a black screen for the three images.

With 50, it’s OK for the big bird, but not all the small birds are shown, and the plane is not fully highlighted:

With many tries, it seems that 30 is a good value for all:

Now, we’re ready to remove these highlighted parts.

First, we have to select them:

  • select the black part of the image, using the Select by Color tool
  • invert the selection
  • grow the selection by 5 pixels

We now have this:

And we can open as layer, on each image, the original with the bird(s) or plane. Our selected parts become:

If we look closer, we can see birds that are not selected in the second image.

I choose to ignore them. Because if they are not selected, it’s because they are small enough, and with little contrast compared to the sky, so we should not see them in the final video.

Notice that it could be solved by using a lower threshold (for example 25 instead of 30), but with such a small value, the risk to select things we don’t want to (for example a cloud moving) increase fast. So I decided to keep 30.

Now, to finally remove the birds and planes, we’ll use the magical tool provided by the “Resynthetizer” Gimp plugin.

On each image, keeping the selections as they are, go to the menu Filters > Enhance > Heal selection

I chose to keep the default values:

After doing this, simply cancel the selection. Now, our objects are removed!

Now that we are happy with the result, we can close Gimp and all the images. We do not save them, because we’ll do it programmatically for the 5100!

If we summarize, what we do to remove an object of an image B is:

  • open image A
  • add image B as layer
  • do what we have to do to remove the objects on this image B

At the end we have two layers:

  • on top, the final image B with the objects removed
  • and below, the one with the difference, in black and white

So if we flatten both layers (or remove the bottom one), we have only one, that we can export as B’ (which is B without birds or planes)

So we can continue:

  • add image C as layer
  • do what we have to do to remove the objects on this image C
  • flatten the images
  • export as image C’
  • add image D as layer
  • do what we have to do to remove the objects on this image D
  • flatten the images
  • export as image D’

The only thing that can change is if there is no object to remove. In this case, there is no need to manipulate the selection and to apply the Heal selection filter.

To detect this, when we select the black part of the difference, then invert the selection, if there is no selection, it’s because there is nothing to remove.

In this case we can simply add the image back as layer, flatten and start again and simply copy the image in the final directory as nothing changed (faster than generating an image that didn’t change, with Gimp)

So here is the script I wrote that will do all of this for us. Save it as clean_sky.py at your timelapse root folder.

Obviously, it cannot be done in parallel, because each image depends on the previous one being cleared.

And, it’s important to notice it, the very first image must be cleared manually! Because it will be the base for the whole process.

Now we are ready to apply the script:

We ask Gimp to run without interface (-i), by running in batch python mode (batch-interpreter=python-fu-eval), with some python code (explained below), then we ask it to quit when done (-b ‘pdb.gimp_quit(0)’). We chose to not display the (verbose) error output but saving it, just in case, in a file ( 2> ../../clean_sky.error.log), and on the standard output, we remove a sentence we don’t need, produced by gimp ( | grep -v ‘Gimp adaption’), to only have our own output.

For the python part (in the first -b part), the way it’s done here is the only way I found to run this script without registering it in Gimp, which I don’t want to.

Here is the output for my first images. The ones marked as “updated” are the ones where something was removed from the sky.

It’s, again, a very long process. But at the end, I have my 5104 images ready to be selected in a video editor to create my timelapse video!

Creating this video is not the subject of this post, which is already very long. Just to let you know, I used blender to create mine.

Here is the final result, with one second of the video for 2 minutes of real time (images shot at one every second)

PS: I’m really not a professional photographer, image editor, or video creator. It was my first time for a lot of things in this post. If you have comments, don’t hesitate, I really want to learn about this subject!