OS X Photos (beta) vs. 70.000 photos

First chapter

I was brave. After installing OS X 10.10.3 beta which includes the beta of the long-waited Photos for OS X app, I made an up-to-date backup and pressed the big, nice import button in the app. Photos is meant to replace my old and beloved iPhoto, and take over control of my 70.000 pictures and 946 videos. It was written from scratch, which I needed so much while using the outdated iPhoto, which I always thought was coded by interns without any Objective-C experience as a starting project at Apple.

The import process took 3–4 hours, after that I was able to browse through my pictures with a moderate speed in Photos’ nice, clean window bezel. It seemed that it has imported all of my images. My several thousand events were also shown as albums. I don’t like that Photos does not support events anymore, but I will get used to it for sure.

It took another 15 hours for the background processes to finish. I think they created the thumbnails for my viewing pleasure. After that the library was running at decent speeds. Not incredibly fast, but at least faster than iPhoto. I can live with that.

Second chapter

I got very excited so I decided why not go further and opted in for the €9.59 package at Apple, which gives 500 GB space on iCloud. It is not even the biggest option. Then I closed my eyes and clicked the “Enable iCloud Photo Library” button in Photos preferences. We’ll see.

I knew that it will take a long time, because 450 GB is 450 GB no matter what. I was prepared for the following. It has worked on upload from Sunday to Tuesday. I was even happy, because all of it is handled by system processes and not the Photos app itself, so I could close the program and let the upload take place in the background, automatically. The peak bandwidth was 6–7 MB/s, which I call pretty cool. Much better than Dropbox or anything similar from here.

The feedback was not as good as the upload speeds. There is only a single number in a single view in the whole Photos app which indicates something is happening (below the photos, at the same place as on iOS), and this number is refreshed too infrequently. I would also prefer to have some indication of the remaining time and the progress of the overall process too.

On a bright Tuesday afternoon the number reached 0, so I could begin the rest.

Third chapter

iOS can’t handle a library with this size. Take this as I say it: I thought that downloading the changes will take several hours then as I scroll through the library it will download the thumbnail needed to display. It does not work like this.

First of all, it seems, that iOS Photos app syncs only when it is running, the device is on WiFi and when it is connected to a charger (I am not sure about this last one). The charger does not hurt tho, because downloading the changes causes so much high CPU usage, that iPhone 6 was hotter that ever and consumed battery bower with a greater intense than the USB charger could recharge it at the same time. The same was the case with my iPad Air 2.

No problem, I was thinking, we will pass this soon and will forget the inconveniences very fast.

Well, we had not pass it. Until Thursday (three days after the upload finished) the iOS devices could not finish downloading the library. I have the “Optimize for iOS Storage” option selected so obviously it needed only to download the meta informations and a few thumbnails. If I opened the Photos app, it had start syncing, then, after a few hundred images it always crashed. Of course crashing makes the syncing to stop also. There was an afternoon, when me and my phone played this scene 40 times. Start, wait, crash, start again… Then I got bored.

The final state of the syncing looked like this: on my 64 GB iPhone (mostly empty) I was able to scroll back until 2008 (my library has the first pictures from 2000). The thumbnails were downloaded until around 2010, but 2010 and 2011 were also missing a few hundred of them. Newer dates looked good. It stated that it has been downloaded 42.000 images and 28.000 is still to go, but this numbers were a little bit erratic and not even summed 70.000 everytime.

On my iPad (16 GB) the whole process stopped around 20.000 images then nothing after that was downloaded. Not even meta infos or gray placeholder images. The tablet was obviously full and crying for free space every five minutes.

Fourth chapter

That’s a thing that downloading the photos takes a week (before somebody asks, I have a 100 Mbit download bandwidth), but we have two much more serious problems here too:

1. A library in this size makes most of your app’s photo pickers whine. A lot of apps on iOS lets you choose some of your photos. Most of them nowdays uses custom image picker implementations to do this. These implementations are not prepared for 70.000 images. Some of the apps wait 15–30 sec before even showing you an image or crash immediately. Facebook Messenger for example is totally unusable, because it even does precaching or what in the background, so there are 10–20 second periods when you can’t scroll, type or do anything in the app.
2. 16 GB devices are outdated. Period. Apple promotes iCloud Photo Library as something which can carry all of your photos (and by all they mean a lifetime worth of photos) on all of your devices. These devices are clearly not 16 GB ones, because the 20.000 synced image meta informations and around 10.000 synced thumbnails took 5–6 GB of space, so they completely filled my iPad. And by “filled” I mean that the iPad is constantly whining for freeing up some space; I can’t install updates, download apps or anything. And I don’t even have my photo library with me!

Fifth chapter

I tought I check the iCloud.com client too, why not. It is also a very problematic case. I seems it is written to download every photos dimensions in advance to be able to do it’s layout. So this downloading took 4–5 hours for the web page. Until then my CPU was burnt.

(I leave it here for a while. How much CPU power is wasted here? Just think about it. I have a 4-core Intel Core i7 at 2 GHz. 4–5 hours of this CPU’s time is enough for… so basically it should be enough for solving any kind of mathematical problem. And it was wasted to download useless image dimensions.)

During this, the web page crashed a few times. I was not aware of the thing that a web application can even internally crash. It is. It displayed a nice HTML error dialog with a retry button which I could push.

One more little nice thing: on friday, without even starting the Photos app on the desktop for days, photolibraryd choosed to do something unidentifiable, and run for 5 to 6 hours using 100–200% CPU power. I don’t know what it did but until then this was not happened again. I hope this was the last time because I don’t want to waste battery power for unseen optimalizations or whatever.

Summary

So I have subscribed for a service for €114 a year to be able to view any of my images from any of my devices but I was not able to view them nor on the web, nor on iOS, but at least had two iOS devices which were basically useless, because apps couldn’t handle the photos in this amount or missed more free space. My Mac was haunted by some photolibraryd daemon.

Thanks, I opt out, find something else.

Well, you say, that it is a beta, and it will be fixed in the 1.0 release? Sorry guys, but I have to disappoint you. In the latest 2 years none of the Apple betas was fixed not for the 1.0 release but at all. Yosemite is still full of bugs, iOS 8 too, iCloud, iMessage, and I can probably all of the things they released lately.

The bigger problem I see is that they have to use algorithms which are exponential instead of constant time or at least linear. So as your library grows they act a multitude worse.

Sixth chapter

I dediced to entertain you with the followups too.

• On Thursday my phone capitulated and stopped syncing at 27.999 remaining images. Seemingly it did not want to download any further images for an unknown reason. It still had a plenty of free space.
• I got enough, so I decided give it a new shot, so I have turned off iCloud Photo Library on my phone to erase all the images and try again from scratch. Not so easy. After turning it off, all my images remained in Photos. I was not able to view them anymore tho: the progress circle stalls at quarter.
• 3–4 hours later the thumbnails were disappeared. I say thumbnails, because all the meta information remained in the library, causing “black” placeholder images to be seen everywhere. It looked like this: http://gk.lka.hu/x/photos.mov
• That night I tried to restore my phone. Created a backup in iCloud then erased all content and settings, then restored it from the backup. Of course this was a whole night process. In the morning I got all of my images back. I mean the black ones, only they were now gray.
• As it seemed clear that it will not heal itself until the end of the times, I was forced to deal with it. So I took my iPhone and deleted photos group by group from the Photos app. This took 2 hours from my life. My library is empty again.
• You think it’s over? No. The empty library also feels slow. So it must have left some garbage somewhere. Maybe an SQLite vaacum is missing?
• For the iPad I thought I will be smarter and plugged in the tablet into my Mac, opened Image Capture selected all the images and hit delete. I could to this, because for some unknown reason all the photos could be seen via Image Capture. I really think this is also a bug, because the program should show only the contents of Camera Roll. My iPad had 28.000 gray images. After deleting 3000 in 2 hours the process stalled and I had to force-quit Image Capture. Restarted the process several times with the same results. The most surprising news was not the slowness and stalling of the delete process, but that it was much faster if I switched Image Capture to display content as list instead of icon grid. Around 5 times faster. Excellent piece of software!
• After seeing that I am faster when doing the delete by hand, I capitulated and did it by hand. By 30 minutes I could finish it in 1500–2000 piece batches. So my library was empty? No. All the images were moved to “Recently deleted” which clearly was not the case previously on iPhone. No problem, I thought, this folder has a “Delete all” button.
• The thing is, that this button can’t handle 17.000 images in one run. It does something for half an hour then crashes. I have tried it several times. During retrials the number of images remained the same.
• After several trials I noticed that every time iOS displays an “iPad is almost full” dialog during the delete process. (Oh, and don’t forget, that the process has zero feedback indication. You press the confirm button and your iPad stalls for half an hour.) I have deleted some big apps, and tried again. After freeing up 2.3 GB of space, it clearly had enough free space on the iPad to delete the images. Ladies and gentleman, deleting needs space.
• So finished, everybody happy, rainbows and unicorns everywhere? No. Of course all of this caused only to have a nice empty Photo Library and an “Other” in the size of the previous Photo Library, and of course the whine for empty space. Basically I had a lot of garbage somewhere.
• I found most of them with iExplorer (a 3rd party iOS browser app for the Mac). Deleting the 2.3 garbage took a whole night for the stuff, but it did it. For the other several gigs of “Other” I have no idea to get rid of.
• Deleting PhotoData folder from iPhone took also almost a day, because iExplorer can do the delete with 2–3 sec/file.
• And finally, the last step: iPhone after the delete was fine (with 1.59 GB “Other”), but iTunes was showing 5.64 GB of “Other” in the case of my iPad. I made an iCloud backup then restored it. This worked, the amount of “Other” decreased to 0.9 GB. I am happy now. Photos apps work fast again, I have my space back, everything works.
• I have no clue yet how to delete images from the cloud, but one day that will be the my next quest. But I have to rest for this job.
• Just for fun: I have tried iOS 8.2 beta and iOS 8.3 beta too. 8.2 is exactly the same as 8.1.3: it crashed, stopped downloading meta data, etc. On the other hand 8.3 seems to be a bit better: it did not crash at all, and managed to download all the 70.000 image meta data. It has also downloaded thumbnails for 2013–2014 and for 2010–2011. I don’t know why it didn’t like 2012 or the years before 2010. After 3 days, the image counter is still yoyoing up and down, I am not sure it will finish at all.