File system features we encountered developing the Mail.Ru Cloud synchronization mechanism

Data synchronization is one of the main functions of the Mail.Ru Cloud desktop client. Its purpose is to fully synchronize selected folders and files on user PCs with their version in the cloud. When developing this mechanism, we found some (at first) rather obvious features of various file and operating systems. However, if you don’t know about them, you might find yourself in hot water (you won’t be able to download or delete the file). In this article we collected information that when you’re aware of, you can work correctly with data on disks and, perhaps, never need an urgent hotfix again.

1. File system events do not guarantee a complete picture of the event

Any directory synchronization mechanism requires the monitoring of file and folder state changes. Fortunately, the API of each operating system can do this for us. We use ReadDirectoryChangesW for Windows, FSEventStream for macOS and inotify for Linux. And here is the first time things start to get unpleasant. The fact is that for macOS it’s impossible to say for sure exactly what kind of event came from the file system. You can easily get CREATED, DELETED, RENAMED and MODIFIED on a file in one event.

And everything seems logical: if there’s a deletion, then there’s no file anymore. However:

$ rm 1.txt && echo “hello” > 1.txt

will be one event:

1.txt: CREATED | REMOVED | MODIFIED

Therefore, it’s necessary to use additional mechanisms of verification for events to understand what happened with the file or directory.

In inotify, the event queue can overflow, and you can start to lose it until you remove some events from the queue. In this case, the lost events are not compensated to you in any way, and you have to perform expensive operations like a disk bypass.

2. You cannot work with symbolic links as if they were regular files

Symbolic links can be looped: A -> B -> C -> B. It’s possible to solve this problem, for example, using the inode number (the unique number of the file or folder in the current disk partition. But we’ll cover this later). In our case, we keep the inode list of symbolic links that we collected to the current directory. If the inode of the current symbolic link is the same as in the list, then we consider it looped and skip it.

A symbolic link can end up being dead. If at some moment the content the symbolic link specified is moved or deleted, the link will become unavailable. It is important to process this moment correctly.

If you subscribe to a directory event where you have symbolic links to other directories, events about the change of content on the symbolic link won’t be delivered.

3. Names of files and folders can be in the wrong UTF-16

There used to be an interesting bug related to this. There was a file in the local tree of the user who reported a problem to us. However, when we tried to read it, we saw that in reality there was no file. This would seem to be a logical situation if the file is deleted at the moment we start our work. But the file was again in place after the next directory listing.

The fact is that you can create a non-valid coding of UTF-16 in Windows. More precisely, the name may contain a non-valid surrogate pair. Converting this name into UTF-8 and then back into UTF-16 by standard means (WideCharToMultiByte, MultiByteToWideChar) will not work. Here is an example:

wchar_t name[] = { 0xDCA9, 0x2E, 0x74, 0x78, 0x74, 0x00 };

Surrogate pairs consist of High and Low values and are necessary to expand the range of coded symbols. High Surrogates are in the range of xD800 — xDB7F. Low Surrogates are in the range of DC00 — DFFF. We took High, but not Low in our name. Thus, we obtained a non-valid UTF-16.

Now we convert this name into UTF-8, then back:

wchar_t name2[] = { 0xFFFD, 0x2E, 0x74, 0x78, 0x74, 0x00 }; // “� .txt”

The symbol representing the beginning of the surrogate pair breaks, and you cannot use that name any more.

Example code

We always work with UTF-8 in the synchronization module. We receive events or listings from the file system and convert names into UTF-8. The server also works with UTF-8. When accessing the file system, we have to convert UTF-8 back into UTF-16. This problem was solved by prohibiting the synchronization of non-valid UTF-16.

4. Traps and pitfalls when working with inodes

The desktop client did not for a long time support the renaming of files and folders. Instead, the renaming event was processed by deleting files in one place and creating them in another. This mechanism worked for a long time, and rather steadily at that. The fact is that deleting a file from the cloud only means deleting the link to this file from the user tree. The file itself, at the same time, remains alive for a while on the server so that you, for example, could restore it from the trash. Thus, deleting and creating the file in another place was managed just by the meta-information on the server, which just deleted the link to the file from one place and created it in another, even without opening the local copy. However, with the advent of shared folders, we began to understand that it was movements we should be processing (to not lose the attribute of the shared folder and not unmount the attached folder).

It’s one thing when the renaming event comes from the file system. In that case, there aren’t any problems. Click-click, and it’s renamed. And if the application isn’t running? We need some information we can use to detect the renaming event. There were several options to detect movements:

  • Compare the hierarchy of files and folders. This is a very difficult process, even knowing that trees are stored in the memory.
  • We need to create hidden files with service information in each folder, so we can understand where the folder has moved or what it was renamed into. However, this causes some difficulties, including the fact that the user can change and edit these utility files, which can lead to unpleasant consequences. And we didn’t want to ‘look over’ every directory either.
  • Inodes. We ended up going with this option.

Inode is an index descriptor. It is denoted by an integer and represents the identifier of the file or folder in a particular file system.

I recommend reading this article to better understand how it works. In POSIX we get inode from stat (st_ino), and in Windows — GetFileInformation (nFileIndex). And it all seems like it’s pretty simple:

  1. the client restarts, and we load the cached representation of the file hierarchy.
  2. Now we compare this to what’s actually on the disk.
  3. We find the nodes where inode numbers are absent where we believed they should be, but are in fact in some other place.
  4. Then we move these nodes.

However, you should be very, very careful with inodes. Here are some of the traps and pitfalls we’ve been facing.

4.1. Hard links

Each link of this type to one file has an identical inode number. We do not detect renaming if there are hard links in the tree. You cannot create a hard link to the folder (or it’s almost impossible), so there are no particular problems here.

4.2. Inodes can work differently than you expect

Inode numbers are not assigned as they should be in some file systems (or rather, as we think they should be). We believe that their numbers don’t change when renaming the file. We also assume that if we delete the last file on FS with inode 9, then the following file will have inode number 10. Unfortunately, some file systems do not agree with this.

New files (not folders) with inode number 9999… are created for macOS on FAT. When renaming these files, the inode number does not change. When editing these files, numbers change to ordinal values, just as we would expect to see:

$ touch 1.txt
$ ls -i
999999999 1.txt
$ echo “hello” > 1.txt
$ ls -i
223 1.txt

Ext 4. The fact here is that if on this file system (which is standard in the majority of Linux distribution kits) we delete the file with inode number 9 in one place, and create the new file in another place, it will have an inode not of 10 or above, but 9.

$ touch 1.txt
$ ls -i
270 1.txt
$ rm 1.txt && touch 2.txt
$ ls -i
270 2.txt

I.e. on this file system, the first free number becomes the inode number. This is a little strange for us. The solution came by itself: if we detected the renaming of the folder, we compare inode numbers for the folders and hash + the size for files for its content. If directories are the same 70% and higher, we rename them. For files — if the hash + the size are the same.

Taking into account that the numbering of inodes in different file systems works differently, we have to check whether inodes work as we expect: when starting the synchronization module, the test behavior for the check is reproduced. If it is as we expect, it’ll be possible to work with inode numbers. Otherwise, we continue without renaming support.

5. Programs store a lot of service files on disks

Operating systems and programs with varying popularity use the service files on the disk, and synchronizing them makes no sense. The list of files and masks that should be ignored by the synchronization mechanism, as we thought, is given below:

Windows:

  • desktop.ini — stores user settings for the current directory;
  • Thumbs.db — caches of image previews;
  • the files beginning with “~$”, or “.~”, or beginning with “~” and ending with “.tmp” are templates of temporary files that are quite commonly used. Microsoft Office also creates files of such templates when editing documents.

macOS:

  • .DS_Store — analogue of desktop.ini for Windows;
  • Icon\r — quite an interesting file; displayed as “Icon?” when listing, stores information about the icon on the directory it’s in;
  • files beginning with “._” — there were quite a lot of templates instead of this, but various software applications prefer to use their own format for temporary files, so we decided to ignore files under this mask.

Linux:

  • .directory — analogue of desktop.ini for Windows and .DS_Store for macOS, relevant for some window managers.

6. Windows paths to files and folders

Paths for Windows, of course, deserve special attention. For paths exceeding MAX_PATH value (260 symbols), it is necessary to use a prefix “\\? \”. This prefix, by the way, should be used for CreateFile if you are planning to open COM-port.

Windows creates short aliases (also called “8.3”) for each file or folder, with names longer than 8 symbols. Aliases are always in the high register and contain the character “~” followed by a number, which increases if this alias is already in use (for example: «C:\PROGRA~1\»). The content of these features is necessary, but ultimately not enough to understand that it is just a common name or a short alias that sits before you. WinApi can convert short paths back into long one (GetFullPathName). However, it is necessary to remember that it will not convert a path to its long representation if such a file no longer exists.

If someone opens the file through CreateFile using a short path and modifies it, then the same short path will come to you in the event from the file system (through ReadDirectoryChangesW). In this regard, we try to convert them into the long version as soon as possible. By the way, you can see the aliases by entering “dir/x” from the necessary directory in the Windows command line.

Another unpleasant feature that is not to be ignored: files and folders with a dot at the end cannot be opened using the explorer (true for Windows 7):

7. Conclusion

The synchronization algorithm had to be adapted to fit each file system. The reconstruction of a test environment, when you start in order to check the folder that the user chooses, was the best solution in our case. And if the tests are not passed we either prohibit the user to work with this folder or deactivate certain features. I hope that what we dealt with can help you avoid some difficulties when working with file systems.

If you have any questions or comments, please feel free to ask them in the comments or write me personally at a.skogorev@corp.mail.ru.