lsyncd Command: The Ultimate File Synchronization Tool

File synchronization is an essential aspect of modern computing, enabling seamless data transfer and backup between various devices and platforms. As the volume of data grows exponentially, ensuring that files are kept up-to-date across systems becomes increasingly challenging. This is where lsyncd, the ultimate file synchronization tool, comes to the rescue. In this blog post, we will explore what lsyncd is, how to set it up for local and remote syncing, and why it stands out as a reliable and efficient solution for file synchronization.

What is lsyncd?

Lsyncd is an abbreviation for “Live Syncing Daemon,” and it lives up to its name. It is an open-source tool designed to monitor multiple directories for changes and propagate these changes to designated target locations in real-time. Whether you need to synchronize files between local directories or across different servers, lsyncd proves to be an ideal choice.

Key Features and Advantages

Real-time Synchronization: Lsyncd excels in real-time synchronization, ensuring that any modifications to files are promptly replicated to the destination

Local and Remote Syncing: Lsyncd supports both local and remote synchronization, allowing you to keep files in sync within your local network or across geographically dispersed servers

Low Resource Consumption: With its efficient design, lsyncd operates with minimal resource consumption, making it suitable for both small-scale systems and high-performance servers.

Scalability: Lsyncd is capable of handling synchronization tasks between multiple sources and destinations simultaneously, providing flexibility and scalability.

Support for Multiple Platforms: Whether you’re running Linux, macOS, or even Windows (via cygwin), lsyncd supports a variety of platforms.

Getting Started with lsyncd

To begin using lsyncd, you’ll need to install it and set up the configuration. Here’s a step-by-step guide to get you started:

Installation: On most Linux distributions, you can install lsyncd using the package manager. For example, on Ubuntu, run:

sudo apt-get install lsyncd

Configuration: Lsyncd relies on a configuration file (usually located at /etc/lsyncd/lsyncd.conf) where you define the synchronization tasks. The configuration file is written in Lua, a simple and powerful scripting language.

Local Syncing Configuration

So, for testing, let’s try a local syncing with lsyncd.
First, let’s create a “source folder”, containing the files we want to sync.

$ mkdir -p $HOME/Linuxexplorers/sync_source

Then, we’ll make a backup folder:

$ mkdir -p $HOME/sync_backup

Just create some file in sync_source directory using touch command:

$ touch $HOME/lsyncd/sync_source/test{1..10}0

Now, make log and status files:

# mkdir /var/log/lsyncd

# touch /var/log/lsyncd.{log,status}

Configuration file goes on

/etc/lsyncd

# mkdir /etc/lsyncd

In there, we’ll create a new config file:

# vim /etc/lsyncd/lsyncd.conf.lua

where we will put the following code:

settings {
    logfile    = "/var/log/lsyncd.log",
    statusFile = "/var/log/lsyncd.status",
}

sync {
    default.rsync,
    source = "/home/linuxexplorers/lsyncd/sync_source/",
    target = "/home/linuxexplorers/sync_backup",    rsync = {        archive = true,
        compress = true,
        verbose = true,
    },
    delay = 1, -- Delay in seconds before syncing changes
}

Save and close the file when you are finished.

This configuration uses rsync, a fast and versatile file synchronization tool, for local synchronization.

Right now, if you look in the backup directory, you’ll find it empty, because the service is not running.

But, to solve this, simply start lsyncd with the following:

# systemctl start lsyncd

# systemctl enable lsyncd

It will start (and keep) syncing, even if you add more files on sync_source. It works completely in automated mode.


You can check the status of lsyncd service using below command:

# systemctl status lsyncd

lsyncd.service - LSB: lsyncd daemon init script
     Loaded: loaded (/etc/init.d/lsyncd; generated)
     Active: active (running) since Sun 2023-07-30 15:15:19 IST; 36min ago
       Docs: man:systemd-sysv-generator(8)
    Process: 19794 ExecStart=/etc/init.d/lsyncd start (code=exited, status=0/SUCCESS)
      Tasks: 1 (limit: 9102)
     Memory: 1.0M
        CPU: 168ms
     CGroup: /system.slice/lsyncd.service
             └─19800 /usr/bin/lsyncd -pidfile /var/run/lsyncd.pid /etc/lsyncd/lsyncd.conf.lua

Jul 30 15:15:19 linuxexplorers systemd[1]: Starting LSB: lsyncd daemon init script...Jul 30 15:15:19 linuxexplorers lsyncd[19794]:  * Starting synchronization daemon lsyncd
Jul 30 15:15:19 linuxexplorers lsyncd[19799]: 15:15:19 Normal: --- Startup, daemonizing ---
Jul 30 15:15:19 linuxexplorers lsyncd[19794]:    ...done.
Jul 30 15:15:19 linuxexplorers systemd[1]: Started LSB: lsyncd daemon init script.

You can check the Lsyncd log file for more details as shown below:

# tail -f /var/log/lsyncd/lsyncd.log

You should get the following output:

Sun Jul 30 15:15:19 2023 Normal: --- Startup, daemonizing ---
Sun Jul 30 15:15:19 2023 Normal: recursive startup rsync: /home/linuxexplorers/lsyncd/sync_source/ -> /home/linuxexplorers/sync_backup/Sun Jul 30 15:15:19 2023 Normal: Startup of /home/linuxexplorers/lsyncd/sync_source/ -> /home/linuxexplorers/sync_backup/ finished.Sun Jul 30 15:15:56 2023 Normal: Calling rsync with filter-list of new/modified files/dirs
/file1

You can also check the syncing status with the following command:

# tail -f /var/log/lsyncd/lsyncd.status

You should be able to see the changes in the /home/linuxexplorers/sync_backup directory with the following command:

# ls /home/linuxexplorers/sync_backup

file1  file2  file3  file4  file5  file6  file7  file8  file9  file10

You should see that all the files and directories from the “/home/linuxexplorers/lsyncd/sync_source/” directory are added to the “/home/linuxexplorers/sync_backup” Directory.

Configure Lsyncd to Synchronize Remote Directories

In this section, we will configure Lsyncd to synchronize /etc/ directory on the local system to the /opt/ directory on the remote system.

Before starting, you will need to setup SSH key-based authentication between the local system and remote server so that the local system can connect to the remote server without password.

On the local system, run the following command to generate a public and private key:

# ssh-keygen -t rsa

Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa
Your public key has been saved in /root/.ssh/id_rsa.pub
The key fingerprint is:
SHA256:mnwGPNegukLA+NClNRUnMDF0J35qG4stxxiVwMK4cXI root@ubuntu22
The key's randomart image is:
+---[RSA 3072]----+
| o oB+*.o        |
|+ E =* *          |
|oB = .+ o         |
|=.o  o + o         |
| +  . O S .         |
|  o  X X            |
| .  = X o           |
|  .  + o              |
|   ..                   |
+----[SHA256]-----+

The above command will generate a private and public key inside ~/.ssh directory.

Now we will copy the public key from local server to remote server. Using below command.

# ssh-copy-id root@remote-server-ip

it will be asked to provide the password of the remote root user as shown below:

root@45.58.33.57's password: 

Number of key(s) added: 1
Now try logging into the machine, with:   "ssh 'root@45.58.33.57'"
and check to make sure that only the key(s) you wanted were added.

Once the user is authenticated, the public key will be appended to the remote user authorized_keys file and connection will be closed.

Now, you should be able log in to the remote server without entering password.

To test it just try to login to your remote server via SSH:

# ssh root@remote-server-ip

If everything went well, you will be logged in immediately.

Next, you will need to edit the Lsyncd configuration file and define the rsyncssh and target host variables:

# vim /etc/lsyncd/lsyncd.conf.lua

settings {
    logfile    = "/var/log/lsyncd.log",
    statusFile = "/var/log/lsyncd.status",
}

sync {
    default.rsync,
    source = "/etc/",
    host = "remote-server-ip",
    target = "/opt/",
        rsync = {        archive = true,
        compress = true,
        verbose = true,
    },
    delay = 1, -- Delay in seconds before syncing changes
}

Save and close the file when you are finished. Then, restart the Lsyncd service to start the sync.

# systemctl restart lsyncd

You can check the status of synchronization with the following command:

# tail -f /var/log/lsyncd/lsyncd.log

You should see the following output:

sun jul  30 04:32:05 2020 Normal: --- Startup, daemonizing ---
sun jul  30 04:32:05 2020 Normal: recursive startup rsync: /etc/ -> 45.58.33.58:/opt/
sun jul  30 04:32:06 2020 Normal: Startup of "/etc/" finished: 0

You should be able to see the changes in the /opt directory on the remote server with the following command:

# ls /opt

You should see that all the files and directories from the /etc directory are added to the remote server’s /opt directory:

Advanced Usage and Customization

Lsyncd offers various synchronization modes, which cater to specific use cases:

• Single Source to Single Target: The default mode, where changes from one source directory are propagated to a single target directory.
• Multiple Sources to Single Target: Monitor multiple source directories and synchronize them into a single target directory.
• Many-to-One: Merge changes from multiple source directories into a single target directory without overwriting existing files.

To further customize lsyncd’s behavior:

Handling File Types: Lsyncd allows you to specify which file types to include or exclude during synchronization.

Fine-tuning Options: Explore the various options in the configuration file, such as handling deletions, syncing permissions, and setting up custom exclude patterns.

Real-world Examples

Let’s explore some real-world examples to illustrate how lsyncd can be implemented:

Local Folder Backup: As a developer working on a project, you can configure lsyncd to synchronize your working directory with a backup folder on an external drive to ensure changes are automatically backed up.

Remote Server Deployment: If you have a web application hosted on a remote server, lsyncd can monitor the source code directory on your local machine and synchronize it to the target directory on the server in real-time, ensuring consistent code deployment.

Conclusion

Lsyncd, the Live Syncing Daemon, has proven to be the ultimate file synchronization tool with its real-time capabilities, low resource consumption, and flexibility. Whether you’re a developer, system administrator, or someone seeking a reliable backup solution, lsyncd is an excellent choice to keep your files in sync across devices and servers. By understanding its configuration and advanced features, you can harness the power of lsyncd for seamless data synchronization and confidently manage your data across different environments. Give it a try, and experience the simplicity and efficiency of lsyncd for yourself!