Setup live file-synchronization between remote hosts

Synchronization between remote hosts is most effectively achieved by using programs like rsync over SSH, possibly under crontab.

We discuss here a couple of alternative ways for live file-sync, namely a continuous synchronization in which a transfer is started whenever a file changes at the source.

Lsyncd

This is available at this URL. In the words of the developers, it is best suited for (one-way, my comment) sync between a local directory tree with low profile of expected changes and a remote mirror. For further information please check the wiki page.

Lsyncd watches a filesystem for changes (via Inotify or fsevents), waits a few seconds to aggregate changes and finally triggers synchronization (via rsync+SSH).

Before you install it:

• Create SSH key:

  ssh-keygen -o
  

• Exchange keys between hosts. On hostB add the content of hostA:/root/.ssh/id_rsa.pub to hostB:/root/.ssh/authorized_keys, and vice-versa.

• Try to connect from hostA to hostB and vice-versa: make sure the Security Groups allow this.

Proceed with installation:

• the package should be already available, quick install with:

  apt-get install -y lsyncd # will bring in dependency lua5.X and liblua5.X
  

• or you can recompile lsyncd:

  cd <some_base_dir>
  git clone https://github.com/axkibe/lsyncd
  # install dependencies as per INSTALL file, for example
  apt-get install -y gcc g++ cmake make binutils rsync
  # Read file INSTALL and compile/install, for example
  mkdir build
  cd build
  cmake ..
  make
  sudo make install
  

Configure:

• look in /etc/init.d/lsyncd and check what is the expected name for the configuration file, in my case (Ubuntu 16.04) it was /etc/lsyncd/lsyncd.conf.lua

• create your own configuration file:

  mkdir /etc/lsyncd
  mkdir /var/log/lsyncd/
  touch /var/log/lsyncd/lsyncd.{log,status}
  cp -p /usr/share/doc/lsyncd/examples/lrsyncssh.lua /etc/lsyncd/lsyncd.conf.lua
  

• edit configuration file as desired, in my case the result was (please check option delete in manual):

  settings {
    logfile = "/var/log/lsyncd/lsyncd.log",
    statusFile = "/var/log/lsyncd/lsyncd.status",
  }
  
  sync {
    default.rsyncssh,
    source = "/home/diskVdb/dirPa",
    host = "<remoteHostIP>",
    targetdir = "/home/diskVdb/dirPa",
    delete = "running",
    rsync = {
      archive = true,
      compress = false,
      update = true,
      whole_file = false
    },
    delay = 5,
  }
  

• restart the program:

  systemctl enable lsyncd
  systemctl restart lsyncd
  

Running lsyncd as two-way mirroring may cause race conditions, and the program may be confused by the appearance/disappearance of temporary files. According to your specific use case, you may consider:

• in light of this message from the author, set temp_dir to a path outside the synchronization tree,

• use exclude or excludeFrom in the sync section of the configuration file, so filenames matching specific patterns are not considered for mirroring.

Mirror

Mirror is available at this URL. The striking features of this application are that it is two-way and that it is triggered by filesystem events (Inotify).

Potential drawbacks are that it is designed to sync a single directory (although you can run multiple instances of the program to deal with multiple directories), that file ownership are not preserved (files are owned by the user/group running mirror) and that synchronization does not leverage TLS encryption: this latter inconvenience can be overcome by running mirror within a SSH tunnel, please see the docs for further information.

Installation:

• install Java 8 with:

  apt install openjdk-8-jre
  

• install watchman according to instructions, please read them carefully. At time of writing the minimal set of steps (but again, check for fancier configurations in the above web page) you need to:

  apt-get install -y gcc g++ cmake make binutils
  apt-get install -y libssl-dev autoconf automake libtool python-dev pkg-config
  mkdir -p <someBaseDir>
  cd <someBaseDir>
  git clone https://github.com/facebook/watchman.git
  cd watchman
  git checkout v4.9.0  # the latest stable release
  ./autogen.sh
  ./configure
  make
  sudo make install
  

• download mirror executables to some directory in your path:

  cd /usr/local/bin
  wget http://repo.joist.ws/mirror-all.jar .
  wget http://repo.joist.ws/mirror .
  chmod u+x mirror
  

• adjust inotify limits (please also see this page):

  echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p
  echo fs.inotify.max_queued_events=50000 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p
  

• elect one of your hosts as the server and start mirror:

  cd /usr/local/bin/
  ./mirror server
  

• on the other host, run the client like this:

  cd /usr/local/bin/
  ./mirror client -h remote-host -l /the/source/dir/ -r /the/remote/dir/