Archive for April, 2014

Arq 4 SFTP critical issue

By

April 8th, 2014

Please read this if you’ve been backing up to an SFTP server with Arq.

Potential Incomplete SFTP Backup Data

If you used Arq versions older than 4.1.1 to back up to an SFTP server, you may be affected by an issue that was fixed in Arq 4.1.1.

Arq 4.0, 4.0.2, 4.0.3, and 4.1 could potentially have written incomplete data files to your SFTP server. This issue would not show up until you tried to restore your files, and that is obviously not the time you want to find out that your backup data aren’t complete!

What Is Affected

Only backups to SFTP created by the above versions of Arq are affected. Backups to S3, Glacier, Greenqloud, DreamObjects, or other S3-compatible servers are not affected.

Steps to Resolve the Issue

If you used Arq versions 4.0, 4.0.2, 4.0.3 or 4.1 to back up to an SFTP server, please do the following:

  1. Open Arq and pick “Preferences” from Arq’s menu to open the preferences window.
  2. Select the “Targets” tab in the preferences window.
  3. Select your SFTP target and click the “-“ button to remove it.
  4. Click the “+” button to add your SFTP target.
  5. In the main window, add your folders to the SFTP target as you did originally.
  6. Click the triangle next to “Other Computers”; select your computer in the list, and click the button to delete the now-unused SFTP data.

Arq will back up your files again from scratch. I sincerely apologize for the inconvenience, which I realize is potentially a massive one if you’ve already uploaded a large amount of data.

The details of the issue and how it’s been fixed are explained below.

- Stefan

 

Details of the Issue and the Fix

The issue was that Arq would write a file to the SFTP server, and if the connection was lost and never recovered, or Arq was restarted, an incomplete file would be left on the SFTP server. Arq assumes that objects on the SFTP server are complete, just as it does for objects written to S3 or Glacier. The difference with S3 and Glacier is that those APIs are “atomic”; either the file is completely stored, or the transaction fails. With SFTP there’s more than one transaction — Arq opens the file, writes many chunks of data to the file, and then closes the file.

The issue never cropped up during a lot of internal testing and several weeks of heavy beta testing. I think this is because SFTP (SSH) re-establishes the network connection whenever it’s lost, and the writes eventually succeed. In some conditions however, such as if the app (or the computer) is restarted, the file could be only partially written.

To resolve the issue, starting with version 4.1.1 Arq writes each file to a temporary directory called “temp” on the SFTP server. The file has a random name (a UUID). When the file is completely written, Arq then moves the file to its final resting place. The move is an atomic operation, so there’s no risk of a partially-written file in the final resting place.