Archive for the ‘s3’ Category

Arq 4 Beta

Tuesday, February 4th, 2014

The Arq 4 beta is here!


UPDATE March 3: Arq 4 is officially released.


I’m really excited about this release! It’s got features that many people have been asking for, and it opens Arq up to a whole new range of options for storing backup data.

PLEASE NOTE: Arq 4 should officially ship sometime before the end of February. Every Arq 3 license purchased on or after December 1, 2013 is eligible for free upgrade to Arq 4.

ALSO: Arq 4 is for OS X 10.7 and later.

New Storage Options

For the first time, Arq can back up to not just Amazon Web Services. You can choose GreenQloud, DreamObjects, Google Cloud Storage, or any other S3-compatible target:

 Target types

You can even choose to back up to an SFTP server! If you have a NAS in your home or office that allows SSH/SFTP access, you can back up to that and pay $0 in monthly storage charges. Or back up to a VPS (virtual private server) like Dreamhost for cheap offsite backup.

More Control

 Several new features are aimed at providing you with more control over your backups:

  • Backup only on selected wireless networks — prevent upload when tethered to your phone, for instance
  • Email notifications — great for monitoring headless/remote Macs and customer Macs 
    Email prefs
  • A unified budget across S3 and S3/Glacier backups (see “A New Approach to Glacier” below)
  • Optionally specify a “window” of time during the day/night when Arq pauses — useful for networks that are underutilized at night, and for ISPs that charge less at certain times of day
    Backup window

Other Features

Arq 4 includes several other features and improvements, including a “Stop Backup” function, the display of the last backup date in the agent’s menu, and less prompting for administrator privileges when restoring. Also, the process for setting up and restoring to a new computer is more straightforward.

A New Approach to Glacier

When Amazon announced their Glacier offering in the fall of 2012, we built Arq 3 to take advantage of it. Some time after Arq 3 was released, Amazon announced an S3 Glacier Lifecycle feature through which Amazon would automatically store certain S3 objects in Glacier. Arq 4 uses this new S3 Glacier Lifecycle feature for Glacier backups (existing Glacier backups made with Arq 3 will continue to use the old Glacier API as Amazon offers no way to move Glacier objects to S3). There are several benefits to using the S3 Glacier Lifecycle feature:

  • S3 objects with Glacier storage class have all the benefits of regular S3 objects — a known location/name, S3 object query abilities
  • No more creating Glacier vaults, which are hard to use and even harder to delete.
  • Restored objects are at known/expected locations, unlike restored Glacier objects which receive a random name.
  • Restored objects can be persisted for much longer, unlike restored Glacier objects which have a fixed 24-hour expiration.
  • Restarting a restore with Arq means only requesting restore of objects which haven’t been requested yet because previously-requested objects are at known locations.
  • Restoring is less complex and much faster — no more creating SQS queues and SNS topics; no more taking the time to read all messages from the SQS queue before beginning to download.
  • Budgeting is possible using the same logic as the S3 budget feature.

Changing to the New S3/Glacier Format

Unfortunately Amazon doesn’t provide a mechanism for creating “pointers” in S3 to existing Glacier objects, so if you want budgeting and easier restoring with Glacier, you’ll have to re-upload your files. I know this is far from ideal, but in the long term I believe it pays dividends in cost and ease of use. 

Arq no longer restricts you from adding the same folder twice, or adding a folder contained by a folder that Arq is already backing up. So, if you’ve been backing up a folder to Glacier with Arq 3, just add the folder to Arq again, and choose “Glacier storage class”:

Adding folder

Arq will back up the folder to S3 in a subdirectory of your S3 bucket called “glacier” which Amazon will automatically archive to Glacier storage class (Arq creates the Glacier lifecycle policy automatically). When it’s done uploading, delete the old folder (the old-style Glacier backups created with Arq 3).

Beta Testing

The Arq 4 beta is available here:

Your Arq 3 license will work with it just fine.

The beta will expire March 17, but Arq 4 should be shipping by then. If not, a new beta version will be available.

Please submit your feedback, questions, and bug reports via Twitter @arqbackup or via email to

I look forward to getting your feedback!

- Stefan

Arq 3.2 is out!

Wednesday, June 12th, 2013

Arq 3.2 is a free update for all Arq 3 users.

The big new feature in 3.2 is restoring onto an existing folder. While it looks like no big deal on the surface (it just asks whether you’d like to overwrite the existing folder or not), under the covers it was a big change in the restore process. Arq now asks you for permission to launch a helper program as “root”, the super user. The helper program needs to be “root” so that it can write files into folders that your regular user account may not have permission for. Arq compares the existing file contents to the backup record and only downloads files that are different or missing. Then it applies all the metadata correctly as before.

Full list of new features:

  • Restore into an existing folder of files, only downloading the files that are different or missing. Restore runs using administrator privileges to avoid permission issues.
  • Added a “pause on battery power” feature — set it in Arq’s preferences.
  • Added a “setthrottle” command-line option to change the transfer rate setting. This is useful for those who wish to change Arq’s throttle setting via a script.

Fixed Bugs:

  • Fixed an issue during very large Glacier restores where Arq would request too many items, which would then expire from AWS before Arq got a chance to download them.
  • Improved memory usage and performance during Glacier restore and S3 restore.
  • Fixed 10.6-related crashes.
  • Fixed broken throttling issue.
  • Removed AddressBook features (pre-populating name and email in crash report forms and in-app purchase) so that users aren’t asked to give Arq permission to access contacts.
  • Added explanation of command-line options to Help documents.

To get Arq 3.2, pick “Check for Updates” from Arq’s menu.

Using Arq with IAM

Thursday, August 23rd, 2012

This post is for system administrators who support Arq on multiple computers. If that’s you, please read on!

IAM and Arq

If you need to install Arq on many computers using the same S3 account but you don’t want Arq to see the other computers’ backup data, use Amazon’s IAM (Identity and Access Management) to restrict what Arq sees.

The easiest way to do this is as follows:

  1. Use your main keys to install and configure Arq on a computer.
  2. Quit Arq and quit Arq Agent.
  3. Create an IAM user and capture its access key ID and secret access key.
  4. Look in (home)/Library/Arq/config/app_config.plist for the localS3BucketName and localComputerUUID values.
  5. Set up an IAM user with a policy that allows full access only to /<localComputerUUID> in the localS3BucketName, as well as “ListBucket” access (see example IAM policy below).
  6. Open the Keychain Access app and change the “Arq S3″ entry’s Account and Password fields to the access key ID and secret access key of that IAM user.
  7. Launch Arq.

Example IAM Policy

For computer with the following values:

  • localS3BucketName = akiaiyuk3n3tme6l4hfa.comhaystacksoftwarearq
  • localComputerUUID = 32D9D7A2-3B3E-4BE7-B85B-0605AF24F570

the IAM policy would look like this:

 "Statement": [
     "Sid": "Stmt1344522941209",
     "Action": [
     "Effect": "Allow",
     "Resource": [
     "Condition": {
       "StringLike": {
         "s3:prefix": "32D9D7A2-3B3E-4BE7-B85B-0605AF24F570/*"
     "Sid": "Stmt1344522997713",
     "Action": [
     "Effect": "Allow",
     "Resource": [

The first part gives “s3:ListBucket” permission for the user’s bucket, but only with a prefix starting with 32D9D7A2-3B3E-4BE7-B85B-0605AF24F570/* (her UUID).

The second part gives permission for all actions for resources starting with akiaiyuk3n3tme6l4hfacomhaystacksoftwarearq/32D9D7A2-3B3E-4BE7-B85B-0605AF24F570/*.

Answer Files and IAM

For information on automating Arq configuration using answer files and IAM, please read the Arq manual’s Configuring Arq Using an Answer File section.

Arq plugin for Sidekick

Saturday, April 28th, 2012

Arq Forum member jmah did some reverse-engineering of Arq and posted a message about a plugin he wrote for Sidekick which tells Arq to back up whenever he returns home.

The source code is on github.

Really clever! I love it.

Arq 2.6.9 is out

Saturday, April 28th, 2012

Arq version 2.6.9 is now available!

This minor update fixes several minor issues, including the issue where some backup sets weren’t appearing under “Other Backup Sets”.

It’s a free update for all Arq users. Pick “Check for Updates” from the Arq menu to get the update.

As always, full release notes for all Arq versions are on the release notes page.

Arq 2.3 is out!

Wednesday, December 7th, 2011

Arq version 2.3 is now available!

This update includes support for the new “us-west-2″ (Oregon) S3 region.

It’s a free update for all Arq users.

As always, full release notes for all Arq versions are on the release notes page.

How I recovered after an OS X reinstall

Sunday, September 19th, 2010

The other day I reinstalled OS X. My computer had become extremely sluggish and I wanted to see if the performance would improve if I reformatted my hard disk and started over. Along the way I learned a few lessons about restoring using Arq. Here’s what I did:

Before Wiping Out My Data

Before I went through with it, I made sure I had all my data backed up. Arq had backed up the following:

  • ~/Library (excluding Logs and Caches)
  • ~/Documents
  • ~/Music
  • ~/Pictures/iPhoto Library (my photos)
  • ~/src (my source code)
  • /Applications
  • /Library/Application Support


I inserted the Snow Leopard installation disk, shut down the computer, and then started it holding down the Option key. I clicked on the DVD and the computer booted from it. I formatted the disk and installed OS X. I created a user with the same name as I was using before.

Next I downloaded and installed Arq. I launched Arq and entered the same S3 keys and encryption password I was using before.

Finally it was time to restore using Arq.

Initial Restore

Instead of waiting for absolutely everything to be restored from S3, I restored files in several steps.

Restoring ~/Library

The first step was to restore ~/Library from my “other computer” (the previous incarnation of my computer). I opened the triangle next to “Other Computers”, found my old computer, opened the triangle next to “Library” and selected the latest backup:


Then I clicked “Restore…” and Arq restored the Library folder to ~/Restored by Arq/Library (because a Library folder already existed).

When that restore was done, I closed all open applications, deleted the contents of ~/Library, and dragged everything from ~/Restored by Arq/Library to ~/Library.

Back in Business

At that point I could use Mail, iCal and Address Book. I selected a few applications in Applications backup folder and restored them as well.

I also wanted to sync my calendars with my iPhone, so I plugged it in and it sync’d. Later I’ll delete the iTunes files in ~/Music and replace them with the backed-up files.

Restoring Everything Else

Now that the computer felt “back to normal”, I restored my “src” folder (where all my work files are). Then I got back to work, restoring the really large folders (Documents, Music and Pictures) at my leisure over the next few days.


The multi-step restore approach was a big time-saver and got me up and running fairly quickly. The Library folder was relatively small (really small in fact, with the exception of Mail).

I learned that reformatting the hard drive helped a little with sluggishness, but the long-term fix is likely the purchase of an Optibay and an SSD.

I also learned that restoring this way is fairly complicated. So I’m thinking about how to make a product that would restore more seamlessly while also allowing people to get back to work before absolutely everything is restored. There’ll be more to come on that.

Deleting other computers’ backups

Saturday, September 18th, 2010

If you’ve transferred your work to a new computer and don’t need the old computer’s backups in your S3 account anymore, you’ll need to delete them. Arq does not currently provide a mechanism for deleting those backups, but you can delete them through the AWS Management Console. Here’s how to do that:

First, open the AWS Management Console (

Next, select the bucket that Arq uses for its backups (named “.com.haystacksoftware.arq”).

Now you’ll have to determine the computer UUID that you want to delete. To do this, look at the computerinfo file within each one:

  1. double-click on a computer UUID
  2. control-click on the file computerinfo and pick “Download”
  3. open the downloaded file with TextEdit
  4. if the “computer name” matches the one you want to delete, you’ve found the right computer UUID.

Here’s an example “computerinfo” file:

<plist version="1.0">
        <string>Stefan Reitshamer’s MacBook Pro</string>

In that example, the computer name is “Stefan Reitshamer’s MacBook Pro”.

Now that you’ve found the right computer UUID to delete, go back and select the bucket itself to see all the computer UUIDs again. Then control-click on the computer UUID you want to delete, and pick “Delete” from the pop-up menu. AWS Management Console will delete all the objects for that computer UUID.

WARNING: This delete cannot be undone! Please be careful when deleting.

How to back up and restore your important Mac OS X files

Saturday, September 18th, 2010

Apple has excellent instructions for backing up and restoring

  • Safari bookmarks
  • Address Book contacts
  • iCal calendars
  • Keychains
  • Mail preferences and messages

Here’s the link:

To back up your Safari bookmarks etc, make sure Arq is backing up these items:

  • ~/Library/Preferences/
  • ~/Library/Preferences/
  • ~/Library/Preferences/
  • ~/Library/Safari/

To back up your Address Book, make sure Arq is backing up these items:

  • ~/Library/Application Support/AddressBook/
  • ~/Library/Preferences/

To back up your iCal calendars, make sure Arq is backing up these items:

  • ~/Library/Calendars/
  • ~/Library/Preferences/
  • ~/Library/Preferences/

To back up your Keychains, make sure Arq is backing up this folder:

  • ~/Library/Keychains/

To back up your Mail, make sure Arq is backing up these items:

  • ~/Library/Mail
  • ~/Library/Preferences/

To restore, follow the instructions in the Apple support article linked above.

How to back up your Mac using Arq

Wednesday, July 21st, 2010

When I started developing Arq it was partly because I couldn’t find an existing online backup offering that gave me enough control. I wanted to control exactly which files would be backed up, and I didn’t want to be constrained by rules that many of the “unlimited backup” offerings had like excluding network drives, excluding applications, etc.

So Arq lets you back up anything you want. But then the question is, what should you back up? The following is my suggestion for a basic backup of your files on your Mac.

Basic Backup Using Arq

When you first install and launch Arq, it asks your for your Amazon S3 “keys” and a few other things. Then it asks if you’d like to choose your own files for backup, or back up your home folder minus a few unnecessary items:

Screen shot 2010-07-21 at 8.02.18 AM.png

If you picked “I’ll manually add folders to back up” and you’ve changed your mind, here’s how to set up Arq to back up your home folder minus the unnecessary items:

1. Add your home folder

Click the + button at the bottom left of the Arq main window.

Screen shot 2010-07-21 at 8.10.25 AM.png

Pick your home folder (/Users/<yourname>) and click OK.

Screen shot 2010-07-21 at 9.27.33 AM.png

2. Add some excludes

Click the “Edit Excludes…” button.

Screen shot 2010-07-21 at 8.08.05 AM.png

Add 3 excludes.

Screen shot 2010-07-21 at 8.15.33 AM.png

Make sure the first 2 are set to “relative path” instead of “name”.

Click OK.

Backing Up Applications Using Arq

If you want to back up your applications, add the Applications folder.

Screen shot 2010-07-21 at 8.28.12 AM.png

Many applications put some of their support files in /Library/Application Support, so add that too.

Screen shot 2010-07-21 at 8.29.02 AM.png

Advanced Backup Using Arq

If you prefer, you pick and choose specific folders to back up instead of backing up your entire home directory.

WARNING: If you choose to do this and you later create a new folder in your home directory and start putting important files in there, you’ll have to remember to add this new folder to Arq or else it won’t be backed up!

I back up the following folders as separate items in Arq:

  • Application Support (/Library/Application Support)
  • Applications (/Applications)
  • Documents
  • Library, excluding files/folders named ‘Caches’ and ‘Logs’
  • Music
  • osaka iPhoto Library (my big iPhoto Library, named after my computer), excluding files/folders named ‘iPod Photo Cache’
  • src (my work files), excluding files/folders named ‘build’ and ‘bin’

Time Machine and Arq

Time Machine and Arq are complementary. Backing up using Time Machine to another disk is cheap and fast. If you’re backing up to a Time Capsule via Wifi it’s very convenient because it just happens; there’s nothing to plug in. If you’re backing up to a USB drive, you’ll have to remember to plug in the USB drive periodically. Restoring is fast because you’re reading from a USB disk physically connected to your Mac, or from a Time Capsule over Wifi.

But Time Machine doesn’t cover all cases. If someone breaks in and steals your computer, they may steal your Time Capsule or USB drive as well, and then your files are gone forever. If fire, flood, or lightning strikes, you may lose both your computer and your backups; files gone forever. And if you travel often, you’ll have to bring along your USB drive or Time Capsule, or backups won’t happen until you get home and stay home long enough for a backup to complete.

Arq covers those cases that Time Machine doesn’t. The backups are off site at Amazon’s servers, safe from your theif and your natural disasters. They’re even safe from disaster at an Amazon site because Amazon replicates your data at several sites. And Arq works whenever there’s an Internet connection, so backups still happen when you’re on the road.