Arq now backs up to Amazon’s new Glacier service, and I’m really excited about it! Glacier storage is super-cheap — just $.01/GB per month!

With Glacier you can store 100GB for just $1/month! Or store a terabyte for just $10/month!

I got hundreds of emails and tweets asking for Glacier support. Turns out it’s a good option for some scenarios (even with the slow restore time and possible extra Amazon charges). People want to use it for big stuff like iPhoto libraries, videos, etc that get too expensive in S3. They use it as a secondary backup, so they don’t expect to actually restore unless their whole house burns down, taking their primary backup with it.

Arq’s been getting pretty popular with independent folks as well as corporate employees. One user described it recently as “the backup utility of choice for the geekier segment of the Mac community.” Glacier support makes Arq a good fit for even more people!

Retrieval Costs

Amazon has designed Glacier for “infrequent retrievals”, according to their FAQ. In the event you need to restore a significant amount of data from Glacier, Amazon may charge you an additional retrieval fee on top of the standard data-transfer-out charges. The formula for calculating this retrieval fee is complicated, but Arq helps figure it out for you. When you select an item to restore, Arq shows you the expected retrieval fee given the detected download rate:

You can adjust the download rate to change the retrieval fee:

For more details on Arq and Glacier, see the Arq product page.

Pricing

Arq 3 is still just $29 per computer. Upgrade from Arq 2 for just $15.

How to Upgrade from Arq 2 to Arq 3

To upgrade from Arq 2 to Arq 3, just pick “Check for Updates” from the Arq menu.

When Arq 3 launches, it’ll prompt you to upgrade your license.

IAM and Arq

If you need to install Arq on many computers using the same S3 account but you don’t want Arq to see the other computers’ backup data, use Amazon’s IAM (Identity and Access Management) to restrict what Arq sees.

The easiest way to do this is as follows:

1. Use your main keys to install and configure Arq on a computer.
2. Quit Arq and quit Arq Agent.
3. Create an IAM user and capture its access key ID and secret access key.
4. Look in (home)/Library/Arq/config/app_config.plist for the localS3BucketName and localComputerUUID values.
5. Set up an IAM user with a policy that allows full access only to /<localComputerUUID> in the localS3BucketName, as well as “ListBucket” access (see example IAM policy below).
6. Open the Keychain Access app and change the “Arq S3″ entry’s Account and Password fields to the access key ID and secret access key of that IAM user.
7. Launch Arq.

Example IAM Policy

For computer with the following values:

• localS3BucketName = akiaiyuk3n3tme6l4hfa.comhaystacksoftwarearq
• localComputerUUID = 32D9D7A2-3B3E-4BE7-B85B-0605AF24F570

the IAM policy would look like this:

{
 "Statement": [
   {
     "Sid": "Stmt1344522941209",
     "Action": [
       "s3:ListBucket"
     ],
     "Effect": "Allow",
     "Resource": [
       "arn:aws:s3:::akiaiyuk3n3tme6l4hfacomhaystacksoftwarearq"
     ],
     "Condition": {
       "StringLike": {
         "s3:prefix": "32D9D7A2-3B3E-4BE7-B85B-0605AF24F570/*"
       }
     }
   },
   {
     "Sid": "Stmt1344522997713",
     "Action": [
       "s3:*"
     ],
     "Effect": "Allow",
     "Resource": [
       "arn:aws:s3:::akiaiyuk3n3tme6l4hfacomhaystacksoftwarearq/32D9D7A2-3B3E-4BE7-B85B-0605AF24F570/*"
     ]
   }
 ]
}

The first part gives “s3:ListBucket” permission for the user’s bucket, but only with a prefix starting with 32D9D7A2-3B3E-4BE7-B85B-0605AF24F570/* (her UUID).

The second part gives permission for all actions for resources starting with akiaiyuk3n3tme6l4hfacomhaystacksoftwarearq/32D9D7A2-3B3E-4BE7-B85B-0605AF24F570/*.

Answer Files and IAM

For information on automating Arq configuration using answer files and IAM, please read the Arq manual’s Configuring Arq Using an Answer File section.

Arq 2.8.2: online backup to Amazon S3

SyncPhotos 2.2.2: sync your iPhoto Libraries across computers

Duplifinder 2.1.1: eliminate duplicate photos

If you do happen to run into an issue with any of the above, please email support@haystacksoftware.com and I’ll address the issue very promptly!

I had switched to using Apple’s new Developer ID scheme for code signing so that when Mountain Lion comes out people will still be able to use Arq and SyncPhotos. I even watched the WWDC video to make sure I was doing it correctly. But apparently I didn’t watch closely enough. As the 3 blog posts linked above explain, the code signatures produced with the new scheme can only be validated by Macintosh computers that have the necessary information to validate them, namely computers running OS X 10.7 or later.

Both Arq and SyncPhotos launched OK on 10.6, 10.7 and 10.8. But on 10.6 they asked for keychain read permission over and over again. This was because OS X couldn’t validate the apps’ signatures and therefore didn’t recognize them.

If you’re working on a Macintosh app and you want to support 10.6, you’ll need to do some extra steps when building it. If you’re not, then the rest of this blog post will be really boring!

For a full explanation of the issue involved, please read the linked blog posts above. They were a great start for me, but didn’t give precise enough step-by-step instructions. In the hope of saving someone else from the hours of trial and error I wet through, I’ve listed the steps I followed below. It worked for me every time. The one crucial difference from the referenced blog posts I think is in step 2, booting into 10.6 before exporting the designated requirement.

Leopard-Compatible Code Signing Using Developer ID

1. Build your app using Xcode 4.3. This may be painful if you’ve been building in Xcode 3, but fix up the errors as best you can to get a successful build.
2. Boot OS X 10.6 (I have it on a second hard drive in my MacBook Pro, which is handy), get the app that you built in step 1, and run the following on it:

       codesign -d -r- <path to app>

3. Find the line in the codesign output that starts with “designated” and store that line in a text file named designated_requirement.txt in the root folder of your project.
4. Boot your normal development environment (for me this is 10.7).
5. Discard all those changes you made to your project in step 1 by doing whatever command your revision control system has for that purpose. 
6. Open your app’s project in Xcode 3. Don’t set code signing as an Xcode build setting. Instead, add a custom build step at the end that looks something like this:

       codesign -f -s 'Developer ID Application: (your company)' \
       -r $SOURCE_ROOT/designated_requirement.txt \
       $BUILD_DIR/$CONFIGURATION$EFFECTIVE_PLATFORM_NAME/<name>.app

7. Build it in Xcode 3.
8. Verify the app’s signature using the codesign utility:

       codesign --verify -vvv <path to app>

9. Boot OS X 10.6 again, get the app you built in step 7, and verify it again:

       codesign --verify -vvv <path to app>

   In both cases you should see output like this:

       Arq.app: valid on disk
       Arq.app: satisfies its Designated Requirement

The source code is on github.

Really clever! I love it.

This minor update fixes several minor issues, including the issue where some backup sets weren’t appearing under “Other Backup Sets”.

It’s a free update for all Arq users. Pick “Check for Updates” from the Arq menu to get the update.

As always, full release notes for all Arq versions are on the release notes page.

Please note: Arq 2.6.8 may upload much more than you expect or want it to the first time it backs up. This is due to a bug that I unfortunately created in version 2.6.3. I apologize for this. Please let it finish backing up.

There is no charge from Amazon for data transfer to Amazon, but I realize that re-uploading a lot of data is a long, frustrating experience.

The full story of what happened starting with Arq 2.6.3

There’s a problem with some of S3′s servers, particularly in the us-west-1 region, but also in other regions. Connections just get “stuck”. Customers were complaining heavily about Arq getting “stuck”. I was initially unable to reproduce the problem, but felt I had to do something to fix it. The Apple networking API that Arq was using had no facility for timing out; plus it was very hard to throttle properly. So I changed Arq to use Apple’s older CFHTTP API. 

In my haste to fix this customer problem, I didn’t do enough testing on 2.6.3-4. I screwed up. There was a bug in those releases where some objects were marked as “compressed” but they weren’t actually compressed. So Arq couldn’t restore properly.

What really made things bad was that at some point in my rush to get the issue fixed I accidentally (because of muscle-memory habit) ran the command to publish 2.6.3 in the “update stream” as an official release of Arq, which meant that everyone who happened to check for updates would get this buggy version. I didn’t realize this for several hours. At that point I could only move forward and fix things as quickly as possible.

Since there’s no way for me to tell which objects were compressed, Arq 2.6.5 (and later) ignores all backups made after March 7 by an Arq version older than 2.6.5. I added a “version” to the backup data so that if this sort of thing ever happens again, I can fix it with a less drastic solution.

I’ve also added some “gates” to my release process so that I don’t inadvertently do this again.

Arq 2.6.6 through 2.6.8 contain very small changes — checks for various network errors that were causing Arq to fail the backup instead of retry.

Please let Arq finish doing what it’s doing. It will clean up all the “bogus” S3 data after it successfully backs up everything.

Also, please consider using more than one backup system, as I blogged about in 7 facets of a good Mac backup strategy.

A Memory For Your Files

Before personal computers, we did our work on paper. The paper approach has a built-in “memory.” If you rewrite a paper document, the old document sticks around until you take action and throw it away. By default you have versioned backup. Even after you throw it away you could still get it back if you’re willing to dumpster-dive and you do it before the garbage truck arrives.

When we switched to personal computers for documents, we lost that “memory.” When you save a new version of your document on your computer, the old version is immediately obliterated — gone forever! It’s unfortunate because having the old versions can be a huge time saver. When you’ve made a mess of things and you want to start over, when you realize you deleted a file that you still need, or when a file becomes unreadable for whatever reason, going “back in time” to get an old version saves the day.

Incomplete Solutions

The “trash can” in Windows and OS X isn’t a complete solution, because it only holds onto files you’ve deleted. If you open a document, make some changes, and then click “Save”, the previous version of the document doesn’t go into the trash can.

Apple has tried to provide a form of versioned backup through OS X Lion’s Auto Save and Versions feature. But it only works in apps that have added support for that feature, so it’s not a complete solution.

Recommendation

So, make sure you include a product in your backup strategy that keeps versions. Time Machine (Apple’s built-in backup app) does it for local backups. Arq does it for online backups. Dropbox does it to some extent (it keeps a few revisions of each file).

Cloning tools like SuperDuper don’t do it (but they have other advantages like quick recovery time).

This update includes a new feature that posts a Growl notification when a backup completes. It’s useful for folks who hide the menubar icon; now they have a way to know when the backup completes without launching the Arq app. It’s optional — you can turn it off in the preferences.

There are also several bug fixes.

It’s a free update for all Arq users. Pick “Check for Updates” from the Arq menu to get the update.

As always, full release notes for all Arq versions are on the release notes page.

1. Assume your hard drive will fail very soon

Expect imminent disk failure no matter how old or new your hard drive is. The other day a customer sent me email saying Arq was reporting input/output errors. I told him it was probably a hardware problem and he should replace his hard drive ASAP. He said it’s an SSD that he installed 2 days ago, so that can’t be it. A few days later he wrote back saying the SSD was the culprit.

SSDs in my opinion are worse than spinning drives because they seem to fail catastrophically more often. Spinning drives often fail more gradually, giving you a chance to copy your data off, which is especially good if you haven’t been doing backups — but you are doing backups, right?

2. Automate it

Any backup approach that requires you to remember something has one big problem: you’ll forget. If you have to plug in an external hard drive for your backup approach, you won’t do it. At least not often enough.

3. Keep it simple

Choose simple backup processes to minimize the opportunity for error. Apple’s Time Machine is a great example of a simple app. Arq asks almost no questions — the defaults are fine. SuperDuper is just as simple — you just click one button and it makes a clone of your hard drive. All of these apps have lots more options, but you can safely ignore them.

4. Use multiple backup systems

This goes against the “keep it simple” advice, but counting on just one backup strategy is risky. When it comes time to recover from failure, you want as many opportunities to get your stuff back as possible. You don’t want to wake up one morning to a disk failure and then find out that you’d accidentally deleted your one backup app 6 months ago and you’ve lost 6 months of work. Or find out that your one online backup provider lost your data, or disappeared altogether.

Speaking of online backup: make sure one of the backup systems you use is off-site, to protect against theft, fire, lightning strike, flood, etc. For example, rotate your clone backup drives keeping one at the office (if your office is in a different location than your home!) or use an online backup service. I use 2 systems — one local and one off-site (explained below).

5. Minimize recovery time if possible

If you need to recover your entire computer from a Time Machine backup, you’re supposed to use Apple’s Migration Assistant app. Migration Assistant can be very slow however, especially when restoring from a Time Capsule over the network. If you have a clone of your hard drive made with an app like SuperDuper, you’ll be back in business in a minute — just plug the clone drive in, hold down the Option key, and boot your computer from the clone.

One potential downside of recovering with a clone is that in your haste to get back to work you may forget all about the fact that you’ve got no clone anymore! This can easily happen if you use a desktop computer — you won’t even notice that you’re running off the external hard drive.

At your earliest convenience you need to get another hard drive and clone to it, in case your clone fails. Having multiple backup systems helps mitigate this problem too.

6. Protect against corruption and “user error”

One of your backup systems should be a “versioning” system. Time Machine and Arq are 2 examples of this. They keep hourly backups of your files for the past 24 hours, daily backups for the past month, and weekly backups until they reach your storage budget (Arq) or the target disk is full (Time Machine).

Clones of your hard drive are great, but they’re only the latest version of your stuff. If a file becomes corrupt, the next time you clone your hard drive you’ll replace your old clone’s copy of the file with the new corrupt one.

One of your backup systems should keep multiple copies of your files over time to guard against corruption as well as the occasional what-was-I-thinking-when-I-deleted-half-that-document moments.

7. Avoid services whose interests aren’t aligned with yours

If you’re choosing an online backup provider, pay close attention to the data retention policies, especially with the “unlimited” offerings. Backblaze, for instance, will delete backups of your external drive if it hasn’t been connected within the past 30 days.

Also consider who has access to your stuff. With Backblaze you can pick your own encryption password, but if you need to restore your stuff you’ll have to give them your password; they decrypt your stuff and leave it in an unencrypted zip file on their servers; if you have them send you a disk with your stuff, your files will be sent through the mail unencrypted on that disk.

Also, any service that offers web access to your backups obviously has the ability to read your stuff (so that they can serve it to you through a web browser).

My Approach

I do all my work on 1 laptop (a MacBook Pro). I clone my laptop’s 2 internal hard drives (an SSD plus a spinning drive) using SuperDuper whenever I think of it. Arq backs up hourly all day long, from wherever I am, as long as there’s an internet connection. My computer doesn’t really go anywhere for very long that doesn’t have an internet connection, so this works for me.

If my SSD boot drive fails, I can’t boot from my Arq backups in S3, but I can get up and running quickly from the clone (which will probably be out-of-date) and then replace my key files with the latest versions from my Arq backups.

I feel good about my data at S3 not going anywhere. It’s in my own S3 account, and Amazon promises 99.999999999% (that’s 11 9s) of durability over 12 months.

In the worst case, if both my computer and my clone are damaged/lost/stolen I can download all my stuff from S3 using Arq, but it’ll take a while.

(SuperDuper and Arq are Mac-only. If you’re on Windows, you could try Acronis True Image for cloning and CloudBerry Backup for backup to Amazon S3.)

I should probably add a third option. Any suggestions? Send me email or post a comment!