Archive for the ‘arq’ Category

« Older Entries

7 facets of a good Mac backup strategy

Monday, January 23rd, 2012

I’ve been studying the computer backup industry for 3 years now and I’ve been selling my own online backup product, Arq, since February 2010. I’ve seen and heard lots of different approaches to backing up one’s computer. Here are some backup lessons I’ve learned.

1. Assume your hard drive will fail very soon

Expect imminent disk failure no matter how old or new your hard drive is. The other day a customer sent me email saying Arq was reporting input/output errors. I told him it was probably a hardware problem and he should replace his hard drive ASAP. He said it’s an SSD that he installed 2 days ago, so that can’t be it. A few days later he wrote back saying the SSD was the culprit.

SSDs in my opinion are worse than spinning drives because they seem to fail catastrophically more often. Spinning drives often fail more gradually, giving you a chance to copy your data off, which is especially good if you haven’t been doing backups — but you are doing backups, right?

2. Automate it

Any backup approach that requires you to remember something has one big problem: you’ll forget. If you have to plug in an external hard drive for your backup approach, you won’t do it. At least not often enough.

3. Keep it simple

Choose simple backup processes to minimize the opportunity for error. Apple’s Time Machine is a great example of a simple app. Arq asks almost no questions — the defaults are fine. SuperDuper is just as simple — you just click one button and it makes a clone of your hard drive. All of these apps have lots more options, but you can safely ignore them.

4. Use multiple backup systems

This goes against the “keep it simple” advice, but counting on just one backup strategy is risky. When it comes time to recover from failure, you want as many opportunities to get your stuff back as possible. You don’t want to wake up one morning to a disk failure and then find out that you’d accidentally deleted your one backup app 6 months ago and you’ve lost 6 months of work. Or find out that your one online backup provider lost your data, or disappeared altogether.

Speaking of online backup: make sure one of the backup systems you use is off-site, to protect against theft, fire, lightning strike, flood, etc. For example, rotate your clone backup drives keeping one at the office (if your office is in a different location than your home!) or use an online backup service. I use 2 systems — one local and one off-site (explained below).

5. Minimize recovery time if possible

If you need to recover your entire computer from a Time Machine backup, you’re supposed to use Apple’s Migration Assistant app. Migration Assistant can be very slow however, especially when restoring from a Time Capsule over the network. If you have a clone of your hard drive made with an app like SuperDuper, you’ll be back in business in a minute — just plug the clone drive in, hold down the Option key, and boot your computer from the clone.

One potential downside of recovering with a clone is that in your haste to get back to work you may forget all about the fact that you’ve got no clone anymore! This can easily happen if you use a desktop computer — you won’t even notice that you’re running off the external hard drive.

At your earliest convenience you need to get another hard drive and clone to it, in case your clone fails. Having multiple backup systems helps mitigate this problem too.

6. Protect against corruption and “user error”

One of your backup systems should be a “versioning” system. Time Machine and Arq are 2 examples of this. They keep hourly backups of your files for the past 24 hours, daily backups for the past month, and weekly backups until they reach your storage budget (Arq) or the target disk is full (Time Machine).

Clones of your hard drive are great, but they’re only the latest version of your stuff. If a file becomes corrupt, the next time you clone your hard drive you’ll replace your old clone’s copy of the file with the new corrupt one.

One of your backup systems should keep multiple copies of your files over time to guard against corruption as well as the occasional what-was-I-thinking-when-I-deleted-half-that-document moments.

7. Avoid services whose interests aren’t aligned with yours

If you’re choosing an online backup provider, pay close attention to the data retention policies, especially with the “unlimited” offerings. Backblaze, for instance, will delete backups of your external drive if it hasn’t been connected within the past 30 days.

Also consider who has access to your stuff. With Backblaze you can pick your own encryption password, but if you need to restore your stuff you’ll have to give them your password; they decrypt your stuff and leave it in an unencrypted zip file on their servers; if you have them send you a disk with your stuff, your files will be sent through the mail unencrypted on that disk.

Also, any service that offers web access to your backups obviously has the ability to read your stuff (so that they can serve it to you through a web browser).

My Approach

I do all my work on 1 laptop (a MacBook Pro). I clone my laptop’s 2 internal hard drives (an SSD plus a spinning drive) using SuperDuper whenever I think of it. Arq backs up hourly all day long, from wherever I am, as long as there’s an internet connection. My computer doesn’t really go anywhere for very long that doesn’t have an internet connection, so this works for me.

If my SSD boot drive fails, I can’t boot from my Arq backups in S3, but I can get up and running quickly from the clone (which will probably be out-of-date) and then replace my key files with the latest versions from my Arq backups.

I feel good about my data at S3 not going anywhere. It’s in my own S3 account, and Amazon promises 99.999999999% (that’s 11 9s) of durability over 12 months.

In the worst case, if both my computer and my clone are damaged/lost/stolen I can download all my stuff from S3 using Arq, but it’ll take a while.

(SuperDuper and Arq are Mac-only. If you’re on Windows, you could try Acronis True Image for cloning and CloudBerry Backup for backup to Amazon S3.)

I should probably add a third option. Any suggestions? Send me email or post a comment!

Posted in arq, backup | 7 Comments »

Arq 2.4 is out!

Thursday, December 29th, 2011

Arq version 2.4 is now available!

This update includes support for the new “sa-east-1″ (São Paulo, Brazil) S3 region.

It also now checks whether Amazon S3 is experiencing long “eventual consistency” delays, abort backup and budget enforcement activities until the next backup interval to avoid potential data loss due to incorrect (old) values being returned from S3.

It’s a free update for all Arq users. Pick “Check for Updates” from the Arq menu to get the update.

As always, full release notes for all Arq versions are on the release notes page.

Posted in arq, backup, releases | No Comments »

Arq 2.3 is out!

Wednesday, December 7th, 2011

Arq version 2.3 is now available!

This update includes support for the new “us-west-2″ (Oregon) S3 region.

It’s a free update for all Arq users.

As always, full release notes for all Arq versions are on the release notes page.

Posted in arq, backup, releases, s3 | No Comments »

Arq 1.6 is out!

Friday, November 26th, 2010

This update includes performance improvements and bug fixes.

To get it, pick “Check for Updates” from the Arq menu, or download it from the product page.

Here are the details:

Release Notes for Arq Backup Version 1.6

Feature Additions

Several performance improvements, including:

  • Much faster collection of referenced objects for a backup version (used in verify-backups phase).
  • Much faster caching of current S3 object list (used when backing up).

Bug Fixes

Several fixes to reduce the size of the cache in Library/Arq/Caches.noindex:

  • Periodically consolidate “packs”, removing unused objects to save space.
  • Delete invalid/obsolete cache files from disk.
  • Changed default verify-backups interval from 1 day to 4 days.

And some other fixes:

  • Fixed an issue where a network error wasn’t passed on to the user, resulting in an incompletely-restored file.
  • Fixed issues with the pause feature.
  • Updated the UI to reflect new S3 pricing.
  • Fixed an issue with positioning the help bubble on the first setup window.

Posted in arq, releases | No Comments »

How I recovered after an OS X reinstall

Sunday, September 19th, 2010

The other day I reinstalled OS X. My computer had become extremely sluggish and I wanted to see if the performance would improve if I reformatted my hard disk and started over. Along the way I learned a few lessons about restoring using Arq. Here’s what I did:

Before Wiping Out My Data

Before I went through with it, I made sure I had all my data backed up. Arq had backed up the following:

  • ~/Library (excluding Logs and Caches)
  • ~/Documents
  • ~/Music
  • ~/Pictures/iPhoto Library (my photos)
  • ~/src (my source code)
  • /Applications
  • /Library/Application Support

Reinstalling

I inserted the Snow Leopard installation disk, shut down the computer, and then started it holding down the Option key. I clicked on the DVD and the computer booted from it. I formatted the disk and installed OS X. I created a user with the same name as I was using before.

Next I downloaded and installed Arq. I launched Arq and entered the same S3 keys and encryption password I was using before.

Finally it was time to restore using Arq.

Initial Restore

Instead of waiting for absolutely everything to be restored from S3, I restored files in several steps.

Restoring ~/Library

The first step was to restore ~/Library from my “other computer” (the previous incarnation of my computer). I opened the triangle next to “Other Computers”, found my old computer, opened the triangle next to “Library” and selected the latest backup:

recover2.png

Then I clicked “Restore…” and Arq restored the Library folder to ~/Restored by Arq/Library (because a Library folder already existed).

When that restore was done, I closed all open applications, deleted the contents of ~/Library, and dragged everything from ~/Restored by Arq/Library to ~/Library.

Back in Business

At that point I could use Mail, iCal and Address Book. I selected a few applications in Applications backup folder and restored them as well.

I also wanted to sync my calendars with my iPhone, so I plugged it in and it sync’d. Later I’ll delete the iTunes files in ~/Music and replace them with the backed-up files.

Restoring Everything Else

Now that the computer felt “back to normal”, I restored my “src” folder (where all my work files are). Then I got back to work, restoring the really large folders (Documents, Music and Pictures) at my leisure over the next few days.

Conclusions

The multi-step restore approach was a big time-saver and got me up and running fairly quickly. The Library folder was relatively small (really small in fact, with the exception of Mail).

I learned that reformatting the hard drive helped a little with sluggishness, but the long-term fix is likely the purchase of an Optibay and an SSD.

I also learned that restoring this way is fairly complicated. So I’m thinking about how to make a product that would restore more seamlessly while also allowing people to get back to work before absolutely everything is restored. There’ll be more to come on that.

Posted in arq, backup, s3 | 2 Comments »

Deleting other computers’ backups

Saturday, September 18th, 2010

If you’ve transferred your work to a new computer and don’t need the old computer’s backups in your S3 account anymore, you’ll need to delete them. Arq does not currently provide a mechanism for deleting those backups, but you can delete them through the AWS Management Console. Here’s how to do that:

First, open the AWS Management Console (https://console.aws.amazon.com/s3/home).

Next, select the bucket that Arq uses for its backups (named “.com.haystacksoftware.arq”).

Now you’ll have to determine the computer UUID that you want to delete. To do this, look at the computerinfo file within each one:

  1. double-click on a computer UUID
  2. control-click on the file computerinfo and pick “Download”
  3. open the downloaded file with TextEdit
  4. if the “computer name” matches the one you want to delete, you’ve found the right computer UUID.

Here’s an example “computerinfo” file:

<plist version="1.0">
    <dict>
        <key>userName</key>
        <string>stefan</string>
        <key>computerName</key>
        <string>Stefan Reitshamer’s MacBook Pro</string>
    </dict>
</plist>

In that example, the computer name is “Stefan Reitshamer’s MacBook Pro”.

Now that you’ve found the right computer UUID to delete, go back and select the bucket itself to see all the computer UUIDs again. Then control-click on the computer UUID you want to delete, and pick “Delete” from the pop-up menu. AWS Management Console will delete all the objects for that computer UUID.

WARNING: This delete cannot be undone! Please be careful when deleting.

Posted in arq, backup, s3 | 14 Comments »

How to back up and restore your important Mac OS X files

Saturday, September 18th, 2010

Apple has excellent instructions for backing up and restoring

  • Safari bookmarks
  • Address Book contacts
  • iCal calendars
  • Keychains
  • Mail preferences and messages

Here’s the link: http://support.apple.com/kb/ht2980

To back up your Safari bookmarks etc, make sure Arq is backing up these items:

  • ~/Library/Preferences/com.apple.Safari.plist
  • ~/Library/Preferences/com.apple.Safari.LSSharedFileListplist
  • ~/Library/Preferences/com.apple.Safari.RSS.plist
  • ~/Library/Safari/

To back up your Address Book, make sure Arq is backing up these items:

  • ~/Library/Application Support/AddressBook/
  • ~/Library/Preferences/com.apple.AddressBook.plist

To back up your iCal calendars, make sure Arq is backing up these items:

  • ~/Library/Calendars/
  • ~/Library/Preferences/com.apple.iCal.plist
  • ~/Library/Preferences/com.apple.iCal.helper.plist

To back up your Keychains, make sure Arq is backing up this folder:

  • ~/Library/Keychains/

To back up your Mail, make sure Arq is backing up these items:

  • ~/Library/Mail
  • ~/Library/Preferences/com.apple.mail.plist

To restore, follow the instructions in the Apple support article linked above.

Posted in arq, backup, s3 | No Comments »

Arq 1.5.12 is out!

Monday, August 23rd, 2010

This is a minor update which adds 1 new significant feature — the ability to select a single file to back up.

To get it, pick “Check for Updates” from the Arq menu, or download it from the product page.

Here are the details:

Release Notes for Arq Backup Version 1.5.12

Feature Additions

  • Select an individual file to add to backups.

Bug Fixes

  • If Arq is in the middle of relaunching after installing an update, don’t show an error about Arq Agent not running.
  • Arq Agent checks for app updates and displays a message when an update is available.
  • Don’t show filename as “download name” in S3 objects.
  • Added help to the initial setup screen that explains what S3 keys are.

Posted in arq, backup, releases | 5 Comments »

How to restore to a new computer using Arq

Tuesday, August 10th, 2010

So you’ve had Arq installed for a while and it’s been backing up your key folders like Documents, Music, and Photos.

But your computer’s hard drive died. So you took it to the Apple store, and they replaced the hard drive. Now how do you get your files back? Here’s how:

Restoring with Arq

First, download Arq from http://www.haystacksoftware.com/arq/ and unzip it.

Next, launch Arq. It’ll ask you for S3 account credentials. Use the same S3 account you used before. If you don’t have your S3 credentials handy you can look them up at the Amazon web site.

Next, Arq will ask you for your encryption key. Use the same key; otherwise Arq won’t be able to decrypt your backups.

Once you’ve entered that information, Arq’s main window appears, and you’ll see a spinning progress indicator next to the title “Other Computers”:

otherComputers1.png

Arq will download all the “index” files it needs to figure out the old computer’s backups. Once that’s done you can click on the triangle next to “Other Computers” to see the backups. Select a backup version on the left; then drag and drop the backed-up folder on the right to a Finder window to start the restore process:

Screen shot 2010-08-10 at 9.49.19 AM.png

Arq displays a progress dialog during the restore process:

Screen shot 2010-08-10 at 9.46.05 AM.png

And that’s it! If you wish you can drag and drop more backed-up items to restore multiple backed-up items in parallel.

Posted in arq, backup | 18 Comments »

How to back up your Mac using Arq

Wednesday, July 21st, 2010

When I started developing Arq it was partly because I couldn’t find an existing online backup offering that gave me enough control. I wanted to control exactly which files would be backed up, and I didn’t want to be constrained by rules that many of the “unlimited backup” offerings had like excluding network drives, excluding applications, etc.

So Arq lets you back up anything you want. But then the question is, what should you back up? The following is my suggestion for a basic backup of your files on your Mac.

Basic Backup Using Arq

When you first install and launch Arq, it asks your for your Amazon S3 “keys” and a few other things. Then it asks if you’d like to choose your own files for backup, or back up your home folder minus a few unnecessary items:

Screen shot 2010-07-21 at 8.02.18 AM.png

If you picked “I’ll manually add folders to back up” and you’ve changed your mind, here’s how to set up Arq to back up your home folder minus the unnecessary items:

1. Add your home folder

Click the + button at the bottom left of the Arq main window.

Screen shot 2010-07-21 at 8.10.25 AM.png

Pick your home folder (/Users/<yourname>) and click OK.

Screen shot 2010-07-21 at 9.27.33 AM.png

2. Add some excludes

Click the “Edit Excludes…” button.

Screen shot 2010-07-21 at 8.08.05 AM.png

Add 3 excludes.

Screen shot 2010-07-21 at 8.15.33 AM.png

Make sure the first 2 are set to “relative path” instead of “name”.

Click OK.

Backing Up Applications Using Arq

If you want to back up your applications, add the Applications folder.

Screen shot 2010-07-21 at 8.28.12 AM.png

Many applications put some of their support files in /Library/Application Support, so add that too.

Screen shot 2010-07-21 at 8.29.02 AM.png

Advanced Backup Using Arq

If you prefer, you pick and choose specific folders to back up instead of backing up your entire home directory.

WARNING: If you choose to do this and you later create a new folder in your home directory and start putting important files in there, you’ll have to remember to add this new folder to Arq or else it won’t be backed up!

I back up the following folders as separate items in Arq:

  • Application Support (/Library/Application Support)
  • Applications (/Applications)
  • Documents
  • Library, excluding files/folders named ‘Caches’ and ‘Logs’
  • Music
  • osaka iPhoto Library (my big iPhoto Library, named after my computer), excluding files/folders named ‘iPod Photo Cache’
  • src (my work files), excluding files/folders named ‘build’ and ‘bin’

Time Machine and Arq

Time Machine and Arq are complementary. Backing up using Time Machine to another disk is cheap and fast. If you’re backing up to a Time Capsule via Wifi it’s very convenient because it just happens; there’s nothing to plug in. If you’re backing up to a USB drive, you’ll have to remember to plug in the USB drive periodically. Restoring is fast because you’re reading from a USB disk physically connected to your Mac, or from a Time Capsule over Wifi.

But Time Machine doesn’t cover all cases. If someone breaks in and steals your computer, they may steal your Time Capsule or USB drive as well, and then your files are gone forever. If fire, flood, or lightning strikes, you may lose both your computer and your backups; files gone forever. And if you travel often, you’ll have to bring along your USB drive or Time Capsule, or backups won’t happen until you get home and stay home long enough for a backup to complete.

Arq covers those cases that Time Machine doesn’t. The backups are off site at Amazon’s servers, safe from your theif and your natural disasters. They’re even safe from disaster at an Amazon site because Amazon replicates your data at several sites. And Arq works whenever there’s an Internet connection, so backups still happen when you’re on the road.

Posted in amazon, arq, backup, s3 | 16 Comments »

« Older Entries