This release fixes a bug that causes high CPU usage after Arq Agent has been running for many hours and there are backup errors (e.g. folder to be backed up isn’t available).
Pick “Check for Updates” from the Arq menu to automatically update to 1.4.4. Or download Arq here.
This release fixes a few bugs:
Pick “Check for Updates” from the Arq menu to automatically update to 1.4.2. Or download Arq here.
I’m the author of Arq, a backup product for the Mac, so I’m certainly biased, but in this post I’m only going to describe the factual results of a backup/restore with CrashPlan. You can verify these results yourself.
Warning: What follows is technical nitty-gritty. If you just want to know why Mac metadata are important in the real world, read The Importance of Metadata on the Mac.
Here’s what CrashPlan says it backs up and restores on the Mac:
CrashPlan states that the following are unsupported:
That list a bit redundant: “other named forks” is the same as “HFS+ extended attributes”, and an “alias” is a special type of file containing some data as well as 2 extended attributes (com.apple.FinderInfo and com.apple.ResourceFork) and some Finder Flags (‘Alias’ and ‘Custom Icon’).
Also, CrashPlan doesn’t mention that they do not support hardlinks, FIFOs, or device files.
I tested CrashPlan 3.8.2010 on a MacBook Pro running OS X 10.6.3. I followed Backup Bouncer‘s test procedure:
Here’s the output of that ‘bbouncer verify’ command:
backup-bouncer-0.2.0 $ sudo ./bbouncer verify /Volumes/Src /Volumes/Dst Verifying: basic-permissions ... ok (Critical) Verifying: timestamps ... ok (Critical) Verifying: symlinks ... ok (Critical) Verifying: symlink-ownership ... FAIL Verifying: hardlinks ... FAIL (Important) Verifying: resource-forks ... Sub-test: on files ... ok (Critical) Sub-test: on hardlinked files ... FAIL (Important) Verifying: finder-flags ... FAIL (Critical) Verifying: finder-locks ... ok Verifying: creation-date ... ok Verifying: bsd-flags ... FAIL Verifying: extended-attrs ... Sub-test: on files ... FAIL (Important) Sub-test: on directories ... FAIL (Important) Sub-test: on symlinks ... FAIL Verifying: access-control-lists ... Sub-test: on files ... ok (Important) Sub-test: on dirs ... ok (Important) Verifying: fifo ... FAIL Verifying: devices ... FAIL Verifying: combo-tests ... Sub-test: xattrs + rsrc forks ... FAIL Sub-test: lots of metadata ... FAIL
The source directory looks like this:
$ ls -l Src/15-symlink-ownership/ total 32 -rw-r--r-- 1 root staff 14 Jun 1 07:07 some-file lrwxr-xr-x 1 _www _www 11 Jun 1 07:07 symlink1@ -> ./some-file lrwxr-xr-x 1 root wheel 11 Jun 1 07:07 symlink2@ -> ./some-file lrwxr-xr-x 1 root staff 10 Jun 1 07:07 symlink3@ -> ./symlink1
The files restored by CrashPlan look like this:
$ ls -l Dst/15-symlink-ownership/ total 32 -rw-r--r-- 1 root staff 14 Jun 1 07:07 some-file lrwxr-xr-x 1 root staff 11 Jun 1 07:11 symlink1@ -> ./some-file lrwxr-xr-x 1 root staff 11 Jun 1 07:11 symlink2@ -> ./some-file lrwxr-xr-x 1 root staff 10 Jun 1 07:11 symlink3@ -> ./symlink1
CrashPlan doesn’t restore symlink owner information, even though they claim to on their web site.
The source directory looks like this:
$ stat -f 'inode:%i file:%N' Src/20-hardlinks/* inode:48 file:Src/20-hardlinks/link1 inode:48 file:Src/20-hardlinks/link2 inode:48 file:Src/20-hardlinks/link3 inode:48 file:Src/20-hardlinks/some-file
Note the ‘inode’ number is the same for each file. That’s what a “hardlink” is. Wikipedia has more information on hardlinks.
The files restored by CrashPlan look like this:
$ stat -f 'inode:%i file:%N' Dst/20-hardlinks/* inode:57 file:Dst/20-hardlinks/link1 inode:59 file:Dst/20-hardlinks/link2 inode:58 file:Dst/20-hardlinks/link3 inode:60 file:Dst/20-hardlinks/some-file
The ‘inode’ numbers are all different in the restored files.
CrashPlan correctly restores almost all the Finder Flags, but Backup Bouncer marks this test as “failed” because CrashPlan fails to restore the Extended Finder Flag “busy”. The source files look like this:
$ GetFileInfo -P -a Src/40-finder-flags/mucho-flags-file AVBSTClINMEDZ
$ GetFileInfo -P -a Src/40-finder-flags/mucho-flags-dir aVbstClInmEDZ
The files restored by CrashPlan look like this:
$ GetFileInfo -P -a Dst/40-finder-flags/mucho-flags-file AVBSTClINMEDz
$ GetFileInfo -P -a Dst/40-finder-flags/mucho-flags-dir aVbstClInmEDz
The ‘z’ flag is “busy” (kExtendedFlagObjectIsBusy) in the Extended Finder Flags.
CrashPlan restores 1 of the 4 BSD flags, UF_NODUMP (1), UF_IMMUTABLE (2), UF_APPEND (4), and UF_OPAQUE (8). The source files look like this:
$ stat -f 'flags:%f name:%N' Src/60-bsd-flags/* flags:15 name:Src/60-bsd-flags/dir-with-flags flags:15 name:Src/60-bsd-flags/file-with-flags
The files restored by CrashPlan look like this:
$ stat -f 'flags:%f name:%N' Dst/60-bsd-flags/* flags:2 name:Dst/60-bsd-flags/dir-with-flags flags:2 name:Dst/60-bsd-flags/file-with-flags
In the output above, the “flags:15″ is all 4 flags (1+2+4+8=15) and “flags:2″ is UF_IMMUTABLE. CrashPlan actually restores the UF_IMMUTABLE flag correctly because it’s the same as the “Locked” Finder Flag in OS X.
Backup Bouncer creates some extended attributes with silly names on a file, a symlink, and a directory. The names of the extended attributes of each file are listed here (‘xattr-test’ has 2 extended attributes):
$ xattr-util l Src/70-extended-attrs/dir-with-xattrs mamma.mia $ xattr-util l Src/70-extended-attrs/symlink-with-xattrs good.grief $ xattr-util l Src/70-extended-attrs/xattr-test message this.that
The files restored by CrashPlan look like this:
$ xattr-util l Dst/70-extended-attrs/dir-with-xattrs $ xattr-util l Dst/70-extended-attrs/symlink-with-xattrs $ xattr-util l Dst/70-extended-attrs/xattr-test
Those 3 ‘xattr-util’ commands show no output because no extended attributes exist on the restored files.
The source file is a “fifo” (a first-in-first-out file, or “named pipe”):
$ file Src/90-fifo/some-fifo Src/90-fifo/some-fifo: fifo (named pipe)
CrashPlan doesn’t restore anything at all in this case:
$ file Dst/90-fifo/some-fifo Dst/90-fifo/some-fifo: cannot open `Dst/90-fifo/some-fifo' (No such file or directory)
Backing up and restoring fifos is quite esoteric and probably of no interest to most people.
The source files are “device” files:
$ file Src/95-devices/devvn0 Src/95-devices/devvn0: block special $ file Src/95-devices/devzero Src/95-devices/devzero: character special
CrashPlan doesn’t restore anything at all in this case:
$ file Dst/95-devices/devvn0 Dst/95-devices/devvn0: cannot open `Dst/95-devices/devvn0' (No such file or directory) $ file Dst/95-devices/devzero Dst/95-devices/devzero: cannot open `Dst/95-devices/devzero' (No such file or directory)
Like fifos, device files are quite esoteric and probably of no interest to most people.
As I said at the beginning of this post, I wrote Arq, which competes with CrashPlan, so I’m biased.
If you’ve analyzed your programs and your files and determined you don’t need the metadata that CrashPlan skips, then you should be OK, at least in terms of metadata.
If you want to be sure all your data (and metadata) are backed up and restored correctly, then Arq and Jungle Disk are the only 2 options I know of. Arq has a Mac-native UI, a one-time software license fee (instead of an endless software subscription fee), and has gotten excellent reviews (see MacUpdate and MacStories).
Your backup product/service should back up your data completely and correctly. This of course includes each file’s contents, but on the Mac it also includes many types of “metadata”. What are “metadata”? The general definition is “data about data”. In the case of Mac files, metadata include the file’s size, its modification date, its owner, who’s allowed to read and/or write it, who’s allowed to run it (if it’s a program), and many others.
Unfortunately, many online backup products, such as Mozy, Carbonite, Backblaze, CrashPlan, and Dropbox, don’t back up and restore all the metadata correctly. Fortunately, Nathan Gray, an independent researcher at Caltech, has created a tool called Backup Bouncer that tests every type of Mac metadata. I’ve run the Backup Bouncer tests through Mozy, Carbonite, Backblaze, CrashPlan and Dropbox, and the results range from perfect to really really bad; a table with links to Backup Bouncer results for those products is on the Arq product page.
But are all the metadata types really important for your situation? Well, it depends. Let’s analyze a few scenarios:
Let’s say you just want to back up your JPEG photo files and your AVI movie files from your home directory. These files typically won’t have any more than the bare minimum of metadata. They’ll be owned by you with default permissions. Any backup product should be able to restore those files well enough; even if they restore the files as owned by you with default permissions, they’ll end up (by coincidence) to be the same as the originals.
One caveat even in this simplest of scenarios is the modification dates of files. If the backup program you’re using can’t even restore modification and creation dates correctly (Backblaze, Mozy and Carbonite can’t), when you restore, it’ll look like all your files were just created. This can be confusing and/or disconcerting in the case of text files and word processing documents. For some files like photos and videos, the “date taken” is stored within the file itself so iPhoto shows the photo with the correct date, but in the Finder the dates will still be off.
How about backing up applications? On the Mac, apps are actually “bundles”, which are folders with the “bundle bit” set. The folders have a fixed structure and contain the actual executable file as well as lots of resources (icons, configuration files, etc). The executable file must have “executable permission”; otherwise OS X silently refuses to run it.
So if you use Dropbox for backing up your applications, for example, and you restore a deleted app (or you use it for sync and sync an app to another Mac), Dropbox will restore the app’s files but won’t set the executable permission. When you double-click on the restored app, nothing happens!
If you’re sufficiently technical you might check the system log and see an error like this:
com.apple.launchd.peruser.501[144] ([0x0-0x36b36b].com.haystacksoftware.iphotosync[18955]): posix_spawn(“/Users/stefan/Dropbox/iPhotoSync.app/Contents/MacOS/iPhotoSync”, …): Permission denied
You might then look at the binary’s permissions using Terminal and see it’s not marked as executable:
$ ls -l iPhotoSync.app/Contents/MacOS/ total 2272 -rw-r--r-- 1 stefan staff 1161472 May 28 15:10 iPhotoSync
You’ll then set the binary’s permissions correctly to be able to start the app:
$ chmod +x iPhotoSync.app/Contents/MacOS/iPhotoSync $ ls -l iPhotoSync.app/Contents/MacOS/ total 2272 -rwxr-xr-x 1 stefan staff 1161472 May 28 15:10 iPhotoSync*
But that’s pretty geeky!
When you download a file using Safari, by default it goes to your Downloads folder. When Safari puts it there, it adds an “extended attribute” called “com.apple.quarantine” to the file indicating it’s potentially unsafe because you downloaded it from the internet (more on extended attributes later). When you attempt to open it, you get a dialog like this:
When you click “Open”, OS X removes the “com.apple.quarantine” extended attribute from the app.
If you restore these files from a backup using software that doesn’t restore extended attributes, you won’t get this security warning. For more details on extended attributes, see below.
Time Machine uses extended attributes heavily. Here’s a list of the extended attributes on one of my Time Machine backup folders:
$ xattr 2010-05-11-080152/ com.apple.backup.SnapshotNumber com.apple.backup.SnapshotVersion com.apple.backupd.SnapshotCompletionDate com.apple.backupd.SnapshotStartDate com.apple.backupd.SnapshotState com.apple.backupd.SnapshotType
This is the list of extended attributes on the “Macintosh HD” folder within that backup folder:
$ xattr 2010-05-11-080152/Macintosh\ HD com.apple.backupd.SnapshotVolumeFSEventStoreUUID com.apple.backupd.SnapshotVolumeLastFSEventID com.apple.backupd.SnapshotVolumeUUID com.apple.backupd.VolumeBytesUsed com.apple.backupd.VolumeIsCaseSensitive com.apple.metadata:_kTimeMachineNewestSnapshot com.apple.metadata:_kTimeMachineOldestSnapshot
Restoring these files from backup using a backup program that doesn’t restore extended attributes isn’t going to work too well; Time Machine wouldn’t find its expected metadata.
Alias files are a somewhat esoteric feature of OS X. If you control-click a file in Finder and choose “Make Alias” from the pop-up menu (or Command+Option+drag the file to another location), you’ll get an alias file that points to the file you had clicked on. The alias acts as a stand-in for the original, and works even if you move the original to another location (on the same filesystem).
Alias files depend on “resource forks” (a type of extended attribute) to function properly. If you restore an alias file from backup using a backup program that doesn’t restore extended attributes, the restored file will be incomplete and unusable. For example, I made an alias to a PDF file in my Dropbox, waited for it to back up, deleted it, and then restored it via Dropbox. When I double-clicked on the restored PDF alias, it opened in TextEdit and looked like this:
That’s not usable.
A symbolic link is a special type of file that refers to another file. Some backup programs are unable to differentiate between a regular file and a symbolic link; they “follow” the link instead of backing up the link itself. So if you restore a folder tree containing a large directory as well as a symlink to that directory, you’ll instead get 2 copies of the large directory.
Some sync programs like Dropbox consider this a feature: even though Dropbox will only backup/sync the “Dropbox” folder, you can place symbolic links within it that point to other outside folders, and those will be backed up as well.
In my opinion a real backup program should accurately back up and restore the contents of your disk, not its interpretation of those contents.
If you’re backing up data from an application that depends on having symbolic links to a file instead of multiple copies of a file, failing to recreate the symbolic links could be a big problem. CrossOver is one program that creates many symbolic links and some complex ownership settings as well.
OS X can store many “flags” with a file. The following flags can be read and written through the “Get Info” window in the Finder (control-click a file and choose “Get Info” from the pop-up menu):
Additional Finder flags include:
“Has bundle” is a bit that makes your iPhoto Library look like a file instead of a folder. So if your backup program can’t correctly restore Finder flags, when you restore your iPhoto Library it’ll look like a regular folder. You can’t double-click on that to open it in iPhoto. You’ll have to launch iPhoto with the Option key held down and navigate to that folder; after you do this once, iPhoto resets the “has bundle” bit on the folder. You’ll find the same thing with a Pages document or a Numbers document, or any bundle that’s meant to look like a single file in the Finder.
“Invisible” is a bit that makes a folder invisible to the Finder. I’m not sure which apps (if any) use it.
“Custom icon” is a bit that Safari sets on a file while it’s being downloaded (the Finder shows a progress bar on the file icon).
“Busy” is a bit that indicates a file is busy or incomplete.
If your backup program can’t restore these flags, things may look a bit different when/if you have to restore from backup.
As described in this TidBITS article, creator codes and type codes are 4-letter codes attached to a file that specify what type of file it is and which application created it. Before OS X this was the way MacOS figured out which application to open when you double-clicked on a file. When OS X arrived (with its Unix base), a conflict arose between using creator codes and using filename extensions.
As of Snow Leopard (OS X 10.6) creator codes are completely ignored for opening files. So whether you should care that your backup program correctly restores creator codes and type codes depends on whether you use 10.6, among other things.
OS X still uses creator code and type code for copying files however, even in 10.6. If you copy a large file from one place to another, the incomplete copy receives a type code of “brok” and creator code of “MACS”; Finder resets these to empty values when the copy is completed.
Extended attributes are small pieces of metadata associated with files and folders. In the “old days” (before OS X) MacOS used “resource forks” heavily; in OS X, a resource fork is just one type of extended attribute.
Using Terminal you can see which files have extended attributes (they get a ‘@’ next to the permissions):
$ ls -l Knox-2.0.1.zip -rw-r--r--@ 1 stefan staff 5210452 May 25 15:37 Knox-2.0.1.zip
The ‘xattr’ utility lists the extended attributes of a zip file I downloaded from the internet:
$ xattr Knox-2.0.1.zip com.apple.metadata:kMDItemWhereFroms com.apple.quarantine
In this example, the file was downloaded with Safari. The “com.appl.metadata:kMDItemWhereFroms” extended attribute contains where the file came from (in a binary format readable by OS X):
$ xattr -pl com.apple.metadata:kMDItemWhereFroms Knox-2.0.1.zip com.apple.metadata:kMDItemWhereFroms: 00000000 62 70 6C 69 73 74 30 30 A2 01 02 5F 10 3B 68 74 |bplist00..._.;ht| 00000010 74 70 3A 2F 2F 61 77 73 2E 63 61 63 68 65 66 6C |tp://aws.cachefl| 00000020 79 2E 6E 65 74 2F 61 77 73 2F 64 6D 67 2F 4B 4E |y.net/aws/dmg/KN| 00000030 4F 58 2F 45 6E 67 6C 69 73 68 2F 4B 6E 6F 78 2D |OX/English/Knox-| 00000040 32 2E 30 2E 31 2E 7A 69 70 5F 10 26 68 74 74 70 |2.0.1.zip_.&http| 00000050 3A 2F 2F 61 67 69 6C 65 77 65 62 73 6F 6C 75 74 |://agilewebsolut| 00000060 69 6F 6E 73 2E 63 6F 6D 2F 64 6F 77 6E 6C 6F 61 |ions.com/downloa| 00000070 64 73 08 0B 49 00 00 00 00 00 00 01 01 00 00 00 |ds..I...........| 00000080 00 00 00 00 03 00 00 00 00 00 00 00 00 00 00 00 |................| 00000090 00 00 00 00 72 |....r|
The “com.apple.quarantine” attribute is an indicator that the file was downloaded by Safari:
$ xattr -pl com.apple.quarantine Knox-2.0.1.zip com.apple.quarantine: 0000;4bfcfa48;Safari;A17E9A1C-F662-4DF0-95AA-18F44791DAFC|com.apple.Safari
If possible, choose a backup program for your Mac that correctly backs up everything. Then you don’t have to worry. Arq is one backup program that backs up and restores everything correctly.
