When I started developing Arq it was partly because I couldn’t find an existing online backup offering that gave me enough control. I wanted to control exactly which files would be backed up, and I didn’t want to be constrained by rules that many of the “unlimited backup” offerings had like excluding network drives, excluding applications, etc.
So Arq lets you back up anything you want. But then the question is, what should you back up? The following is my suggestion for a basic backup of your files on your Mac.
Basic Backup Using Arq
When you first install and launch Arq, it asks your for your Amazon S3 “keys” and a few other things. Then it asks if you’d like to choose your own files for backup, or back up your home folder minus a few unnecessary items:
If you picked “I’ll manually add folders to back up” and you’ve changed your mind, here’s how to set up Arq to back up your home folder minus the unnecessary items:
1. Add your home folder
Click the + button at the bottom left of the Arq main window.
Pick your home folder (/Users/<yourname>) and click OK.
2. Add some excludes
Click the “Edit Excludes…” button.
Add 3 excludes.
Make sure the first 2 are set to “relative path” instead of “name”.
Click OK.
Backing Up Applications Using Arq
If you want to back up your applications, add the Applications folder.
Many applications put some of their support files in /Library/Application Support, so add that too.
Advanced Backup Using Arq
If you prefer, you pick and choose specific folders to back up instead of backing up your entire home directory.
WARNING: If you choose to do this and you later create a new folder in your home directory and start putting important files in there, you’ll have to remember to add this new folder to Arq or else it won’t be backed up!
I back up the following folders as separate items in Arq:
- Application Support (/Library/Application Support)
- Applications (/Applications)
- Documents
- Library, excluding files/folders named ‘Caches’ and ‘Logs’
- Music
- osaka iPhoto Library (my big iPhoto Library, named after my computer), excluding files/folders named ‘iPod Photo Cache’
- src (my work files), excluding files/folders named ‘build’ and ‘bin’
Time Machine and Arq
Time Machine and Arq are complementary. Backing up using Time Machine to another disk is cheap and fast. If you’re backing up to a Time Capsule via Wifi it’s very convenient because it just happens; there’s nothing to plug in. If you’re backing up to a USB drive, you’ll have to remember to plug in the USB drive periodically. Restoring is fast because you’re reading from a USB disk physically connected to your Mac, or from a Time Capsule over Wifi.
But Time Machine doesn’t cover all cases. If someone breaks in and steals your computer, they may steal your Time Capsule or USB drive as well, and then your files are gone forever. If fire, flood, or lightning strikes, you may lose both your computer and your backups; files gone forever. And if you travel often, you’ll have to bring along your USB drive or Time Capsule, or backups won’t happen until you get home and stay home long enough for a backup to complete.
Arq covers those cases that Time Machine doesn’t. The backups are off site at Amazon’s servers, safe from your theif and your natural disasters. They’re even safe from disaster at an Amazon site because Amazon replicates your data at several sites. And Arq works whenever there’s an Internet connection, so backups still happen when you’re on the road.
Thanks for this great tool for online backup.
Now, I have started testing with some sub folders of the documents folder. What happens if I now change my settings and include my home folder? How do I go about the older backups of the sub folders? Will they be integrated in the home folder backups or do I need to look at them specifically for the backup dates?
Any hint will be appreciated before I switch my backup to the home folder. Thank you.
Comment by Daniel — July 26, 2010 @ 5:42 pm
If you want to back up your entire home folder, you’ll have to remove the sub folders from Arq and then add the home folder. Your old backup versions of the subfolders won’t be visible once you do this.
When you remove a folder from Arq, Arq doesn’t go and delete all the referenced S3 files right away. Instead, Arq Agent (the background process) goes through all the S3 files about once per day and deletes S3 files that aren’t referenced by any backups. Also, as long as Arq is running, Arq Agent won’t start deleting.
So, if you remove the sub folders and then add the home folder right away, the files that were already uploaded to S3 won’t be re-uploaded.
Comment by Stefan Reitshamer — July 26, 2010 @ 6:24 pm
Wow, that is really a great functionality. That way I can easily extend my backup whenever I want.
Thanks again for this great tool.
Comment by Daniel — July 27, 2010 @ 5:18 pm
I just installed Arq, and the data is really flying to my brand new S3 account, so I’m thrilled with the speed. But I do have questions:
1) Daniel, I really wanted to know the same thing you asked, but I didn’t completely follow Stefan’s answer. So let me explain how I intended to use Arq, and Stephen, you can tell me if it will or won’t work.
Because I have so much data on my computer that I would like to backup, I want to start with one folder (my most important data), and gradually add more folders, until I feel the crucial data is all backed-up, at which point I would back-up the rest by adding my home folder. This way I can start with the most crucial back-ups, and eventually end up with all my data. But I would like to end up eventually adding the home folder, rather than a long list of sub-folders, because that will keep my entire home folder backed up (and seems less prone to user error than trying to remember to add every relevant sub-folder). Hopefully you can see why this makes sense from a user’s perspective.
So, will this work? If not, how would you suggest I slowly add folders to my backup (in an order that makes sense), and eventually end up with the entire home folder (so I won’t ever have to ever add sub-folders after that point)?
Also, if my original idea will not work, do you think a future version of Arq would accommodate my preferred way to work with Arq, of starting with my most important subfolders, and ending up adding the entire user folder (which, I suspect, might be the way many users would like to backup their data as they get the process started)?
2) I use 1Password, not my standard Mac keychain, but I see all my Arq data is stored on my Mac keychain. Is there any way or plan to use something other than the Mac keychain?
3) If I choose to use my new S3 account to deliver web content (using CloudFront), are there any issues I should be aware of, or potential conflicts using Arq with the same account?
Comment by Rob — July 28, 2010 @ 7:30 pm
Hi Rob,
1) Yes, you can definitely add folders one by one, and then remove them all and add the home folder (which contains all the other folders). As long as you do this without quitting Arq in between, Arq won’t delete and then re-upload the data.
2) I’m not a 1Password user, so I’ll have to look into how that works.
3) I don’t think you’ll have issues with using CloudFront with other data in your S3 account. You should put the other data in other S3 buckets to avoid conflicts with Arq and its bucket.
Comment by Stefan Reitshamer — July 28, 2010 @ 9:08 pm
Stefan, I imagine it will take a week or so before I get to my entire home folder, and expect I will have cause to restart my Mac in the interim. So, with that in mind, it sounds like I need to take a different approach? I’d hate to upload tens of GB of data, only to have to redo the entire process (or be stuck having to always add new folders). I guess the process of adding different sets and subsets of folders as time goes by doesn’t seem straight forward to me, especially if quitting Arq limits what can be done in subsequent sessions (w/o having to re-upload data). Maybe I should bite the bullet and add my entire user folder (with some exceptions)?
Comment by Rob — July 28, 2010 @ 10:31 pm
Shoot, sorry to clog your post with support questions… As I’m backing up another few folders, I can’t seem to find my way to the screen that shows my progress (I saw it the first time). At the bottom of the screen I currently see “Backing up FolderName Data” but no other sense of progress.
Comment by Rob — July 28, 2010 @ 10:48 pm
Double-shoot, I figured it out right after I hit the “submit” button. I see each folder has it’s own progress when you select it… I do love how fast it is compared to Carbonite, my only other online backup trial.
Comment by Rob — July 28, 2010 @ 10:50 pm
Rob,
If you restart your computer while Arq is doing the initial backup, when Arq restarts it will start the backup from the beginning, but it won’t upload files that it has already uploaded. So for example, if you get 50% of the way through your initial backup and then restart, Arq will start from the beginning but you’ll see the progress bar move extremely quickly for the first 50% of the backup because it has already uploaded those files to S3.
Once there is an initial backup, Arq uses that as a baseline (comparing file sizes and last-modified times of the files to the data in the backup), so subsequent backups go much much faster.
Comment by Stefan Reitshamer — July 28, 2010 @ 11:59 pm
@Stefan:
The current version of Arq shows an error message if you try to backup a folder with one or more sub-folders already included in the backup. Couldn’t Arq just handle this issue automatically without an error message and the need for manual deletion?
Manual deletion is not user-friendly, in particular since Arq lacks a tree view showing all the folders already included in the backup.
(That is an area where I hope for major improvements anyway. I like to see which directories and files get saved and which not.)
@Rob:
1Password cannot fully replace the Mac OS X keychain. You can of course add your Amazon S3 credentials to your 1Password keychain, however, it is still not the keychain where Mac OS X applications get their passwords, etc. from.
I use my S3 account for various purposes including Arq. There should be no issues as long as you use different buckets.
Comment by Martin — July 29, 2010 @ 8:57 am
Thanks for the detailed responses… I just ran across the issue where I tried to add an enclosing folder to be backed up, but got the error message Martin mentioned (and I don’t believe I’ve quite Arq since I started using it). Message says:
“Failed to add folder for backup
/Volumes/DriveName/Users/UserName/Documents/Clients contain the folder “ClientOneName” which is already being backed up”
(general names inserted)
So now I’m at the fork in the road where I have to decide, will I keep adding folders one-by-one? Or should I trash the current backup, and start again with the entire home folder (with exclusions)? I assume I can remove exclusions at any time? Is there any chance this limitation (not being able to add enclosing folders) will be worked around in future version of Arq? Any advice on how to continue?
Comment by Rob — July 29, 2010 @ 4:10 pm
A future version of Arq will have a more flexible folder chooser, but not in the short term, sorry.
Whether you want to back up several folders individually or 1 folder with exclusions is mostly up to you. I use the several-folders approach myself.
Comment by Stefan Reitshamer — July 29, 2010 @ 4:18 pm
Arq looks great! Is there any chance you will include a feature to create multiple backup jobs that can be scheduled to run separately?
Comment by Christopher York — September 16, 2010 @ 2:31 pm
Yes, eventually. Not sure when though.
Comment by Stefan Reitshamer — September 16, 2010 @ 4:13 pm
Is it possible to exclude files and folders by regular expression. So for example if I want to exclude all mp3 and mp4 files can I exclude *.mp*
Comment by John — December 4, 2010 @ 9:54 pm
Sorry, regular expressions aren’t supported currently.
You could exclude where name contains ‘.mp’
Comment by Stefan Reitshamer — December 5, 2010 @ 4:08 pm