Arq 2.6.8 is available. To update, pick “Check for Updates” from Arq’s menu.
Please note: Arq 2.6.8 may upload much more than you expect or want it to the first time it backs up. This is due to a bug that I unfortunately created in version 2.6.3. I apologize for this. Please let it finish backing up.
There is no charge from Amazon for data transfer to Amazon, but I realize that re-uploading a lot of data is a long, frustrating experience.
The full story of what happened starting with Arq 2.6.3
There’s a problem with some of S3′s servers, particularly in the us-west-1 region, but also in other regions. Connections just get “stuck”. Customers were complaining heavily about Arq getting “stuck”. I was initially unable to reproduce the problem, but felt I had to do something to fix it. The Apple networking API that Arq was using had no facility for timing out; plus it was very hard to throttle properly. So I changed Arq to use Apple’s older CFHTTP API.
In my haste to fix this customer problem, I didn’t do enough testing on 2.6.3-4. I screwed up. There was a bug in those releases where some objects were marked as “compressed” but they weren’t actually compressed. So Arq couldn’t restore properly.
What really made things bad was that at some point in my rush to get the issue fixed I accidentally (because of muscle-memory habit) ran the command to publish 2.6.3 in the “update stream” as an official release of Arq, which meant that everyone who happened to check for updates would get this buggy version. I didn’t realize this for several hours. At that point I could only move forward and fix things as quickly as possible.
Since there’s no way for me to tell which objects were compressed, Arq 2.6.5 (and later) ignores all backups made after March 7 by an Arq version older than 2.6.5. I added a “version” to the backup data so that if this sort of thing ever happens again, I can fix it with a less drastic solution.
I’ve also added some “gates” to my release process so that I don’t inadvertently do this again.
Arq 2.6.6 through 2.6.8 contain very small changes — checks for various network errors that were causing Arq to fail the backup instead of retry.
Please let Arq finish doing what it’s doing. It will clean up all the “bogus” S3 data after it successfully backs up everything.
Also, please consider using more than one backup system, as I blogged about in 7 facets of a good Mac backup strategy.
Actually, since the last backup and S3 budget enforcement has been done, I lost all my previous backups…
Hopefully, I’ve still my Time Machine backups if I want to recover from older than 2 days but I’m very disappointed by this behavior…
Comment by Joris — April 3, 2012 @ 6:54 pm
So am I. I’m not going to put out another release until I test it for weeks, no matter what the issue is.
Comment by Stefan Reitshamer — April 3, 2012 @ 8:30 pm
Thank you for the honest explanation. At the end of the day, Arq is still the best solution for untethered, versioned backup.
Comment by Nicholas Pike — April 3, 2012 @ 11:26 pm
Agreed! I like this software and this is why I try to be useful for its improvement
Comment by Joris — April 4, 2012 @ 1:44 pm
All software has bugs. What separates the great developers from the rest is what they do when a bug is discovered. “I screwed up, and here’s what I’ve done to fix it” is the best possible outcome. Arq rocks.
Comment by John — April 5, 2012 @ 10:55 pm
This is remarkable to read an honest explanation and it is very appreciated. This guy Stephan is really sweating things out and he deserves support and commendation.
I am in the long upload process right now. I got a bandwidth warning from the ISP on Thursday and that triggered me to take notice. I am not that stressed, as I use 5 backup methods. No one should use just one.
Comment by William — April 14, 2012 @ 10:24 am
Good work, Stefan!
Comment by Henrik — April 14, 2012 @ 11:04 am
Had to reupload 30 GB, but I use multiple backup systems of course so no loss of data occured. I always assume that the backup software and/or hard drives will fail eventually.
I still thinks Arq is the best online solution for Macs – I hate it when competitors use the words “unlimited storage” for a limited fee, when clearly that is not the case. Arq (or Amazon really) is honest and offers a fixed price per GB, I like that.
Also, I’m gonna be more careful when installing software updates in the future.
Comment by Kurt — April 15, 2012 @ 12:22 pm
Thanks Kurt. I’m going to be a lot more careful when publishing updates. I won’t ship another update without weeks of testing, no matter how minor the changes seem.
Comment by Stefan Reitshamer — April 16, 2012 @ 2:56 pm
Kudos for openness and honesty – things go wrong, and when they do that’s all anyone can ask of you. Thanks Stefan.
Comment by John Noble — April 17, 2012 @ 12:20 am
Thanks for the info. This sort of thing happens which is one of the reasons I usually don’t upgrade any software right away. I happen to still be on 2.6 (Software works great right now, no need to change). The only thing I worry about is when does my version either become unsupported or have issues taking a new update. When you get time it might be nice to have a range of versions to upgrade (Instead of current) as well as “your stuff is getting old, time to move on”. Thanks for the great software, I have been very happy with the product.
Comment by Marty Kincaid — April 18, 2012 @ 1:11 am
Thankyou for your honesty!
Comment by Paul Richards — April 22, 2012 @ 12:35 pm
Thanks for the honest appraisal of what happened … Quite a few GB re-uploaded and I’m still an Arq fan!
We all make mistakes from time to time, it’s how we handle those mistakes that really matters. The integrity shown by you in handling this bug openly and publicly is a sign of the integrity of the product as a whole. You could have buried this issue, but you knew that would be wrong and you opted for the right solution rather than the easy one.
Comment by Dave — May 2, 2012 @ 2:17 am
Wow, after reading this full, open and honest statement, and reading security guru Steve Gibson’s tweet,
see below:
@SGgrc: “ARQ” Cloud backup for Mac to Amazon S3 w/TNO Crypto & iOS viewer. The deeper I dig, the more I wish I was a Mac user: http://t.co/D1hmt8Dt
I will now be purchasing your Arq backup software for my 3 Macs, thank you Haystack Software.
Comment by KJ — May 2, 2012 @ 1:48 pm
Thanks for the explanation, it just reaffirms my opinion that Arq is a wonderful piece of software!
Comment by Mike Rundle — May 4, 2012 @ 1:26 pm
As developers, we all have blemishes on our records. Kudos for taking responsibility (too bad most of our corporate and government leaders don’t have the same integrity!). I’m sure it was a hard thing to have to explain this to your customers, but I think it work in your favor in the long term.
I just started with Arq on April 8th and I’m running 2.6.3, so I think I’m gong to lose all of my backups
But, hey, at least I *know* I’m going to lose them and why.
Comment by Mark Kirk — May 10, 2012 @ 5:31 pm
Thanks, Stefan. I appreciate your honesty as well. Arq has been working great for me over the past two years. It’s the backup of my backup, so this isn’t a major problem for me. If need be, I can start the backups over as well.
Comment by Steven — May 11, 2012 @ 9:30 am
Those S3 buckets can be persnickety.
I love Arq tho, and I hadn’t upgraded, apparently, since 2.5.1, so bully for me.
Glad you fixed the issue and were upfront about it. The transparency of the whole Arq system is the reason I like it. I still haven’t read those format specs like I thought I would two plus years ago when I started using Arq…
Comment by DCam — May 11, 2012 @ 7:03 pm