DLo
Member
Hello,
Arq 2.6.4 was giving me errors when trying to back up...
Example: http://dl.dropbox.com/u/1931507/error.jpg
Now Arq 2.6.5 is trying to do a full backup instead of incremental. I have verified the files its trying to backup have not been changed or modified.
Please advise,
DLo
mrblack
Member
I am having a similar problem.
After the update to 2.6.5 mine is doing a full backup instead of incremental... plus there is no progress bar.
Screenshot: http://imgur.com/0BtXk
Did you have backups from before March 7 (when I shipped the buggy release)?
If so, Arq will reuse that S3 data and it shouldn't take so long.
If you only had backups since March 7, unfortunately those backups have an issue. The problem that I fixed is that some objects were marked as "compressed" in the backups when they weren't actually compressed. There was no way for me to tell which objects were compressed, so I had to ignore any backups since March 7. I added a "version" to the backup data so that if this sort of thing happens again, I can tell which parts of the backup are problematic.
I'm sorry for the hassle. I know uploading is a long, frustrating experience.
jmah
Member
I have a similar issue; after updating from Arq 2.6.2 to 2.6.5 this morning (I was surprised to see that I missed a couple of updates), Arq wants to upload more than usual - around 3.7GB (instead of perhaps 300MB?). If the delta is since 2012-03-07, that seems reasonable (though unfortunate).
However, when the backup tries to complete, folders keep failing with error -9810 (perhaps errSSLInternal). I've tried it a couple of times; I'm now trying again with a more stable connection. I'll post the results shortly.
DLo
Member
I have confirmed again as an example, that my movies folder hasn't been modified but its still backing it up. I have backups going back to February 10, 2012, with the movies not being modified since February 26, 2012. It looks like whatever happened in version 2.6.5 is causing me to start a brand new full backup.
jmah
Member
Well, I'm now up to 5.5GB, and long ago lost the progress bar. Looking at the open files in Activity Monitor, it seems arqcommitter is going over EVERY file in my Documents directory. Since it has barely begun, I'm going back to an older version until this is sorted out. (Does 2.6.2 have the "compressed" bug?)
@jmah I can't reproduce the error -9810.
Which S3 region are you using? What kind of network are you using?
Even though I can't reproduce it, I added a check for any SSL errors and Arq will retry the request instead of failing the whole backup.
It's version 2.6.6. I haven't put it into the update stream yet (because I don't want to inflict everyone with yet another update), but you can download it here and try it:
Homer712
Member
Even better. In the time it took me to write the above post Arq 2.6.6 finished the backup. Seems something has changed again and 2.6.6 will not be doing a complete (re)backup (from scratch) as version 2.6.5 would. Was this something that got fixed in 2.6.6? I had backups from later than March 7th but I've been using the program for many months (flawlessly!) so I had backups from before March 7th also.
mrblack
Member
I have also tried the 2.6.6 update and still the same issue. It looks like it is doing a full backup although it is hard to tell without a progress bar.
I have backups from before March 7.
It sits at ~5.9GB to be uploaded for a long time and looking at my quota usage it is definitely uploading the whole 70GB again.
DLo
Member
As a test, I removed 2.6.5 and installed 2.6.2, the backup tried to start and backup 10GB instead of a full backup.
Stefan - At this point what do you recommend we do? I'm not sure if everyone is having the same issues I am, but I am willing to delete everything on S3 and start over with a full backup. Please advise.
Homer712
Member
I for one would like to know if there is a simple one or two step process for deleting everything we have backed up on-line and starting fresh. For those of us that simply back up 1 or 2GB of documents to our S3 account for safe keeping this would be a welcome feature.
@DLo I explained earlier in this thread that due to a bug in versions 2.6.3 and 2.6.4, backups created after March 7 are bad and will be discarded by Arq 2.6.5 and later. So if you didn't have any backups from before March 7, Arq is going to start over uploading. I'm sorry.
There's no need to delete everything on S3 and start over. Just get the latest Arq (pick "Check for Updates" from the menu) and let it do its thing. It will clean up the bad S3 data when it does its "budget enforcement" process (by default every 4 days, and only upon successful backup).
If you really really want to start over, you can pick "Change S3 Backup Set" from Arq's menu and create a new backup set. Then you'll have to go under "Other Backup Sets" in Arq's main window, find your previous backup set, and delete it.
DLo
Member
Hello Stefan,
Thanks for the reply! My issue is that I do have backups from before March 7, but its still doing a full backup, and my S3 size has grown from around 55GB to 87GB for some strange reason. I just need it to get back to the correct size but am not sure how to proceed.
I am worried that my backups might be so messed up or corrupted that the safest solution might be to start over.
Any thoughts?
@DLo It has grown because it's uploading the new backup data. It will delete the old, bad backup data later, as I explained in my previous post. Just please leave it alone and let it finish. Thanks.
@jmah Yes, because I have no way of knowing which versions of Arq have backed up what.
jmah
Member
@stefan Ok. I presume you record that now, otherwise 2.6.5 won't know where the good backups start again, right? (Unless it assumes everyone has updated by a certain date... which wouldn't really work.)
Could you perhaps provide default, e.g. "LastKnownGoodBackupDate" that you register to be 2012-03-07 but I can override? That way when I move off 2.6.2, I can keep my history.
jmah
Member
I hadn't been running Arq since I quit it and went back to 2.6.2 yesterday. About an hour ago I rebooted my machine, and it launched automatically and started backing up (as expected). I only noticed it had started 10-15 minutes later, and I opened the Arq app to see what was going on.
It seems that the backup of my Desktop had finished, and it was currently on another folder. But I looked at my Desktop's history, and the only entry was for today. Did Arq really just delete the entire backup history for that folder, and not even notify me? This is not good. I tried to look at the list for some other folders — the ones that successfully backed up with 2.6.5 — and received an "invalid Commit header" error. That's understandable, if 2.6.5 is now recording the Arq version (or something) in the header, but arq_restore also gives the same error. This is not good!
I'm rather angry right now, and I'm sorry about that. Arq established its reliability through great Backup Bouncer results, an open-source restore tool, and a stable third-party (Amazon S3) for storage. This mess has, unfortunately, significantly damaged the "trustworthy online backup" status.
If Arq is going to delete my data (2.6.5 deleting backups since 2012-03-07; 2.6.2 apparently deleting backups made by 2.6.5 along with all previous ones) — TELL ME ABOUT IT. If you've changed the storage format, update arq_restore at the same time.
I'm not really sure what to do now. I don't want to run Arq 2.6.2, as it will delete the backup history for the other folders that finished with 2.6.5. I don't want to run 2.6.5, as it will delete backups since 2012-03-07, and it looked like it was uploading a lot more than seemed necessary. And none of my data is bad, since I never even ran 2.6.3 or 2.6.4!
Again, sorry for my tone. I wish there was something I could do to help. If there is, please let me know! Perhaps I can help you with the code over the weekend — complete non-disclosure, and I'll relinquish all copyright to you. I don't care about Arq's code, I care about my data! If you honestly could use any help, send me an email (it's in my profile, and you'll find it order #6801). @wilshipley will vouch for me, if that matters at all.
@jmah I'm sorry you're frustrated. I feel very badly about how things have been going this past week, and I'm working nonstop to try to straighten it out.
Here's the full story of what happened starting with Arq 2.6.3:
There's a problem with some of S3's servers, particularly in the us-west-1 region, but also in other regions. Connections just get stuck. Customers were complaining heavily about Arq getting stuck. The Apple networking API that Arq was using had no facility for timing out; plus it was very hard to throttle properly. So I changed Arq to use Apple's older CFHTTP API.
In my haste to fix this customer problem, I didn't do enough testing on 2.6.3-4. I screwed up.
(In fact, I never intended to publish 2.6.3 to everyone. I was sending updates via email in response to several people's complaints of Arq getting "stuck". But one time when I ran my build script late at night I accidentally ran the part that puts it into the update stream. I didn't realize it until the next morning, and by that time I couldn't go backwards.)
There was a bug in 2.6.3-4 where some objects were marked as "compressed" but they weren't actually compressed. So Arq couldn't restore properly. So, since there's no way for me to tell which objects were compressed, Arq 2.6.5 ignores all backups made after March 7 by an Arq version < 2.6.5. I added a "version" to the backup data so that if this sort of thing happens again, I can fix it with a less drastic solution.
In addition, 2.6.5 fails on some transient network errors that I couldn't reproduce. But from customer log data I think I've tracked down all the possible error cases and Arq will retry for those transient errors. Arq 2.6.7 will have fixes for them, but I'm not going to release it until I'm comfortable that it works properly, probably within a few days.
So, please hang on for another day or two. I will publish a release when it's very stable, and it takes time to do enough testing to make sure that's the case. I also need to figure out a way to avoid accidentally publishing a release to the update stream.
By the way, downgrading isn't something I test, and Arq assumes that it's replacing an older version, not a newer version. If anyone else is considering going to an older version of Arq, please don't. Contact me here or via email at support@haystacksoftware.com. I'm pretty responsive via email.
@jmah Pick "Back Up Now". It should fix itself.
Lans
Member
@mrblack I think that depends on the version you have now. If you now use an older version of Arq, a version before 2.6.3, you should be OK with 2.6.5 already.
As Stefan pointed out: "due to a bug in versions 2.6.3 and 2.6.4, backups created after March 7 are bad and will be discarded by Arq 2.6.5 and later."
If you use 2.6.3 or 2.6.4, waiting on 2.6.7 is not going to solve the issue, because (Stefan quote):"There was a bug in 2.6.3-4 where some objects were marked as "compressed" but they weren't actually compressed. So Arq couldn't restore properly. So, since there's no way for me to tell which objects were compressed, Arq 2.6.5 ignores all backups made after March 7 by an Arq version < 2.6.5."
optolythic
Member
I'm having the same problem as some other people here. After updating to 2.6.5 it seems that Arq is uploading my whole folder instead of just the files updated/uploaded between March 7 and now. I've updated about 500mb of files in the last month, but the current data uploaded is 6GB and still rising now.
The list of currently backing up files doesn't reflect this though, because in this list only the 500mb of updated files show and nothing more. So it seems Arq is ignoring this list and uploading much more than it has to.
I've deactivated Arq agent for now and I'm hoping this issue will be resolved soon.
