Legacy Arq Forum

This forum has been superseded. Please to go support.haystacksoftware.com

This legacy forum is read-only. Please use the new Arq forum.

1. 

   tharpold
   Member

   "I'm having the same problem as some other people here. After updating to 2.6.5 it seems that Arq is uploading my whole folder instead of just the files updated/uploaded between March 7 and now..."

   Whatever it is that happened in the last couple of versions of Arq, the results can be, well, terrible. I was mostly away from my computer after upgrading to 2.6.5, and didn't catch what was happening as it went merrily on its way: *every* backup I had for the last 18 months was wiped out, as Arq (I can only guess here) reuploaded fresh copies of nearly everything. Arq then trimmed the data to within my storage budget (100 GB).

   I now have no backups prior to March 7. Nothing. 18 months of incremental backups wiped clean, leaving only incremental backups for March 7, 28, 29, 30, and 31. My guess is that the same thing has happened on my spouse's machine, but I'm not home to be able to check that or stop the slaughter. I expect our Amazon S3 bill this month to inflate accordingly.

   That's, er, pretty irritating. Not catastrophic: Arq is one of my backup-of-backup methods. I have Time Machine backups for all of this data which are probably sound for this period. And it's unlikely that I'll need access to a version of a file changed umpteen times since last October. But I'll confess that I'm a little less confident of the Arq/S3 model than I was before this AM.

   Posted 1 year ago #
2. 

   cdchase
   Member

   Just wanted to Stefan and everyone some *good* news: After being patient.... VERY patient for the re-sync to occur, everything is back to normal. Yeah! I have over 140GB of data synced, so you can understand what I mean by VERY patient. it took a long time, but it eventually completed. And at the end, it did the 'budget trim' and brought my S3 storage down to a little more that I remember it having been, but that's expected, as I had added some items lately.

   As everyone should do occasionally, I did a test Restore from each of my backup sets for three different dates, the oldest in the set, the newest in the set, and March 6. :-) All where fine for the random files I used, both binary & text-based.

   While it may make you nervous, be assured that Stefan has done a good job keeping our data safe. Thanks!

   FYI, this was with Arq v2.6.5.

   -Dan Chase

   Posted 1 year ago #
3. 

   jmah
   Member

   My backups finally ran to completion, and my budget enforced. Like tharpold, I now have no backups before 2012-03-07; the first full backup performed by 2.6.6 seemed to treat everything as modified, triggering a large upload which caused almost all old backups to be wiped out. Or something like that.

   Stefan, when you said, "If so, Arq will reuse that S3 data and it shouldn't take so long", that's not always true. "View Items Being Backed Up" only showed items changed since the 2012-03-07 backup, but after those complete the progress bar disappears and the entirety of the folder is re-uploaded.

   Posted 1 year ago #
4. 

   optolythic
   Member

   Yes, exactly like jmah said. I've just started Arq again, just to see what really happens. First it shows all the right files (changed/added since 07-03-2012) in the "View Items Being Backed Up" and then shows the X MB from 888MB uploaded. But after a while the complete progress bar disappears and all my files start re-uploading. There is no progress bar anymore, but it just says X GB uploaded. The "View Items Being Backup Up" still shows the right files though, not all my files.

   A "solution" would be to just let it run the whole backup again. Some people here have done it, and with success. But my concern is that Stefan hasn't said anything about this being a known bug. He said the only problem was that everything from March 7 until the update had to be re-uploaded, but nothing about all data being re-uploaded. So if this problem isn't acknowledged, it probably still exists and therefore possibly can happen again. And that's what I'm personally afraid of. Uploading all my data again is an option, but only if I'm certain I only have to do it once. If this was a one time fluke.

   Posted 1 year ago #
5. 

   steve3001
   Member

   Had been hoping it would sort itself out but have been watching and waiting for a few days with no luck.
   I have downloaded and installed 2.6.6 (following the advice above) AND been an Arq owner since end of last year.

   Anyhow it now tells me it is backing up my media folder (which is huge but has not changed) yet when I inspect the log then Arq is not working on files in this folder. It appears from the log to be working on the correct files. However, it is not managing to make backups of these as often as it used to (set to hourly). There is often a day or more now between backups.
   This is a massive shame as I had been using Arq as sort of version tool.

   Really not sure what is being backed up now.

   Nor how much this is going to cost me in upload/download fees?

   And no progress bar , and it does't display properly the top part of the Arq window.

   Posted 1 year ago #
6. 

   optolythic
   Member

   Stefan, could you please reply in this thread? Should the people who have these problems upload all previous data again and will it be fixed then? Was/is there some kind of bug in the current version which results in this behavior? What should we do to keep our data safe again?

   Posted 1 year ago #
7. 

   steve3001
   Member

   Seconding optolythic's request

   Posted 1 year ago #
8. 

   paulnorman
   Member

   I have lost all my backups prior to build 2.6.8 i can live with this as I have my time machine backups and only started using ARQ in Jan this year. But all now seems to be working ok. I can only hope all runs smoth wth ARQ from now on.Im sure Stefan is working very hard to make sure everything will be fine for all of us after all you do not get this kind of service and support from any other setup. I think stefan deserves a medal for this most other setups would just fob you off.

   Posted 1 year ago #
9. 

   stefan
   Key Master

   @optolythic @steve3001 Sorry, I thought I had explained it pretty thoroughly already in this thread.

   Please update to Arq 2.6.8 (pick "Check for Updates" from the Arq menu). Then please wait patiently until it's done. It will upload what it needs to upload.

   There was a bug in 2.6.3. It's fixed now. I explained the full history of the problem in a blog post here.

   Posted 1 year ago #
10. 

    optolythic
    Member

    Your explanation in this thread and now also the explanation in the blog post explains why all data uploaded/updated after March 7 needs to be re-uploaded. I get that, you can't see which files are indeed compressed and which files aren't, so everything must be uploaded again. So I understand the problem and the solution. About one month of uploads/updates needs to be re-uploaded, no problemo.

    What I and other people have been talking about is that ALL DATA is being re-uploaded again. I use Arq to backup manually, once a day. So I know exactly which files I have added/changed and which files I haven't touched since March 7. Arq seems to recognize those files and starts uploading about 500MB of files at first (and also shows them in the backup now list), but after a while it seems to ignore these files and Arq just starts uploading EVERYTHING in all folders. So also whole directories of files which I haven't touched for months. After about 6GB of uploading I decided to stop it, so I could figure out what was going on...

    That is what I have been trying to do here. I wanted to know why it is also re-uploading files from way before March 7. Is it a side effect from the faulty update? Or is some new bug which you should be made aware of? Bottom line, we just want to know why it is doing that?

    Every time you reply the bottom line is "it needs to re-upload all files which where changed after March 7, just be patient". We reply "Yes, but why is it uploading files we haven't touched in months?" And you reply with the same answer, which isn't satisfying. :(

    Arq is a really awesome program and I really don't mind that some bug accidentally screwed up some backups. I'm a programmer myself, so I know these things happen and I really don't blame you for anything. But that being said, I am a little bit annoyed that I can't seem to just get an answer to a question I have been repeatedly asking. We are talking about backups here, so people are trying to make sure everything is as it is supposed to be. It's a sensitive subject. :)

    So, I will try it one final time. Let's try it the multiple choice way:

    Question : Why is Arq also updating lots of data which wasn't even touched between March 7 and now? Sometimes resulting in having to upload ALL data again?

    Answers:

    A. Well, that's another weird side effect of the bug in 2.6.3. Some people will need to upload everything again, because of some funky setting. Yes, in some cases this can even mean ALL data ever uploaded. But don't worry, everything is fixed now...you guys upload ahead. :)
    B. Huh?! That is not something it is supposed to do! Let me see if another bug creeped in and I will let you guys know.
    C. I like bunny rabbits.

    Ok, what is it? A, B or C? :D

    Posted 1 year ago #
11. 

    stefan
    Key Master

    I tested on computers that had backups from several months ago. They were using Arq pre-2.6.5. I updated them to 2.6.3 and ran some backups. Then I updated them to 2.6.8. Arq 2.6.8 creates a new backup. Then it starts with the newest previous backup and "drops" each one until it finds one that is either created by Arq 2.6.5 or later. It makes that the "parent" of the newly-created backup.

    @optolythic I don't know your exact situation.
    I don't know exactly what backup data you had beforehand.
    I don't know exactly which versions of Arq you've used or not used.
    I don't know when the last time was that Arq ran its budget enforcement process in your case.
    Given all the things I don't know about you and what you've done, I can't possibly tell you exactly what Arq is doing on your computer right now. This is one of the drawbacks of providing software where I have no access to customer data. If you want to send me debug logging from Arq I will try to give you a more specific explanation of what it's doing.

    Posted 1 year ago #
12. 

    steve3001
    Member

    Hi Stefan
    Well downloaded 2.6.8 and so far so good. It's ignoring the old stuff and gone to the new.
    Fingers crossed all fixed.
    I do think Arq is great so really pleased that it looks like you have sorted this out.
    Thanks
    Steve

    Posted 1 year ago #
13. 

    optolythic
    Member

    @Stefan I didn't mean any disrespect. Arq is by far the best S3 solution on any platform, and believe me, I've tried them all. Your doing a great job and I hope to be using Arq for a very long time.

    I guess it was answer A, so whatever was wrong is probably fixed now. I will update to the latest version and just let it roll the full backup.

    Posted 1 year ago #
14. 

    askbjoernhansen
    Member

    I was using 2.6.2 and recently - in the last few weeks - upgraded to 2.6.5 (I think it was). Now after upgrading to 2.6.8 Arq is busy re-uploading by ~200GB backup, despite all directories in the backup set having backups from before March 7. Very frustrating.

    Also seems dumb/inconsiderate to just bulk delete all backups back to March 7. I understand there wasn't a programmatic way to tell which backups were bad, but at least Arq should have given me an option if I wanted to delete the maybe-bad backups or not. I could have looked in my logs/mental memory/whatever and determined that most of March was okay because I hadn't upgraded.

    I will change the log level to debug and send some debug logs if that helps determining why it's re-uploading so much.

    Posted 1 year ago #
15. 

    askbjoernhansen
    Member

    Another question:

    "There was a bug in those releases where some objects were marked as “compressed” but they weren’t actually compressed. So Arq couldn’t restore properly."

    Except for files compressed with the same compression system, couldn't Arq "just" (I realize nothing is "just") check if the file can be uncompressed and "guess" in those cases. That'd have been a way to let us preserve our snapshots from the last month, even if some of the files might not restore correctly.

    Posted 1 year ago #
16. 

    Plumeria
    Member

    Stephan: I'm running 2.6.8. Its resolved these issues in all folders except my iPhoto backup, where the size of the backup continues to grow. Also, when I attempt to calculate the size of S3 (Singapore), I still get the message 'The operation couldn’t be completed. Connection reset by peer'

    Posted 1 year ago #
17. 

    LilliK
    Member

    One more unhappy camper here. Had a similar problem: I've set up Arq to backup every night at 1 in the morning. Then I started to notice it was active in the morning. I was busy, so just hit pause. Then when I finally looked at it a few days a go, I panicked: the last full backup was from the end of March! Yikes! And Arq indeed was reuploading EVERYTHING, including years-old photos and files I haven't touched in a decade... 180 gigs of stuff. And it kept freezing, and I reinstalled it, and it finally got the whole backup done today.

    My question differs a little: I've set my monthly budget as $15, which so far has covered all files changed and new files getting uploaded. Now with this bug, ARQ reuploaded everything. Does that mean that I am getting a huge bill from Amazon? I don't quite understand how the budgeting works in a situation like this. But wouldn't seem fair to (re-)pay for storage I had already paid for earlier, right?

    Not a huge deal, it'll be a business expense, but would like to know. It's tax month, so money is tight.

    Posted 1 year ago #
18. 

    stefan
    Key Master

    @askbjoernhansen I actually tried that, but the problem was that it would restore a file that was compressed to begin with, and it would decompress it. Git repositories, for instance, have compressed files in them. So that didn't work.

    Posted 1 year ago #
19. 

    stefan
    Key Master

    @LilliK Arq will delete unreferenced S3 objects every 4 days when it enforces the budget. So at most, Amazon will charge you for the extra space for 4 days.

    Posted 1 year ago #
20. 

    jmah
    Member

    For what it's worth, my S3 bill for March wasn't any higher than it typically is (within about a dollar).

    Posted 1 year ago #
21. 

    DLo
    Member

    I figured I would copy what jmah's last post had...

    My S3 Bill February - $6.06
    My S3 Bill March - $7.21

    Not a big deal at all.

    Posted 1 year ago #
22. 

    fletch
    Member

    Just to add to the comments so far in case it helps. I'm now running 2.6.8 after experiencing similar issues and my whole backup set is still re-uploading.

    So far I've done 60GB so I'm half way through and it's really really frustrating. I suppose it does go to show the fragility of having your stuff "in the cloud" whizzed off somewhere without a fast connection to it.

    It's definitely reuploading things from way before 7th March. The progress bar has disappeared but the "items being backed up" list does seem to show show the correct files..

    Weird.

    Posted 1 year ago #
23. 

    askbjoernhansen
    Member

    @fletch it worked the same for me, but after uploading all 200GB (and growing the S3 bucket to ~370GB, not quite double but close) it's done.

    It only took about a week for me fortunately as I have a fast connection at work. Waiting to see what happens when the budget enforcement kicks in.

    The data set that was definitely 100% re-uploaded is an external "not always connected" disk, I wonder if that has anything to do with it. The other directories are relatively small and changes more, so I didn't notice if they re-uploaded everything, too.

    Posted 1 year ago #
24. 

    fletch
    Member

    Thanks for the feedback @askbjoernhansen :-) first time I've ben annoyed that I moved to an iMac! Not quite as easy to take into work! Bah.

    Posted 1 year ago #
25. 

    paulgrav
    Member

    I’m having some similar issues. It looks like Arq is re-sending **all** 120GB again, not just the stuff that changed since 7 March. Also “View Items Being Backed Up” is grossly under reporting the number items actually being sent; it doesn’t correspond with what I’m seeing in the logs. No progress bar either.

    A few days ago Arq managed to get 60GB uploaded, since then I think my iMac rebooted (don’t know why) overnight and Arq has forgotten about the 60GB and reverted to 0. Arrrgggg.

    The files that it’s sending the Amazon are files that live on an AFP mounted drive.

    The annoying thing is that I can see in the list of backups all of the files that Arq is currently re-sending.

    Posted 1 year ago #
26. 

    askbjoernhansen
    Member

    It's ~5 days since Arq finished the re-upload and my S3 Size is still roughly twice the size it used to be (~410GB now, all my data is about 200GB). I've tried picking "Enforce Budget Now" from the menu to see if that'd clear out the old objects, but nothing happens.

    Do I have to make the budget smaller first? I have it artificially "too high" because I had an early experience when I started using Arq where it deleted a bunch of old backups I didn't want deleted.

    Ask

    Posted 1 year ago #
27. 

    askbjoernhansen
    Member

    @paulgrav For me the files that were re-uploaded were on an external not-always-connected disk and I had the same experience.

    Posted 1 year ago #

RSS feed for this topic