"Agent Error: deleted 1 head(s) due to missing objects in S3 arq"

Legacy Arq Forum » Backing Up and Restoring

"Agent Error: deleted 1 head(s) due to missing objects in S3 arq"

(65 posts) (17 voices)

Started 1 year ago by dansimau
Latest reply from afbroman

12 3 Next »

This legacy forum is read-only. Please use the new Arq forum.

dansimau
Member

When Arq was last running a check, I got the following error message in the main Arq window and log:

"Agent Error: deleted 1 head(s) due to missing objects in S3 arq"

Now none of my historial backups appear, but I'm still using 124.5GB of storage.

Can anyone shed any light on what has happened, and what I should do next?

Note that the storage I am using is reduced-redundancy, and I have set it up to email me if I ever lose objects (but I haven't had an email from Amazon as yet).

Thanks in advanced for your help.

Posted 1 year ago #
stefan
Key Master

Have you recently updated your version of Arq?
Is some other program doing things to your S3 data?
Did you ever use Arq version 2.0.x? I sent an email to all customers about a bad bug in that version that could cause data loss.

Posted 1 year ago #
dansimau
Member

I update Arq as soon as there is an update. The error appeared only a couple of days ago and I'm pretty sure 2.6.2 was released a while ago (so no, I haven't updated recently - relative to the error).

No other applications touch my S3 account.

When was 2.0.x released? I may have used it at some point (I can't remember when I purchased my Arq license) but definitely not recently.

Anyway, I deleted the entire S3 bucket to remove existing data and I'm doing a backup again.

Could this be caused by lost objects in S3 (ie. if Amazon loses objects because the storage is reduced redundancy)?

Posted 1 year ago #
jacobgardiner
Member

I've also just seen this error. It happened almost immediately after a new full backup on v2.7. This will be a bit annoying if I have to start fresh with a clean bucket.

I have at no point had v2.0 installed.

/me waves @ simmo

Posted 11 months ago #
jeffstearns
Member

This error also just happened for me. It occurred after an hourly incremental backup. The error messages in the console log suggest that the problem occurred during budget enforcement.

Nothing unusual was happening before the error; the computer has been operating just fine before, during, and after the problem. Only Arq has anything to complain about.

I am running Arq 2.8.2.

Arq's error message says that one missing block. Arq now says that *all* of my backup sets are gone. This is a significant problem.

Posted 8 months ago #
stefan
Key Master

I agree it's a significant problem. I apologize. I'm working on it.

Posted 8 months ago #
ChrisM6
Member

This same error occurred yesterday for me as well- similar to above posts, but Amazon indicates I am still using the amount of space equal to my backup, yet no backups visible...

Posted 7 months ago #
Robert
Member

The same error occurred yesterday for me as well. I am recently new user and started using Arq with 2.8.2 - so no issues with the older versions. For me, I backup two folders, one seems fine and one no longer has any of the backup sets.

Posted 7 months ago #
stefan
Key Master

I'm testing a fix for this. I will ship a 2.8.3 as soon as it's very stable and thoroughly tested.

Posted 7 months ago #
David
Member

I have this error as well, and nothing appears in the backup sets. Arq 2.8.2

Posted 7 months ago #
jvp
Member

FYI, I just hit the same error last night. Also on Arq 2.8.2. Before this, it's been pretty reliable. It's a bit disconcerting to lose your whole backup set in one event.

Edit: Now that I'm backing up again, it's clear the set isn't completely lost. It's moving relatively fast in terms of progress, but very little upload happening. I assume it's reconciling the backup data with what's stored in S3, and hopefully repairing the connection.

Posted 7 months ago #
stefan
Key Master

In the morning I'll post a very detailed explanation of what Arq does and what the error means (it's late Sunday night so I can't do it right now). Give me a few hours.

Posted 7 months ago #
jeffstearns
Member

Folks, I have a suggestion.

If you still have backup sets and if they're valuable to you, my recommendation is to immediately turn off Arq Agent, and leave it off until this problem is resolved.

I didn't do that, and I regret it. Arq has now failed a second time. It's now emitting complaints about missing blob trees, and trying to upload tens of GB of data. All of my historical backup sets are gone, and I certainly can't trust the newly-created backups with their complaints about tree errors. Leaving Arq running has probably made a bad problem worse.

Posted 7 months ago #
stefan
Key Master

Here's what Arq does:

Every 96 hours (by default) Arq "enforces your budget", which is actually several steps:
1. It runs a backup.
2. Then it checks to make sure every object referenced by your backups is still available in S3.
3. Then it drops the oldest backup versions until it gets under budget (or until there's only 1 backup version left, even if that's over budget).

When Arq finds a missing object in step 2, it drops that backup version and every backup version before it. This is pretty terrible behavior. I should have addressed it before, but it's really a case that should never happen. It should only happen if there's a bug in Arq (I'm pretty sure there isn't!), or if S3 drops an object (it's really not supposed to!).

The error message "deleted 1 head(s)…" indicates the last backup version (the "head commit") had a missing object, so Arq dropped that backup version and all the ones before it. The next time it does a backup, it will create a new backup version, but it won't re-upload the files that are already in S3.

I'm working on an update that replaces the terrible behavior with much better behavior. It will rewrite the backup metadata to show which files are missing, and keep the rest. It will show a warning icon next to all backup versions that have missing objects. Arq can't recreate the past, but the next time Arq backs up, it will of course upload missing files.

I'm going to put it out for beta testing as soon as I possibly can. It's code-complete, but I'm not done with my tests yet.

Again, I apologize for the terrible behavior. An improvement is coming very soon.

Posted 7 months ago #
samuel
Member

Just wanted to chime and say this happened to me today and it seems to have wiped out all my backups.

One thing I don't understand from your explanation: I'm not using RRS so it seems like files should be (and I quote) 99.999999999% reliable. Does this mean a file slipped passed this?

Posted 7 months ago #
samuel
Member

Just thought through it a bit more and realize what may have caused it:

- Arq started backing up in the morning. I was on wifi, so I quit it. It was only partially complete.
- Started Arq again in the afternoon.
- Hit by the bug.

Might it be related to these cases where a backup is only partially complete?

Posted 7 months ago #
stefan
Key Master

@samuel Thanks for that info. That's interesting. I'll try to reproduce it by quitting in the middle.

Regarding the 99.99999999% durability: it could be an object was lost by Amazon. I don't know. The Arq update will make it a lot clearer what's going on, and the rest of the data will still be accessible.

Posted 7 months ago #
David
Member

For now I have stopped Arq agent, waiting for the update. Since Arq shows no backups available at all I just want to wait for the new version before running any more backups. I still don't understand if the S3 is still out there (and being paid for) or what or if I'll have to upload the whole thing all over again, pay the transfer fees etc. I'm just confused about the state of the S3 store and worried about my Amazon AWS bill.

Posted 7 months ago #
David
Member

For the record, I didn't stop any backups or do anything different before this happened in my case. The Mac where it happened never sleeps and runs the backup every night at 1am. I suppose there could have been a network outage one night that I wasn't aware of.

Posted 7 months ago #
nickm
Member

I've seen this error multiple times with Arq 2.8.2. In each case, all my backups were wiped out. I use a laptop and probably frequently interrupt backups, but it's difficult to say.

Posted 7 months ago #
stefan
Key Master

@David Your files are still in S3, but Arq unfortunately dropped the newest backup record (the metadata) and all the ones before it because it couldn't find the newest backup record.

The next update won't have that terrible behavior. It will keep as much as it can, and mark missing things with an error icon.

Posted 7 months ago #
David
Member

So does that mean the S3 files are now "orphaned" and cannot be recovered? Do I need to do something to remove them?

Posted 7 months ago #
Robert
Member

@stefan. First, thanks for keeping us informed and working hard to resolve.

You said "I should have addressed it before, but it's really a case that should never happen. It should only happen if there's a bug in Arq (I'm pretty sure there isn't!), or if S3 drops an object (it's really not supposed to!). "

One thing I have just done is use S3 Notifications so that I am texted a message to my phone if an error is detected in the Arq bucket. The Amazon S3 text says "Enabling notifications causes a message to be published to an Amazon Simple Notification Service (SNS) Topic when Amazon S3 detects that a Reduced Redundancy Storage object stored in this bucket is lost."

This should help determine if S3 is experiencing a recent rash of lost objects or there is some subtle bug in Arq.

Posted 7 months ago #
dansimau
Member

I always had S3 set up for to notify me of lost objects and I never received a notification so I don't think it was "lost" objects (ie. lost in the sense that Amazon knows about it).

To be honest it could be a bug in Arq, network connectivity, S3 errors, etc., and while it would be great to track down the source of the problem, it's probably a pretty tough one to debug. It's more important that we can deal lost objects and errors gracefully (because they will happen at some point).

As a side note, other people are talking about S3's error rates here:
http://stackoverflow.com/questions/31466/does-amazon-s3-fail-sometimes

Dan

Posted 7 months ago #
stefan
Key Master

I finished Arq 2.9 with a fix for this problem. Instead of dropping backup versions that have missing objects, it rewrites the backup record to show which objects are missing, keeping everything else.

Would anyone be willing to beta test this for me? I'd really appreciate it.
Here's what to do:
1. delete your Arq app
2. download this one: Arq2.9
3. launch Arq
4. (optional, if you don't want to mess with your current backups) write down the "UUID" shown in the main window; pick "Change S3 Backup Set" from Arq's menu and configure a new backup set; add some folders for backup, etc
5. quit Arq
6. pick "Enforce Budget Now" from Arq Agent's menu and wait for it to finish
7. launch Arq and look for red error icons next to backup versions
8. pick "Back Up Now" from the menu and wait for the backup to finish
9. go to the AWS Management Console and delete an object (or several) from the "objects" folder within your computer's folder
10. pick "Enforce Budget Now" from Arq Agent's menu and wait for it to finish
11. launch Arq and look for red error icons next to backup versions
12. (optional, if you followed step 4) pick "Change S3 Backup Set" and choose your old backup set (with the UUID you wrote down in step 4)

Please post your results here or email support@haystacksoftware.com.
If no problems are found I'll put it in the update file for everybody.
Thank you!

Posted 7 months ago #
Lee Joramo
Member

I am currently testing the Arq2.9 fix. Some initial notes:

After installing 2.9 the Arq Agent menu reported that an update was available. I again quit the Agent and started the app. The agent no longer reported that an update was available.

Per the instructions, I quit the app.

From the Agent menu I selected Enforce Budget. The confirmation dialog indicated that it would begin a backup first and then do the Enforce Budget. I am not sure if this is doing an incremental backup or a full backup; however, at the rate that backup percentage is moving it looks like an incremental. Of course, I stopped backups 7 days ago as a precaution waiting for this fix, so it will take more time than normal.

I will report back when the Backup/Enforce Budget is complete.

Posted 7 months ago #
askbjoernhansen
Member

I haven't been running 2.8 since these issues came up on the forum (I was bitten by it in the spring sometime, too - wasn't fun), so Arq is uploading ~30GB new stuff this afternoon. I'll fuss with a new backup set and deleting some objects when the real backup has been updated.

Posted 7 months ago #
Lee Joramo
Member

UPDATE: It looks like Arq Agent was able to do an incremental backup. Since it had been a week since I had backed up, it took much longer than normal, but not long enough for a full backup. Arq Agent is currently running "Checking for missing RRS objects". Total time is now over 5 hours.

The amount of time this takes is no doubt related to the amount of data saved. I have Arq set to use 100GB of data on Amazon S3. Not sure if the RRS check is bandwidth limited.

Posted 7 months ago #
askbjoernhansen
Member

Mine stopped at "Validating S3 data". That is, the whole computer stopped. :-) I was alerted to something being wrong by a message about OS X not having room on the disk for more application memory. Looking at Activity Viewer arqcommitter was using 48GB of virtual memory (8GB of real memory).

A minute later while I was looking in the logs for more information it was just under 50GB virtual memory and the system locked up.

Posted 7 months ago #
askbjoernhansen
Member

I tried it again this morning after rebooting and the arqcommitter process (while "Validating S3 data") grows 5-15MB/second.

Posted 7 months ago #

RSS feed for this topic

12 3 Next »