Backup Mismatch Scenarios | Zimbra

When performing backup operations, it is possible to experience mismatches in production data, in the backup itself, or between the two environments. In this article we are going to introduce you some of the most common situations that can occur, showing you how you can perform restore operations through the tools of Zextras modules.

1st CASE

In the first case we want to analyze, we have a situation with missing blob in Backup (only Metadata has been copied), while on DB/Store both Metadata and Blob are present, like in the scheme below:

To solve this problem the right command is the backup doSmartScan used with the deep true option.

The process could be I/O intensive, so to speed it up, if you already know the involved account, you can use the backup doAccountScan command, even with the deep true option.

2nd CASE

In this situation, unlike the previous one, we end up with the file not present, often reported by the user that receives a “Missing Blob” error in the web interface:

It could bring to different sub scenarios with as many solutions:

1. File is completely missing: in this case executing backup doRestoreBlobs command will restore the missing blob from the backup
2. File is empty: in this situation it is necessary to manually delete the file in order to go back to a situation like the one in the first case and then apply the relative restore operation

3rd CASE

This situation is similar to 1st case, but in this one what is missing in backup is Metadata. It could happen, for example, if something has damaged the account folder of the backup path.

Again, to solve this problem the right command is the backup doSmartScan used with the deep true option, and again to speed it up you can use the backup doAccountScan command

4th CASE

“Unexpected BLOB”. Unexpected BLOBS often occurs when the purge operation deletes the Database rows but left the blobs in the store directory (maybe because they were opened or locked or unaccessible). Below we can see three different scenarios with the proper solution to be applied:

1. It has been carried out a mailbox move using only the BLOB stage. You don’t have to worry, that’s correct. As soon as the mailbox move is completed, the orphaned blobs will be deleted.
2. BLOB exists but on a different volume from the one to which the metadata points. In this case, the cause was probably an unaccessible file during a Volume to Volume or HSM operation. The right command for you is powerstore doRemoveOrphanedBlobs. It will remove all the orphaned blobs found in the volume. If in doubt, run it using the dry_run option. That will only report the file that would be deleted.
3. BLOB exists but the metadata has been deleted by purge. In this case, the cause was probably an unaccessible file during the bulkDelete operation. You can solve it in the same way as in case 2.
4. The blob is valid but the metadata in the database has been lost. Due to a corrupted table, or a manual operation on the MariaDB. For example, consider the situation where the message_345’s mail_item row has been deleted. The blob and the backup data are valid. Running the command backup doItemRestore 345 will create a new Item, eg 900, using the metadata and blobs from the backup. It will also create new metadata in the backup, so It will be necessary to delete by hand the previous blobs (345-123.msg) and backup metadata orphaned (345) by the corresponding BLOB.

5th CASE

In this case we have missing both Metadata and BLOB on the Database/Store:

To solve this situation, all you have to do is to execute the command zxsuite backup doItemRestore which will restore both the missing blob and metadata.

6th CASE

In this case, unlike the previous one, Metadata and BLOB are both missing in the backup:

Here we can solve the problem by performing zxsuite backup doSmartScan start. It will execute the backup of both metadata and blob.

Please note that if the data is not recent, it will be necessary to launch smartscan with deep true option enabled

7th CASE

This is a particular case where blob is not on the backup nor on the database/store. In this situation, doing a coherency check will always return an error:

This case is not solvable but can be mitigated. All you have to do is create a fake blob using the command zxsuite backup doSmartScan start create_fake_blob true

Pay attention that in this case the command zxsuite powerstore doCheckBlobs start will still return an error, because the blobs will still be missing from the store.

As an alternative, you can delete the orphaned metadata using the command zxsuite powerstore doCheckBlobs missing_blob_delete_item true

8th CASE

This case, similar to the previous one, shows you a situation where metadata is not on the backup nor on the database/store. In this situation, doing a coherency check will always return an error:

At this point the only operation you can perform is the manual deletion of the blob corresponding to the orphaned metadata. As an alternative, you can import the blob file with the command zmmailbox -z -m user@domain.ltd addMessage /inbox/subdir /path_to_blobfile. This will create a new metadata linked to the imported blob file.

The following two cases refer to situations where we operate on an offline backup (eg an additional copy or an export of the backup_path).

9th CASE

In this case the backup path contains the metadata but doesn’t contain the blobs, for example due to incomplete sync or a corrupted items folder. In this case, the metadata’s states could refer to blobs – using the blob digest – that are no more available:

The command to solve this situation is the ChoerencyCheck with the fixBackup option. There can be 3 sub-cases:

1. Only some state refers to unavailable blob. In this case the command remove the corrupted states
2. All metadata states refer to unavailable blob. In this case the command generate a “fake” blob using the information stored in the metadata
3. Only the last digest is missing, In this case the command remove the corrupted states

The command will also move a corrupted copy of the metadata into the corruptedMetadata folder

10th CASE

In this last case, the backup is missing of item metadata:

Also in this situation you can make the backup consistent by executing the following command:

zxsuite backup doCoherencyCheck fixbackup true

This will move blobs that are not referred by any items in a dedicated orphaned folder