When performing backup operations, it is possible to experience mismatches in production data, in the backup itself, or between the two environments. In this article we are going to introduce you some of the most common situations that can occur, showing you how you can perform restore operations through the tools of Zextras modules.
In the first case we want to analyze, we have a situation with missing blob in Backup (only Metadata has been copied), while on DB/Store both Metadata and Blob are present, like in the scheme below:
To solve this problem the right command is the
backup doSmartScan used with the
deep true option.
The process could be I/O intensive, so to speed it up, if you already know the involved account, you can use the
backup doAccountScan command, even with the
deep true option.
In this situation, unlike the previous one, we end up with the file not present, often reported by the user that receives a “Missing Blob” error in the web interface:
It could bring to different sub scenarios with as many solutions:
- File is completely missing: in this case executing
backup doRestoreBlobscommand will restore the missing blob from the backup
- File is empty: in this situation it is necessary to manually delete the file in order to go back to a situation like the one in the first case and then apply the relative restore operation
This situation is similar to 1st case, but in this one what is missing in backup is Metadata. It could happen, for example, if something has damaged the account folder of the backup path.
Again, to solve this problem the right command is the
backup doSmartScan used with the
deep true option, and again to speed it up you can use the
backup doAccountScan command
“Unexpected BLOB”. Unexpected BLOBS often occurs when the purge operation deletes the Database rows but left the blobs in the store directory (maybe because they were opened or locked or unaccessible). Below we can see three different scenarios with the proper solution to be applied:
- It has been carried out a mailbox move using only the BLOB stage. You don’t have to worry, that’s correct. As soon as the mailbox move is completed, the orphaned blobs will be deleted.
- BLOB exists but on a different volume from the one to which the metadata points. In this case, the cause was probably an unaccessible file during a Volume to Volume or HSM operation. The right command for you is
powerstore doRemoveOrphanedBlobs. It will remove all the orphaned blobs found in the volume. If in doubt, run it using the
dry_runoption. That will only report the file that would be deleted.
- BLOB exists but the metadata has been deleted by purge. In this case, the cause was probably an unaccessible file during the
bulkDeleteoperation. You can solve it in the same way as in case 2.
- The blob is valid but the metadata in the database has been lost. Due to a corrupted table, or a manual operation on the MariaDB. For example, consider the situation where the message_345’s mail_item row has been deleted. The blob and the backup data are valid. Running the command
backup doItemRestore 345will create a new Item, eg 900, using the metadata and blobs from the backup. It will also create new metadata in the backup, so It will be necessary to delete by hand the previous blobs (345-123.msg) and backup metadata orphaned (345) by the corresponding BLOB.
In this case we have missing both Metadata and BLOB on the Database/Store:
To solve this situation, all you have to do is to execute the command
zxsuite backup doItemRestore which will restore both the missing blob and metadata.
In this case, unlike the previous one, Metadata and BLOB are both missing in the backup:
Here we can solve the problem by performing zxsuite
backup doSmartScan start. It will execute the backup of both metadata and blob.
Please note that if the data is not recent, it will be necessary to launch
deep trueoption enabled
This is a particular case where blob is not on the backup nor on the database/store. In this situation, doing a coherency check will always return an error:
This case is not solvable but can be mitigated. All you have to do is create a fake blob using the command
zxsuite backup doSmartScan start create_fake_blob true
Pay attention that in this case the command
zxsuite powerstore doCheckBlobs startwill still return an error, because the blobs will still be missing from the store.
As an alternative, you can delete the orphaned metadata using the command
zxsuite powerstore doCheckBlobs missing_blob_delete_item true
This case, similar to the previous one, shows you a situation where metadata is not on the backup nor on the database/store. In this situation, doing a coherency check will always return an error:
At this point the only operation you can perform is the manual deletion of the blob corresponding to the orphaned metadata. As an alternative, you can import the blob file with the command
zmmailbox -z -m email@example.com addMessage /inbox/subdir /path_to_blobfile. This will create a new metadata linked to the imported blob file.
The following two cases refer to situations where we operate on an offline backup (eg an additional copy or an export of the backup_path).
In this case the backup path contains the metadata but doesn’t contain the blobs, for example due to incomplete sync or a corrupted items folder. In this case, the metadata’s states could refer to blobs – using the blob digest – that are no more available:
The command to solve this situation is the ChoerencyCheck with the
fixBackup option. There can be 3 sub-cases:
- Only some state refers to unavailable blob. In this case the command remove the corrupted states
- All metadata states refer to unavailable blob. In this case the command generate a “fake” blob using the information stored in the metadata
- Only the last digest is missing, In this case the command remove the corrupted states
The command will also move a corrupted copy of the metadata into the
In this last case, the backup is missing of item metadata:
Also in this situation you can make the backup consistent by executing the following command:
zxsuite backup doCoherencyCheck fixbackup true
This will move blobs that are not referred by any items in a dedicated orphaned folder