Our zimbra installation is as follows:
Zimbra 8.0.4 OSE, ZeXtras 1.8.11, CentOS 6 x86_64, 17 Zimbra servers of which 9 are mailbox servers.
ZxBackup stores everything in a remote NFS export, mounted on all mailbox servers.
After an unexpected NFS server crash, we get the following error from daily Smart Scans on two of our mailbox stores:
This is an automated notification from ZxBackup about Smart Scan.
Operation Smart Scan failed.
Operation Id: bc7de78f-ece3-45bc-a83d-fd3382ffacad
Operation Host: zm-mbox-06.cloud.forthnet.prvScheduled operation
ZeXtras Version: 1.8.11
ZeXtras commit: a7c892d274217bc312945152e49013aadc8c2b35
Zimbra version: 8.0.4_GA_5737 20130524120036 20130524-1204 FOSS- exception -
Exception: unable to read file: /mail-backup/zm-mbox-06.cloud.forthnet.prv/accounts/b6f22268-19d6-4f0c-b189-75347639aecd/account_stat
at com.zextras.backup.ZEBackupMetadata.readStat ( ZEBackupMetadata.java:305 )
at com.zextras.backup.ZEBackupMetadata.assertStat ( ZEBackupMetadata.java:90 )
at com.zextras.backup.ZEBackupMetadata.getStatField ( ZEBackupMetadata.java:47 )
at com.zextras.backup.ZEBAccount.getLastSequence ( ZEBAccount.java:1892 )
at com.zextras.op.backup.ZEScanOperation.doAccountScan ( ZEScanOperation.java:392 )
at com.zextras.op.backup.ZEScanOperation.doDomainScan ( ZEScanOperation.java:105 )
at com.zextras.op.backup.ZESmartScanOperation.doOperation ( ZESmartScanOperation.java:220 )
at com.zextras.op.ZEOperation.exec ( ZEOperation.java:652 )
at com.zextras.backup.ZEBackupThread.execOperation ( ZEBackupThread.java:225 )
at com.zextras.backup.ZEBackupThread.run ( ZEBackupThread.java:114 )
Caused by: com.zextras.lib.json.JSONException: Error read JSON file: /mail-backup/zm-mbox-06.cloud.forthnet.prv/accounts/b6f22268-19d6-4f0c-b189-75347639aecd/account_stat
at com.zextras.lib.ZEUtils.readJSONObject ( ZEUtils.java:450 )
at com.zextras.backup.ZEBackupMetadata.readStat ( ZEBackupMetadata.java:296 )
at com.zextras.backup.ZEBackupMetadata.assertStat ( ZEBackupMetadata.java:90 )
at com.zextras.backup.ZEBackupMetadata.getStatField ( ZEBackupMetadata.java:47 )
at com.zextras.backup.ZEBAccount.getLastSequence ( ZEBAccount.java:1892 )
at com.zextras.op.backup.ZEScanOperation.doAccountScan ( ZEScanOperation.java:392 )
at com.zextras.op.backup.ZEScanOperation.doDomainScan ( ZEScanOperation.java:105 )
at com.zextras.op.backup.ZESmartScanOperation.doOperation ( ZESmartScanOperation.java:220 )
at com.zextras.op.ZEOperation.exec ( ZEOperation.java:652 )
at com.zextras.backup.ZEBackupThread.execOperation ( ZEBackupThread.java:225 )
at com.zextras.backup.ZEBackupThread.run ( ZEBackupThread.java:114 )
Caused by: com.fasterxml.jackson.core.JsonParseException: Illegal character ((CTRL-CHAR, code 31)): only regular white space (r, n, t) is allowed between tokens
at [Source: /mail-backup/zm-mbox-06.cloud.forthnet.prv/accounts/b6f22268-19d6-4f0c-b189-75347639aecd/account_stat; line: 1, column: 2]
at com.fasterxml.jackson.core.JsonParser._constructError ( null:-1 )
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError ( null:-1 )
at com.fasterxml.jackson.core.base.ParserMinimalBase._throwInvalidSpace ( null:-1 )
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._skipWSOrEnd ( null:-1 )
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken ( null:-1 )
at com.fasterxml.jackson.databind.ObjectMapper._initForReading ( null:-1 )
at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose ( null:-1 )
at com.fasterxml.jackson.databind.ObjectMapper.readValue ( null:-1 )
at com.zextras.lib.ZEUtils.readJSONObject ( ZEUtils.java:446 )
at com.zextras.backup.ZEBackupMetadata.readStat ( ZEBackupMetadata.java:296 )
at com.zextras.backup.ZEBackupMetadata.assertStat ( ZEBackupMetadata.java:90 )
at com.zextras.backup.ZEBackupMetadata.getStatField ( ZEBackupMetadata.java:47 )
at com.zextras.backup.ZEBAccount.getLastSequence ( ZEBAccount.java:1892 )
at com.zextras.op.backup.ZEScanOperation.doAccountScan ( ZEScanOperation.java:392 )
at com.zextras.op.backup.ZEScanOperation.doDomainScan ( ZEScanOperation.java:105 )
at com.zextras.op.backup.ZESmartScanOperation.doOperation ( ZESmartScanOperation.java:220 )
at com.zextras.op.ZEOperation.exec ( ZEOperation.java:652 )
at com.zextras.backup.ZEBackupThread.execOperation ( ZEBackupThread.java:225 )
at com.zextras.backup.ZEBackupThread.run ( ZEBackupThread.java:114 )- debug info -
-- parameters --
dataPath = /mail-backup/zm-mbox-06.cloud.forthnet.prv/
OpStartTime = 1384056060420
origin = Cron
-- context --
numCheckedAccounts = 1
numPurgedItems = 0
lastAccountId = b6f22268-19d6-4f0c-b189-75347639aecd
failed = 1
numBackuppedAccounts = 3
numCheckedItems = 0
numDeletedItems = 0
numActiveBackuppedAccounts = 1
numNewItems = 0
numPurgedFiles = 0
numDeletedAccounts = 0
numPurgedAccounts = 0
numUpItems = 0
items4s = 1
completed = 1
numContentFiles = 0
numTotalAccounts = 2
Last progress info:
--
Account scanned: 0/2
Account updated/new: 0/0
Account skipped(disabled by COS): 0
Item scanned: 0
Item updated/new/unique: 0/0/0
Indeed, the file mentioned looks corrupted
[zimbra@zm-mbox-06 zm-mbox-06.cloud.forthnet.prv]$ hexdump -c accounts/b6f22268-19d6-4f0c-b189-75347639aecd/account_stat
0000000 037 213 b 355 033 k s 333 6
0000010 362 363 e &
0000014
[zimbra@zm-mbox-06 zm-mbox-06.cloud.forthnet.prv]$ ls -l accounts/b6f22268-19d6-4f0c-b189-75347639aecd/account_stat
-rw-r
1 nobody nobody 20 Nov 2 06:01 accounts/b6f22268-19d6-4f0c-b189-75347639aecd/account_stat
[zimbra@zm-mbox-06 zm-mbox-06.cloud.forthnet.prv]$
We tried doCoherencyCheck, doFixOrphans, doFullScan, they all fail..
Any thoughts on how can we fix this?
Is it safe to delete the whole accounts/b6f22268-19d6-4f0c-b189-75347639aecd/ directory?