Background:
Migrating from ZCS 7.2.7 on CentOS 5/x86. Exported mail with zxmig. No issues with the export.
Migrating to ZCS 8.6.0
a precise description of the issue
The initial rate of message import was reasonable; approx 10,000 messages per hour. It has degraded to about 1 thousand messages per hour.
Disk, CPU and RAM vitals appear fine (reasonable latency, plenty of CPU time, enough RAM to have a good size page cache [2GB]).
The import rate is slow, but consistent. Could there be some SOAP DOS prevention being triggered? I didn't see any evidence of this in the logs.
At the current rate of import, it appears the process will not complete for at least another 100 hours, which is unacceptable for my customer. I'm open to suggestions. I have been tuning the VM subsystem to try to alleviate disk load and keep the page cache large, but this appears to have no tangible effect. It seems more like lock contention, throttling or similar.
steps to reproduce the issue
NA
your server's specs (cpu, ram, storage etc...)
6GB of RAM (not great, but overlooked during provisioning and now can't interrupt the restore due to criticality of email to customer).
16 x E7340 Xeon cores.
Paravirt disk drivers (raw throughput reads ~280MB/s, writes ~27MB/s).
I/O latency is reasonable (1 second intervals):
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
xvda 0.00 0.00 72.00 3.00 2488.00 11.00 66.64 7.45 103.23 107.22 7.33 4.56 34.20
xvda 0.00 0.00 0.00 34.00 0.00 551.50 32.44 0.16 4.59 0.00 4.59 2.71 9.20
xvda 0.00 0.00 5.00 228.00 24.00 9952.00 85.63 5.00 20.91 8.00 21.20 0.74 17.30
xvda 0.00 0.00 72.00 17.00 3076.00 43.00 70.09 5.67 65.06 74.11 26.71 5.54 49.30
xvda 0.00 0.00 10.00 245.00 68.00 10078.50 79.58 2.91 11.41 8.30 11.54 1.82 46.40
xvda 0.00 0.00 1.00 6.00 4.00 19.00 6.57 0.05 7.29 0.00 8.50 7.29 5.10
CPU load is fluctuating, but around 80-100%.
your Zimbra version
8.6.0
your ZeXtras Suite/ZxMig version ( as the zimbra user run: "zxsuite/zxmig core getVersion" )
zextras_version 1.10.2
zextras_commit 5fa7c70acf715a78c825bdac8babbe223627b40b
zal_version 1.5.2
zal_commit 5fa7c70acf715a78c825bdac8babbe223627b40b
your OS, its version and architecture
CentOS 7/x64.
the number of servers in your Zimbra infrastructure
1.
the number of mailbox server in your Zimbra infrastructure
1.
for GUI related issues, the browser you were using when experiencing the issue and its version
NA
for ZxChat Zimlet issues, the ZxChat Zimlet version
NA
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
xvda 0.00 0.00 72.00 3.00 2488.00 11.00 66.64 7.45 103.23 107.22 7.33 4.56 34.20
xvda 0.00 0.00 0.00 34.00 0.00 551.50 32.44 0.16 4.59 0.00 4.59 2.71 9.20
xvda 0.00 0.00 5.00 228.00 24.00 9952.00 85.63 5.00 20.91 8.00 21.20 0.74 17.30
xvda 0.00 0.00 72.00 17.00 3076.00 43.00 70.09 5.67 65.06 74.11 26.71 5.54 49.30
xvda 0.00 0.00 10.00 245.00 68.00 10078.50 79.58 2.91 11.41 8.30 11.54 1.82 46.40
xvda 0.00 0.00 1.00 6.00 4.00 19.00 6.57 0.05 7.29 0.00 8.50 7.29 5.10
CPU load is fluctuating, but around 80-100%.
your Zimbra version
8.6.0
your ZeXtras Suite/ZxMig version ( as the zimbra user run: "zxsuite/zxmig core getVersion" )
zextras_version 1.10.2
zextras_commit 5fa7c70acf715a78c825bdac8babbe223627b40b
zal_version 1.5.2
zal_commit 5fa7c70acf715a78c825bdac8babbe223627b40b
your OS, its version and architecture
CentOS 7/x64.
the number of servers in your Zimbra infrastructure
1.
the number of mailbox server in your Zimbra infrastructure
1.
for GUI related issues, the browser you were using when experiencing the issue and its version
NA
for ZxChat Zimlet issues, the ZxChat Zimlet version
NA