@anahuac Looks like the site is down.... Thanks for all your good work.... Let us know if you need some hosting; we have a fair amount of bandwidth so can set you up for free..
@hodfords hello there... tyvm for the offer it's great to find people willing to help others 😀
My site is not down and it wasn't down. Maybe some connection issue?
I do have some GeoIP blocks on maybe you're being blocked by it?
Where are you from?
Telegram: https://t.me/CarbonioMail
From Hong Kong it shows that I'm blocked:-
FRom Vietnam I see this:-
Ok.. I just let Hong Kong to go =)
Also the URL changed to https://www.anahuac.eu/zimbra-to-carbonio-z2c/
I am trying to migrate from Zimbra OSE to Carbonio following the article https://community.zextras.com/how-to-use-script-to-migrate-zimbra-to-carbonio-carbonio-ce/
All has gone well until the end. When I go;
for i in `cat /opt/backups/zmigrate/emails.txt`; do zmmailbox -z -m $i -t 0 postRestURL "/?fmt=tgz&resolve=skip" /opt/backups/zmigrate/MBOX/$i.tgz ; echo "$i -- finished "; done
I get the following failure for each domain
ERROR: service.FAILURE (system failure: unable to read SQL statements from /opt/zextras/mailboxd/../db/create_database.sql)
Where did I go wrong?
Additionally to the above, whilst the accounts transferred over, they were not accessible to view. When you log in you can see a greyed over blank template with the message
When I check the message count with
:/opt/backups/zmigrate/MBOX$ for j in $( carbonio prov -l gaa | egrep -v "^(spam|ham)"); do total=0; echo -n "Total for $j = "; for i in $( zmmailbox -z -m "$j" gaf | awk '{print $4}' | egrep -o "[0-9]+" ); do total=$((total + i )); done; echo "$total"; done
Total for zextras@manually-added-domain.com = 28
Total for galsync.vkvk_udb@manually-added-domain.com = 2
Total for accounts@another-manually-added-domain.com = 8
wheras the imported domains
Total for admin@imported-domain.com = ERROR: service.FAILURE (system failure: unable to read SQL statements from /opt/zextras/mailboxd/../db/create_database.sql)
0
Total for virus-quarantine.utvishdp@another-imported-domain.com = ERROR: service.FAILURE (system failure: unable to read SQL statements from /opt/zextras/mailboxd/../db/create_database.sql)
I am tempted to try and delete the imported domains and manually add them but If I did that, I think the imported emails will get added to the wron account id.
any help would be gratefully received
As these problems are slowly resolving without any changes, they must be DNS cacheing issues
Thanks for your very valuable input! I started out using the “official” script method for migration but found that it doesn’t move quite some content I’d like to see on the new server, like lists and other settings, so I’d like to try with your method.
A few questions:
- Can I safely use the scripts when the accounts are already there on the target server, just to copy over additional settings? I also already recreated the aliases. I’m not familiar with ldapadd, so I don’t know if I might end up with doubled accounts.
- Why do you export Calendars and Contacts etc. separately? They are part of mailbox.tgz, anyway, aren’t they? To be able to import them separately if required?
- Will (with your method and/or the official script method) calendars and address books still be shared to others after the import or does that have to be recreated manually?
Thanks a lot, to you and also to Zextras for the great work!
Hello,
Answering your questions:
- Can I safely use the scripts when the accounts are already there on the target server, just to copy over additional settings? I also already recreated the aliases. I’m not familiar with ldapadd, so I don’t know if I might end up with doubled accounts.
Yes you can run it over it will not duplicate anything. But it also will not add anything new, meaning that once LDAP entried are done it will not be updated anymore. If you want to upgrade users and passwords and the things that are stored in LDAP you must delete it and start over.
The mailboxes restoring works adding new content because it uses the "skip" option, so you can tun it over and over and it will only add new data.
- Why do you export Calendars and Contacts etc. separately? They are part of mailbox.tgz, anyway, aren’t they? To be able to import them separately if required?
You got it. To be able to importe them separately.
- Will (with your method and/or the official script method) calendars and address books still be shared to others after the import or does that have to be recreated manually?
Shares are not exported or imported. So you have to recreate it manually on Carbonio.
You are very welcome... I'm glad my tools and articles are helping people to use Carbonio.
Just a heads up for everybody who is trying that, too:
I had the problem that mails were missing. That happened for every user, sometimes many (like 20,000 of 24,000 messages missing!), sometimes only a few. It happened even with small mailboxes, like one with 313 messages, 38 of which were migrated.
No error messages anywhere.
After digging a while, I found that the messages were already missing in the export on the Zimbra side. Strangely, the messages showed up when I switched from tgz format to zip format (in the /?fmt=tgz parameter, changed to /?fmt=zip). So I'm exporting again with ZIP format now.
I have no idea what the source of the problem is. But it might make sense to switch all the instructions to zip for more reliability.
As I can't edit anymore above, a new post:
With ?fmt=zip, &meta=1 must be added, because otherwise the metadata is left out and it's impossible to import the ZIPs, so /?fmt=zip&meta=1 altogether. Otherwise, there will be no error while importing on the command line, but in mailbox.log there is a Stream Closed exception from the Formatter, and nothing or only a few items are imported.
What an odyssey! I thought that max values for mail sizes were the problem when zip imports still didn't import everything, chose too high values, and afterwards mailboxd wasn't running correctly and I couldn't use zmprov anymore to reduce the values again. Not knowing where they are actually saved, I found the jetty config files with the values, which were overwritten when I restarted mailboxd, then found the .in files for them and could repair mailboxd by setting fixed values in order to be able to use zmprov again to set better values ... Puh.
Still, the count shows higher numbers for Zimbra than for Carbonio. That's trash and spam on the one hand, but that's not enough by number. As I couldn't find any item for my test user that was available in Zimbra but not in Carbonio, I'm just shrugging now and hoping that everything will be transferred when I now start it again for all users.
Here's a list of stuff I learned on the way that might be helpful to others who do the migration:
Stuff that will not be migrated:
- Mail aliases. Will be migrated only if you use anahuac's Z2C scripts.
- Distribution lists. Will be migrated only if you use anahuac's Z2C scripts.
- Tasks. Those are in the export, but will not be imported. Error in the logs: "misc - ArchiveFormatter addError:Cannot invoke "com.zimbra.cs.mailbox.MailItem.getType()" because "mi" is null: path=/Tasks/"
- Briefcase documents. Same error as with tasks with different path.
- Address book shares. There are still shares active after the migration, but they point nowhere, I think they use UUIDs from the Zimbra server that are not valid on the Carbonio server anymore.
- Calendar shares. Same as with address book shares.
- User-created mail filters.
- User-created personas (alternative from addresses).
- Contact lists created in iOS. Whatever format iOS uses to create these lists, they are not visible in UI (neither in Zimbra nor in Carbonio). The lists will be created during migration, but they will not be populated, i.e. contain no contacts. (Address books created in Zimbra/Carbonio will be synced to iOS as contact lists, and these stay intact, like the Emailed Contacts address book, e.g.. Not contact lists created in iOS, though.)
Potential problems:
- This might be a problem special to my Zimbra server that was originally created with Zimbra 6 and has seen various moves to other servers and OS updates, but: The script export method with fmt=tgz didn't contain all the data, in fact it was missing a lot of the data: Contacts, calendar items, emails. For one of my users, only about 4000 out of roughly 24000 objects were contained in the export, whereas for others, it was half or around 99% or anything in between. That was reproducible, i.e. the export always had the same size.
- When I used "/?fmt=zip&meta=1" instead of "/?fmt=tgz", everything was there. The "&meta=1" is important, because otherwise, metadata will be left out in ZIP format export (for zip, the default for meta is 0, but for tar and tgz, it is 1), and the result cannot be imported.
Other useful information that would have helped me, had I known:
- It is possible to import multiple times without creating duplicate data.
- Well, almost. Some contacts were duplicated, others weren't. There seems to be some attribute Carbonio uses for disambiguation that wasn't set for all of my or my users' contacts. For those contacts, the log says "mailop - adding contact null", and they will be duplicated every time the data is imported. In my case, this seems to have applied to contacts that were created in Zimbra very long ago, and to contacts that were created in iOS.
- Import errors that only apply to single elements will not lead to any hint that something has gone wrong on the command line.
- The import is logged in /opt/zextras/log/mailbox.log.
- PublicServiceProtocol must be set to https manually for all domains where user data is imported. This does not happen automatically during the domain creation process.
- The message counts will never match in real world scenarios because trash and spam folders will be counted, too, but not exported (though anahuac's Z2C scripts include one that exports trash). Also, I think that calendars and address books that were shared with the account are counted, too, that would explain the rather large differences I found for some users where I couldn't find any actually missing items on the Carbonio server after the import.
What I would have done differently, had I known what would happen:
I would have tested data export and import before I switched over the MX. :o) As it was, I imported all the users and domains to Carbonio, then switched the MX record, in order to transfer data when no mails would be arriving at Zimbra anymore. So I had the problem that when stuff went awry, I couldn't just roll back to a snapshot because that would have meant loss of emails that had arrived meanwhile. I just trusted it would work, but also didn't really think it through, TBH. 🙂
So, my advice is: Don't only import users and domains on Carbonio before you switch the MX, import everything so you can check if it works and if it is complete while you still can do rollbacks. If everything looks good, switch the MX to Carbonio, and after the DNS TTL has run out, export and import the data again.
Correction:
It seems that contacts that were duplicated in multiple imports were those that had no email address, it has nothing to do with iOS or old Zimbra. Additionally, contacts that were imported when I tried to import zip files that were created without meta=1 lead to duplicates (this import process didn't fail immediately, possibly only when the first email item was encountered).