RFE: Downgrade script from ianw1974 builds

[adriangibanelbtactic]

ianw1974 builds are made from develop branch which I will refer as zimbra-next.

Well, an script that eases a downgrade from zimbra-next to normal zimbra based on versions built on tags could be handy for admins that wish to switch to Maldua builds.

The script would accept a target Maldua version as an argument.

A rough draft of what the script would do:

• Have an array Maldua 10.0.8 ldap config attributes
• Dump installed Zimbra ldap config attributes
• Compare them to see what ldap config attributes are the new ones
• Ask if the user is sure
• Create ldap commands to remove the attributes
• Remove those attributes from ldap
• Advise the user to upgrade as soon as possible

Useful links:

• ngvannguyen's removal snippet
• zmcontrol's hacky method to downgrade

[adriangibanelbtactic]

@halfgaar I've just seen your script on the Zimbra forums.

Well, my implementation can be found here: tagged-zimbra-downgrade.sh.

Differences with your script:

• I do not enforce zimbraLDAPSchemaVersion change because I cannot find an explanation on why it's needed.
• I use more temporary files than you do.
• I embed 10.0.7 config (which I assume are the same ones as the 10.0.8 ones) inside the same script so that you only have to download one script.
• My script stops Zimbra before applying the change to ldap so that those values are not saved back into ldap while Zimbra is working there.
• My script asks if you want to restart Zimbra or not

This is still a work-in-progress and I pretend to add an option for -v10.0.7, -v10.0.8 and so on so that we can add more config values in the future.

I have also thought about:

• Adding an ldap dump
• Showing the actual changes that are about to be done

It would be nice if you could test the current version on your snapshot VM, here there is how you would use it:

sudo su - zimbra
cd /tmp
wget 'https://github.com/maldua/zimbra-foss-builder/raw/17d7736e7806bfa5358590f6e6ee70120805d2b1/tools/tagged-zimbra-downgrade.sh'
chmod +x tagged-zimbra-downgrade.sh
./tagged-zimbra-downgrade.sh

Thank you!

[halfgaar]

Yeah, I can test it. I made a backup of my snapshot pre-changes, for easy retrying. I'll get to it.

Your script is more elaborate than mine, nice work. I just put an hour of work in it when I saw one can source the ldap passwords into the env.

Some comments, relevant or not:

• zimbraLDAPSchemaVersion. I have not checked the script, but in general these versions are used by migrators to be deterministic. They look at it to determine which migration patches need to be applied, so it doesn't matter what your upgrade route is.
• It's not really a concern here, but remember the predictable temp file name exploits. mktemp is safer.
• Embedding the attrs is better of course. Your name print_10.0.7_config is a bit misleading though. It's the 'anti-config'.
• One thing that's hard to know/check, is what version somebody runs. Otherwise you can check that an installation is too new and the embedded attribute list is too old. I mean, had I installed from the dev branch today, new attributes may have appeared. There may be a way to detect that. One way is to embed all valid attributes, but that's huge (1792 in 10.0.8).

I really like the idea of a preview of changes, and/or making a backup first. I know people should have a full machine backup, but still, it's a nice addition.

Edit: oh, and I recommend shellcheck on shell scripts. Nice linter. It finds some issues.

[adriangibanelbtactic]

Yeah, I can test it. I made a backup of my snapshot pre-changes, for easy retrying. I'll get to it.

Your script is more elaborate than mine, nice work. I just put an hour of work in it when I saw one can source the ldap passwords into the env.

Some comments, relevant or not:

* `zimbraLDAPSchemaVersion`. I have not checked the script, but in general these versions are used by migrators to be deterministic. They look at it to determine which migration patches need to be applied, so it doesn't matter what your upgrade route is.

Hopefully that's the case.

* It's not really a concern here, but remember the predictable temp file name exploits. `mktemp` is safer.

Yeah.

* Embedding the attrs is better of course. Your name `print_10.0.7_config` is a bit misleading though. It's the 'anti-config'.

Well, I could rename it to get_10.0.7_config .

* One thing that's hard to know/check, is what version somebody runs.

Otherwise you can check that an installation is too new and the embedded attribute list is too old.
I mean, had I installed from the dev branch today, new attributes may have appeared.
There may be a way to detect that. One way is to embed all valid attributes, but that's huge (1792 in 10.0.8).

Each time I release a new version I should update add an get_10.0.8_config with whatever that is. Also documentation so that you now use --v10.0.8 when updating (or --v10.0.7 if nothing relevant has changed).

I really like the idea of a preview of changes, and/or making a backup first. I know people should have a full machine backup, but still, it's a nice addition.

Ok.

Edit: oh, and I recommend shellcheck on shell scripts. Nice linter. It finds some issues.

Nice to know.

[adriangibanelbtactic]

I'm currently working on a set of instructions that let you recover your borked system after ./install.sh fails because right now the only way of avoiding these problems is removing those additional attributes before ./install.sh.

I'm not sure I will be able to get away with it.

[adriangibanelbtactic]

So I have managed to fix this after the installation fails thanks to an updated tagged-zimbra-downgrade.

Here's how you use before I make public all of this in the main branch:

sudo su - zimbra
cd /tmp
wget 'https://github.com/maldua/zimbra-foss-builder/raw/dev_v051/tools/tagged-zimbra-downgrade/tagged-zimbra-downgrade.sh'
chmod +x tagged-zimbra-downgrade.sh
./tagged-zimbra-downgrade.sh
# Say 'y' to the question and press enter
exit

and, well, then you are supposed to restart the installation/upgrade (./install.sh).

Feedback is welcome!

[halfgaar]

My note about print_10.0.7_config actually missed the mark. What I meant didn't apply: you actually do a diff of the current, so that's always going to produce a list of attributes to be removed.

I tested the first one you asked (17d7736e7806bfa5358590f6e6ee70120805d2b1). I have a 10.0.6 install, but/and the installation did succeed. Still, having the accurate config list is preferable. And it can probably detect which version is required by asking the current Zimbra version. Otherwise your idea about --v10.0.6 works for me / most people.

Neither of our scripts touch /opt/zimbra/conf/zimbra-attrs-schema. I wonder if we should. Installing the 10.0.8 build did set it back to 1673397105.

The fixBorkedLdap fix I'll have to test later. It's interesting though, that grepping out lines with capitals ZIMBRA fixes it? I never saw such attributes.

[adriangibanelbtactic]

The fixBorkedLdap fix I'll have to test later. It's interesting though, that grepping out lines with capitals ZIMBRA fixes it? I never saw such attributes.

What I understand (without actually knowing too much about ldap) is that when the installation is done the Tagged Zimbra Ldap Schema is forced. Those capital letters attributes might be unrecognized attributes (instead of the usual non-capitalised letter ones) which are output as such (in capital letters).

However you cannot re-import them as such that's why you have to removed them.

[halfgaar]

Something else. Our scripts only modifies cn=config,cn=zimbra. But as an example new attribute, zimbraFeatureAllowUsernameInPassword also occurs here:

• dn: cn=default,cn=cos,cn=zimbra
• dn: uid=wiebe,ou=people,dc=halfgaar,dc=net
• dn: cn=defaultExternal,cn=cos,cn=zimbra

Do you think we need to go over all trees to look for these attributes? I'm assuming most if not all code will ignore attribute values for my user that don't correspond with a setting, but I can't be sure.

[adriangibanelbtactic]

Something else. Our scripts only modifies cn=config,cn=zimbra. But as an example new attribute, zimbraFeatureAllowUsernameInPassword also occurs here:

* dn: cn=default,cn=cos,cn=zimbra

* dn: uid=wiebe,ou=people,dc=halfgaar,dc=net

* dn: cn=defaultExternal,cn=cos,cn=zimbra

Do you think we need to go over all trees to look for these attributes? I'm assuming most if not all code will ignore attribute values for my user that don't correspond with a setting, but I can't be sure.

Why don't you set it to true in one account and you set it to false in another account, then upgrade and see what happens?

Also do those extra settings for those accounts disappear after the upgrade?

[halfgaar]

I tried making a slapcat dump after the upgrade:

$ /opt/zimbra/libexec/zmslapcat /tmp/zz
66566310 UNKNOWN attributeDescription "ZIMBRADOMAINLOGINPAGEENABLED" inserted.
66566310 UNKNOWN attributeDescription "ZIMBRAMODERNWEBCLIENTDISABLED" inserted.
66566310 UNKNOWN attributeDescription "ZIMBRAFEATUREMAILRECALLTIME" inserted.
66566310 UNKNOWN attributeDescription "ZIMBRAFEATUREMAILRECALLENABLED" inserted.
66566310 UNKNOWN attributeDescription "ZIMBRACOUNTACCOUNTSENABLED" inserted.
66566310 UNKNOWN attributeDescription "ZIMBRASOLRBATCHDELETIONINTERVAL" inserted.
66566310 UNKNOWN attributeDescription "ZIMBRAMAXSOLRBATCHDELETIONSIZE" inserted.
66566310 UNKNOWN attributeDescription "ZIMBRAEVENTINDEXREPLICATIONFACTOR" inserted.
66566310 UNKNOWN attributeDescription "ZIMBRAPREFSLACKCALENDARREMINDERENABLED" inserted.
66566310 UNKNOWN attributeDescription "ZIMBRAUSERTYPE" inserted.
66566310 UNKNOWN attributeDescription "ZIMBRATWOFACTORAUTHMETHODALLOWED" inserted.
66566310 UNKNOWN attributeDescription "ZIMBRADELAYEDINDEXINACTIVEACCOUNTAGE" inserted.
66566310 UNKNOWN attributeDescription "ZIMBRAFEATUREZULIPCHATENABLED" inserted.
66566310 UNKNOWN attributeDescription "ZIMBRATWOFACTORCODELIFETIMEFOREMAIL" inserted.
66566310 UNKNOWN attributeDescription "ZIMBRAFEATUREALLOWUSERNAMEINPASSWORD" inserted.
66566310 UNKNOWN attributeDescription "ZIMBRAFEATURERETENTIONPOLICYENABLED" inserted.

So, I think we do have to remove these.

The commands /opt/zimbra/libexec/zmslapcat -c and /opt/zimbra/libexec/zmslapcat -a don't give errors.

[adriangibanelbtactic]

I tried making a slapcat dump after the upgrade:

$ /opt/zimbra/libexec/zmslapcat /tmp/zz
66566310 UNKNOWN attributeDescription "ZIMBRADOMAINLOGINPAGEENABLED" inserted.
66566310 UNKNOWN attributeDescription "ZIMBRAMODERNWEBCLIENTDISABLED" inserted.
66566310 UNKNOWN attributeDescription "ZIMBRAFEATUREMAILRECALLTIME" inserted.
66566310 UNKNOWN attributeDescription "ZIMBRAFEATUREMAILRECALLENABLED" inserted.
66566310 UNKNOWN attributeDescription "ZIMBRACOUNTACCOUNTSENABLED" inserted.
66566310 UNKNOWN attributeDescription "ZIMBRASOLRBATCHDELETIONINTERVAL" inserted.
66566310 UNKNOWN attributeDescription "ZIMBRAMAXSOLRBATCHDELETIONSIZE" inserted.
66566310 UNKNOWN attributeDescription "ZIMBRAEVENTINDEXREPLICATIONFACTOR" inserted.
66566310 UNKNOWN attributeDescription "ZIMBRAPREFSLACKCALENDARREMINDERENABLED" inserted.
66566310 UNKNOWN attributeDescription "ZIMBRAUSERTYPE" inserted.
66566310 UNKNOWN attributeDescription "ZIMBRATWOFACTORAUTHMETHODALLOWED" inserted.
66566310 UNKNOWN attributeDescription "ZIMBRADELAYEDINDEXINACTIVEACCOUNTAGE" inserted.
66566310 UNKNOWN attributeDescription "ZIMBRAFEATUREZULIPCHATENABLED" inserted.
66566310 UNKNOWN attributeDescription "ZIMBRATWOFACTORCODELIFETIMEFOREMAIL" inserted.
66566310 UNKNOWN attributeDescription "ZIMBRAFEATUREALLOWUSERNAMEINPASSWORD" inserted.
66566310 UNKNOWN attributeDescription "ZIMBRAFEATURERETENTIONPOLICYENABLED" inserted.

So, I think we do have to remove these.

The commands /opt/zimbra/libexec/zmslapcat -c and /opt/zimbra/libexec/zmslapcat -a don't give errors.

You are going through the same experience I did.
In the end you might end up with dev_v051's tagged-zimbra-downgrade.sh which I linked you above ;).

[halfgaar]

I don't quite understand what state LDAP is in at that point. When I do this on my current live server (with zimbraFeatureAllowUsernameInPassword as example attribute):

ldapsearch -x -H "$ldap_master_url" -D "uid=zimbra,cn=admins,cn=zimbra" -w "$zimbra_ldap_password"  '(zimbraFeatureAllowUsernameInPassword=*)'

I get all dn entries in which this entry occurs. So, I thought I'd automate deletion in all dn entries, but my copied test server doesn't yield these dn entries anymore. Yet, the zmslapcat does error about them.

There may be a specific order they need to be deleted in. cn=config,cn=zimbra probably last. But, it's a bit of a guess. I'll have to revert my server copy to try.

[halfgaar]

So I have managed to fix this after the installation fails thanks to an updated tagged-zimbra-downgrade.

Here's how you use before I make public all of this in the main branch:

sudo su - zimbra
cd /tmp
wget 'https://github.com/maldua/zimbra-foss-builder/raw/dev_v051/tools/tagged-zimbra-downgrade/tagged-zimbra-downgrade.sh'
chmod +x tagged-zimbra-downgrade.sh
./tagged-zimbra-downgrade.sh
# Say 'y' to the question and press enter
exit

and, well, then you are supposed to restart the installation/upgrade (./install.sh).

Feedback is welcome!

I still haven't been able to fix the UNKNOWN attributeDescription "ZIMBRAxxxxxxxxxxxx", but its behavior is interesting. They don't show when making a zmslapcat after fixing ldap on my 10.0.6-dev. But, they appear after the installation of zcs-10.0.8_GA_4200000.UBUNTU18_64.20240422174845.

For me, when I run zmslapcat, the resulting files don't contain the ZIMBRAxxxxxxxxxxxx attributes (I just get warnings), so when I do the actions of your script minus the grep -v ZIMBRA, it also fixes it. In other words: just dump/reload.

Reinstalling zcs-10.0.8_GA_4200000.UBUNTU18_64.20240422174845 did not make the unknown attributes return. I'm confused. They appear to be coming out of nothing.