Plan for replacing an existing Leaf release with a new version

[artgoldberg]

Hello Leaf eaters

We're deploying a new production release of Leaf. We're implementing this by creating a VM, configuring the new release in the VM, and then changing DNS to point our publicly known Leaf domain name (leaf.mssm.edu) to the new VM.

The final phase of this process will replace our production Leaf (release 3.9) with the new release (3.11). This phase will be done while Leaf is down for maintenance. It must be prepared for the possibility that the new release does not work, in which case the existing release must remain in production. I've developed the plan below for this phase. Please let me know if the plan makes mistakes, forgets important steps, or has other problems.

1. Leaf maintenance period starts
2. Shut down services (leaf_api httpd shibd) for Leaf 3.9 production
3. Shut down services for Leaf 3.11 development
4. Point DNS for leaf.mssm.edu to the Leaf 3.11 VM
5. Make a backup copy of the production Leaf database
6. Migrate the production Leaf database to 3.11
7. Make other changes needed by 3.11 in the production Leaf database
8. Put the production Leaf database in Leaf Maintenance mode
9. Configure leaf_api.service.conf for Leaf 3.11 to use the production Leaf database
10. Copy the logs (Leaf and Apache) from the Leaf 3.9 VM to the Leaf 3.11 VM
11. Start Leaf 3.11 services
12. Have some admin users with various clients test Leaf 3.11
13. If the system is working well, make Leaf 3.11 publicly available by setting ServerState.IsUp true in the production Leaf database
14. Announce the new release

If these steps fail at some point and the new release cannot be deployed during the scheduled maintenance period, then Leaf 3.9 must be put back into production. These steps would be necessary then.

1. Restore the backup of the Leaf database as the production Leaf database
2. Restore Leaf 3.9 as the production server by pointing DNS for leaf.mssm.edu to the Leaf 3.9 VM
3. Start services for Leaf 3.9 production
4. Test Leaf 3.9 production
5. Announce that maintenance has ended and that the new release will be deployed later

Can you think of any steps that are wrong, out of order, or missing?

Thanks
Arthur

[ndobb]

@artgoldberg these steps make sense to me. 2 questions:

1. Do you have a working Leaf v3.11 dev instance whose features behave as expected?
2. How are you planning to copy the LeafDB data over to the new SQL Server? (eg doing a backup/restore, ETL'ing table by table, row by row, etc)

Best,
-nic

[artgoldberg]

Thanks for your response and questions @ndobb

1. Yes, I have a "working Leaf v3.11 dev instance whose features behave as expected", except that it is using a development version of the clinical database which differs substantially from the production version. Before I execute the final phase of the upgrade described above, I will connect the Leaf v3.11 dev instance with the production instance of the clinical database and a copy of the production instance of the LeafDB to further verify that 3.11 behaves as expected (as best as one can testing the GUI for a while).
2. This upgrade does not involve a new SQL Server. (I could have been more explicit about this above.) We have production and development instances of the clinical data warehouse (which follow the omop/OHDSI CDM standard). These host production and development instances of the Leaf database, respectively.

Expanding on my steps 1 - 12 above, we will:

1. Take Leaf 3.9 offline so its database stops being updated
2. Shutdown services for Leaf 3.9 so its logs also stop being updated
3. Shutdown Leaf 3.11
4. Point the DNS for our production Leaf site to the VM running Leaf 3.11
5. Make a backup copy of the production Leaf database that has been used by Leaf 3.9; this is on the production SQL Server. The backup is needed in case the 3.11 deployment does not work and we need to rollback to 3.9.
6. Using all the Leaf database migration steps between from release 3.9 to release 3.11, migrate the production Leaf database that has been used by Leaf 3.9 so it is ready for Leaf 3.11
7. Our Leaf 3.11 release is using the Authorization.Mechanism: APPDB, so I will add admin users to the migrated production Leaf database who will help me test the 3.11 release before I make it available to the public
8. Put the production Leaf database in Leaf maintenance mode
9. Configure leaf_api.service.conf for Leaf 3.11 to use the production Leaf database
10. Set the logs (Leaf and Apache) in the Leaf 3.11 VM aside, and copy the logs in the Leaf 3.9 VM to the Leaf 3.11 VM; the logs need to be copied so that we have a continuous log history on the production VM
11. Start Leaf 3.11
12. Have some admin users with various clients and queries to run test Leaf 3.11

I believe these steps will give our users continuity (except that we have a known bug in our saved queries) as we upgrade from release 3.9 to release 3.11.

Your thoughts?

Thanks
Arthur

[artgoldberg]

Hi @ndobb . I'm checking on whether I've answered your questions. Our release is tomorrow Tues., Aug. 9.
Thanks
Arthur

[ndobb]

Got it, thanks for the detailed explanation, Arthur. (also apologies for my delayed reply; I'm taking time off work for half of each week for the remainder of the summer).

I'm trying to think of edge cases or possible areas where you might run into trouble, but this seems like a solid plan. The only possible issue I can think of is that if you have different Leaf JWT signing certs/keys on the respective v3.9 and v3.11 VMs, the new v3.11 Leaf API will not recognize previously issued user tokens and will reject them. However the web client should simply request a new token without users needing to clear their browser caches, etc., so I think this shouldn't be a problem.

[artgoldberg]

@ndobb Thanks! I'm using the same Leaf JWT signing certs/keys, so that should be fine.