Update Mobile UCR Versions for Domains (Script) #35158

zandre-eng · 2024-10-01T09:34:30Z

Product Description

No user-facing changes.

Technical Summary

Link to ticket here.

Please note: This PR should not be merged and is only created for review purposes. The script contained in this PR will be run as a once-off script.

This PR is the code for a once-off script that will be run to update the Mobile UCR versions of all domains from V1 to V2. This will only be done for domains that do not have any manual references to V1 fixtures. Domains that do contain V1 fixtures will be flagged and will need to be handled separately.

Feature Flag

MOBILE_UCR

Safety Assurance

Safety story

Local testing

Automated test coverage

N/A

QA Plan

No QA planned.

Labels & Review

Risk label is set correctly
The set of people pinged as reviewers is appropriate for the level of risk of the change

kaapstorm

Just three suggestions, nothing blocking. I am happy with your approach.

kaapstorm · 2024-10-01T09:56:33Z

corehq/mobile_ucr_v2_update_script.py

+V1_FIXTURE_IDENTIFIER = 'src="jr://fixture/commcare:reports'
+V1_FIXTURE_PATTERN = r'<.*src="jr://fixture/commcare:reports.*>'
+V1_REFERENCES_PATTERN = r"<.*instance\('reports'\)/reports/.*>"
+V1_ALL_REFERENCES = f"{V1_FIXTURE_PATTERN}|{V1_REFERENCES_PATTERN}"


Because you're using this pattern multiple times, I'd compile it.

... using re.compile() and saving the resulting regular expression object for reuse is more efficient when the expression will be used several times in a single program.
-- https://docs.python.org/3/library/re.html#re.compile

Suggested change

V1_ALL_REFERENCES = f"{V1_FIXTURE_PATTERN}|{V1_REFERENCES_PATTERN}"

RE_V1_ALL_REFERENCES = re.compile(f"{V1_FIXTURE_PATTERN}|{V1_REFERENCES_PATTERN}")

You can then use findall() or search() as a method:

RE_V1_ALL_REFERENCES.search(form.source)

This is useful to know, thanks! Addressed in 0392d0f.

kaapstorm · 2024-10-01T09:58:38Z

corehq/mobile_ucr_v2_update_script.py

+        save_in_log(log_file, f"Processing Form: {domain}: {form.name}")
+        # The second condition should always be False if the first one is
+        # but just as a precaution we check for it
+        if V1_FIXTURE_IDENTIFIER in form.source or re.findall(V1_ALL_REFERENCES, form.source):


findall() will return all matches, but we don't really care about all the matches, we just what to know whether any exist. If you use search() instead then it will stop searching as soon as it's found the first match.

Yeah this sounds like the better approach. Addressed in 0392d0f.

kaapstorm · 2024-10-01T10:02:56Z

corehq/mobile_ucr_v2_update_script.py

+    try:
+        processed_domains = read_ndjson_file(PROCESSED_DOMAINS_PATH)
+    except FileNotFoundError:
+        processed_domains = []


Nit: Better to use a set: If you're checking for membership of a group, a set will be O(1) but a list will be O(n). This applies to skip_domains too.

This is a good idea. Addressed in b3e4252.

gherceg · 2024-10-01T10:34:11Z

Excited to see this happening! I see there are some tickets (here and here) already created to default to V2, but I can't tell if those changes have been deployed/what the current behavior on production is. It would make sense to me to ensure we default to V2 prior to running this, so just making sure that is the plan or if not, there is a reason why.

zandre-eng · 2024-10-01T10:39:19Z

Thanks for sharing the concern @gherceg! I definitely agree this would be important to do beforehand and can confirm that we do have a plan outlined to first ensure that domains aren't able to go back to V1 before running the migration script from this PR. This also includes the related code changes from this Jira ticket to handle when users are reverting/publishing apps with V1 references.

mkangia

Non blocking comments.

Would be nice to have a -dry-run option to the script so it can be run without doing actual updates to see which apps it would update or not update eventually. I have always found that useful, specifically on production

mkangia · 2024-10-03T11:03:04Z

corehq/mobile_ucr_v2_update_script.py

+
+    mobile_ucr_domains = find_domains_with_toggle_enabled(MOBILE_UCR)
+
+    log_file = open(LOG_FILE, 'a')


I believe this would leave the connection to the file open even after the script exits abruptly, so we should this with with option. This way you also don't need to worry about explicitly calling close.
with open(LOG_FILE, 'a') as log_file:

Does mode 'a' create the file if its missing?

Looking through the code, I can remove the need for passing the log file around to the various helper functions. Since we already know what the file path is, we can just use this directly in the helper function for adding to the log. Addressed in 27c1c25.

Does mode 'a' create the file if its missing?

It does. Here is an article for more info on the append mode.

mkangia · 2024-10-03T11:04:43Z

corehq/mobile_ucr_v2_update_script.py

+        apps = get_apps_by_id(domain, app_ids)
+        for app in apps:
+            try:
+                # Don't look at app.is_released since the latest version might not be released yet


Confirming that we are just picking app and not releases.
I believe one way to confirm that is by checking copy_of property on the application. Is it blank for applications and not blank for releases. it would be a good check to have though I assume you are anyway only picking up applications and not releases.

Confirming that we are just picking app and not releases.

Could you please elaborate on what you mean by app vs release? Are you referring to app versions that have been marked as "Released" or are you referring to the app versions themselves?

This comment refers that we are picking the latest version of each app and we don't want to consider app.is_released since the latest version of an app might still be set to "Test" and not "Released"

mkangia · 2024-10-03T11:08:54Z

corehq/mobile_ucr_v2_update_script.py

+V1_FIXTURE_IDENTIFIER = 'src="jr://fixture/commcare:reports'
+V1_FIXTURE_PATTERN = r'<.*src="jr://fixture/commcare:reports.*>'
+V1_REFERENCES_PATTERN = r"<.*instance\('reports'\)/reports/.*>"
+RE_V1_ALL_REFERENCES = re.compile(f"{V1_FIXTURE_PATTERN}|{V1_REFERENCES_PATTERN}")


I trust this has been tested so it matches as needed.

These regex patterns come from this script which has been used to analyze all the domains with V1 references. Based on the results of the script that was run, these regex patterns have successfully picked up domains with V1 references.

These references were taken from this Confluence page, which shows what needs to change to migrate from V1->V2.

mkangia · 2024-10-03T11:11:46Z

corehq/mobile_ucr_v2_update_script.py

+skip_domains = set()
+
+
+def process():


nit: the flow of code in the file made it a bit hard to follow along. Just reminding on keeping things in a file in logical order. I am big fan on vertical formatting

I can see how this might be difficult with all the helper functions at the top. I've refactored this to have the main process function at the top instead eac2bcf.

zandre-eng · 2024-10-03T13:46:08Z

Would be nice to have a -dry-run option to the script so it can be run without doing actual updates to see which apps it would update or not update eventually.

@mkangia I think this is a great idea. I have added a dry run option in 20d9401.

zandre-eng added 2 commits October 1, 2024 11:26

implement script

5184652

logging for apps containing v1 refs

30a6a13

zandre-eng added the product/invisible Change has no end-user visible impact label Oct 1, 2024

zandre-eng requested review from kaapstorm, mkangia, Charl1996 and ajeety4 October 1, 2024 09:34

zandre-eng added the Open for review: do not merge A work in progress label Oct 1, 2024

kaapstorm approved these changes Oct 1, 2024

View reviewed changes

zandre-eng added 2 commits October 1, 2024 12:15

compile re expression

0392d0f

use sets instead of lists

b3e4252

kaapstorm approved these changes Oct 1, 2024

View reviewed changes

mkangia approved these changes Oct 3, 2024

View reviewed changes

mkangia reviewed Oct 3, 2024

View reviewed changes

zandre-eng added 3 commits October 3, 2024 15:14

refactor logging

27c1c25

reorganize script flow

eac2bcf

add option for dry run

20d9401

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update Mobile UCR Versions for Domains (Script) #35158

Update Mobile UCR Versions for Domains (Script) #35158

zandre-eng commented Oct 1, 2024

kaapstorm left a comment

kaapstorm Oct 1, 2024 •

edited

Loading

zandre-eng Oct 1, 2024

kaapstorm Oct 1, 2024

zandre-eng Oct 1, 2024

kaapstorm Oct 1, 2024

zandre-eng Oct 1, 2024

gherceg commented Oct 1, 2024

zandre-eng commented Oct 1, 2024

mkangia left a comment

mkangia Oct 3, 2024

zandre-eng Oct 3, 2024

mkangia Oct 3, 2024

zandre-eng Oct 3, 2024

mkangia Oct 3, 2024

zandre-eng Oct 3, 2024

mkangia Oct 3, 2024

zandre-eng Oct 3, 2024

zandre-eng commented Oct 3, 2024

	V1_ALL_REFERENCES = f"{V1_FIXTURE_PATTERN}\|{V1_REFERENCES_PATTERN}"
	RE_V1_ALL_REFERENCES = re.compile(f"{V1_FIXTURE_PATTERN}\|{V1_REFERENCES_PATTERN}")


		mobile_ucr_domains = find_domains_with_toggle_enabled(MOBILE_UCR)

		log_file = open(LOG_FILE, 'a')

		skip_domains = set()


		def process():

Update Mobile UCR Versions for Domains (Script) #35158

Are you sure you want to change the base?

Update Mobile UCR Versions for Domains (Script) #35158

Conversation

zandre-eng commented Oct 1, 2024

Product Description

Technical Summary

Feature Flag

Safety Assurance

Safety story

Automated test coverage

QA Plan

Labels & Review

kaapstorm left a comment

Choose a reason for hiding this comment

kaapstorm Oct 1, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gherceg commented Oct 1, 2024

zandre-eng commented Oct 1, 2024

mkangia left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zandre-eng commented Oct 3, 2024

kaapstorm Oct 1, 2024 •

edited

Loading