Expand list of supported Transifex languages to push/pull in Makefile

[adamstankiewicz]

Transifex is a third-party service used to manage our translations for localization of the platform. Essentially, messages tagged for translations in a repo are extracted on an automated, recurring basis and pushed to Transifex. Translators can then translate those English messages to various languages. Generally, once a week, the i18n automation pulls reviewed translations from Transifex, which populates the JSON files containing translations (e.g., https://github.com/openedx/frontend-app-learning/tree/master/src/i18n/messages).

However, in many (most) repos, there are more JSON files to languages than are actually pulled by Transifex during the automation (aka we're not pulling in all the languages we seem to officially support).

Taking frontend-app-learning as an example, it has 13 JSON files for presumably 13 supported languages. However, because the transifex_utils is only configured for 4 of these languages, we are only pulling translations for 4 of those 13 languages, even if there are existing, reviewed translations for the other languages.

In the Makefile for frontend-app-learning:

transifex_langs = "ar,fr,es_419,zh_CN"

# Pulls translations from Transifex.
pull_translations:
	tx pull -f --mode reviewed --languages=$(transifex_langs)

Here is another similar example in frontend-app-profile.

We should ensure this issue of not including all supported language codes is consistent and fixed across all repos that integrate with Transifex.

[abdullahwaheed]

Should we pull all languages or specify the required languages?

[adamstankiewicz]

@abdullahwaheed Looking over some docs about Transifex here, it specifies:

If the languages you want to pull down differ from the default set, you can update the transifex_langs variable in your local Makefile.

I guess I'd be curious to know what the "default set" of languages is if transifex_langs isn't provided 🤔

Generally, we should ensure that we're pulling all the languages for which there are JSON files for in ./i18n/messages for each repo.

(from linked docs above) You must commit an empty language file for each language, or the pull job will fail.

If we started pulling all languages and don't have a JSON file for it created already in ./i18n/messages, will the job fail?

[adamstankiewicz]

@abdullahwaheed Speaking of, do you have the ability to access the Transifex jobs in Jenkins?

[adamstankiewicz]

@abdullahwaheed I believe Open edX officially supports the following 10 languages:

• Spanish (es-419)
• French (CA)
• Arabic
• Italian
• Portuguese (PT)
• German
• Hindi
• Mandarin
• Russian
• Ukrainian

However, it appears most repos have ~13 JSON files in their ./src/i18n/messages directories for each language code...

I'd propose we align towards the languages depicted in the linked blog post above since that was reasonably recent (as of this month):

1. Ensure each repo only has JSON files in ./src/i18n/messages for the above 10 languages.
2. Ensure transifex_langs also reflects the above 10 languages in each repo's Makefile.

I wonder if there is way to automate these changes in any way across all the repos with a custom script? 👀 For example, if we decide to add a newly supported language or remove a language, it might be nice to have an automation script to make future changes like this easier, too.

[abdullahwaheed]

@abdullahwaheed Speaking of, do you have the ability to access the Transifex jobs in Jenkins?

I can't access this Jenkins, I have to setup VPN probably for this.

[abdullahwaheed]

Repos Checklist:

• https://github.com/openedx/frontend-app-publisher
• https://github.com/openedx/frontend-app-gradebook
• https://github.com/openedx/frontend-app-admin-portal
• https://github.com/openedx/frontend-app-profile
• https://github.com/openedx/frontend-app-ecommerce
• https://github.com/openedx/frontend-app-account
• https://github.com/openedx/frontend-app-payment
• https://github.com/openedx/frontend-app-learner-portal-programs
• https://github.com/openedx/frontend-app-learning
• https://github.com/edx/prospectus
• https://github.com/openedx/frontend-app-program-console
• https://github.com/openedx/frontend-app-learner-portal-enterprise
• https://github.com/openedx/studio-frontend
• https://github.com/openedx/frontend-app-course-authoring
• https://github.com/openedx/frontend-app-library-authoring
• https://github.com/openedx/frontend-app-programs-dashboard
• https://github.com/edx/frontend-app-learner-record
• https://github.com/openedx/frontend-app-enterprise-public-catalog
• https://github.com/edx/frontend-app-communications
• https://github.com/openedx/paragon
• https://github.com/openedx/frontend-platform
• https://github.com/openedx/frontend-build
• https://github.com/openedx/frontend-component-header
• https://github.com/edx/frontend-component-header-edx
• https://github.com/openedx/frontend-component-cookie-policy-banner
• https://github.com/edx/frontend-component-footer-edx
• https://github.com/openedx/frontend-component-footer
• https://github.com/openedx/frontend-enterprise
• https://github.com/openedx/eslint-config
• https://github.com/openedx/stylelint-config-edx
• https://github.com/openedx/frontend-template-application

[abdullahwaheed]

@adamstankiewicz i had a discussion with "translations-working-group". They have mentioned that Chinese (zh_CN) is currently in use rather than Mandarin. Also, both French variants should be used i.e. fr and fr_CA. it makes around 11 languages.

Here is the thread of discussions

[abdullahwaheed]

As you have suggested, I am working on this script runner created by arbi-bom, it will need a script of needed changes and will create PRs with respective changes.

[adamstankiewicz]

@abdullahwaheed

I can't access this Jenkins, I have to setup VPN probably for this.

Yes, Jenkins is behind the VPN.

i had a discussion with "translations-working-group".

Great! Thanks for the link to the discussion thread. Yeah, it seems there's some discrepencies between what's supported based on documentation versus reality as implemented in code. If we can align both of these to be the same throughout docs and implementation, that'd be the ideal situation 😄

As you have suggested, I am working on this script runner created by arbi-bom, it will need a script of needed changes and will create PRs with respective changes.

Sounds good!

[abdullahwaheed]

@adamstankiewicz the script didn't work due to token so i have manually created PRs and updated there status here. I have marked somethings for Discussion in this sheet. There were some missing configurations and studio-frontend's translation pull gives 401 for old languages(ar,fr,es_419,zh_CN,). Need to contact owning squad regarding this.

[arbrandes]

Merged some, re-requested reviews on others. Some require rebasing and conflict resolution.

[arbrandes]

@abdullahwaheed, where do we stand, here? I see many were merged, but are we done?

[abdullahwaheed]

@arbrandes yes i think we are done with these changes

[arbrandes]

@abdullahwaheed, awesome, thanks!