Add script to extract strings for translation

.gitignore

@@ -17,6 +17,9 @@ server/web/static/dist/
 logs/
 *.log
 
+# Ignore translation exports
+translations/
+
 #python
 venv/
 *pyc

.tx/config

@@ -1,7 +1,7 @@
 [main]
 host = https://www.transifex.com
 
-[tasking-manager-3.master]
+[tasking-manager.master]
 source_file = client/locale/en.json
 source_lang = en
 trans.ar = client/locale/ar.json
@@ -10,11 +10,11 @@ trans.cs = client/locale/cs.json
 trans.da = client/locale/da.json
 trans.de = client/locale/de.json
 trans.es = client/locale/es.json
-trans.fi = client/locale/fi.json
 trans.fa_IR = client/locale/fa_IR.json
+trans.fi = client/locale/fi.json
 trans.fr = client/locale/fr.json
-trans.hu = client/locale/hu.json
 trans.gl = client/locale/gl.json
+trans.hu = client/locale/hu.json
 trans.id = client/locale/id.json
 trans.it = client/locale/it.json
 trans.ja = client/locale/ja.json
@@ -34,3 +34,4 @@ trans.uk = client/locale/uk.json
 trans.vi = client/locale/vi.json
 trans.zh_TW = client/locale/zh_TW.json
 type = KEYVALUEJSON
+

Makefile

@@ -21,6 +21,9 @@ down:
 list:
 	docker-compose ps
 
+export-translation-file:
+	python3 devops/translation/generate_translation_strings.py
+
 tests:test-client test-server
 
 test-client:

devops/translation/README.md

@@ -0,0 +1,24 @@
+# Translation script
+
+These scripts facilitate the management of translations for the Tasking Manager.
+
+## Usual work flow:
+
+* Extract the strings from the Tasking Manager with `generate_translation_strings.py`.
+* Upload the file manually to Transifex.
+* Invite people to translate them.
+* Use the local `tx` client to pull down any changes.
+* Open a PR to submit the new ones added in.
+
+## Scripts
+
+* `generate_translation_strings.py` - Generates a JSON file of English translation strings
+
+## Howto extract strings
+
+* `pip3 install glob2` - install dependencies
+* Make sure you are in the root directory of the tasking manager and run the script: `python3 devops/translation/generate_translation_strings.py`
+* You get three files:
+  * `translations`/`export`/`en.json` (Strings for translation from the current code base)
+  * `translations`/`export`/`en_new.json` (Strings that are new)
+  * `translations`/`export`/`en_removed.json` (Strings that are not used anymore)

devops/translation/generate_translation_strings.py

@@ -0,0 +1,82 @@
+#!/usr/bin/python3
+
+# Generates a JSON file of English translation strings
+# Furhter it exports two JSON files containing only the new and the removed strings
+#
+
+import os
+import re
+import glob2  # Eventually I can move this off of glob2 and instead use os's dir walk
+
+# Source files containing translation strings
+files = glob2.glob('./client/app/**')
+files.append('./client/index.html')
+
+# Regular expression to detect translation strings
+translate_string = re.compile('\{\{\s+[\'\"]([^\|]*)[\'\"]\s+\|\s+translate\s+\}\}')
+
+# Create directory for translations
+try:
+    os.makedirs("translations/export")
+except FileExistsError:
+    pass
+
+# Assemble the data
+strings = []
+for f in files:
+
+    # Read in frontend files
+    try:
+        text = open(f, 'r').readlines()
+    except Exception:
+        continue
+    for line in text:
+
+        # Identify translation string
+        matches = translate_string.findall(line)
+        if len(matches) > 0:
+            flag = 1
+
+            # Add translation string to strings array
+            for match in matches:
+                if match not in strings:
+                    strings.append(match)
+
+# Write results to disk
+output = open('translations/export/en.json', 'w')
+output.write('{\n')
+for string in sorted(strings):
+    output.write('\t"%s": "%s",\n' % (string, string))
+output.write('}\n')
+output.close()
+
+# Read in the just generated export of strings
+output_new = open('translations/export/en.json', 'r')
+new = output_new.readlines()[1:-1]
+output_new.close()
+
+# Read in the strings from the code base
+output_old = open('client/locale/en.json', 'r')
+old = output_old.readlines()[1:-1]
+output_old.close()
+
+# Compare strings between old and new stage create diff files
+output_missing = open('translations/export/en_removed.json', 'w')
+output_new = open('translations/export/en_new.json', 'w')
+cn = 0
+co = 0
+for word in new:
+    if word not in old:
+        output_new.write(word)
+        cn += 1
+for word in old:
+    if word not in new:
+        output_missing.write(word)
+        co += 1
+
+# Write files with string comparison to disk
+output_missing.close()
+output_new.close()
+
+# Print out some stats
+print(cn, ' have been added, ', co, ' are no longer with is.', len(strings), ' were extracted in total.')

docs/contributing-translation.md

@@ -1,32 +1,48 @@
 # Contributions to the localization
 
-The Tasking Manager is localised using the Transifex. Transifex is included in the requirements.txt so you should be
-able to use the Transifex commands once you have set up your server side code. The Tasking Manager is using Transifex's
-[CLI client](https://docs.transifex.com/client/introduction/). Commands are documented there.
+## Translators
 
-Tasking Manager is using Angular Translate to display the translated strings. It works with key/value pairs in .json
-format and this is also the format that is used to store the translations in Transifex.
+The Tasking Manager is localised using our [Transifex repository](https://www.transifex.com/hotosm/tasking-manager/dashboard/).
+This is super easy. If you are interested click yourself an account and apply to join the `hotosm-translator` team.
+Everybody is welcome to support mapping through the Transifex website.
 
-Our [transifex repository](https://www.transifex.com/hotosm/tasking-manager-3/dashboard/).
+## Developers
+
+For developers Transifex offers a [CLI client](https://docs.transifex.com/client/introduction/) and the Tasking
+Manager is offering commands to interact with it. The client is already included in the requirements.txt so you should 
+have the Transifex commands installed, once you have set up your server side code.
+
+The Tasking Manager is using Angular Translate to display the translated strings. It works with key/value pairs
+in .json format and this is also the format that is used to store the translations in Transifex.
 
 ### Setting up Transifex locally
-https://docs.transifex.com/client/init
-To use Transifex client, you'll need a Transifex account. In the project top level directory, initialize Transifex
-service: tx init. The init process will ask for service URL (leave the default suggestion by hitting enter) and your
-Transifex username/password.
+
+To [setup the Transifex client](https://docs.transifex.com/client/init), you'll need a Transifex account and API key.
+In the project top level directory, initialize Transifex service: tx init. The init process will ask for service URL
+(leave the default suggestion by hitting enter) and your Transifex username/password.
 
 The .tx folder contains the Transifex config file. This is where you can find the mappings to local translation files.
 
 ### Pushing translations
+
 Use Transifex's ```tx push -s -t``` to push all local changes to Transifex.
+
 * Argument ```-s``` pushes source files (English in our case)
 * Argument ```-t``` pushes all translation files
 
 ### Pulling translations
+
 Use Transifex's ```tx pull``` to get all translations from Transifex.
 
 ### Adding a new language
+
 In this example we are adding support for German.
+
 * Add a new .json file with the appropriate language code as the name of the file, so in this case de.json.
-* Configure local mapping by using Transifex's set command: ```tx set -r tasking-manager-3.master -l de client/locale/de.json```
+* Configure local mapping by using Transifex's set command: ```tx set -r tasking-manager.master -l de client/locale/de.json```
 * Add the new language and language code to the config file so it shows up in dropdowns etc. in server/config.py
+
+### Extract language files from the Tasking Manager
+
+* `make export-translation-file` - This generates JSON files that can be uploaded to Transifex inside the `translations`
+directory.