[tor-commits] [Git][tpo/applications/tor-browser][tor-browser-153.6.0esr-16.0-1] 4 commits: fixup! Tor Browser localization migration scripts.

["henry (@henry) via tor-commits" <tor-commits@xxxxxxxxxxxxxxxxxxxx>]

Title: GitLab

henry pushed to branch tor-browser-153.6.0esr-16.0-1 at The Tor Project / Applications / Tor Browser

Commits:

• 0d103a1e
  by Henry Wilkes at 2026-11-03T15:03:48+00:00

  fixup! Tor Browser localization migration scripts.
  
  TB 45433: Change the script to be more versatile for test runs.
  
  Instead of passing in a python module name for the recipe, you now pass
  in the file path to the recipe script.
  
  You can now change the reference directory with the
  --reference-directory argument, which is useful for testing scripts
  before the changes have reached the translation repository.
  

• 7e33b574
  by Henry Wilkes at 2026-11-04T14:51:14+00:00

  fixup! Tor Browser localization migration scripts.
  
  TB 45433: Refactor the WeblateMetadata class.
  
  We are stricter about checking the return types from the weblate API,
  and making sure it doesn't make requests to unexpected URLs.
  
  We also limit the components to the browser components, and we slightly
  alter the `is_translated` public method.
  

• 90fb8c89
  by Henry Wilkes at 2026-11-04T14:51:16+00:00

  fixup! Tor Browser localization migration scripts.
  
  TB 45433: Add type hints to the script.
  

• fdcaef33
  by Henry Wilkes at 2026-11-04T14:51:17+00:00

  fixup! Tor Browser localization migration scripts.
  
  TB 45433: Update the migrate.py documentation.
  

1 changed file:

• tools/torbrowser/l10n/migrate.py

Changes:

tools/torbrowser/l10n/migrate.py

---

1	1	"""
2		-Migration script for running migration recipes.
	2	+Migration script for running Tor Browser migration recipes.
3	3
4		-Requires the mozilla fluent.migrate package (available with pip).
	4	+Based on the Firefox migration script, adapted for Tor Browser. See
	5	+https://firefox-source-docs.mozilla.org/l10n/migrations/ for more context and
	6	+information on how to write the migration recipes.
5	7
6		-NOTE: This will likely send queries to the weblate API as part of the run.
7		-Weblate may have a query limit of 100 requests per day for unauthorised users.
8	8
	9	+## Concepts
	10	+
	11	++ String. Here, when we say "string", it refers to a piece of text that can be
	12	+ translated. In practice, you can think of it as a being defined by a triple:
	13	+ a file path relative to a locale's directory, an ID, and some content. For
	14	+ example `("./tor-browser.ftl", "my-id", "my-id =\\n .label=Content")`.
	15	+ When a string is translated the file path and ID stay the same, whilst the
	16	+ translatable part of the content can vary.
	17	++ Reference files. These are the en-US source Fluent files that contain the
	18	+ *manual* string changes made by a developer. For most migrations, there is
	19	+ only one reference file.
	20	++ Migration recipe. A collection of instructions that take some string inputs
	21	+ and transforms them into new strings. These are meant to mirror the manual
	22	+ changes that were made in the reference files, in a way that can be applied
	23	+ to any locale.
	24	++ Translation repository. This refers to a local copy of the `translation`
	25	+ repository (https://gitlab.torproject.org/tpo/translation). This repository
	26	+ contains a different branch for different collections of strings. For
	27	+ Tor Browser we usually use `tor-browser`, `base-browser` and
	28	+ `fenix-torbrowserstringsxml`. Each branch will contain an `en-US` directory,
	29	+ which contains the "source" files for translators to translate. These source
	30	+ files are taken from the `tor-browser` repository. The branches also contain
	31	+ a directory for each locale that is to be translated, in a file structure that
	32	+ reflects the source `en-US` directory. This is where the migration script will
	33	+ write changes to the translated files.
	34	+
	35	+
	36	+## Usage
	37	+
	38	+You will likely want to run this with `./mach python`, so that it uses the
	39	+`fluent.migrate` and `fluent.syntax` packages provided with the `tor-browser`
	40	+repository.
	41	+
	42	+For example, you might run this as
9	43
10		-Based on the firefox migration script, adapted for tor browser. See
11		-https://firefox-source-docs.mozilla.org/l10n/migrations/ for more context and
12		-information on how to write the migration recipes.
	44	+```
	45	+ ./mach python migrate.py --translation-git /home/user/translation --locales 'es-ES sr' ./migrations/bug-xxxx-my-migration-recipe.py
	46	+```
	47	+
	48	+Here `/home/user/translation` is a local translation repository with one of its
	49	+branches checked out (see below). This will run the migration recipe given in
	50	+`./migrations/bug-xxxx-my-migration-recipe.py` for the `es-ES` and `sr`
	51	+sub-directories within the `translation/` directory.
	52	+
	53	+### Changing the reference location
	54	+
	55	+You can optionally run the script with `--reference_directory` to read the
	56	+reference en-US files from somewhere other than the `translation` directory.
	57	+This can be useful when your manual changes have yet to reach the translation
	58	+repository, so you want to point the script to read the reference from the
	59	+`tor-browser` working directory instead. E.g.
	60	+
	61	+```
	62	+ --reference-directory /home/user/tor-browser/toolkit/locales/en-US/toolkit/global/
	63	+```
	64	+
	65	+
	66	+## Recipes
	67	+
	68	+Migration recipes are meant to mirror some manual change that was made to the
	69	+en-US source strings, in a way that can be reproduced in another locale.
13	70
	71	+### Example 1
14	72
15		-Usage with an example:
	73	+For this example, a developer wants to remove the string content from
	74	+`old-strings.properties`:
16	75
17		-We have some en-US file "updated.ftl" in the tor-browser repository that we have
18		-added new strings to. Some of these strings are copies or concatenations of some
19		-already existing strings in a `.dtd`, `.properties` or `.ftl` file (including
20		-strings within "updated.ftl").
	76	+```
	77	+my.old.string=Some string content
	78	+my.old.title=This is the tooltip
	79	+```
	80	+
	81	+And add them to `tor-browser.ftl` instead, as:
21	82
22		-So we also write a migration recipe in
23		-`./l10n_migrations/bug-xxxx-my-migration-recipe.py`
24		-which includes a `migrate(ctx)` method. This method takes the old strings and
25		-create new Messages or Terms for "updated.ftl" using `ctx.add_transforms`. These
26		-should match what was already done manually in "updated.ftl".
	83	+```
	84	+my-new-string = Some string content
	85	+ .title = This is the tooltip
	86	+```
	87	+
	88	+For the en-US source strings, the developer simply deletes the two lines in the
	89	+`old-strings.properties` file (or the whole file, if these are the last two).
	90	+And they manually write in the content of `my-new-string` to `tor-browser.ftl`.
	91	+These manual changes will then eventually reach the `translation` repository to
	92	+be translated.
	93	+
	94	+But to save translators making the same manual changes, or re-translating, the
	95	+developer can write a migration recipe that mirror's their manual changes. In
	96	+this case, this migration recipe would contain:
	97	+
	98	+```
	99	+ctx.add_transforms(
	100	+ "tor-browser.ftl", # The path to write to.
	101	+ "tor-browser.ftl", # The path to the reference file.
	102	+ transforms_from(
	103	+ \"\"\"
	104	+my-new-string = { COPY(path, "my.old.string") }
	105	+ .title = { COPY(path, "my.old.title") }
	106	+\"\"\",
	107	+ path="old-strings.properties", # The path to read from.
	108	+ ),
	109	+)
	110	+```
	111	+
	112	+Note, the first argument of `add_transforms` is the target path, which is
	113	+`tor-browser.ftl` in this case. This means that the migration script will try to
	114	+write the resulting `my-new-string` to the locale's `./tor-browser.ftl` file.
	115	+For example, to `translation/es-ES/tor-browser.ftl`.
	116	+
	117	+The second argument refers to the *reference* path. In almost all cases the
	118	+reference file will have the same name and path as the target path, but it has a
	119	+different role. Here the path is relative to the reference directory, which
	120	+contains the en-US source files. In this case, by default this means that
	121	+`translation/en-US/tor-browser.ftl` will act as the reference, but the exact
	122	+path could be changed with the `--reference-directory` argument. The existing
	123	+`my-new-string` will be read from this file to confirm that the migration
	124	+recipe produces the correct result.
	125	+
	126	+The file `old-strings.properties` is used as an input. This means that the
	127	+existing strings in the locale's `old-strings.properties` file will be read to
	128	+construct the new string `my-new-string`. For example, it will be read from
	129	+`translation/es-ES/old-strings.properties`.
	130	+
	131	+The `COPY` acts like a Fluent function, which asks the transformation to copy
	132	+the content from the translated string directly.
	133	+
	134	+### Example 2
	135	+
	136	+For this example, a developer wants to swap a string in `tor-browser.ftl` from:
	137	+
	138	+```
	139	+my-old-string2 = Some string content
	140	+ .title = This is the tooltip
	141	+```
	142	+
	143	+to
	144	+
	145	+```
	146	+my-new-string2 =
	147	+ .label = Some string content
	148	+ .title = This is the tooltip
	149	+```
27	150
28		-Then we copy the en-US file to our local translation repository, say at
29		-`/home/user/translation/en-US/updated.ftl`. The en-US file is needed as a
30		-reference during migration to know where in the new file the new Messages or
31		-Terms should be placed. Moreover, the migration can be verified against the
32		-en-US file to ensure it is working as intended.
	151	+The developer removes `my-old-string2` from the file and adds `my-new-string2`
	152	+instead. This manual change will then eventually reach the `translation`
	153	+repository for `my-new-string2` to be translated.
33	154
34		-Then we run the migration for individual locales:
	155	+The developer also writes a migration recipe to mirror this change:
35	156
36	157	```
37		- python migrate_l10n.py --translation-git /home/user/translation --locales 'es-ES sr' l10n_migrations.bug-xxxx-my-migration-recipe
	158	+ctx.add_transforms(
	159	+ "tor-browser.ftl",
	160	+ "tor-browser.ftl",
	161	+ transforms_from(
	162	+ \"\"\"
	163	+my-new-string2 =
	164	+ .label = { COPY_PATTERN(path, "my-old-string2") }
	165	+ .title = { COPY_PATTERN(path, "my-old-string2.title") }
	166	+\"\"\",
	167	+ path="tor-browser.ftl",
	168	+ ),
	169	+)
38	170	```
39	171
40		-This will first run a mock migration using the en-US localization files. This is
41		-to help ensure that the migration will work as intended. I.e. if we ran the
42		-migration recipe for en-US, using it as both the source of old strings and as a
43		-reference, then each transformation using these old strings should produce a
44		-Message or Term that is identical to the one in the reference file.
	172	+Note that in this case the locale's `tor-browser.ftl` file is being used as both
	173	+the input and the output.
	174	+
	175	+Note that for a Fluent-to-Fluent migration we use `COPY_PATTERN` rather than
	176	+`COPY`. The second argument provides an ID that either point's to the
	177	+Fluent Messages' value (roughly, the text content) by giving the Message's ID,
	178	+or one of its attributes by adding a corresponding suffix to the ID.
	179	+
	180	+### Other examples
	181	+
	182	+Migration recipes can support a lot more, including more advanced
	183	+transformations like removing or replacing HTML tags from content. There are
	184	+some existing recipes in the `./migrations` directory that can be used as a
	185	+reference.
	186	+
	187	+
	188	+## translation directory
	189	+
	190	+In most cases, you will just need to check out a `translation` branch that
	191	+contains the translated strings used by the migration recipe, as well as the
	192	+Fluent file that will be written to. Usually, this will be a branch based on
	193	+the origin/upstream `tor-browser` or `base-browser` branches.
	194	+
	195	+However, if the migration recipe requires source files that are in a different
	196	+branch to the Fluent file that is being written to, then you will need to
	197	+manually copy those files into the working directory before running the script.
	198	+
	199	+
	200	+## Script procedure
45	201
46		-Then this will run `migrate(ctx)` for the es-ES and sr locales. Some of the
47		-transformations will not be completed for a locale if that locale does not
48		-contain a translated version of the old strings the transformation requires
49		-(because there is no string to copy).
	202	+### Test en-US migration
50	203
51		-If one of the old strings is from a `.dtd` or `.properties` file, this will
52		-check whether they are marked as translated on weblate using its API. This
53		-prevents copying "placeholder" en-US values into the locale's fluent files.
	204	+The script will first run a test migration using the en-US strings. It will run
	205	+the migration using the files from `translation/en-US/` as the input. In the
	206	+previous example 1, this will read `my.old.string` and `my.old.title` from
	207	+`translation/en-US/old-strings.properties`. And in example 2, this will read
	208	+`my-old-string2` from `translation/en-US/tor-browser.ftl`.
54	209
55		-Any warnings or errors will be logged to stderr in the console.
	210	+If any of the input strings are missing, this indicates that there is likely a
	211	+mistake in the recipe or the translation directory and the script will be
	212	+aborted without running the migration for the other locales.
	213	+
	214	+Otherwise, it then compares the result of this migration with the strings found
	215	+in the reference files. In the previous example 1, this will read the entry for
	216	+`my-new-string` from `<reference-dir>/tor-browser.ftl`, where `<reference-dir>`
	217	+is `translation/en-US/` unless `--reference-directory` is given to use a
	218	+different directory. If the reference entry for `my-new-string` differs from the
	219	+entry that the recipe automatically produced, then this indicates that there is
	220	+a mistake in the recipe and the script will be aborted.
	221	+
	222	+### Locale migrations
	223	+
	224	+If the test migration passed, then the recipe will be re-run for each of the
	225	+specified locales. If the locale is missing some strings or files that a single
	226	+transformation requires, then that transformation will be skipped and the
	227	+corresponding entry won't be written to the Fluent file. Otherwise the new
	228	+entries will be appended to the existing Fluent files.
	229	+
	230	+#### Non-Fluent strings and Weblate queries
	231	+
	232	+Some non-Fluent formats, like DTD and properties, will fill untranslated strings
	233	+with the source en-US content. And we don't want these untranslated strings to
	234	+be used for migrations. There is no metadata within the files to determine this,
	235	+and simply being the same as the en-US string is not a reliable indicator.
	236	+
	237	+So for these strings the script will additionally make a remote query to the
	238	+Weblate API to fetch the metadata that indicates whether the string is
	239	+considered translated or not. If a transformation requires a string that is
	240	+marked as untranslated, it will be skipped.
	241	+
	242	+NOTE: The Weblate metadata is not always reliable, so you should still manually
	243	+review the changes.
	244	+
	245	+NOTE: When testing before, the Weblate API was limited to 100 requests per day
	246	+per IP address for unauthorised users. If this becomes an issue you may want to
	247	+run the migrations in limited batches of locales with new IP addresses. Or, you
	248	+can create a Weblate account and change this script to pass an authorisation
	249	+token with the request.
56	250	"""
57	251
58	252	import argparse
...	...	@@ -63,142 +257,299 @@ import logging
63	257	import os
64	258	import sys
65	259	import urllib.request
	260	+from collections.abc import Iterator
	261	+from typing import Any, NamedTuple, Protocol, cast
66	262
67	263	from fluent.migrate.context import MigrationContext
68	264	from fluent.migrate.errors import EmptyLocalizationError, UnreadableReferenceError
69	265	from fluent.syntax import ast, parse, serialize
70	266
71	267
72		-def in_red(string):
	268	+def in_red(string: str) -> str:
73	269	"""
74	270	Wrap a string so that it is shown as red in the console.
	271	+ :param string: The string to wrap.
	272	+ :returns: The wrapped string.
75	273	"""
76	274	return f"\x1b[1;91m{string}\x1b[0m"
77	275
78	276
	277	+def assert_keys(
	278	+ check_dict: dict[Any, Any], expected_keys: tuple[str, ...], url: str
	279	+) -> None:
	280	+ """
	281	+ Verify that the dictionary contains *at least* the expected keys,
	282	+ otherwise this throws.
	283	+ :param check_dict: The dictionary to check.
	284	+ :param url: The url where the dictionary was fetched from.
	285	+ """
	286	+ for key in expected_keys:
	287	+ if key not in check_dict:
	288	+ raise ValueError(f"Missing '{key}' in response for {url}")
	289	+
	290	+
	291	+class WeblateUnit:
	292	+ """
	293	+ Represents a Weblate "unit": roughly corresponds to a translated string.
	294	+
	295	+ See https://docs.weblate.org/en/latest/api.html#units.
	296	+ """
	297	+
	298	+ def __init__(self, data: dict[Any, Any], url: str) -> None:
	299	+ """
	300	+ :param data: The Weblate data response.
	301	+ :param url: The URL that the data came from.
	302	+ """
	303	+ assert_keys(data, ("context", "translated"), url)
	304	+ string_id = data["context"]
	305	+ translated = data["translated"]
	306	+ if not isinstance(string_id, str):
	307	+ raise ValueError(f"Unexpected 'context' type in response for {url}")
	308	+ if not isinstance(translated, bool):
	309	+ raise ValueError(f"Unexpected 'translated' type in response for {url}")
	310	+
	311	+ self.string_id = string_id
	312	+ self.translated = translated
	313	+
	314	+
	315	+class WeblateTranslation:
	316	+ """
	317	+ Represents a Weblate "translation": roughly corresponds to the translation
	318	+ of a file for a given locale/language.
	319	+
	320	+ See https://docs.weblate.org/en/latest/api.html#translations.
	321	+ """
	322	+
	323	+ def __init__(self, data: dict[Any, Any], url: str) -> None:
	324	+ """
	325	+ :param data: The Weblate data response.
	326	+ :param url: The URL that the data came from.
	327	+ """
	328	+ assert_keys(data, ("filename", "units_list_url"), url)
	329	+ locale_path = data["filename"]
	330	+ units_url = data["units_list_url"]
	331	+ if not isinstance(locale_path, str):
	332	+ raise ValueError(f"Unexpected 'filename' type in response for {url}")
	333	+ if not isinstance(units_url, str):
	334	+ raise ValueError(f"Unexpected 'units_list_url' type in response for {url}")
	335	+
	336	+ self.locale_path = locale_path
	337	+ self.units_url = units_url
	338	+ # The units, indexed by their string ID. Lazy-loaded and `None` whilst
	339	+ # unfilled.
	340	+ self.units: None | dict[str, WeblateUnit] = None
	341	+
	342	+
	343	+class WeblateComponent:
	344	+ """
	345	+ Represents a Weblate "component": roughly corresponds to a source (en-US)
	346	+ file that can be translated.
	347	+
	348	+ See https://docs.weblate.org/en/latest/api.html#components.
	349	+ And https://docs.weblate.org/en/latest/user/basic.html.
	350	+ """
	351	+
	352	+ def __init__(self, data: dict[Any, Any], url: str) -> None:
	353	+ """
	354	+ :param data: The Weblate data response.
	355	+ :param url: The URL that the data came from.
	356	+ """
	357	+ assert_keys(data, ("template", "translations_url", "branch"), url)
	358	+ template_path = data["template"]
	359	+ translations_url = data["translations_url"]
	360	+ branch = data["branch"]
	361	+ if not isinstance(template_path, str):
	362	+ raise ValueError(f"Unexpected 'template' type in response for {url}")
	363	+ if not isinstance(translations_url, str):
	364	+ raise ValueError(
	365	+ f"Unexpected 'translations_url' type in response for {url}"
	366	+ )
	367	+ if not isinstance(branch, str):
	368	+ raise ValueError(f"Unexpected 'branch' type in response for {url}")
	369	+
	370	+ self.branch = branch
	371	+ # The path to the source (en-US) file.
	372	+ self.template_path = template_path
	373	+ # The URL to fetch the component's translations from.
	374	+ self.translations_url = translations_url
	375	+ # The translations data, indexed by the locale file paths. Lazy-loaded
	376	+ # and `None` whilst unfilled.
	377	+ self.translations: None | dict[str, WeblateTranslation] = None
	378	+
	379	+
79	380	class WeblateMetadata:
80	381	"""
81		- Manages and fetches string metadata from weblate.
	382	+ Manages and fetches string metadata from Weblate.
82	383	"""
83	384
84		- def __init__(self):
	385	+ # The translation branches that we expect to be used for Weblate.
	386	+ _BROWSER_BRANCHES = ["tor-browser", "base-browser", "fenix-torbrowserstringsxml"]
	387	+ _WEBLATE_API_URL = "https://hosted.weblate.org/api/"
	388	+ _WEBLATE_API_TOR_COMPONENTS_URL = _WEBLATE_API_URL + "projects/tor/components/"
	389	+
	390	+ def __init__(self) -> None:
	391	+ """
	392	+ Create a new instance for fetching Weblate metadata.
	393	+ """
85	394	self.logger = logging.getLogger("weblate-metadata")
86	395	self.logger.setLevel(logging.INFO)
87	396
88		- # The _components property will be lazy-loaded with weblate data.
89		- #
90		- # On weblate, for monolingual formats, the component will have a
91		- # "template" file, which matches the en-US file path in the
92		- # "translation" git repo. So we can find a component through a matching
93		- # template.
94		- #
95		- # Each translation in that component will have a "filename" that will
96		- # match the locale's file path in the "translation" git repo. So we can
97		- # find a translation through a matching filename.
98		- #
99		- # Each translation can be queried for its units, including metadata.
100		- # Each unit's "context" will match the string's id in the corresponding
101		- # file.
102		- #
103		- # Expect the final structure to be:
104		- # {
105		- # template: {
106		- # "translations-url": str, # Used for API translations query.
107		- # "translations": {
108		- # filename: {
109		- # "units-url": str, # Used for API units query.
110		- # "units": {
111		- # context: {
112		- # "translated": bool,
113		- # },
114		- # },
115		- # },
116		- # },
117		- # },
118		- # }
119		- self._components = None
120		-
121		- def _get_weblate_response(self, url):
	397	+ # The _components property will be lazy-loaded with Weblate data.
	398	+ self._components: dict[str, WeblateComponent] | None = None
	399	+
	400	+ def _get_weblate_response(self, url: str) -> Any:
	401	+ """
	402	+ Make a single query to Weblate.
	403	+ :param url: The URL to fetch.
	404	+ :returns: The JSON response.
	405	+ """
	406	+ if not url.startswith(self._WEBLATE_API_URL):
	407	+ raise ValueError(f"Unexpected URL request: {url}")
	408	+
122	409	weblate_request = urllib.request.Request(
123	410	url,
124	411	# NOTE: can set the request header:
125	412	# "Authorization": "Token WEBLATE_TOKEN"
126		- # to get around rate-limiting
	413	+ # to get around rate-limiting.
	414	+ # NOTE: We empty the User-Agent to override the default
	415	+ # value of "Python-urllib/{major}.{minor}".
127	416	headers={"User-Agent": ""},
128	417	)
129	418
130		- with urllib.request.urlopen(weblate_request, timeout=20) as response:
	419	+ with urllib.request.urlopen(weblate_request) as response:
131	420	return json.load(response)
132	421
133		- def _get_from_weblate(self, url):
134		- ret = []
	422	+ def _get_from_weblate(self, url: str) -> list[dict[Any, Any]]:
	423	+ """
	424	+ Fetch a list of results from Weblate.
	425	+ :param url: The initial URL to fetch a response from.
	426	+ :returns: The returned list of results.
	427	+ """
	428	+ ret: list[dict[Any, Any]] = []
135	429	while url:
136	430	response = self._get_weblate_response(url)
	431	+ if not isinstance(response, dict):
	432	+ raise ValueError(f"Unexpected response type for {url}")
	433	+ assert_keys(response, ("results", "next"), url)
	434	+
	435	+ res_val = response["results"]
	436	+ if not isinstance(res_val, list):
	437	+ raise ValueError(f"Unexpected 'results' type for {url}")
	438	+ for res_item in res_val:
	439	+ if not isinstance(res_item, dict):
	440	+ raise ValueError(f"Unexpected 'results' item type for {url}")
	441	+ ret.append(res_item)
	442	+
137	443	# Continue to fetch the next page, if it is present.
138		- # Expect the "next" url to be empty
	444	+ # Expect the "next" url to be null
139	445	# or the same url with "?page=2", etc.
140		- url = response["next"]
141		- ret.extend(response["results"])
	446	+ next_val = response["next"]
	447	+ if next_val is not None and not isinstance(next_val, str):
	448	+ raise ValueError(f"Unexpected 'next' type in response for {url}")
	449	+ url = "" if next_val is None else next_val
142	450	return ret
143	451
144		- def _get_components(self):
	452	+ def _get_components(self) -> dict[str, WeblateComponent]:
	453	+ """
	454	+ Get all the Tor Browser Weblate components.
	455	+
	456	+ We identify each Weblate component by its template path: the en-US
	457	+ file path in the translation repository.
	458	+
	459	+ NOTE: We assume that the template path uniquely identifies the
	460	+ component, even though this is not guaranteed in general by Weblate.
	461	+
	462	+ :returns: A dictionary of components, indexed by their en-US template
	463	+ path.
	464	+ """
	465	+ url = self._WEBLATE_API_TOR_COMPONENTS_URL
145	466	if self._components is None:
146	467	self._components = {
147		- comp["template"]: {
148		- "translations": None,
149		- "translations-url": comp["translations_url"],
150		- }
151		- for comp in self._get_from_weblate(
152		- "https://hosted.weblate.org/api/projects/tor/components/"
	468	+ comp.template_path: comp
	469	+ for comp in (
	470	+ WeblateComponent(data, url) for data in self._get_from_weblate(url)
153	471	)
154		- if comp["template"]
	472	+ if comp.template_path and comp.branch in self._BROWSER_BRANCHES
155	473	}
156	474	return self._components
157	475
158		- def _get_translations(self, template):
159		- component = self._get_components().get(template, None)
	476	+ def _get_translations(
	477	+ self, template_path: str
	478	+ ) -> dict[str, WeblateTranslation] | None:
	479	+ """
	480	+ Get the Weblate translations for a given Weblate component.
	481	+
	482	+ We identify each translation by its file path: the locale-specific path
	483	+ that the translated file is written to.
	484	+
	485	+ :param template_path: The component's en-US template path.
	486	+ :returns: A dictionary of translations, indexed by their locale file
	487	+ paths. Or `None` if no components were found.
	488	+ """
	489	+ component = self._get_components().get(template_path, None)
160	490	if not component:
161		- self.logger.warning(f"No component in weblate for {template}.")
	491	+ self.logger.warning(f"No component in Weblate for {template_path}.")
162	492	return None
163		- if component["translations"] is None:
164		- component["translations"] = {
165		- trans["filename"]: {
166		- "units": None,
167		- "units-url": trans["units_list_url"],
168		- }
169		- for trans in self._get_from_weblate(component["translations-url"])
	493	+ if component.translations is None:
	494	+ url = component.translations_url
	495	+ component.translations = {
	496	+ trans.locale_path: trans
	497	+ for trans in (
	498	+ WeblateTranslation(data, url)
	499	+ for data in self._get_from_weblate(url)
	500	+ )
170	501	}
171		- return component["translations"]
	502	+ return component.translations
	503	+
	504	+ def _get_units(
	505	+ self, template_path: str, locale_path: str
	506	+ ) -> dict[str, WeblateUnit] | None:
	507	+ """
	508	+ Get the Weblate units for a given Weblate translation.
	509	+
	510	+ We identify each unit by its string ID.
172	511
173		- def _get_units(self, template, file):
174		- translation_dict = self._get_translations(template)
	512	+ :param template_path: The containing component's en-US template path.
	513	+ :param locale_path: The translation's locale file path.
	514	+ :returns: A dictionary of units, indexed by their string ID. Or `None`
	515	+ if no translations could be found.
	516	+ """
	517	+ translation_dict = self._get_translations(template_path)
175	518	if translation_dict is None:
176	519	return None
177		- translation = translation_dict.get(file, None)
	520	+ translation = translation_dict.get(locale_path, None)
178	521	if translation is None:
179		- self.logger.warning(f"No translation in weblate for {file}.")
	522	+ self.logger.warning(f"No translation in Weblate for {locale_path}.")
180	523	return None
181		- if translation["units"] is None:
182		- translation["units"] = {
183		- unit["context"]: {
184		- "translated": unit["translated"],
185		- }
186		- for unit in self._get_from_weblate(translation["units-url"])
	524	+ if translation.units is None:
	525	+ url = translation.units_url
	526	+ translation.units = {
	527	+ unit.string_id: unit
	528	+ for unit in (
	529	+ WeblateUnit(data, url) for data in self._get_from_weblate(url)
	530	+ )
187	531	}
188		- return translation["units"]
	532	+ return translation.units
189	533
190		- def is_translated(self, template_path, locale_path, string_id):
	534	+ def is_translated(self, file_path: str, locale: str, string_id: str) -> bool:
191	535	"""
192		- Whether the given string is marked as translated on weblate.
	536	+ Test whether the given string is marked as translated on Weblate.
	537	+
	538	+ :param file_path: The file to check the translation of, relative to the
	539	+ locale's directory.
	540	+ :param locale: The locale to check for.
	541	+ :param string_id: The ID of the string to check.
193	542	"""
	543	+ template_path = os.path.join("en-US", file_path)
	544	+ locale_path = os.path.join(locale, file_path)
194	545	unit_dict = self._get_units(template_path, locale_path)
195	546	if unit_dict is None:
196	547	return False
197	548	unit = unit_dict.get(string_id, None)
198	549	if unit is None:
199		- self.logger.warning(f"No unit in weblate for {locale_path}:{string_id}.")
	550	+ self.logger.warning(f"No unit in Weblate for {locale_path}:{string_id}.")
200	551	return False
201		- return unit["translated"]
	552	+ return unit.translated
202	553
203	554
204	555	class TorBrowserMigrationContext(MigrationContext):
...	...	@@ -206,12 +557,18 @@ class TorBrowserMigrationContext(MigrationContext):
206	557	Extension which adds some extra methods to use for tor-browser.
207	558	"""
208	559
209		- def __init__(self, locale, reference_dir, localization_dir):
	560	+ def __init__(self, locale: str, reference_dir: str, localization_dir: str) -> None:
210	561	super().__init__(locale, reference_dir, localization_dir)
211	562
212		- def _fluent_keys(self, resource):
	563	+ def _fluent_keys(self, resource: ast.Resource) -> Iterator[str]:
	564	+ """
	565	+ Generate all the keys within a Fluent resource, where a key is an
	566	+ identifier for a Term, a Message, a Term attribute, or a Message
	567	+ attribute.
	568	+ :yield: The Fluent keys.
	569	+ """
213	570	# ast.Resource, want to extract all Message and Term identifiers, as
214		- # well as their
	571	+ # well as their attributes.
215	572	for entry in resource.body:
216	573	if not isinstance(entry, (ast.Term, ast.Message)):
217	574	continue
...	...	@@ -223,12 +580,13 @@ class TorBrowserMigrationContext(MigrationContext):
223	580	for attr in entry.attributes:
224	581	yield f"{key}.{attr.id.name}"
225	582
226		- def tb_get_available_strings(self):
	583	+ def tb_get_available_strings(self) -> set[tuple[str, str]]:
227	584	"""
228		- Return all the (path, string_id) pairs for all loaded localization
	585	+ Get all the (path, string_id) pairs for all loaded localization
229	586	resources.
	587	+ :returns: The string pairs.
230	588	"""
231		- all_strings = set()
	589	+ all_strings: set[tuple[str, str]] = set()
232	590	# ctx.localization_resources is a dict containing all the loaded
233	591	# localization_resources that have been added during add_transforms.
234	592	# { localization_path: resource }
...	...	@@ -237,14 +595,13 @@ class TorBrowserMigrationContext(MigrationContext):
237	595	all_strings.update((path, key) for key in self._fluent_keys(resource))
238	596	else:
239	597	# dictionary of { identifies: value } in legacy resource.
240		- # For tor-browser,
241		- # Only include references that differ from the en-US strings.
242	598	all_strings.update((path, key) for key in resource.keys())
243	599	return all_strings
244	600
245		- def tb_get_missing_resources(self):
	601	+ def tb_get_missing_resources(self) -> set[str]:
246	602	"""
247		- Return the missing localization resources.
	603	+ Get the missing localization resources.
	604	+ :returns: The set of resource paths.
248	605	"""
249	606	return set(
250	607	path
...	...	@@ -253,25 +610,51 @@ class TorBrowserMigrationContext(MigrationContext):
253	610	if path not in self.localization_resources
254	611	)
255	612
256		- def tb_get_transform(self, target_path, transform_id):
	613	+ def tb_get_transform(
	614	+ self, target_path: str, transform_id: str
	615	+ ) -> ast.Message | ast.Term | None:
257	616	"""
258		- Find the transformation node with the given id for the given path.
	617	+ Find a transformation node.
	618	+ :param target_path: The target path of the transformation.
	619	+ :param transform_id: The ID of the transformation, AKA the string ID.
	620	+ :returns: The matching transformation node, or `None` if there is no
	621	+ match.
259	622	"""
260	623	for node in self.transforms[target_path]:
261	624	if node.id.name == transform_id:
262	625	return node
263	626	return None
264	627
265		- def tb_get_reference_entry(self, target_path, entry_id):
	628	+ def tb_get_reference_entry(
	629	+ self, target_path: str, transform_id: str
	630	+ ) -> ast.Message | ast.Term | None:
266	631	"""
267		- Find the reference node that would be used for the given id and path.
	632	+ Find a reference Message or Term for a given transformation.
	633	+ :param target_path: The target path of the transformation.
	634	+ :param transform_id: The ID of the transformation, AKA the string ID.
	635	+ :returns: The matching reference entry for the transformation, or `None`
	636	+ if there is no match.
268	637	"""
269	638	for entry in self.reference_resources[target_path].body:
270		- if isinstance(entry, (ast.Term, ast.Message)) and entry.id.name == entry_id:
	639	+ if (
	640	+ isinstance(entry, (ast.Term, ast.Message))
	641	+ and entry.id.name == transform_id
	642	+ ):
271	643	return entry.clone()
272	644	return None
273	645
274	646
	647	+class RecipeModule(Protocol):
	648	+ def migrate(self, context: MigrationContext) -> None: ...
	649	+
	650	+
	651	+class FluentSyntaxError(NamedTuple):
	652	+ path: str
	653	+ message: str
	654	+ line: int
	655	+ sample: str
	656	+
	657	+
275	658	class TorBrowserMigrator:
276	659	"""
277	660	Performs a tor-browser migration.
...	...	@@ -279,26 +662,38 @@ class TorBrowserMigrator:
279	662
280	663	def __init__(
281	664	self,
282		- en_US_dir,
283		- locale_dirs,
284		- migrate_module,
285		- weblate_metadata,
286		- ):
	665	+ en_US_ref_dir: str,
	666	+ en_US_target_dir: str,
	667	+ locale_dirs: dict[str, str],
	668	+ recipe_module: RecipeModule,
	669	+ weblate_metadata: WeblateMetadata,
	670	+ ) -> None:
	671	+ """
	672	+ :param en_US_ref_dir: The path to the en-US directory where reference
	673	+ Fluent entries for the recipe can be found.
	674	+ :param en_US_target_dir: The path to the en-US directory where existing
	675	+ strings used by the recipe can be found.
	676	+ :param locale_dirs: The collection of locales to run the migrations for.
	677	+ :param recipe_module: The python module that contains the recipe
	678	+ instructions.
	679	+ :param weblate_metadata: The object to fetch Weblate metadata from.
	680	+ """
287	681	self.logger = logging.getLogger("tor-browser-migrator")
288	682	self.logger.setLevel(logging.INFO)
289		- self.en_US_dir = en_US_dir
	683	+ self.en_US_ref_dir = en_US_ref_dir
	684	+ self.en_US_target_dir = en_US_target_dir
290	685	self.locale_dirs = locale_dirs
291		- self.migrate_module = migrate_module
	686	+ self.recipe_module = recipe_module
292	687	self.weblate_metadata = weblate_metadata
293	688
294		- def run(self):
	689	+ def run(self) -> None:
295	690	"""
296	691	Run the migration.
297	692	"""
298	693	if not self._check_en_US_resources():
299	694	sys.exit(1)
300	695
301		- fluent_errors = []
	696	+ fluent_errors: list[FluentSyntaxError] = []
302	697	for locale, locale_dir in self.locale_dirs.items():
303	698	if locale == "en-US":
304	699	print("", file=sys.stderr)
...	...	@@ -316,12 +711,18 @@ class TorBrowserMigrator:
316	711	"Migration does not need to be run again, but the following "
317	712	"syntax errors should be fixed manually.\n"
318	713	+ "\n".join(
319		- f"{in_red(full_path)}: line {line}: {message}: [[{sample}]]"
320		- for full_path, message, line, sample in fluent_errors
	714	+ f"{in_red(err.path)}: line {err.line}: {err.message}: [[{err.sample}]]"
	715	+ for err in fluent_errors
321	716	)
322	717	)
323	718
324		- def _run_locale(self, locale, locale_dir):
	719	+ def _run_locale(self, locale: str, locale_dir: str) -> list[FluentSyntaxError]:
	720	+ """
	721	+ Run a migration for a given locale.
	722	+ :param locale: The name of the locale.
	723	+ :param locale_dir: The path to the locale's directory.
	724	+ :returns: A list of Fluent errors in the written files.
	725	+ """
325	726	print("\n\n", file=sys.stderr)
326	727	self.logger.info(f"Migrating '{in_red(locale)}' locale\n")
327	728
...	...	@@ -358,7 +759,7 @@ class TorBrowserMigrator:
358	759
359	760	available_strings = ctx.tb_get_available_strings()
360	761	wrote_file = False
361		- errors = []
	762	+ errors: list[FluentSyntaxError] = []
362	763
363	764	for target_path, reference in ctx.reference_resources.items():
364	765	translated_ids = [
...	...	@@ -422,13 +823,13 @@ class TorBrowserMigrator:
422	823	break
423	824	# Strings in legacy formats might have an entry in the file
424	825	# that is just a copy of the en-US strings.
425		- # For these we want to check the weblate metadata to ensure
	826	+ # For these we want to check the Weblate metadata to ensure
426	827	# it is a translated string.
427	828	if not path.endswith(
428	829	".ftl"
429	830	) and not self.weblate_metadata.is_translated(
430		- os.path.join("en-US", path),
431		- os.path.join(locale, path),
	831	+ path,
	832	+ locale,
432	833	string_id,
433	834	):
434	835	can_transform = False
...	...	@@ -436,7 +837,7 @@ class TorBrowserMigrator:
436	837	f"Skipping transform {target_path}:{transform_id} "
437	838	f"for '{locale}' locale because the string "
438	839	f"{path}:{string_id} has not been translated on "
439		- "weblate."
	840	+ "Weblate."
440	841	)
441	842	break
442	843	if not can_transform:
...	...	@@ -463,7 +864,9 @@ class TorBrowserMigrator:
463	864	wrote_file = True
464	865	# Collect any fluent parsing errors from the newly written file.
465	866	errors.extend(
466		- (full_path, message, line, sample)
	867	+ FluentSyntaxError(
	868	+ path=full_path, message=message, line=line, sample=sample
	869	+ )
467	870	for message, line, sample in self._fluent_errors(full_content)
468	871	)
469	872
...	...	@@ -471,27 +874,42 @@ class TorBrowserMigrator:
471	874	self.logger.info(f"No files written for '{locale}' locale.")
472	875	return errors
473	876
474		- def _fluent_errors(self, fluent):
	877	+ def _fluent_errors(self, fluent_content: str) -> Iterator[tuple[str, int, str]]:
475	878	"""
476		- Verify that the given fluent string can be parsed correctly.
	879	+ Parse the given content as a Fluent file and generate all the syntax
	880	+ errors.
	881	+ :param fluent_content: The file content to parse.
	882	+ :yield: Each encountered Fluent syntax error as a triple of the message,
	883	+ the line number, and a sample.
477	884	"""
478		- resource = parse(fluent)
	885	+ resource = parse(fluent_content)
479	886	for entry in resource.body:
480	887	if not isinstance(entry, ast.Junk):
481	888	continue
482	889	for annotation in entry.annotations:
483		- line = fluent[0 : annotation.span.start].count("\n") + 1
484		- sample_start = max(annotation.span.start - 15, 0)
485		- sample = "…" + fluent[sample_start : sample_start + 30] + "…"
486		- yield annotation.message, line, sample
487		-
488		- def _get_migration_context(self, locale, locale_dir):
489		- prev_missing_resources = set()
	890	+ start = 0
	891	+ if annotation.span:
	892	+ start = annotation.span.start
	893	+ line = fluent_content[0:start].count("\n") + 1
	894	+ sample_start = max(start - 15, 0)
	895	+ sample = "…" + fluent_content[sample_start : sample_start + 30] + "…"
	896	+ yield annotation.message or "", line, sample
	897	+
	898	+ def _get_migration_context(
	899	+ self, locale: str, locale_dir: str
	900	+ ) -> TorBrowserMigrationContext:
	901	+ """
	902	+ Create a new migration context.
	903	+ :param locale: The name of the locale.
	904	+ :param locale_dir: The directory path for the locale.
	905	+ :returns: The new context.
	906	+ """
	907	+ prev_missing_resources: set[str] = set()
490	908	while True:
491		- ctx = TorBrowserMigrationContext(locale, self.en_US_dir, locale_dir)
	909	+ ctx = TorBrowserMigrationContext(locale, self.en_US_ref_dir, locale_dir)
492	910
493	911	try:
494		- self.migrate_module.migrate(ctx)
	912	+ self.recipe_module.migrate(ctx)
495	913	except EmptyLocalizationError:
496	914	# This case will be handled by missing_resources below.
497	915	# NOTE: At the time of writing, add_transforms only throws if
...	...	@@ -543,20 +961,28 @@ class TorBrowserMigrator:
543	961	# before localization_resources was fully populated.
544	962	prev_missing_resources = missing_resources
545	963
546		- def _remove_if_empty(self, path):
	964	+ def _remove_if_empty(self, path: str) -> None:
	965	+ """
	966	+ Remove a file if it is empty.
	967	+ :param path: The path to remove.
	968	+ """
547	969	if os.stat(path).st_size:
548	970	self.logger.warning(f"{path} is no longer empty. Not deleting.")
549	971	return
550	972	os.remove(path)
551	973
552		- def _check_en_US_resources(self):
553		- # We pass in the en-US directory as the localization directory, as well
554		- # as the reference directory.
555		- ctx = TorBrowserMigrationContext("en-US", self.en_US_dir, self.en_US_dir)
	974	+ def _check_en_US_resources(self) -> bool:
	975	+ """
	976	+ Verify that the migration recipe works as expected using en-US strings.
	977	+ :returns: Whether the recipe worked as expected.
	978	+ """
	979	+ ctx = TorBrowserMigrationContext(
	980	+ "en-US", self.en_US_ref_dir, self.en_US_target_dir
	981	+ )
556	982
557	983	have_error = False
558	984	try:
559		- self.migrate_module.migrate(ctx)
	985	+ self.recipe_module.migrate(ctx)
560	986	except EmptyLocalizationError:
561	987	# Handle with localization_resources check.
562	988	# NOTE: This throwing may have prevented further add_transforms from
...	...	@@ -566,7 +992,9 @@ class TorBrowserMigrator:
566	992	have_error = True
567	993	except UnreadableReferenceError:
568	994	# Reference filename is printed before this.
569		- self.logger.error("Missing an en-US reference file.")
	995	+ self.logger.error(
	996	+ "Missing an en-US reference file. Is --reference-directory correct?"
	997	+ )
570	998	return False
571	999
572	1000	# Check each transform would create the same entry in the target as the
...	...	@@ -602,7 +1030,8 @@ class TorBrowserMigrator:
602	1030	reference_entry = ctx.tb_get_reference_entry(target_path, transform_id)
603	1031	if reference_entry is None:
604	1032	self.logger.error(
605		- f"Missing en-US reference entry for transform {transform_name}"
	1033	+ f"Missing en-US reference entry for transform {transform_name}. "
	1034	+ "Is --reference-directory correct?"
606	1035	)
607	1036	have_error = True
608	1037	continue
...	...	@@ -625,9 +1054,50 @@ class TorBrowserMigrator:
625	1054	return not have_error
626	1055
627	1056
	1057	+def main(recipe_path: str, translation_dir: str, locales: str, ref_dir: str) -> None:
	1058	+ """
	1059	+ The main method.
	1060	+ :param recipe_path: The path to the recipe script.
	1061	+ :param translation_dir: The path to the translation repository directory.
	1062	+ :param locales: The locales to run the recipe for, separated by spaces.
	1063	+ :param ref_dir: The path to the reference directory.
	1064	+ """
	1065	+ if not os.path.isfile(recipe_path):
	1066	+ print(f"{in_red(recipe_path)} is not a file.", file=sys.stderr)
	1067	+ sys.exit(1)
	1068	+
	1069	+ recipe_module_spec = importlib.util.spec_from_file_location("_recipe", recipe_path)
	1070	+ recipe_module = cast(
	1071	+ RecipeModule, importlib.util.module_from_spec(recipe_module_spec)
	1072	+ )
	1073	+ recipe_module_spec.loader.exec_module(recipe_module)
	1074	+
	1075	+ def check_dir(path: str) -> str:
	1076	+ if not os.path.isdir(path):
	1077	+ print(f"{in_red(path)} is not a directory.", file=sys.stderr)
	1078	+ sys.exit(1)
	1079	+ return path
	1080	+
	1081	+ translation_dir = check_dir(translation_dir)
	1082	+ en_US_target_dir = check_dir(os.path.join(translation_dir, "en-US"))
	1083	+ en_US_ref_dir = en_US_target_dir if ref_dir is None else check_dir(ref_dir)
	1084	+
	1085	+ TorBrowserMigrator(
	1086	+ en_US_ref_dir,
	1087	+ en_US_target_dir,
	1088	+ {
	1089	+ locale: check_dir(os.path.join(translation_dir, locale))
	1090	+ for locale in (l.strip() for l in locales.split(" "))
	1091	+ if locale
	1092	+ },
	1093	+ recipe_module,
	1094	+ WeblateMetadata(),
	1095	+ ).run()
	1096	+
	1097	+
628	1098	parser = argparse.ArgumentParser(
629	1099	description="Run a tor-browser localization migration. "
630		- "See documentation within migrate_l10n.py."
	1100	+ "See documentation within migrate.py."
631	1101	)
632	1102	parser.add_argument(
633	1103	"--translation-git",
...	...	@@ -638,35 +1108,30 @@ parser.add_argument(
638	1108	parser.add_argument(
639	1109	"--locales",
640	1110	required=True,
641		- metavar="<locale1> <locale2> ...",
642		- help="Set of locales to restrict the migration to, separated by space.",
	1111	+ metavar="'<locale1> <locale2> ...'",
	1112	+ help="Set of locales to run the migration for, separated by space.",
643	1113	)
644	1114	parser.add_argument(
645		- "migration",
646		- help="Migration to run, given as a python module. "
647		- 'E.g. "l10n_migrations.my-migration-script".',
	1115	+ "--reference-directory",
	1116	+ metavar="<dir>",
	1117	+ help="Location of the en-US directory that contains the reference changes "
	1118	+ "that the recipe corresponds to. E.g. if the recipe constructs the string "
	1119	+ "`my-new-string` in `./tor-browser.ftl`, it should point to a directory "
	1120	+ "that contains the en-US `tor-browser.ftl` with `my-new-string` already "
	1121	+ "within it. Defaults to the `en-US/` sub-directory within the "
	1122	+ "translation-git directory.",
	1123	+)
	1124	+parser.add_argument(
	1125	+ "recipe",
	1126	+ help="The path to the migration recipe to run.",
648	1127	)
649	1128
650	1129
651	1130	parsed_args = parser.parse_args()
652	1131
653		-
654		-def check_dir(path):
655		- if not os.path.isdir(path):
656		- print(f"{in_red(path)} is not a directory.", file=sys.stderr)
657		- sys.exit(1)
658		- return path
659		-
660		-
661		-translation_dir = check_dir(os.path.abspath(parsed_args.translation_git))
662		-
663		-TorBrowserMigrator(
664		- check_dir(os.path.join(translation_dir, "en-US")),
665		- {
666		- locale: check_dir(os.path.join(translation_dir, locale))
667		- for locale in (l.strip() for l in parsed_args.locales.split(" "))
668		- if locale
669		- },
670		- importlib.import_module(parsed_args.migration),
671		- WeblateMetadata(),
672		-).run()
	1132	+main(
	1133	+ parsed_args.recipe,
	1134	+ parsed_args.translation_git,
	1135	+ parsed_args.locales,
	1136	+ parsed_args.reference_directory,
	1137	+)

—
View it on GitLab.
You're receiving this email because of your account on gitlab.torproject.org. Manage all notifications · Help Notification message regarding https://gitlab.torproject.org/tpo/applications/tor-browser/-/compare/1f8c6d9758e139aee1da08d170fe59a5b4580e87...fdcaef33a80a79431e303b13fdcb21f4b79fb6d9 at be890c07a27f3d379fc60b409db787fc

_______________________________________________
tor-commits mailing list -- tor-commits@xxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to tor-commits-leave@xxxxxxxxxxxxxxxxxxxx