[Author Prev][Author Next][Thread Prev][Thread Next][Author Index][Thread Index]

[tor-commits] [Git][tpo/applications/tor-browser-build][main] 2 commits: Bug 41195: Simplify the go update in relprep.py.

Title: GitLab

Pier Angelo Vendrame pushed to branch main at The Tor Project / Applications / tor-browser-build

Commits:

  • 1fb6aaae
    by Pier Angelo Vendrame at 2024-07-30T11:10:15+02:00 
    Bug 41195: Simplify the go update in relprep.py.

Since we do not support Windows 7 anymore, we can use only one version
of Go, and simplify the function that looks for updates in relprep.py.
  • f51f78b2
    by Pier Angelo Vendrame at 2024-07-30T11:10:28+02:00 
    Bug 41200: Remove the tools for allowed_addons.json.

For tor-browser#42618 and tor-browser#42619, we stopped using the
allowed_addons.json, but initially we kept the tools to update it.
However, the file creates some confusion, because it is used only for
Andorid, but should be updated also when doing desktop-only releases
when they update NoScript, or Android nightly builds might fail.
So, since as a matter of fact we do not use that file anymore, we
decided to stop updating it and remove the related tools.

9 changed files:

Changes:

  • .gitattributes deleted
    1 
    -projects/browser/allowed_addons.json -diff

  • .gitlab/issue_templates/Release Prep - Tor Browser Alpha.md
    ... ... @@ -59,8 +59,6 @@ Tor Browser Alpha (and Nightly) are on the `main` branch
    59 59 
         - [ ] `fenix_version` : update to match alpha `firefox-android` build tag
    60 60 
         - [ ] `browser_branch` : update to match alpha `firefox-android` build tag
    61 61 
         - [ ] `browser_build` : update to match alpha `firefox-android` build tag
    62 
    -  - [ ] Update allowed_addons.json by running (from `tor-browser-build` root):
    63 
    -    - `./tools/fetch_allowed_addons.py > projects/browser/allowed_addons.json`
    64 62 
     - [ ] Update `projects/translation/config`:
    65 63 
       - [ ] run `make list_translation_updates-alpha` to get updated hashes
    66 64 
       - [ ] `steps/base-browser/git_hash` : update with `HEAD` commit of project's `base-browser` branch

  • .gitlab/issue_templates/Release Prep - Tor Browser Stable.md
    ... ... @@ -60,8 +60,6 @@ Tor Browser Stable lives in the various `maint-$(TOR_BROWSER_MAJOR).$(TOR_BROWSE
    60 60 
         - [ ] `browser_branch` : update to match stable `firefox-android` build tag
    61 61 
         - [ ] `browser_build` : update to match stable `firefox-android` build tag
    62 62 
       variant: Beta
    63 
    -  - [ ] Update allowed_addons.json by running (from `tor-browser-build` root):
    64 
    -    - `./tools/fetch_allowed_addons.py > projects/browser/allowed_addons.json`
    65 63 
     - [ ] Update `projects/translation/config`:
    66 64 
       - [ ] run `make list_translation_updates-release` to get updated hashes
    67 65 
       - [ ] `steps/base-browser/git_hash` : update with `HEAD` commit of project's `base-browser` branch

  • projects/browser/allowed_addons.json deleted The diff for this file was not included because it is too large.
  • projects/browser/build.android
    ... ... @@ -39,15 +39,6 @@ unzip ../omni.ja
    39 39 
           }) %]
    40 40 
     popd
    41 41
    42 
    -
    43 
    -[% IF c("var/verify_allowed_addons") %]
    44 
    -  # Check that allowed_addons.json contains the right versions of our bundled extension(s).
    45 
    -  # If so, replace the default allowed_addons.json by ours in the apk assets folder.
    46 
    -  $rootdir/verify_allowed_addons.py "$rootdir/allowed_addons.json" "$noscript_path"
    47 
    -[% END %]
    48 
    -
    49 
    -mv $rootdir/allowed_addons.json $assets_dir/allowed_addons.json
    50 
    -
    51 42 
     mkdir apk
    52 43 
     pushd apk
    53 44 
     7zz x "$apk"

  • projects/browser/config
    ... ... @@ -49,7 +49,6 @@ targets:
    49 49 
       android:
    50 50 
         build: '[% INCLUDE build.android %]'
    51 51 
         var:
    52 
    -      verify_allowed_addons: 1
    53 52 
           arch_deps:
    54 53 
             - 7zip
    55 54 
             - openjdk-17-jdk-headless
    ... ... @@ -160,10 +159,6 @@ input_files:
    160 159 
         enable: '[% c("var/namecoin") %]'
    161 160 
       - filename: namecoin.patch
    162 161 
         enable: '[% c("var/namecoin") %]'
    163 
    -  - filename: allowed_addons.json
    164 
    -    enable: '[% c("var/android") %]'
    165 
    -  - filename: verify_allowed_addons.py
    166 
    -    enable: '[% c("var/android") && c("var/verify_allowed_addons") %]'
    167 162 
       - project: manual
    168 163 
         name: manual
    169 164 
         enable: '[% ! c("var/android") && c("var/tor-browser") %]'

  • projects/browser/verify_allowed_addons.py deleted
    1 
    -#!/usr/bin/env python3
    2 
    -
    3 
    -import json
    4 
    -import sys
    5 
    -import hashlib
    6 
    -import zipfile
    7 
    -
    8 
    -def find_addon(addons, addon_id):
    9 
    -  results = addons['results']
    10 
    -  for x in results:
    11 
    -    addon = x['addon']
    12 
    -    if addon['guid'] == addon_id:
    13 
    -      return addon
    14 
    -  sys.exit("Error: cannot find addon " + addon_id)
    15 
    -
    16 
    -def verify_extension_version(addons, addon_id, version):
    17 
    -  addon = find_addon(addons, addon_id)
    18 
    -  expected_version = addon['current_version']['version']
    19 
    -  if version != expected_version:
    20 
    -    sys.exit("Error: version " + version + " != " + expected_version)
    21 
    -
    22 
    -def verify_extension_hash(addons, addon_id, hash):
    23 
    -  addon = find_addon(addons, addon_id)
    24 
    -  expected_hash = addon["current_version"]["files"][0]["hash"]
    25 
    -  if hash != expected_hash:
    26 
    -    sys.exit("Error: hash " + hash + " != " + expected_hash)
    27 
    -
    28 
    -def read_extension_manifest(path):
    29 
    -  return json.loads(zipfile.ZipFile(path, 'r').read('manifest.json'))
    30 
    -
    31 
    -def main(argv):
    32 
    -  allowed_addons_path = argv[0]
    33 
    -  noscript_path = argv[1]
    34 
    -
    35 
    -  addons = None
    36 
    -  with open(allowed_addons_path, 'r') as file:
    37 
    -    addons = json.loads(file.read())
    38 
    -
    39 
    -  noscript_hash = None
    40 
    -  with open(noscript_path, 'rb') as file:
    41 
    -    noscript_hash = "sha256:" + hashlib.sha256(file.read()).hexdigest()
    42 
    -
    43 
    -  noscript_version = read_extension_manifest(noscript_path)["version"]
    44 
    -
    45 
    -  verify_extension_hash(addons, '{73a6fe31-595d-460b-a920-fcc0f8843232}', noscript_hash)
    46 
    -  verify_extension_version(addons, '{73a6fe31-595d-460b-a920-fcc0f8843232}', noscript_version)
    47 
    -
    48 
    -if __name__ == "__main__":
    49 
    -   main(sys.argv[1:])

  • tools/fetch_allowed_addons.py deleted
    1 
    -#!/usr/bin/env python3
    2 
    -
    3 
    -import urllib.request
    4 
    -import json
    5 
    -import base64
    6 
    -import sys
    7 
    -
    8 
    -NOSCRIPT = "{73a6fe31-595d-460b-a920-fcc0f8843232}"
    9 
    -
    10 
    -
    11 
    -def fetch(x):
    12 
    -    with urllib.request.urlopen(x) as response:
    13 
    -        return response.read()
    14 
    -
    15 
    -
    16 
    -def find_addon(addons, addon_id):
    17 
    -    results = addons["results"]
    18 
    -    for x in results:
    19 
    -        addon = x["addon"]
    20 
    -        if addon["guid"] == addon_id:
    21 
    -            return addon
    22 
    -
    23 
    -
    24 
    -def fetch_and_embed_icons(addons):
    25 
    -    results = addons["results"]
    26 
    -    for x in results:
    27 
    -        addon = x["addon"]
    28 
    -        icon_data = fetch(addon["icon_url"])
    29 
    -        addon["icon_url"] = "data:image/png;base64," + str(
    30 
    -            base64.b64encode(icon_data), "utf8"
    31 
    -        )
    32 
    -
    33 
    -
    34 
    -def fetch_allowed_addons(amo_collection=None):
    35 
    -    if amo_collection is None:
    36 
    -        amo_collection = "83a9cccfe6e24a34bd7b155ff9ee32"
    37 
    -    url = f"https://services.addons.mozilla.org/api/v4/accounts/account/mozilla/collections/{amo_collection}/addons/"
    38 
    -    data = json.loads(fetch(url))
    39 
    -    fetch_and_embed_icons(data)
    40 
    -    data["results"].sort(key=lambda x: x["addon"]["guid"])
    41 
    -    return data
    42 
    -
    43 
    -
    44 
    -def main(argv):
    45 
    -    data = fetch_allowed_addons(argv[0] if len(argv) > 1 else None)
    46 
    -    # Check that NoScript is present
    47 
    -    if find_addon(data, NOSCRIPT) is None:
    48 
    -        sys.exit("Error: cannot find NoScript.")
    49 
    -    print(json.dumps(data, indent=2, ensure_ascii=False))
    50 
    -
    51 
    -
    52 
    -if __name__ == "__main__":
    53 
    -    main(sys.argv[1:])

  • tools/relprep.py
    ... ... @@ -4,7 +4,6 @@ from collections import namedtuple
    4 4 
     import configparser
    5 5 
     from datetime import datetime, timezone
    6 6 
     from hashlib import sha256
    7 
    -import json
    8 7 
     import locale
    9 8 
     import logging
    10 9 
     from pathlib import Path
    ... ... @@ -16,7 +15,6 @@ from git import Repo
    16 15 
     import requests
    17 16 
     import ruamel.yaml
    18 17
    19 
    -from fetch_allowed_addons import NOSCRIPT, fetch_allowed_addons, find_addon
    20 18 
     import fetch_changelogs
    21 19 
     from update_manual import update_manual
    22 20
    ... ... @@ -303,28 +301,27 @@ class ReleasePreparation:
    303 301 
             logger.info("Updating addons")
    304 302 
             config = self.load_config("browser")
    305 303
    306 
    -        amo_data = fetch_allowed_addons()
    307 
    -        logger.debug("Fetched AMO data")
    308 
    -        if self.android:
    309 
    -            with (
    310 
    -                self.base_path / "projects/browser/allowed_addons.json"
    311 
    -            ).open("w") as f:
    312 
    -                json.dump(amo_data, f, indent=2)
    313 
    -
    314 
    -        noscript = find_addon(amo_data, NOSCRIPT)
    315 304 
             logger.debug("Updating NoScript")
    316 
    -        self.update_addon_amo(config, "noscript", noscript)
    305 
    +        self.update_addon_amo(
    306 
    +            config, "noscript", "{73a6fe31-595d-460b-a920-fcc0f8843232}"
    307 
    +        )
    317 308 
             if self.mullvad_browser:
    318 309 
                 logger.debug("Updating uBlock Origin")
    319 
    -            ublock = find_addon(amo_data, "uBlock0@xxxxxxxxxxxxxxx")
    320 
    -            self.update_addon_amo(config, "ublock-origin", ublock)
    310 
    +            self.update_addon_amo(
    311 
    +                config, "ublock-origin", "uBlock0@xxxxxxxxxxxxxxx"
    312 
    +            )
    321 313 
                 logger.debug("Updating the Mullvad Browser extension")
    322 314 
                 self.update_mullvad_addon(config)
    323 315
    324 316 
             self.save_config("browser", config)
    325 317
    326 
    -    def update_addon_amo(self, config, name, addon):
    327 
    -        addon = addon["current_version"]["files"][0]
    318 
    +    def update_addon_amo(self, config, name, addon_id):
    319 
    +        r = requests.get(
    320 
    +            f"https://services.addons.mozilla.org/api/v4/addons/addon/{addon_id}"
    321 
    +        )
    322 
    +        r.raise_for_status()
    323 
    +        amo_data = r.json()
    324 
    +        addon = amo_data["current_version"]["files"][0]
    328 325 
             assert addon["hash"].startswith("sha256:")
    329 326 
             addon_input = self.find_input(config, name)
    330 327 
             addon_input["URL"] = addon["url"]
    ... ... @@ -448,9 +445,9 @@ class ReleasePreparation:
    448 445 
                     major = major[2:]
    449 446 
                 return major
    450 447
    448 
    +        logger.info("Updating Go")
    451 449 
             config = self.load_config("go")
    452 
    -        # TODO: When Windows 7 goes EOL use config["version"]
    453 
    -        major = get_major(config["var"]["go_1_21"])
    450 
    +        major = get_major(config["version"])
    454 451
    455 452 
             r = requests.get("https://go.dev/dl/?mode=json")
    456 453 
             r.raise_for_status()
    ... ... @@ -463,7 +460,7 @@ class ReleasePreparation:
    463 460 
             if not data:
    464 461 
                 raise KeyError("Could not find information for our Go series.")
    465 462 
             # Skip the "go" prefix in the version.
    466 
    -        config["var"]["go_1_21"] = data["version"][2:]
    463 
    +        config["version"] = data["version"][2:]
    467 464
    468 465 
             sha256sum = ""
    469 466 
             for f in data["files"]:
    ... ... @@ -472,15 +469,7 @@ class ReleasePreparation:
    472 469 
                     break
    473 470 
             if not sha256sum:
    474 471 
                 raise KeyError("Go source package not found.")
    475 
    -        updated_hash = False
    476 
    -        for input_ in config["input_files"]:
    477 
    -            if "URL" in input_ and "var/go_1_21" in input_["URL"]:
    478 
    -                input_["sha256sum"] = sha256sum
    479 
    -                updated_hash = True
    480 
    -                break
    481 
    -        if not updated_hash:
    482 
    -            raise KeyError("Could not find a matching entry in input_files.")
    483 
    -
    472 
    +        self.find_input(config, "go")["sha256sum"] = sha256sum
    484 473 
             self.save_config("go", config)
    485 474
    486 475 
         def update_manual(self):
    ... ... @@ -556,18 +545,11 @@ class ReleasePreparation:
    556 545 
                 # Sometimes this might be incorrect for alphas, but let's
    557 546 
                 # keep it for now.
    558 547 
                 kwargs["firefox"] += "esr"
    559 
    -        self.check_update_simple(kwargs, prev_tag, "tor")
    560 
    -        self.check_update_simple(kwargs, prev_tag, "openssl")
    561 
    -        self.check_update_simple(kwargs, prev_tag, "zlib")
    562 
    -        self.check_update_simple(kwargs, prev_tag, "zstd")
    563 
    -        try:
    564 
    -            self.check_update(kwargs, prev_tag, "go", ["var", "go_1_21"])
    565 
    -        except KeyError as e:
    566 
    -            logger.warning(
    567 
    -                "Go: var/go_1_21 not found, marking Go as not updated.",
    568 
    -                exc_info=e,
    569 
    -            )
    570 
    -            pass
    548 
    +        self.check_update(kwargs, prev_tag, "tor")
    549 
    +        self.check_update(kwargs, prev_tag, "openssl")
    550 
    +        self.check_update(kwargs, prev_tag, "zlib")
    551 
    +        self.check_update(kwargs, prev_tag, "zstd")
    552 
    +        self.check_update(kwargs, prev_tag, "go")
    571 553 
             self.check_update_extensions(kwargs, prev_tag)
    572 554 
             logger.debug("Changelog arguments for %s: %s", tag_prefix, kwargs)
    573 555 
             cb = fetch_changelogs.ChangelogBuilder(
    ... ... @@ -587,7 +569,7 @@ class ReleasePreparation:
    587 569 
             with (self.base_path / path).open("w") as f:
    588 570 
                 f.write(changelogs + "\n" + last_changelogs + "\n")
    589 571
    590 
    -    def check_update(self, updates, prev_tag, project, key):
    572 
    +    def check_update(self, updates, prev_tag, project, key=["version"]):
    591 573 
             old_val = self.load_old_config(prev_tag.tag, project)
    592 574 
             new_val = self.load_config(project)
    593 575 
             for k in key:
    ... ... @@ -596,9 +578,6 @@ class ReleasePreparation:
    596 578 
             if old_val != new_val:
    597 579 
                 updates[project] = new_val
    598 580
    599 
    -    def check_update_simple(self, updates, prev_tag, project):
    600 
    -        self.check_update(updates, prev_tag, project, ["version"])
    601 
    -
    602 581 
         def check_update_extensions(self, updates, prev_tag):
    603 582 
             old_config = self.load_old_config(prev_tag, "browser")
    604 583 
             new_config = self.load_config("browser")

    • —
    View it on GitLab.
    You're receiving this email because of your account on gitlab.torproject.org. Manage all notifications · Help 

    _______________________________________________
tor-commits mailing list
tor-commits@xxxxxxxxxxxxxxxxxxxx
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits