[Author Prev][Author Next][Thread Prev][Thread Next][Author Index][Thread Index]

[tor-commits] [Git][tpo/applications/tor-browser-bundle-testsuite][main] Bug 40091: Add CI for testing Nightly builds



Title: GitLab

brizental pushed to branch main at The Tor Project / Applications / tor-browser-bundle-testsuite

Commits:

  • 2e6652ed
    by Beatriz Rizental at 2026-04-27T18:18:48+02:00
    Bug 40091: Add CI for testing Nightly builds
    
    This commit creates the base jobs for running tests in multiple
    platforms as well as adds a first job that runs marionette tests.
    

7 changed files:

Changes:

  • .gitlab-ci.yml
    1
    +spec:
    
    2
    +  include:
    
    3
    +    - local: '.gitlab/_inputs.yml'
    
    4
    +---
    
    5
    +stages:
    
    6
    +  - test-marionette
    
    7
    +
    
    8
    +default:
    
    9
    +  artifacts:
    
    10
    +    # Regardless of test outcome, all files inside the `logs` folder are saved.
    
    11
    +    when: always
    
    12
    +    paths:
    
    13
    +      - logs/
    
    14
    +    expire_in: 1 week
    
    15
    +
    
    16
    +include:
    
    17
    +  - local: '.gitlab/_base.yml'
    
    18
    +    inputs:
    
    19
    +      mozharness_url: $[[ inputs.mozharness_url ]]
    
    20
    +      android_x86_64_installer_url: $[[ inputs.android_x86_64_installer_url ]]
    
    21
    +      android_x86_64_package_name: $[[ inputs.android_x86_64_package_name ]]
    
    22
    +      android_x86_64_artifacts_url: $[[ inputs.android_x86_64_artifacts_url ]]
    
    23
    +      android_x86_64_sha256sums_url: $[[ inputs.android_x86_64_sha256sums_url ]]
    
    24
    +      debian_x86_64_installer_url: $[[ inputs.debian_x86_64_installer_url ]]
    
    25
    +      debian_x86_64_artifacts_url: $[[ inputs.debian_x86_64_artifacts_url ]]
    
    26
    +      debian_x86_64_sha256sums_url: $[[ inputs.debian_x86_64_sha256sums_url ]]
    
    27
    +      macos_x86_64_installer_url: $[[ inputs.macos_x86_64_installer_url ]]
    
    28
    +      macos_x86_64_artifacts_url: $[[ inputs.macos_x86_64_artifacts_url ]]
    
    29
    +      macos_x86_64_sha256sums_url: $[[ inputs.macos_x86_64_sha256sums_url ]]
    
    30
    +      windows_x86_64_installer_url: $[[ inputs.windows_x86_64_installer_url ]]
    
    31
    +      windows_x86_64_artifacts_url: $[[ inputs.windows_x86_64_artifacts_url ]]
    
    32
    +      windows_x86_64_sha256sums_url: $[[ inputs.windows_x86_64_sha256sums_url ]]
    
    33
    +  - local: '.gitlab/test_marionette.yml'

  • .gitlab/README.md
    1
    +# Tor Browser Nightly CI
    
    2
    +
    
    3
    +This directory holds the the GitLab CI configuration for the Tor Browser nightlies test pipeline.
    
    4
    +
    
    5
    +The top-level entry point is [`.gitlab-ci.yml`](../.gitlab-ci.yml). That file defines the pipeline stages and includes the files in this directory.
    
    6
    +
    
    7
    +## Files
    
    8
    +
    
    9
    +- [`_inputs.yml`](./_inputs.yml): Shared `spec:inputs` definitions. Both the top-level pipeline and [`_base.yml`](./_base.yml) include this file so they agree on the input schema.
    
    10
    +- [`_base.yml`](./_base.yml): Hidden base jobs for each platform. These jobs define runner tags, `rules`, setup steps, and platform-specific variables.
    
    11
    +- [`test_*.yml`](./test_*.yml): The actual test jobs. These are small jobs that extend the platform base jobs from [`_base.yml`](./_base.yml) and add the test command to run.
    
    12
    +
    
    13
    +All jobs here use Mozharness. Mozharness is Mozilla's Python-based automation harness, and Mozilla keeps test runner scripts like `marionette.py` and `android_emulator_unittest.py` under `testing/mozharness/scripts`.
    
    14
    +
    
    15
    +> ### How Inputs Flow
    
    16
    +>
    
    17
    +> 1. [`../.gitlab-ci.yml`](../.gitlab-ci.yml) includes [`_inputs.yml`](./_inputs.yml) in its `spec`.
    
    18
    +> 2. [`../.gitlab-ci.yml`](../.gitlab-ci.yml) includes [`_base.yml`](./_base.yml) and forwards all input values with `include:inputs`.
    
    19
    +> 3. [`_base.yml`](./_base.yml) also includes [`_inputs.yml`](./_inputs.yml) in its own `spec`, so it can use `$[[ inputs.* ]]`.
    
    20
    +> 4. Jobs in [`test_marionette.yml`](./test_marionette.yml) extend the hidden base jobs from [`_base.yml`](./_base.yml), inheriting the platform setup and variables like `INSTALLER_URL` and `ARTIFACTS_URL`.
    
    21
    +>
    
    22
    +> This explicit forwarding is required because inputs are scoped to the file that declares them.
    
    23
    +
    
    24
    +## Running A Job
    
    25
    +
    
    26
    +These jobs are meant to be started as triggered pipelines. The base jobs in [`_base.yml`](./_base.yml) only run when `CI_PIPELINE_SOURCE == "trigger"` and the platform-specific inputs are present.
    
    27
    +
    
    28
    +To create a trigger token in GitLab:
    
    29
    +
    
    30
    +1. Open the project in GitLab.
    
    31
    +2. Go to `Settings` > `CI/CD`.
    
    32
    +3. Expand `Pipeline trigger tokens`.
    
    33
    +4. Create a new token and copy the value.
    
    34
    +
    
    35
    +GitLab's trigger API accepts an `inputs` map, which matches the names defined in [`_inputs.yml`](./_inputs.yml). A typical JSON request looks like this:
    
    36
    +
    
    37
    +```bash
    
    38
    +curl --request POST \
    
    39
    +  --header "Content-Type: application/json" \
    
    40
    +  --data '{
    
    41
    +    "inputs": {
    
    42
    +      "mozharness_url": "https://example.org/mozharness.zip",
    
    43
    +      "debian_x86_64_installer_url": "https://example.org/tor-browser-linux-x86_64.tar.xz",
    
    44
    +      "debian_x86_64_artifacts_url": "https://example.org/debian-artifacts",
    
    45
    +      "debian_x86_64_sha256sums_url": "https://example.org/sha256sums.txt"
    
    46
    +    }
    
    47
    +  }' \
    
    48
    +  "https://gitlab.example.org/api/v4/projects/$PROJECT_ID/trigger/pipeline?token=$TRIGGER_TOKEN&ref=$BRANCH_NAME"
    
    49
    +```
    
    50
    +
    
    51
    +You can also trigger multiple platform jobs in a single pipeline by providing more than one platform's input set in the same request:
    
    52
    +
    
    53
    +```bash
    
    54
    +curl --request POST \
    
    55
    +  --header "Content-Type: application/json" \
    
    56
    +  --data '{
    
    57
    +    "inputs": {
    
    58
    +      "mozharness_url": "https://example.org/mozharness.zip",
    
    59
    +      "debian_x86_64_installer_url": "https://example.org/tor-browser-linux-x86_64.tar.xz",
    
    60
    +      "debian_x86_64_artifacts_url": "https://example.org/debian-artifacts",
    
    61
    +      "debian_x86_64_sha256sums_url": "https://example.org/debian-sha256sums.txt",
    
    62
    +      "macos_x86_64_installer_url": "https://example.org/TorBrowser.dmg",
    
    63
    +      "macos_x86_64_artifacts_url": "https://example.org/macos-artifacts",
    
    64
    +      "macos_x86_64_sha256sums_url": "https://example.org/macos-sha256sums.txt",
    
    65
    +      "windows_x86_64_installer_url": "https://example.org/tor-browser-install.exe",
    
    66
    +      "windows_x86_64_artifacts_url": "https://example.org/windows-artifacts",
    
    67
    +      "windows_x86_64_sha256sums_url": "https://example.org/windows-sha256sums.txt"
    
    68
    +    }
    
    69
    +  }' \
    
    70
    +  "https://gitlab.example.org/api/v4/projects/$PROJECT_ID/trigger/pipeline?token=$TRIGGER_TOKEN&ref=$BRANCH_NAME"
    
    71
    +```
    
    72
    +
    
    73
    +### Accepted Inputs
    
    74
    +
    
    75
    +The pipeline accepts the following inputs:
    
    76
    +
    
    77
    +- `mozharness_url`: URL to the `mozharness.zip` archive used by every job.
    
    78
    +- `debian_x86_64_installer_url`: URL to the Debian Tor Browser build to test.
    
    79
    +- `debian_x86_64_artifacts_url`: URL to the Debian test artifacts, including `target.test_packages.json`.
    
    80
    +- `debian_x86_64_sha256sums_url`: URL to the Debian `sha256sums.txt` file. Optional. When provided, the job checks this URL is reachable before running the full test suite.
    
    81
    +- `macos_x86_64_installer_url`: URL to the macOS Tor Browser build to test.
    
    82
    +- `macos_x86_64_artifacts_url`: URL to the macOS test artifacts, including `target.test_packages.json`.
    
    83
    +- `macos_x86_64_sha256sums_url`: URL to the macOS `sha256sums.txt` file. Optional. When provided, the job checks this URL is reachable before running the full test suite.
    
    84
    +- `windows_x86_64_installer_url`: URL to the Windows Tor Browser build to test.
    
    85
    +- `windows_x86_64_artifacts_url`: URL to the Windows test artifacts, including `target.test_packages.json`.
    
    86
    +- `windows_x86_64_sha256sums_url`: URL to the Windows `sha256sums.txt` file. Optional. When provided, the job checks this URL is reachable before running the full test suite.
    
    87
    +- `android_x86_64_installer_url`: URL to the Android APK to test.
    
    88
    +- `android_x86_64_artifacts_url`: URL to the Android test artifacts, including `target.test_packages.json`.
    
    89
    +- `android_x86_64_sha256sums_url`: URL to the Android `sha256sums.txt` file. Optional. When provided, the job checks this URL is reachable before running the full test suite.
    
    90
    +- `android_x86_64_package_name`: Android package name to test.
    
    91
    +
    
    92
    +All inputs default to the empty string. The platform-specific inputs are optional when triggering the pipeline as a whole, but the installer URL, artifacts URL, and (for Android) package name must be provided together for that platform's job to run.
    
    93
    +
    
    94
    +### How Job Selection Works
    
    95
    +
    
    96
    +Each hidden base job in [`_base.yml`](./_base.yml) has a `rules` section that checks two things:
    
    97
    +
    
    98
    +- the pipeline was started with `CI_PIPELINE_SOURCE == "trigger"`
    
    99
    +- the inputs required for that platform are not empty
    
    100
    +
    
    101
    +That means the set of jobs in the pipeline depends on the inputs you provide:
    
    102
    +
    
    103
    +- If you provide `mozharness_url`, `debian_x86_64_installer_url`, and `debian_x86_64_artifacts_url`, the Debian Marionette job is included.
    
    104
    +- If you provide the macOS installer and artifacts inputs, the macOS Marionette job is included.
    
    105
    +- If you provide the Windows installer and artifacts inputs, the Windows Marionette job is included.
    
    106
    +- If you provide the Android installer URL, artifacts URL, and package name, the Android Marionette job is included.
    
    107
    +
    
    108
    +If a platform-specific input is missing or left as an empty string, that platform's job is skipped entirely rather than starting and failing later. In other words, one triggered pipeline can run one platform, several platforms, or all platforms, depending on which input set you send.

  • .gitlab/_base.yml
    1
    +spec:
    
    2
    +  include:
    
    3
    +    - local: '.gitlab/_inputs.yml'
    
    4
    +---
    
    5
    +variables:
    
    6
    +  MOZHARNESS_URL: "$[[ inputs.mozharness_url ]]"
    
    7
    +
    
    8
    +.common:
    
    9
    +  before_script:
    
    10
    +    - python3 .gitlab/scripts/before_script.py
    
    11
    +
    
    12
    +  after_script:
    
    13
    +    - python3 .gitlab/scripts/after_script.py
    
    14
    +
    
    15
    +.debian-x86_64:
    
    16
    +  extends: .common
    
    17
    +
    
    18
    +  tags:
    
    19
    +    - debian-trixie-x86_64
    
    20
    +
    
    21
    +  rules:
    
    22
    +    - if: $CI_PIPELINE_SOURCE == "trigger" && "$[[ inputs.mozharness_url ]]" != "" && "$[[ inputs.debian_x86_64_installer_url ]]" != "" && "$[[ inputs.debian_x86_64_artifacts_url ]]" != ""
    
    23
    +      when: on_success
    
    24
    +    - when: never
    
    25
    +
    
    26
    +  variables:
    
    27
    +    MOZ_FETCHES_DIR: "/home/gitlab-runner/fetches"
    
    28
    +    MINIDUMP_STACKWALK_URL: "https://github.com/rust-minidump/rust-minidump/releases/download/v0.26.1/minidump-stackwalk-x86_64-unknown-linux-gnu.tar.xz"
    
    29
    +    MINIDUMP_STACKWALK_SHA256: "f789997f086dab6e46c46ab4560fc28676124613770ddac3b89a6e2e0e4963d8"
    
    30
    +    INSTALLER_URL: "$[[ inputs.debian_x86_64_installer_url ]]"
    
    31
    +    ARTIFACTS_URL: "$[[ inputs.debian_x86_64_artifacts_url ]]"
    
    32
    +    SHA256SUMS_URL: "$[[ inputs.debian_x86_64_sha256sums_url ]]"
    
    33
    +
    
    34
    +.macos-x86_64:
    
    35
    +  extends: .common
    
    36
    +
    
    37
    +  tags:
    
    38
    +    - macos-sonoma-x86_64
    
    39
    +
    
    40
    +  rules:
    
    41
    +    - if: $CI_PIPELINE_SOURCE == "trigger" && "$[[ inputs.mozharness_url ]]" != "" && "$[[ inputs.macos_x86_64_installer_url ]]" != "" && "$[[ inputs.macos_x86_64_artifacts_url ]]" != ""
    
    42
    +      when: on_success
    
    43
    +    - when: never
    
    44
    +
    
    45
    +  variables:
    
    46
    +    MOZ_FETCHES_DIR: "/Users/gitlab-runner/fetches"
    
    47
    +    MINIDUMP_STACKWALK_URL: "https://github.com/rust-minidump/rust-minidump/releases/download/v0.26.1/minidump-stackwalk-x86_64-apple-darwin.tar.xz"
    
    48
    +    MINIDUMP_STACKWALK_SHA256: "7a3e274a09bc35ccc35e9d3dcd60e738ab14e54ea3306be94a4a8dedaf7a468d"
    
    49
    +    INSTALLER_URL: "$[[ inputs.macos_x86_64_installer_url ]]"
    
    50
    +    ARTIFACTS_URL: "$[[ inputs.macos_x86_64_artifacts_url ]]"
    
    51
    +    SHA256SUMS_URL: "$[[ inputs.macos_x86_64_sha256sums_url ]]"
    
    52
    +
    
    53
    +.windows-x86_64:
    
    54
    +  extends: .common
    
    55
    +
    
    56
    +  tags:
    
    57
    +    - windows-11-x86_64
    
    58
    +
    
    59
    +  rules:
    
    60
    +    - if: $CI_PIPELINE_SOURCE == "trigger" && "$[[ inputs.mozharness_url ]]" != "" && "$[[ inputs.windows_x86_64_installer_url ]]" != "" && "$[[ inputs.windows_x86_64_artifacts_url ]]" != ""
    
    61
    +      when: on_success
    
    62
    +    - when: never
    
    63
    +
    
    64
    +  variables:
    
    65
    +    MOZ_FETCHES_DIR: 'C:\windoes\fetches'
    
    66
    +    MINIDUMP_STACKWALK_URL: 'https://github.com/rust-minidump/rust-minidump/releases/download/v0.26.1/minidump-stackwalk-x86_64-pc-windows-msvc.zip'
    
    67
    +    MINIDUMP_STACKWALK_SHA256: "dc6da411047ef15b784bffe0f9bf7ecb9221186c4c8fd655e3d8a3fbba0a9c61"
    
    68
    +    INSTALLER_URL: "$[[ inputs.windows_x86_64_installer_url ]]"
    
    69
    +    ARTIFACTS_URL: "$[[ inputs.windows_x86_64_artifacts_url ]]"
    
    70
    +    SHA256SUMS_URL: "$[[ inputs.windows_x86_64_sha256sums_url ]]"
    
    71
    +
    
    72
    +.android-x86_64:
    
    73
    +  extends: .debian-x86_64
    
    74
    +
    
    75
    +  rules:
    
    76
    +    - if: '$CI_PIPELINE_SOURCE == "trigger" && "$[[ inputs.mozharness_url ]]" != "" && "$[[ inputs.android_x86_64_installer_url ]]" != "" && "$[[ inputs.android_x86_64_artifacts_url ]]" != "" && "$[[ inputs.android_x86_64_package_name ]]" != ""'
    
    77
    +      when: on_success
    
    78
    +    - when: never
    
    79
    +
    
    80
    +  variables:
    
    81
    +    # TODO (tor-browser-bundle-testsuite#40101): Stop hardcoding the filename here.
    
    82
    +    ANDROID_SDK_REPACK_URL: "https://tb-build-06.torproject.org/~tb-builder/tor-browser-build/out/android-sdk/android-sdk-36.1.0-f31d7b.tar.zst"
    
    83
    +    INSTALLER_URL: "$[[ inputs.android_x86_64_installer_url ]]"
    
    84
    +    ARTIFACTS_URL: "$[[ inputs.android_x86_64_artifacts_url ]]"
    
    85
    +    PACKAGE_NAME: "$[[ inputs.android_x86_64_package_name ]]"
    
    86
    +    SHA256SUMS_URL: "$[[ inputs.android_x86_64_sha256sums_url ]]"

  • .gitlab/_inputs.yml
    1
    +inputs:
    
    2
    +  mozharness_url:
    
    3
    +    type: string
    
    4
    +    default: ""
    
    5
    +    description: "Location of the mozharness.zip archive"
    
    6
    +
    
    7
    +  android_x86_64_installer_url:
    
    8
    +    type: string
    
    9
    +    default: ""
    
    10
    +    description: "Location of the Android APK to test"
    
    11
    +  android_x86_64_package_name:
    
    12
    +    type: string
    
    13
    +    default: ""
    
    14
    +    description: "Name of the Android package to test"
    
    15
    +  android_x86_64_artifacts_url:
    
    16
    +    type: string
    
    17
    +    default: ""
    
    18
    +    description: "Location of Android test artifacts"
    
    19
    +  android_x86_64_sha256sums_url:
    
    20
    +    type: string
    
    21
    +    default: ""
    
    22
    +    description: "Location of the Android sha256sums.txt file"
    
    23
    +
    
    24
    +  debian_x86_64_installer_url:
    
    25
    +    type: string
    
    26
    +    default: ""
    
    27
    +    description: "Location of the Debian build to test"
    
    28
    +  debian_x86_64_artifacts_url:
    
    29
    +    type: string
    
    30
    +    default: ""
    
    31
    +    description: "Location of Debian test artifacts"
    
    32
    +  debian_x86_64_sha256sums_url:
    
    33
    +    type: string
    
    34
    +    default: ""
    
    35
    +    description: "Location of the Debian sha256sums.txt file"
    
    36
    +
    
    37
    +  macos_x86_64_installer_url:
    
    38
    +    type: string
    
    39
    +    default: ""
    
    40
    +    description: "Location of the macOS build to test"
    
    41
    +  macos_x86_64_artifacts_url:
    
    42
    +    type: string
    
    43
    +    default: ""
    
    44
    +    description: "Location of macOS test artifacts"
    
    45
    +  macos_x86_64_sha256sums_url:
    
    46
    +    type: string
    
    47
    +    default: ""
    
    48
    +    description: "Location of the macOS sha256sums.txt file"
    
    49
    +
    
    50
    +  windows_x86_64_installer_url:
    
    51
    +    type: string
    
    52
    +    default: ""
    
    53
    +    description: "Location of the Windows build to test"
    
    54
    +  windows_x86_64_artifacts_url:
    
    55
    +    type: string
    
    56
    +    default: ""
    
    57
    +    description: "Location of Windows test artifacts"
    
    58
    +  windows_x86_64_sha256sums_url:
    
    59
    +    type: string
    
    60
    +    default: ""
    
    61
    +    description: "Location of the Windows sha256sums.txt file"

  • .gitlab/scripts/after_script.py
    1
    +import shutil
    
    2
    +from pathlib import Path
    
    3
    +
    
    4
    +if __name__ == "__main__":
    
    5
    +    # This directory will be saved as an artifact after each job.
    
    6
    +    logs_dir = Path("logs")
    
    7
    +    logs_dir.mkdir(parents=True, exist_ok=True)
    
    8
    +
    
    9
    +    # This is the directory created by mozharness with a lot of useful debug things.
    
    10
    +    blobber_dir = Path("build") / "blobber_upload_dir"
    
    11
    +    if blobber_dir.exists():
    
    12
    +        shutil.copytree(blobber_dir, logs_dir, dirs_exist_ok=True)
    
    13
    +        print(f"Copied {blobber_dir} -> {logs_dir}")
    
    14
    +    else:
    
    15
    +        print(f"No blobber_upload_dir found at {blobber_dir}, skipping")

  • .gitlab/scripts/before_script.py
    1
    +import hashlib
    
    2
    +import os
    
    3
    +import shutil
    
    4
    +import subprocess
    
    5
    +import time
    
    6
    +import urllib.request
    
    7
    +import zipfile
    
    8
    +from pathlib import Path
    
    9
    +
    
    10
    +SPECULATIVE_DOWNLOAD_RETRY_ATTEMPTS = 3
    
    11
    +SPECULATIVE_DOWNLOAD_RETRY_DELAY = 300  # 5min
    
    12
    +
    
    13
    +# Mozilla hardcodes this version in their testing code and
    
    14
    +# expects the emulator to have a this exact android version.
    
    15
    +EMULATOR_ANDROID_VERSION = 34
    
    16
    +
    
    17
    +
    
    18
    +def download_file(url: str, dest: Path, sha256: str = "") -> None:
    
    19
    +    print(f"Downloading {url} -> {dest}")
    
    20
    +    urllib.request.urlretrieve(url, dest)
    
    21
    +    if sha256:
    
    22
    +        actual = hashlib.sha256(dest.read_bytes()).hexdigest()
    
    23
    +        if actual != sha256:
    
    24
    +            raise ValueError(
    
    25
    +                f"SHA256 mismatch for {dest}: expected {sha256}, got {actual}"
    
    26
    +            )
    
    27
    +
    
    28
    +
    
    29
    +def extract_tar(tar_path: Path, dest_dir: Path) -> None:
    
    30
    +    subprocess.run(
    
    31
    +        # --strip-components=1: all the archives we fetch have contents nested inside a
    
    32
    +        # top-level directory rather than at the root, so we strip it on extraction.
    
    33
    +        ["tar", "-xf", str(tar_path), "-C", str(dest_dir), "--strip-components=1"],
    
    34
    +        check=True,
    
    35
    +    )
    
    36
    +
    
    37
    +
    
    38
    +def check_sha256sums_url(sha256sums_url: str) -> None:
    
    39
    +    if not sha256sums_url:
    
    40
    +        return
    
    41
    +
    
    42
    +    for attempt in range(SPECULATIVE_DOWNLOAD_RETRY_ATTEMPTS):
    
    43
    +        try:
    
    44
    +            dest = Path("sha256sums")
    
    45
    +            download_file(sha256sums_url, dest)
    
    46
    +            print(
    
    47
    +                f"SHA256SUMS downloaded: {len(dest.read_bytes().splitlines())} entries"
    
    48
    +            )
    
    49
    +            return
    
    50
    +        except Exception as e:
    
    51
    +            print(f"Attempt {attempt + 1}: {e}")
    
    52
    +
    
    53
    +        if attempt < SPECULATIVE_DOWNLOAD_RETRY_ATTEMPTS - 1:
    
    54
    +            print(f"Retrying in {SPECULATIVE_DOWNLOAD_RETRY_DELAY}s...")
    
    55
    +            time.sleep(SPECULATIVE_DOWNLOAD_RETRY_DELAY)
    
    56
    +
    
    57
    +    raise RuntimeError(f"SHA256SUMS not available after retries: {sha256sums_url}")
    
    58
    +
    
    59
    +
    
    60
    +def download_mozharness(mozharness_url: str) -> None:
    
    61
    +    zip_path = Path("mozharness.zip")
    
    62
    +    download_file(mozharness_url, zip_path)
    
    63
    +    with zipfile.ZipFile(zip_path) as zf:
    
    64
    +        zf.extractall("mozharness")
    
    65
    +    print("Extracted mozharness/")
    
    66
    +
    
    67
    +
    
    68
    +def download_minidump_stackwalk(
    
    69
    +    minidump_stackwalk_url: str, moz_fetches_dir: str, sha256: str = ""
    
    70
    +) -> None:
    
    71
    +    dest_dir = Path(moz_fetches_dir) / "minidump-stackwalk"
    
    72
    +    dest_dir.mkdir(parents=True, exist_ok=True)
    
    73
    +
    
    74
    +    tar_path = Path("minidump-stackwalk.tar.xz")
    
    75
    +    download_file(minidump_stackwalk_url, tar_path, sha256)
    
    76
    +    extract_tar(tar_path, dest_dir)
    
    77
    +    print(f"Extracted minidump-stackwalk -> {dest_dir}")
    
    78
    +
    
    79
    +
    
    80
    +def setup_android_sdk(
    
    81
    +    moz_fetches_dir: str,
    
    82
    +    android_sdk_repack_url: str,
    
    83
    +) -> None:
    
    84
    +    android_sdk_dir = Path(moz_fetches_dir) / "android-sdk-linux"
    
    85
    +    android_sdk_dir.mkdir(parents=True, exist_ok=True)
    
    86
    +
    
    87
    +    tar_zst_path = Path("android-sdk.tar.zst")
    
    88
    +    download_file(android_sdk_repack_url, tar_zst_path)
    
    89
    +    extract_tar(tar_zst_path, android_sdk_dir)
    
    90
    +
    
    91
    +    # Remove fake emulator binary added by tor-browser-build
    
    92
    +    emulator_dir = android_sdk_dir / "emulator"
    
    93
    +    if emulator_dir.exists():
    
    94
    +        shutil.rmtree(emulator_dir)
    
    95
    +        print("Removed fake emulator binary")
    
    96
    +
    
    97
    +    bin_dir = next((android_sdk_dir / "cmdline-tools").glob("*/bin"))
    
    98
    +    sdkmanager = bin_dir / "sdkmanager"
    
    99
    +    avdmanager = bin_dir / "avdmanager"
    
    100
    +    system_image = (
    
    101
    +        f"system-images;android-{EMULATOR_ANDROID_VERSION};google_apis;x86_64"
    
    102
    +    )
    
    103
    +
    
    104
    +    print("Installing emulator and system images via sdkmanager...")
    
    105
    +    subprocess.run(
    
    106
    +        [str(sdkmanager), "emulator", system_image],
    
    107
    +        # 10 is an arbitrary number. We just want to repeat the "y"
    
    108
    +        # as many times as needed to accept all licenses.
    
    109
    +        input="y\n" * 10,
    
    110
    +        text=True,
    
    111
    +        stdout=subprocess.DEVNULL,
    
    112
    +        check=False,
    
    113
    +    )
    
    114
    +
    
    115
    +    android_device_dir = Path(moz_fetches_dir) / "android-device"
    
    116
    +    android_device_dir.mkdir(parents=True, exist_ok=True)
    
    117
    +
    
    118
    +    # The AVD must be named exactly like this, otherwise upstream test code can't find it.
    
    119
    +    avd_name = f"mozemulator-android{EMULATOR_ANDROID_VERSION}-x86_64"
    
    120
    +    print(f"Creating AVD: {avd_name}")
    
    121
    +    subprocess.run(
    
    122
    +        [str(avdmanager), "create", "avd", "-n", avd_name, "-k", system_image],
    
    123
    +        # At the end of this command it will prompt the user for a custom
    
    124
    +        # configuration, inputting "no" will let it know we don't want it.
    
    125
    +        input="no\n",
    
    126
    +        text=True,
    
    127
    +        stdout=subprocess.DEVNULL,
    
    128
    +        check=False,
    
    129
    +    )
    
    130
    +
    
    131
    +    avd_source = Path.home() / ".android" / "avd"
    
    132
    +    shutil.move(str(avd_source), str(android_device_dir))
    
    133
    +    print(f"Moved AVD -> {android_device_dir / 'avd'}")
    
    134
    +
    
    135
    +
    
    136
    +def before_script(
    
    137
    +    mozharness_url: str,
    
    138
    +    minidump_stackwalk_url: str,
    
    139
    +    minidump_stackwalk_sha256: str,
    
    140
    +    moz_fetches_dir: str,
    
    141
    +    sha256sums_url: str = "",
    
    142
    +    android_sdk_repack_url: str = "",
    
    143
    +) -> None:
    
    144
    +    check_sha256sums_url(sha256sums_url)
    
    145
    +    download_mozharness(mozharness_url)
    
    146
    +    download_minidump_stackwalk(
    
    147
    +        minidump_stackwalk_url, moz_fetches_dir, minidump_stackwalk_sha256
    
    148
    +    )
    
    149
    +
    
    150
    +    if android_sdk_repack_url:
    
    151
    +        setup_android_sdk(
    
    152
    +            moz_fetches_dir,
    
    153
    +            android_sdk_repack_url,
    
    154
    +        )
    
    155
    +
    
    156
    +
    
    157
    +if __name__ == "__main__":
    
    158
    +    before_script(
    
    159
    +        mozharness_url=os.environ["MOZHARNESS_URL"],
    
    160
    +        minidump_stackwalk_url=os.environ["MINIDUMP_STACKWALK_URL"],
    
    161
    +        minidump_stackwalk_sha256=os.environ.get("MINIDUMP_STACKWALK_SHA256", ""),
    
    162
    +        moz_fetches_dir=os.environ["MOZ_FETCHES_DIR"],
    
    163
    +        sha256sums_url=os.environ.get("SHA256SUMS_URL", ""),
    
    164
    +        # Android-only stuff.
    
    165
    +        android_sdk_repack_url=os.environ.get("ANDROID_SDK_REPACK_URL", ""),
    
    166
    +    )

  • .gitlab/test_marionette.yml
    1
    +.marionette-desktop:
    
    2
    +  stage: test-marionette
    
    3
    +  script:
    
    4
    +    - >
    
    5
    +      python3 "$(pwd)/mozharness/scripts/marionette.py"
    
    6
    +      --config-file "$(pwd)/mozharness/configs/marionette/prod_config.py"
    
    7
    +      --installer-url "$INSTALLER_URL"
    
    8
    +      --test-packages-url "$ARTIFACTS_URL/target.test_packages.json"
    
    9
    +      --tag tor
    
    10
    +      --headless
    
    11
    +
    
    12
    +debian-x86_64_marionette:
    
    13
    +  extends:
    
    14
    +    - .debian-x86_64
    
    15
    +    - .marionette-desktop
    
    16
    +
    
    17
    +macos-x86_64_marionette:
    
    18
    +  extends:
    
    19
    +    - .macos-x86_64
    
    20
    +    - .marionette-desktop
    
    21
    +
    
    22
    +windows-x86_64_marionette:
    
    23
    +  extends:
    
    24
    +    - .windows-x86_64
    
    25
    +    - .marionette-desktop
    
    26
    +
    
    27
    +android-x86_64_marionette:
    
    28
    +  extends: .android-x86_64
    
    29
    +  stage: test-marionette
    
    30
    +  script:
    
    31
    +    - >
    
    32
    +      python3 "$(pwd)/mozharness/scripts/android_emulator_unittest.py"
    
    33
    +      --config-file "$(pwd)/mozharness/configs/android/android_common.py"
    
    34
    +      --config-file "$(pwd)/mozharness/configs/android/android14-x86_64.py"
    
    35
    +      --test-suite=marionette
    
    36
    +      --installer-url "$INSTALLER_URL"
    
    37
    +      --test-packages-url "$ARTIFACTS_URL/target.test_packages.json"
    
    38
    +      --package-name "$PACKAGE_NAME"
    
    39
    +      --tag tor

  • _______________________________________________
    tor-commits mailing list -- tor-commits@xxxxxxxxxxxxxxxxxxxx
    To unsubscribe send an email to tor-commits-leave@xxxxxxxxxxxxxxxxxxxx