Commits:
-
4be76d68
by Beatriz Rizental at 2025-12-09T17:27:56+01:00
Bug 41485: Add a script to retrieve gradle dependencies.
-
5bbb5bef
by Pier Angelo Vendrame at 2025-12-09T17:29:02+01:00
Bug 41485: Automate the Gradle deps update (Glean).
-
0e05b1e9
by Pier Angelo Vendrame at 2025-12-09T17:29:24+01:00
Bug 41485: Automate the Gradle deps update (Application Services).
-
09f06adb
by Pier Angelo Vendrame at 2025-12-09T17:29:42+01:00
Bug 41485: Automate the Gradle deps update (GeckoView).
-
40f3459b
by Pier Angelo Vendrame at 2025-12-09T17:35:18+01:00
Bug 41485: Documentation for the Gradle deps process.
12 changed files:
Changes:
doc/how-to-create-gradle-dependencies-list.txt
deleted
|
1
|
|
-### Updating Gradle Dependencies
|
|
2
|
|
-If additional Android dependencies are required by the project's build, then
|
|
3
|
|
-the Gradle build will fail due to missing dependencies. To find out what the
|
|
4
|
|
-missing dependencies are take the following steps.
|
|
5
|
|
-
|
|
6
|
|
-1) Updating gradle dependencies for `application-services`
|
|
7
|
|
-
|
|
8
|
|
-The following makefile rules can be used:
|
|
9
|
|
-
|
|
10
|
|
- make get_gradle_dependencies_list-application-services
|
|
11
|
|
-
|
|
12
|
|
-which should create the gradle-dependencies-list.txt files in the
|
|
13
|
|
-corresponding out/$project directory, using nightly's branch.
|
|
14
|
|
-
|
|
15
|
|
-
|
|
16
|
|
-2) Updating gradle dependencies for `geckoview`
|
|
17
|
|
-
|
|
18
|
|
-The `geckoview` project doesn't have (yet) a makefile rule to generate the
|
|
19
|
|
-gradle-dependencies-list.txt file, so a few more steps are needed.
|
|
20
|
|
-
|
|
21
|
|
-For the geckoview project, comment out the following line in the project's
|
|
22
|
|
-build_common file:
|
|
23
|
|
-
|
|
24
|
|
- export GRADLE_MAVEN_REPOSITORIES="file://$gradle_repo","file://$gradle_repo/maven2"
|
|
25
|
|
-
|
|
26
|
|
-Also modify the gradle flags to include the debug option so the download logs will show up:
|
|
27
|
|
-
|
|
28
|
|
- export GRADLE_FLAGS="--no-daemon --debug"
|
|
29
|
|
-
|
|
30
|
|
-Otherwise, when calling gradle in the project's build script replace the
|
|
31
|
|
-`--offline` flag with `--debug` and remove any `-Dmaven.repo.local` arguments.
|
|
32
|
|
-
|
|
33
|
|
-Finally, allow network access during the build by setting
|
|
34
|
|
-`container/disable_network/build` to `0` in rbm.conf, and rerun the build.
|
|
35
|
|
-
|
|
36
|
|
-Dependent artifacts will show up as downloads in the logs. You can pull out
|
|
37
|
|
-these dependencies into a list by passing the log file to the gradle dependency
|
|
38
|
|
-list script in the projects/common directory:
|
|
39
|
|
-
|
|
40
|
|
-`projects/common/gen_gradle_deps_file.sh /path/to/log/file`
|
|
41
|
|
-
|
|
42
|
|
-
|
|
43
|
|
-3) Copying the resulting `gradle-dependencies-list.txt`
|
|
44
|
|
-
|
|
45
|
|
-Copy the resulting `gradle-dependencies-list.txt` (from step 1. or 2.)
|
|
46
|
|
-over the one in the respective project. Then, in the project's config
|
|
47
|
|
-file, increment the `var/gradle_dependencies_version` and make sure to
|
|
48
|
|
-restore the project's build file back to original if you modified it. |
doc/how-to-update-gradle-dependencies-list.md
|
|
1
|
+# Gradle Dependencies
|
|
|
2
|
+
|
|
|
3
|
+## TL; DR
|
|
|
4
|
+
|
|
|
5
|
+1. Open `projects/<project name>/config`.
|
|
|
6
|
+2. Look for `# generate_gradle_dependencies_list: 1` and uncomment it.
|
|
|
7
|
+3. Run `rbm/rbm build <project name> --target nightly --target torbrowser-android-<arch>`.
|
|
|
8
|
+ For GeckoView, make sure to run a single-arch build to catch all dependencies
|
|
|
9
|
+ at once.
|
|
|
10
|
+ Testbuilds (i.e., `make torbrowser-testbuild-android-<arch>`) are single-arch
|
|
|
11
|
+ by default, but you can also set `var/android_single_arch`.
|
|
|
12
|
+4. Find the corresponding output: `ls -lt out/<project name> | head`.
|
|
|
13
|
+5. Extract it.
|
|
|
14
|
+6. Move the `gradle-dependencies-list.txt` you just extracted to
|
|
|
15
|
+ `projects/<project name>/`.
|
|
|
16
|
+7. Bump `var/gradle_dependencies_version`.
|
|
|
17
|
+ The preferred format is `<project version>-<list version>`.
|
|
|
18
|
+
|
|
|
19
|
+Theoretically, it should be also possible to set
|
|
|
20
|
+`generate_gradle_dependencies_list: 1` in `rbm.local.conf`, run a full build and
|
|
|
21
|
+update all dependencies at once.
|
|
|
22
|
+
|
|
|
23
|
+## Rationale
|
|
|
24
|
+
|
|
|
25
|
+Many of our Android projects use Gradle to build, which also handles
|
|
|
26
|
+dependencies.
|
|
|
27
|
+
|
|
|
28
|
+For reproducibility purposes, we need to stabilize them, so we version a list of
|
|
|
29
|
+dependencies that are known for working in a file called
|
|
|
30
|
+`gradle-dependencies-list.txt`.
|
|
|
31
|
+
|
|
|
32
|
+This file has a simple format:
|
|
|
33
|
+
|
|
|
34
|
+```
|
|
|
35
|
+# Lines starting with '#' are comments
|
|
|
36
|
+<sha265> | <URL>
|
|
|
37
|
+```
|
|
|
38
|
+
|
|
|
39
|
+In addition to guaranteeing reproducibility, this makes sure we do not get
|
|
|
40
|
+poisoned dependencies in subsequent builds.
|
|
|
41
|
+
|
|
|
42
|
+## Creating and/or updating the list
|
|
|
43
|
+
|
|
|
44
|
+Gradle does not have official tooling to create such lists.
|
|
|
45
|
+
|
|
|
46
|
+Instead, we pass `--info` to make sure Gradle logs information about downloads.
|
|
|
47
|
+Therefore, these logs need to be captured to a file. Most commonly, Gradle is
|
|
|
48
|
+invoked directly and in that case appending something like
|
|
|
49
|
+`2>&1 | tee -a gradle-output.log` to the command line is enough.
|
|
|
50
|
+
|
|
|
51
|
+After that, we parse the logs with the `projects/common/gen-gradle-deps-file.py`
|
|
|
52
|
+script:
|
|
|
53
|
+
|
|
|
54
|
+```shell
|
|
|
55
|
+gen-gradle-deps-file.py gradle-output.log
|
|
|
56
|
+```
|
|
|
57
|
+
|
|
|
58
|
+This generates a new `gradle-dependencies-list.txt`.
|
|
|
59
|
+
|
|
|
60
|
+This procedure can be run outside a tor-browser-build container, but for best compatibility,
|
|
|
61
|
+it should run inside it.
|
|
|
62
|
+
|
|
|
63
|
+Most of our Android projects have been equipped with a flag
|
|
|
64
|
+(`var/generate_gradle_dependencies_list`) to do this automatically.
|
|
|
65
|
+
|
|
|
66
|
+Therefore, it is possible to just enable this flag, and grab an updated file
|
|
|
67
|
+from the output artifact.
|
|
|
68
|
+
|
|
|
69
|
+New Android projects should be setup to do this.
|
|
|
70
|
+
|
|
|
71
|
+> **Important**: dependencies are keyed and cached. So, after updating the list,
|
|
|
72
|
+> you must also bump `var/gradle_dependencies_version`.
|
|
|
73
|
+
|
|
|
74
|
+This used to be an integer, but to avoid clashes between the main and
|
|
|
75
|
+maintenance branches, we decided to switch to a
|
|
|
76
|
+`<project version>-<list version>` format.
|
|
|
77
|
+Usually, the list version will be `1`.
|
|
|
78
|
+
|
|
|
79
|
+## Consuming the list
|
|
|
80
|
+
|
|
|
81
|
+`projects/common/fetch-gradle-dependencies` is the consumer of the list.
|
|
|
82
|
+
|
|
|
83
|
+It ignores the hash of the `.pom` files, as they might change, but it checks the
|
|
|
84
|
+hashes of all the other artifacts.
|
|
|
85
|
+
|
|
|
86
|
+You can wire it in your project with code that looks like this:
|
|
|
87
|
+
|
|
|
88
|
+```yaml
|
|
|
89
|
+- filename: 'gradle-dependencies-[% c("var/gradle_dependencies_version") %]'
|
|
|
90
|
+ name: gradle-dependencies
|
|
|
91
|
+ exec: '[% INCLUDE "fetch-gradle-dependencies" %]'
|
|
|
92
|
+ enable: '[% !c("var/generate_gradle_dependencies_list") %]'
|
|
|
93
|
+```
|
|
|
94
|
+
|
|
|
95
|
+Then, you will have to patch your project to look for offline dependencies, too.
|
|
|
96
|
+But this is project-specific.
|
|
|
97
|
+
|
|
|
98
|
+Generally speaking, you will have to add a `mavenLocal()` entry to the
|
|
|
99
|
+`repositories` section in some `build.gradle` file. |
projects/application-services/build
| ... |
... |
@@ -24,7 +24,7 @@ export JAVA_HOME=/usr/lib/jvm/java-1.17.0-openjdk-amd64 |
|
24
|
24
|
|
|
25
|
25
|
[% INCLUDE 'fake-git' %]
|
|
26
|
26
|
|
|
27
|
|
-[% IF !c('var/fetch_gradle_dependencies') %]
|
|
|
27
|
+[% IF !c('var/generate_gradle_dependencies_list') -%]
|
|
28
|
28
|
tar -xf [% c('input_files_by_name/glean') %]
|
|
29
|
29
|
|
|
30
|
30
|
gradle_repo=/var/tmp/dist/gradle-dependencies
|
| ... |
... |
@@ -33,7 +33,7 @@ export JAVA_HOME=/usr/lib/jvm/java-1.17.0-openjdk-amd64 |
|
33
|
33
|
cp -rl $gradle_repo/dl/android/maven2/* $gradle_repo || true
|
|
34
|
34
|
cp -rl $gradle_repo/m2/* $gradle_repo || true
|
|
35
|
35
|
cp -rl $gradle_repo/maven2/* $gradle_repo || true
|
|
36
|
|
-[% END %]
|
|
|
36
|
+[% END -%]
|
|
37
|
37
|
|
|
38
|
38
|
tar -C /var/tmp/build -xf [% project %]-[% c('version') %].tar.[% c('compress_tar') %]
|
|
39
|
39
|
|
| ... |
... |
@@ -41,24 +41,26 @@ mv /var/tmp/build/[% project %]-[% c('version') %] $builddir |
|
41
|
41
|
|
|
42
|
42
|
cd $builddir
|
|
43
|
43
|
|
|
44
|
|
-# Prepare the offline build.
|
|
|
44
|
+[% IF !c('var/generate_gradle_dependencies_list') -%]
|
|
|
45
|
+ # Prepare the offline build.
|
|
45
|
46
|
|
|
46
|
|
-tar -xf $rootdir/[% c('input_files_by_name/glean-wheels') %]
|
|
47
|
|
-export GLEAN_PYTHON_WHEELS_DIR=$builddir/glean-wheels
|
|
48
|
|
-# Setting GLEAN_PYTHON_WHEELS_DIR is not enough: a Rust build script will still
|
|
49
|
|
-# try to install Glean with pip.
|
|
50
|
|
-cat > $rootdir/pip.conf << 'EOF'
|
|
|
47
|
+ tar -xf $rootdir/[% c('input_files_by_name/glean-wheels') %]
|
|
|
48
|
+ export GLEAN_PYTHON_WHEELS_DIR=$builddir/glean-wheels
|
|
|
49
|
+ # Setting GLEAN_PYTHON_WHEELS_DIR is not enough: a Rust build script will
|
|
|
50
|
+ # still try to install Glean with pip.
|
|
|
51
|
+ cat > $rootdir/pip.conf << 'EOF'
|
|
51
|
52
|
[global]
|
|
52
|
53
|
find-links = /var/tmp/build/application-services/glean-wheels
|
|
53
|
54
|
index-url =
|
|
54
|
55
|
no-index = yes
|
|
55
|
56
|
EOF
|
|
56
|
|
-export PIP_CONFIG_FILE=$rootdir/pip.conf
|
|
57
|
|
-# Move the directory for hardcoding the path in .cargo/config.
|
|
58
|
|
-tar -C $builddir -xf $rootdir/[% c('input_files_by_name/cargo_vendor') %]
|
|
59
|
|
-# Make sure our vendored crates are used for offline builds.
|
|
60
|
|
-cat vendor/cargo-config.toml >> .cargo/config.toml
|
|
61
|
|
-rm vendor/cargo-config.toml
|
|
|
57
|
+ export PIP_CONFIG_FILE=$rootdir/pip.conf
|
|
|
58
|
+ # Move the directory for hardcoding the path in .cargo/config.
|
|
|
59
|
+ tar -C $builddir -xf $rootdir/[% c('input_files_by_name/cargo_vendor') %]
|
|
|
60
|
+ # Make sure our vendored crates are used for offline builds.
|
|
|
61
|
+ cat vendor/cargo-config.toml >> .cargo/config.toml
|
|
|
62
|
+ rm vendor/cargo-config.toml
|
|
|
63
|
+[% END -%]
|
|
62
|
64
|
|
|
63
|
65
|
tar -xf $rootdir/[% c('input_files_by_name/uniffi-rs') %]
|
|
64
|
66
|
|
| ... |
... |
@@ -68,30 +70,33 @@ ln -s $rootdir/[% c("input_files_by_name/nss") %] ./ |
|
68
|
70
|
./build-all.sh android
|
|
69
|
71
|
popd
|
|
70
|
72
|
|
|
71
|
|
-[% IF c('var/fetch_gradle_dependencies') %]
|
|
72
|
|
- # XXX: `assemble` is still not enough to see all fetched dependencies via
|
|
73
|
|
- # Gradle's --debug. See: tor-browser-build#40056.
|
|
74
|
|
- gradle --debug --no-daemon assemble > gradle.log 2>&1 || ( cat gradle.log; exit 1 )
|
|
75
|
|
- $rootdir/gen_gradle_deps_file.sh gradle.log
|
|
76
|
|
- mv gradle-dependencies-list.txt '[% dest_dir _ "/" _ c("filename") %]'
|
|
77
|
|
-[% ELSE %]
|
|
78
|
|
- # We need to set `LC_ALL` and `LANG` to something that is not ASCII as encoding
|
|
79
|
|
- # otherwise `click` barfs. See: https://click.palletsprojects.com/python3/
|
|
80
|
|
- export LC_ALL=C.UTF-8
|
|
81
|
|
- export LANG=C.UTF-8
|
|
82
|
|
- gradle_flags="--offline --no-daemon -PuniffiBindgenNoop=$builddir/uniffi-rs/uniffi-bindgen -PgradleDependenciesUrl=file:///var/tmp/dist/gradle-dependencies"
|
|
83
|
|
- gradle $gradle_flags assembleRelease
|
|
84
|
|
- gradle $gradle_flags publish
|
|
85
|
|
- mv build/maven $distdir
|
|
86
|
|
-
|
|
87
|
|
- pushd components/support/nimbus-fml
|
|
88
|
|
- cargo build --release
|
|
89
|
|
- popd
|
|
90
|
|
- cp target/release/nimbus-fml $distdir
|
|
91
|
|
-
|
|
92
|
|
- cd /var/tmp/dist
|
|
93
|
|
- [% c('tar', {
|
|
94
|
|
- tar_src => [ project ],
|
|
95
|
|
- tar_args => '-caf ' _ dest_dir _ '/' _ c('filename'),
|
|
|
73
|
+# We need to set `LC_ALL` and `LANG` to something that is not ASCII as encoding
|
|
|
74
|
+# otherwise `click` barfs. See: https://click.palletsprojects.com/python3/
|
|
|
75
|
+export LC_ALL=C.UTF-8
|
|
|
76
|
+export LANG=C.UTF-8
|
|
|
77
|
+[% IF !c('var/generate_gradle_dependencies_list') -%]
|
|
|
78
|
+ deps_flags="--offline -PgradleDependenciesUrl=file:///var/tmp/dist/gradle-dependencies"
|
|
|
79
|
+[% ELSE -%]
|
|
|
80
|
+ deps_flags="--info"
|
|
|
81
|
+ gradle_logs=/var/tmp/build/gradle-logs.log
|
|
|
82
|
+[% END -%]
|
|
|
83
|
+gradle_flags="$deps_flags --no-daemon -PuniffiBindgenNoop=$builddir/uniffi-rs/uniffi-bindgen"
|
|
|
84
|
+gradle $gradle_flags assembleRelease [% IF c('var/generate_gradle_dependencies_list') %]2>&1 | tee -a $gradle_logs[% END %]
|
|
|
85
|
+gradle $gradle_flags publish
|
|
|
86
|
+mv build/maven $distdir
|
|
|
87
|
+
|
|
|
88
|
+pushd components/support/nimbus-fml
|
|
|
89
|
+cargo build --release
|
|
|
90
|
+popd
|
|
|
91
|
+cp target/release/nimbus-fml $distdir
|
|
|
92
|
+
|
|
|
93
|
+[% IF c('var/generate_gradle_dependencies_list') -%]
|
|
|
94
|
+ cd $distdir
|
|
|
95
|
+ $rootdir/gen-gradle-deps-file.py $gradle_logs
|
|
|
96
|
+[% END -%]
|
|
|
97
|
+
|
|
|
98
|
+cd /var/tmp/dist
|
|
|
99
|
+[% c('tar', {
|
|
|
100
|
+ tar_src => [ project ],
|
|
|
101
|
+ tar_args => '-caf ' _ dest_dir _ '/' _ c('filename'),
|
|
96
|
102
|
}) %] |
|
97
|
|
-[% END %] |
projects/application-services/config
| ... |
... |
@@ -14,6 +14,8 @@ gpg_keyring: |
|
14
|
14
|
git_submodule: 1
|
|
15
|
15
|
container:
|
|
16
|
16
|
use_container: 1
|
|
|
17
|
+ disable_network:
|
|
|
18
|
+ build: '[% !c("var/generate_gradle_dependencies_list") %]'
|
|
17
|
19
|
|
|
18
|
20
|
var:
|
|
19
|
21
|
build_number: 1
|
| ... |
... |
@@ -23,6 +25,9 @@ var: |
|
23
|
25
|
nss_version: '3.117'
|
|
24
|
26
|
nspr_version: '4.37'
|
|
25
|
27
|
cargo_vendor_include_config: 1
|
|
|
28
|
+ # Uncomment this to run an online build to grab an updated
|
|
|
29
|
+ # gradle-dependencies-list.txt.
|
|
|
30
|
+ # generate_gradle_dependencies_list: 1
|
|
26
|
31
|
|
|
27
|
32
|
targets:
|
|
28
|
33
|
nightly:
|
| ... |
... |
@@ -61,7 +66,7 @@ steps: |
|
61
|
66
|
- project: glean
|
|
62
|
67
|
name: glean
|
|
63
|
68
|
pkg_type: build
|
|
64
|
|
- enable: '[% !c("var/fetch_gradle_dependencies") %]'
|
|
|
69
|
+ enable: '[% !c("var/generate_gradle_dependencies_list") %]'
|
|
65
|
70
|
# Only Application Services currently requires build tools 35.
|
|
66
|
71
|
# So, download them only here, rather than adding them to the shared
|
|
67
|
72
|
# toolchain.
|
| ... |
... |
@@ -75,28 +80,22 @@ steps: |
|
75
|
80
|
- filename: 'gradle-dependencies-[% c("var/gradle_dependencies_version") %]'
|
|
76
|
81
|
name: gradle-dependencies
|
|
77
|
82
|
exec: '[% INCLUDE "fetch-gradle-dependencies" %]'
|
|
78
|
|
- enable: '[% !c("var/fetch_gradle_dependencies") %]'
|
|
|
83
|
+ enable: '[% !c("var/generate_gradle_dependencies_list") %]'
|
|
79
|
84
|
- URL: '[% pc("glean-parser", "var/glean_wheels_url") %]'
|
|
80
|
85
|
name: glean-wheels
|
|
81
|
86
|
sha256sum: '[% pc("glean-parser", "var/glean_wheels_sha256sum") %]'
|
|
82
|
|
- enable: '[% !c("var/fetch_gradle_dependencies") %]'
|
|
|
87
|
+ enable: '[% !c("var/generate_gradle_dependencies_list") %]'
|
|
83
|
88
|
- name: cargo_vendor
|
|
84
|
89
|
project: application-services
|
|
85
|
90
|
pkg_type: cargo_vendor
|
|
86
|
91
|
norec:
|
|
87
|
92
|
sha256sum: 8a93a189985ad28c334970dc5ab386c81d4a415f3777c2941248f836f4af44fd
|
|
88
|
|
- enable: '[% !c("var/fetch_gradle_dependencies") %]'
|
|
89
|
|
- - filename: gen_gradle_deps_file.sh
|
|
90
|
|
- enable: '[% c("var/fetch_gradle_dependencies") %]'
|
|
|
93
|
+ enable: '[% !c("var/generate_gradle_dependencies_list") %]'
|
|
|
94
|
+ - filename: gen-gradle-deps-file.py
|
|
|
95
|
+ enable: '[% c("var/generate_gradle_dependencies_list") %]'
|
|
91
|
96
|
|
|
92
|
97
|
list_toolchain_updates:
|
|
93
|
98
|
git_hash: 'v[% c("version") %]'
|
|
94
|
99
|
input_files: []
|
|
95
|
100
|
container:
|
|
96
|
101
|
use_container: 0 |
|
97
|
|
-
|
|
98
|
|
- get_gradle_dependencies_list:
|
|
99
|
|
- filename: 'gradle-dependencies-list-[% c("version") %].txt'
|
|
100
|
|
- get_gradle_dependencies_list: '[% INCLUDE build %]'
|
|
101
|
|
- var:
|
|
102
|
|
- fetch_gradle_dependencies: 1 |
projects/common/gen-gradle-deps-file.py
|
|
1
|
+#!/usr/bin/env python3
|
|
|
2
|
+
|
|
|
3
|
+import argparse
|
|
|
4
|
+import hashlib
|
|
|
5
|
+import re
|
|
|
6
|
+import sys
|
|
|
7
|
+import urllib.request
|
|
|
8
|
+from concurrent.futures import ThreadPoolExecutor, as_completed
|
|
|
9
|
+
|
|
|
10
|
+
|
|
|
11
|
+def download_and_hash(url):
|
|
|
12
|
+ try:
|
|
|
13
|
+ sha256_hash = hashlib.sha256()
|
|
|
14
|
+ with urllib.request.urlopen(url) as f:
|
|
|
15
|
+ sha256_hash.update(f.read())
|
|
|
16
|
+ return url, sha256_hash.hexdigest(), None
|
|
|
17
|
+ except Exception as e:
|
|
|
18
|
+ return url, None, str(e)
|
|
|
19
|
+
|
|
|
20
|
+
|
|
|
21
|
+def parse_arguments():
|
|
|
22
|
+ parser = argparse.ArgumentParser(
|
|
|
23
|
+ description="""
|
|
|
24
|
+This script generates a `gradle-dependencies-list.txt` file as expected by
|
|
|
25
|
+Tor Browser Build to download dependencies for offline builds.
|
|
|
26
|
+
|
|
|
27
|
+It parses a log file from a Gradle build, determines which dependencies were
|
|
|
28
|
+downloaded and from where, re-downloads them, and calculates their SHA256 hash.
|
|
|
29
|
+Downloads are parallelized — by default, 4 workers are used, but this can be
|
|
|
30
|
+configured via the optional `--workers` argument.
|
|
|
31
|
+
|
|
|
32
|
+---
|
|
|
33
|
+
|
|
|
34
|
+Usage:
|
|
|
35
|
+
|
|
|
36
|
+First, run the exact Gradle command that will be used inside the offline
|
|
|
37
|
+tor-browser-build container. Make sure to set the log level to `info` and
|
|
|
38
|
+capture the full output to a log file.
|
|
|
39
|
+
|
|
|
40
|
+`--refresh-dependencies` can be pass to Gradle to force all dependencies to be
|
|
|
41
|
+re-downloaded if needed (e.g., if running this script outside containers or
|
|
|
42
|
+outside tor-browser-build).
|
|
|
43
|
+
|
|
|
44
|
+Example:
|
|
|
45
|
+
|
|
|
46
|
+ ./gradlew --info clean assembleRelease 2>&1 | tee -a out.log
|
|
|
47
|
+
|
|
|
48
|
+Then run this script with the generated log:
|
|
|
49
|
+
|
|
|
50
|
+ ./gen-gradle-deps-file.py out.log
|
|
|
51
|
+
|
|
|
52
|
+This will create a `gradle-dependencies-list.txt` file in the current
|
|
|
53
|
+directory.
|
|
|
54
|
+""",
|
|
|
55
|
+ formatter_class=argparse.RawDescriptionHelpFormatter,
|
|
|
56
|
+ )
|
|
|
57
|
+
|
|
|
58
|
+ parser.add_argument(
|
|
|
59
|
+ "log_file", help="Path to the Gradle log file (e.g., out.log)"
|
|
|
60
|
+ )
|
|
|
61
|
+ parser.add_argument(
|
|
|
62
|
+ "--workers",
|
|
|
63
|
+ type=int,
|
|
|
64
|
+ default=4,
|
|
|
65
|
+ help="Number of parallel download workers (default: 4).",
|
|
|
66
|
+ )
|
|
|
67
|
+ parser.add_argument(
|
|
|
68
|
+ "--output",
|
|
|
69
|
+ default="gradle-dependencies-list.txt",
|
|
|
70
|
+ help="Output file path (default: gradle-dependencies-list.txt).",
|
|
|
71
|
+ )
|
|
|
72
|
+
|
|
|
73
|
+ return parser.parse_args()
|
|
|
74
|
+
|
|
|
75
|
+
|
|
|
76
|
+def extract_urls(log_file):
|
|
|
77
|
+ pattern = re.compile(rf"Downloading (https://\S+) to ")
|
|
|
78
|
+ urls = []
|
|
|
79
|
+ with open(log_file, "r") as file:
|
|
|
80
|
+ for line in file:
|
|
|
81
|
+ match = pattern.search(line.strip())
|
|
|
82
|
+ if match:
|
|
|
83
|
+ urls.append(match.group(1))
|
|
|
84
|
+ return urls
|
|
|
85
|
+
|
|
|
86
|
+
|
|
|
87
|
+def main():
|
|
|
88
|
+ args = parse_arguments()
|
|
|
89
|
+
|
|
|
90
|
+ urls = extract_urls(args.log_file)
|
|
|
91
|
+ if not urls:
|
|
|
92
|
+ print("⚠️ No matching download lines found in the log file.")
|
|
|
93
|
+ return
|
|
|
94
|
+
|
|
|
95
|
+ print(
|
|
|
96
|
+ f"🔍 Found {len(urls)} URLs. Starting downloads using {args.workers} workers...\n"
|
|
|
97
|
+ )
|
|
|
98
|
+
|
|
|
99
|
+ results = []
|
|
|
100
|
+ with ThreadPoolExecutor(max_workers=args.workers) as executor:
|
|
|
101
|
+ futures = {
|
|
|
102
|
+ executor.submit(download_and_hash, url): url for url in urls
|
|
|
103
|
+ }
|
|
|
104
|
+ for future in as_completed(futures):
|
|
|
105
|
+ url, sha256, error = future.result()
|
|
|
106
|
+ if error:
|
|
|
107
|
+ print(f"[ERROR] {url}\n Reason: {error}")
|
|
|
108
|
+ else:
|
|
|
109
|
+ print(f"[OK] {url}\n SHA256: {sha256}")
|
|
|
110
|
+ results.append((sha256, url))
|
|
|
111
|
+
|
|
|
112
|
+ if results:
|
|
|
113
|
+ with open(args.output, "w") as out_file:
|
|
|
114
|
+ out_file.write(
|
|
|
115
|
+ "# Don't forget to update var/gradle_dependencies_version when modifying this file\n"
|
|
|
116
|
+ )
|
|
|
117
|
+ out_file.write("sha256sum | url\n")
|
|
|
118
|
+ for sha256, url in sorted(results, key=lambda x: x[1]):
|
|
|
119
|
+ out_file.write(f"{sha256} | {url}\n")
|
|
|
120
|
+ print(f"\n✅ {len(urls)} dependencies written to: {args.output}")
|
|
|
121
|
+ else:
|
|
|
122
|
+ print("\n⚠️ No successful downloads to write.")
|
|
|
123
|
+ sys.exit(1)
|
|
|
124
|
+
|
|
|
125
|
+
|
|
|
126
|
+if __name__ == "__main__":
|
|
|
127
|
+ main() |
projects/common/gen_gradle_deps_file.sh
deleted
|
1
|
|
-#!/bin/bash
|
|
2
|
|
-
|
|
3
|
|
-# Copyright (c) 2020, The Tor Project, Inc.
|
|
4
|
|
-#
|
|
5
|
|
-# Redistribution and use in source and binary forms, with or without
|
|
6
|
|
-# modification, are permitted provided that the following conditions are
|
|
7
|
|
-# met:
|
|
8
|
|
-#
|
|
9
|
|
-# * Redistributions of source code must retain the above copyright
|
|
10
|
|
-# notice, this list of conditions and the following disclaimer.
|
|
11
|
|
-#
|
|
12
|
|
-# * Redistributions in binary form must reproduce the above
|
|
13
|
|
-# copyright notice, this list of conditions and the following disclaimer
|
|
14
|
|
-# in the documentation and/or other materials provided with the
|
|
15
|
|
-# distribution.
|
|
16
|
|
-#
|
|
17
|
|
-# * Neither the names of the copyright owners nor the names of its
|
|
18
|
|
-# contributors may be used to endorse or promote products derived from
|
|
19
|
|
-# this software without specific prior written permission.
|
|
20
|
|
-#
|
|
21
|
|
-# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
|
|
22
|
|
-# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
|
|
23
|
|
-# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
|
|
24
|
|
-# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
|
|
25
|
|
-# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
|
|
26
|
|
-# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
|
|
27
|
|
-# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
|
|
28
|
|
-# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
|
|
29
|
|
-# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
|
|
30
|
|
-# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
|
|
31
|
|
-# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
|
32
|
|
-
|
|
33
|
|
-# Usage:
|
|
34
|
|
-# 1) Point to a log file with all the dependency download attempts (for its
|
|
35
|
|
-# generation see: doc/how-to-create-gradle-dependencies-list.txt)
|
|
36
|
|
-# 2) Double-check that you get the same SHA-256 sums when downloaded from a
|
|
37
|
|
-# different network location. E.g. by using `torsocks` with this script after
|
|
38
|
|
-# having made a copy of `gradle-dependencies-list.txt` from 1) and comparing
|
|
39
|
|
-# the two .txt files.
|
|
40
|
|
-
|
|
41
|
|
-export LC_ALL=C
|
|
42
|
|
-log="$1"
|
|
43
|
|
-
|
|
44
|
|
-# Step 1: Extract all the download attempts out of the log file, ignore the ones
|
|
45
|
|
-# for maven-metadata.xml and module files. We don't need those.
|
|
46
|
|
-cat $log | grep "Performing HTTP" | grep -o "https://.*" | \
|
|
47
|
|
- grep -vE "\.module|maven-metadata\.xml" | sort | uniq > dl-attempts
|
|
48
|
|
-
|
|
49
|
|
-# Step 2: Fetch all the dependencies and calculate the SHA-256 sum
|
|
50
|
|
-declare -A URLs
|
|
51
|
|
-function dl_url {
|
|
52
|
|
- local url="$1"
|
|
53
|
|
- test -n "${URLs[$url]}" && return 0
|
|
54
|
|
- URLs[$url]=1
|
|
55
|
|
- wget -U "" $url
|
|
56
|
|
- test $? -eq 0 || return 1
|
|
57
|
|
- local fn=$(basename "$url")
|
|
58
|
|
- local sha256=`sha256sum $fn | cut -d ' ' -f 1`
|
|
59
|
|
- echo "$sha256 | $url" >> deps
|
|
60
|
|
- rm $fn
|
|
61
|
|
-}
|
|
62
|
|
-
|
|
63
|
|
-while read line
|
|
64
|
|
-do
|
|
65
|
|
- dl_url "$line"
|
|
66
|
|
- # If we downloaded a .pom file, also try to get corresponding .jar and
|
|
67
|
|
- # .aar files
|
|
68
|
|
- bname=$(basename "$line" .pom)
|
|
69
|
|
- dname=$(dirname "$line")
|
|
70
|
|
- if test "$line" != "$dname/$bname"
|
|
71
|
|
- then
|
|
72
|
|
- dl_url "$dname/$bname.aar"
|
|
73
|
|
- dl_url "$dname/$bname.jar"
|
|
74
|
|
- fi
|
|
75
|
|
-done < dl-attempts
|
|
76
|
|
-
|
|
77
|
|
-# Step 3: Add the header at the beginning of the final dependency file.
|
|
78
|
|
-echo "# On how to update dependencies see doc/how-to-create-gradle\
|
|
79
|
|
--dependencies-list.txt" > gradle-dependencies-list.txt
|
|
80
|
|
-echo "# Don't forget to update var/gradle_dependencies_version when modifying \
|
|
81
|
|
-this file" >> gradle-dependencies-list.txt
|
|
82
|
|
-echo "sha256sum | url" >> gradle-dependencies-list.txt
|
|
83
|
|
-
|
|
84
|
|
-# Step 4: Keep only successfully downloaded artifacts, remove duplicates, and
|
|
85
|
|
-# sort based on download URL.
|
|
86
|
|
-grep ^[a-f0-9] deps | rev | sort -t/ -u -k1,4 | rev | \
|
|
87
|
|
- sort -k 3 >> gradle-dependencies-list.txt
|
|
88
|
|
-
|
|
89
|
|
-# Step 5: Clean up
|
|
90
|
|
-rm dl-attempts
|
|
91
|
|
-rm deps
|
|
92
|
|
-
|
|
93
|
|
-exit 0 |
projects/geckoview/build
| ... |
... |
@@ -23,7 +23,7 @@ echo "Starting ./mach configure $(date)" |
|
23
|
23
|
[% IF !c("var/rlbox") -%]--without-wasm-sandboxed-libraries[% END %]
|
|
24
|
24
|
|
|
25
|
25
|
echo "Starting ./mach build $(date)"
|
|
26
|
|
-./mach build --verbose
|
|
|
26
|
+./mach build --verbose [% IF c('var/generate_gradle_dependencies_list') %]2>&1 | tee -a $gradle_logs[% END %]
|
|
27
|
27
|
|
|
28
|
28
|
[% IF c("var/dev_artifacts") -%]
|
|
29
|
29
|
echo "Building development artifacts"
|
| ... |
... |
@@ -50,6 +50,11 @@ echo "Build finished, copying the AAR to the to the destination directory $(date |
|
50
|
50
|
[% IF !c("var/android_single_arch") -%]
|
|
51
|
51
|
mkdir -p "$outdir/[% project %]"
|
|
52
|
52
|
cp obj-*/gradle/target.maven.zip $outdir/[% project %]/
|
|
|
53
|
+
|
|
|
54
|
+ [% IF c('var/generate_gradle_dependencies_list') -%]
|
|
|
55
|
+ cd $outdir/[% project %]
|
|
|
56
|
+ $rootdir/gen-gradle-deps-file.py $gradle_logs
|
|
|
57
|
+ [% END -%]
|
|
53
|
58
|
[% ELSE -%]
|
|
54
|
59
|
[% INCLUDE 'build_ac_fenix' %]
|
|
55
|
60
|
[% END -%] |
projects/geckoview/build_ac_fenix
| ... |
... |
@@ -8,7 +8,7 @@ objdir=$(cd obj-* && pwd) |
|
8
|
8
|
|
|
9
|
9
|
echo "Building Android Components $(date)"
|
|
10
|
10
|
pushd mobile/android/android-components
|
|
11
|
|
-gradle $GRADLE_FLAGS assembleGecko -x lint
|
|
|
11
|
+gradle $GRADLE_FLAGS assembleGecko -x lint [% IF c('var/generate_gradle_dependencies_list') %]2>&1 | tee -a $gradle_logs[% END %]
|
|
12
|
12
|
popd
|
|
13
|
13
|
|
|
14
|
14
|
# The build might fail with "file exists" otherwise.
|
| ... |
... |
@@ -36,7 +36,7 @@ variant='[% c("var/variant") %]' |
|
36
|
36
|
version_name="[% c('var/torbrowser_version') %] ([% c('var/geckoview_version') %])"
|
|
37
|
37
|
|
|
38
|
38
|
echo "Building $variant Fenix APK"
|
|
39
|
|
-gradle $GRADLE_FLAGS -PversionName="$version_name" "assemble$variant"
|
|
|
39
|
+gradle $GRADLE_FLAGS -PversionName="$version_name" "assemble$variant" [% IF c('var/generate_gradle_dependencies_list') %]2>&1 | tee -a $gradle_logs[% END %]
|
|
40
|
40
|
|
|
41
|
41
|
echo "Build finished, copying the APK(s) to the destination directory $(date)"
|
|
42
|
42
|
mkdir -p $outdir/[% project %]
|
| ... |
... |
@@ -46,7 +46,7 @@ echo "Building non optimized $variant Fenix APK for testing" |
|
46
|
46
|
gradle $GRADLE_FLAGS -PversionName="$version_name" -PdisableOptimization "assemble$variant"
|
|
47
|
47
|
|
|
48
|
48
|
echo "Building Fenix instrumentation tests"
|
|
49
|
|
-gradle $GRADLE_FLAGS -PversionName="$version_name" -PtestBuildType="$variant" -PdisableOptimization assembleAndroidTest
|
|
|
49
|
+gradle $GRADLE_FLAGS -PversionName="$version_name" -PtestBuildType="$variant" -PdisableOptimization assembleAndroidTest [% IF c('var/generate_gradle_dependencies_list') %]2>&1 | tee -a $gradle_logs[% END %]
|
|
50
|
50
|
|
|
51
|
51
|
echo "Test build finished, copying the APKs to the destination directory $(date)"
|
|
52
|
52
|
|
| ... |
... |
@@ -55,3 +55,8 @@ mv $objdir/gradle/build/mobile/android/fenix/app/outputs/apk/fenix/$variant/*.ap |
|
55
|
55
|
mv $objdir/gradle/build/mobile/android/fenix/app/outputs/apk/androidTest/fenix/$variant/*.apk $outdir/[% project %]/tests
|
|
56
|
56
|
|
|
57
|
57
|
popd
|
|
|
58
|
+
|
|
|
59
|
+[% IF c('var/generate_gradle_dependencies_list') -%]
|
|
|
60
|
+ cd $outdir/[% project %]
|
|
|
61
|
+ $rootdir/gen-gradle-deps-file.py $gradle_logs
|
|
|
62
|
+[% END -%] |
projects/geckoview/build_common
| ... |
... |
@@ -14,8 +14,6 @@ export PATH="/var/tmp/dist/node/bin:$PATH" |
|
14
|
14
|
export LC_ALL=C.UTF-8
|
|
15
|
15
|
export LANG=C.UTF-8
|
|
16
|
16
|
|
|
17
|
|
-tar -xf [% c('input_files_by_name/glean') %]
|
|
18
|
|
-
|
|
19
|
17
|
export TOR_EXPERT_BUNDLE_AAR=$rootdir/[% c('input_files_by_name/tor-expert-bundle-aar') %]/tor-expert-bundle.aar
|
|
20
|
18
|
|
|
21
|
19
|
tar -C /var/tmp/dist -xf [% c('input_files_by_name/application-services') %]
|
| ... |
... |
@@ -29,25 +27,34 @@ tar -C /var/tmp/build -xf [% project %]-[% c('version') %].tar.[% c('compress_ta |
|
29
|
27
|
[% c("var/set_MOZ_BUILD_DATE") %]
|
|
30
|
28
|
|
|
31
|
29
|
export JAVA_HOME=/usr/lib/jvm/java-1.17.0-openjdk-amd64
|
|
32
|
|
-gradle_repo=/var/tmp/dist/gradle-dependencies
|
|
33
|
|
-export GRADLE_MAVEN_REPOSITORIES="file://$gradle_repo","file://$gradle_repo/maven2"
|
|
34
|
|
-# Set the Maven local repository because Gradle ignores our overriding of $HOME.
|
|
35
|
|
-# It is only used for the local pubblication of single-arch AARs.
|
|
36
|
|
-export GRADLE_FLAGS="--no-daemon --offline -Dmaven.repo.local=$distdir/[% project %]"
|
|
37
|
|
-# Move the Gradle repo to a hard-coded location. The location is embedded in
|
|
38
|
|
-# the file chrome/toolkit/content/global/buildconfig.html so it needs to be
|
|
39
|
|
-# standardized for reproducibility.
|
|
40
|
|
-mv $rootdir/[% c('input_files_by_name/gradle-dependencies') %] $gradle_repo
|
|
41
|
|
-cp -rl $rootdir/glean/maven/* $gradle_repo
|
|
42
|
|
-cp -rl $gradle_repo/dl/android/maven2/* $gradle_repo || true
|
|
43
|
|
-cp -rl $gradle_repo/m2/* $gradle_repo || true
|
|
44
|
|
-cp -rl $gradle_repo/maven2/* $gradle_repo || true
|
|
45
|
|
-
|
|
46
|
|
-tar -xf [% c('input_files_by_name/glean-wheels') %]
|
|
47
|
|
-export GLEAN_PYTHON_WHEELS_DIR=$rootdir/glean-wheels
|
|
48
|
|
-
|
|
49
|
|
-tar -xf [% c('input_files_by_name/oss-licenses-plugin') %]
|
|
50
|
|
-cp -rl oss-licenses-plugin/. $gradle_repo
|
|
|
30
|
+
|
|
|
31
|
+[% IF !c('var/generate_gradle_dependencies_list') -%]
|
|
|
32
|
+ gradle_repo=/var/tmp/dist/gradle-dependencies
|
|
|
33
|
+ export GRADLE_MAVEN_REPOSITORIES="file://$gradle_repo","file://$gradle_repo/maven2"
|
|
|
34
|
+ # Set the Maven local repository because Gradle ignores our overriding of $HOME.
|
|
|
35
|
+ # It is only used for the local pubblication of single-arch AARs.
|
|
|
36
|
+ export GRADLE_FLAGS="--no-daemon --offline -Dmaven.repo.local=$distdir/[% project %]"
|
|
|
37
|
+ # Move the Gradle repo to a hard-coded location. The location is embedded in
|
|
|
38
|
+ # the file chrome/toolkit/content/global/buildconfig.html so it needs to be
|
|
|
39
|
+ # standardized for reproducibility.
|
|
|
40
|
+ mv $rootdir/[% c('input_files_by_name/gradle-dependencies') %] $gradle_repo
|
|
|
41
|
+
|
|
|
42
|
+ tar -xf [% c('input_files_by_name/glean') %]
|
|
|
43
|
+ cp -rl $rootdir/glean/maven/* $gradle_repo
|
|
|
44
|
+
|
|
|
45
|
+ cp -rl $gradle_repo/dl/android/maven2/* $gradle_repo || true
|
|
|
46
|
+ cp -rl $gradle_repo/m2/* $gradle_repo || true
|
|
|
47
|
+ cp -rl $gradle_repo/maven2/* $gradle_repo || true
|
|
|
48
|
+
|
|
|
49
|
+ tar -xf [% c('input_files_by_name/glean-wheels') %]
|
|
|
50
|
+ export GLEAN_PYTHON_WHEELS_DIR=$rootdir/glean-wheels
|
|
|
51
|
+
|
|
|
52
|
+ tar -xf [% c('input_files_by_name/oss-licenses-plugin') %]
|
|
|
53
|
+ cp -rl oss-licenses-plugin/. $gradle_repo
|
|
|
54
|
+[% ELSE -%]
|
|
|
55
|
+ gradle_logs=/var/tmp/build/gradle-logs.log
|
|
|
56
|
+ export GRADLE_FLAGS="--no-daemon --info"
|
|
|
57
|
+[% END -%]
|
|
51
|
58
|
|
|
52
|
59
|
# We unbreak mach, see: https://bugzilla.mozilla.org/show_bug.cgi?id=1656993.
|
|
53
|
60
|
export MACH_BUILD_PYTHON_NATIVE_PACKAGE_SOURCE=system
|
projects/geckoview/config
| ... |
... |
@@ -48,6 +48,9 @@ var: |
|
48
|
48
|
has_l10n: '[% !c("var/testbuild") && !c("var/locales").empty %]'
|
|
49
|
49
|
l10n-changesets: '[% exec("git --no-pager show " _ c("git_hash") _ ":browser/locales/l10n-changesets.json", { exec_noco => 1 }) %]'
|
|
50
|
50
|
mozconfig_file: 'mozconfig-[% c("var/osname") %]'
|
|
|
51
|
+ # Uncomment this to run an online build to grab an updated
|
|
|
52
|
+ # gradle-dependencies-list.txt.
|
|
|
53
|
+ # generate_gradle_dependencies_list: 1
|
|
51
|
54
|
|
|
52
|
55
|
targets:
|
|
53
|
56
|
release:
|
| ... |
... |
@@ -87,9 +90,11 @@ steps: |
|
87
|
90
|
- project: glean
|
|
88
|
91
|
name: glean
|
|
89
|
92
|
pkg_type: build
|
|
|
93
|
+ enable: '[% !c("var/generate_gradle_dependencies_list") %]'
|
|
90
|
94
|
- filename: 'gradle-dependencies-[% c("var/gradle_dependencies_version") %]'
|
|
91
|
95
|
name: gradle-dependencies
|
|
92
|
96
|
exec: '[% INCLUDE "fetch-gradle-dependencies" %]'
|
|
|
97
|
+ enable: '[% !c("var/generate_gradle_dependencies_list") %]'
|
|
93
|
98
|
- name: geckoview_armv7
|
|
94
|
99
|
project: geckoview
|
|
95
|
100
|
pkg_type: build
|
| ... |
... |
@@ -123,15 +128,19 @@ steps: |
|
123
|
128
|
- URL: '[% pc("glean-parser", "var/glean_wheels_url") %]'
|
|
124
|
129
|
name: glean-wheels
|
|
125
|
130
|
sha256sum: '[% pc("glean-parser", "var/glean_wheels_sha256sum") %]'
|
|
|
131
|
+ enable: '[% !c("var/generate_gradle_dependencies_list") %]'
|
|
126
|
132
|
- name: tor-expert-bundle-aar
|
|
127
|
133
|
project: tor-expert-bundle-aar
|
|
128
|
134
|
pkg_type: build
|
|
129
|
135
|
- project: oss-licenses-plugin
|
|
130
|
136
|
name: oss-licenses-plugin
|
|
131
|
137
|
pkg_type: build
|
|
|
138
|
+ enable: '[% !c("var/generate_gradle_dependencies_list") %]'
|
|
132
|
139
|
- URL: 'https://github.com/google/bundletool/releases/download/[% c("var/bundletool_version") %]/bundletool-all-[% c("var/bundletool_version") %].jar'
|
|
133
|
140
|
name: bundletool
|
|
134
|
141
|
sha256sum: '[% c("var/bundletool_hash") %]'
|
|
|
142
|
+ - filename: gen-gradle-deps-file.py
|
|
|
143
|
+ enable: '[% c("var/generate_gradle_dependencies_list") %]'
|
|
135
|
144
|
|
|
136
|
145
|
list_toolchain_updates:
|
|
137
|
146
|
git_hash: FIREFOX_NIGHTLY_146_END
|
| ... |
... |
@@ -167,14 +176,15 @@ input_files: |
|
167
|
176
|
- filename: 'gradle-dependencies-[% c("var/gradle_dependencies_version") %]'
|
|
168
|
177
|
name: gradle-dependencies
|
|
169
|
178
|
exec: '[% INCLUDE "fetch-gradle-dependencies" %]'
|
|
|
179
|
+ enable: '[% !c("var/generate_gradle_dependencies_list") %]'
|
|
170
|
180
|
- URL: '[% pc("glean-parser", "var/glean_wheels_url") %]'
|
|
171
|
181
|
name: glean-wheels
|
|
172
|
182
|
sha256sum: '[% pc("glean-parser", "var/glean_wheels_sha256sum") %]'
|
|
173
|
183
|
- project: glean
|
|
174
|
184
|
name: glean
|
|
175
|
|
- pkg_type: build
|
|
176
|
185
|
- project: oss-licenses-plugin
|
|
177
|
186
|
name: oss-licenses-plugin
|
|
|
187
|
+ enable: '[% !c("var/generate_gradle_dependencies_list") %]'
|
|
178
|
188
|
- name: tor-expert-bundle-aar
|
|
179
|
189
|
project: tor-expert-bundle-aar
|
|
180
|
190
|
- project: firefox-l10n
|
| ... |
... |
@@ -192,3 +202,5 @@ input_files: |
|
192
|
202
|
- URL: 'https://github.com/google/bundletool/releases/download/[% c("var/bundletool_version") %]/bundletool-all-[% c("var/bundletool_version") %].jar'
|
|
193
|
203
|
name: bundletool
|
|
194
|
204
|
sha256sum: '[% c("var/bundletool_hash") %]'
|
|
|
205
|
+ - filename: gen-gradle-deps-file.py
|
|
|
206
|
+ enable: '[% c("var/generate_gradle_dependencies_list") %]' |
projects/glean/build
| ... |
... |
@@ -28,7 +28,7 @@ export JAVA_HOME=/usr/lib/jvm/java-1.17.0-openjdk-amd64 |
|
28
|
28
|
|
|
29
|
29
|
[% INCLUDE 'fake-git' %]
|
|
30
|
30
|
|
|
31
|
|
-[% IF !c('var/fetch_gradle_dependencies') %]
|
|
|
31
|
+[% IF !c('var/generate_gradle_dependencies_list') -%]
|
|
32
|
32
|
gradle_repo=/var/tmp/dist/gradle-dependencies
|
|
33
|
33
|
mv $rootdir/[% c('input_files_by_name/gradle-dependencies') %] $gradle_repo
|
|
34
|
34
|
cp -rl $gradle_repo/dl/android/maven2/* $gradle_repo || true
|
| ... |
... |
@@ -42,23 +42,24 @@ mv /var/tmp/build/[% project %]-[% c('version') %] $builddir |
|
42
|
42
|
|
|
43
|
43
|
cd $builddir
|
|
44
|
44
|
|
|
45
|
|
-# Prepare the offline build.
|
|
46
|
|
-
|
|
47
|
|
-tar -xf $rootdir/[% c('input_files_by_name/glean-wheels') %]
|
|
48
|
|
-export GLEAN_PYTHON_WHEELS_DIR=$builddir/glean-wheels
|
|
49
|
|
-# Setting GLEAN_PYTHON_WHEELS_DIR is not enough: a Rust build script will still
|
|
50
|
|
-# try to install Glean with pip.
|
|
|
45
|
+[% IF !c('var/generate_gradle_dependencies_list') -%]
|
|
|
46
|
+ # Prepare the offline build.
|
|
|
47
|
+ tar -xf $rootdir/[% c('input_files_by_name/glean-wheels') %]
|
|
|
48
|
+ export GLEAN_PYTHON_WHEELS_DIR=$builddir/glean-wheels
|
|
|
49
|
+ # Setting GLEAN_PYTHON_WHEELS_DIR is not enough: a Rust build script will still
|
|
|
50
|
+ # try to install Glean with pip.
|
|
51
|
51
|
cat > $rootdir/pip.conf << 'EOF'
|
|
52
|
52
|
[global]
|
|
53
|
53
|
find-links = /var/tmp/build/glean/glean-wheels
|
|
54
|
54
|
index-url =
|
|
55
|
55
|
no-index = yes
|
|
56
|
56
|
EOF
|
|
57
|
|
-export PIP_CONFIG_FILE=$rootdir/pip.conf
|
|
58
|
|
-# Move the directory for hardcoding the path in .cargo/config.
|
|
59
|
|
-tar -C $builddir -xf $rootdir/[% c('input_files_by_name/cargo_vendor') %]
|
|
60
|
|
-# Make sure our vendored crates are used for offline builds.
|
|
61
|
|
-cat >> .cargo/config.toml << 'EOF'
|
|
|
57
|
+
|
|
|
58
|
+ export PIP_CONFIG_FILE=$rootdir/pip.conf
|
|
|
59
|
+ # Move the directory for hardcoding the path in .cargo/config.
|
|
|
60
|
+ tar -C $builddir -xf $rootdir/[% c('input_files_by_name/cargo_vendor') %]
|
|
|
61
|
+ # Make sure our vendored crates are used for offline builds.
|
|
|
62
|
+ cat >> .cargo/config.toml << 'EOF'
|
|
62
|
63
|
|
|
63
|
64
|
[source.crates-io]
|
|
64
|
65
|
replace-with = "vendored-sources"
|
| ... |
... |
@@ -70,16 +71,27 @@ directory = "/var/tmp/build/glean/vendor" |
|
70
|
71
|
offline=true
|
|
71
|
72
|
EOF
|
|
72
|
73
|
|
|
|
74
|
+ deps_flags="--offline -PgradleDependenciesUrl=file:///var/tmp/dist/gradle-dependencies"
|
|
|
75
|
+[% ELSE -%]
|
|
|
76
|
+ deps_flags="--info"
|
|
|
77
|
+ gradle_logs=/var/tmp/build/gradle-logs.log
|
|
|
78
|
+[% END -%]
|
|
|
79
|
+
|
|
73
|
80
|
tar -xf $rootdir/[% c('input_files_by_name/uniffi-rs') %]
|
|
74
|
81
|
|
|
75
|
82
|
patch -p1 < $rootdir/use-custom-gradle-repo.diff
|
|
76
|
83
|
patch -p1 < $rootdir/use-uniffi-noop.diff
|
|
77
|
84
|
|
|
78
|
|
-gradle_flags="--offline --no-daemon -PuniffiBindgenNoop=$builddir/uniffi-rs/uniffi-bindgen -PgradleDependenciesUrl=file:///var/tmp/dist/gradle-dependencies"
|
|
79
|
|
-gradle $gradle_flags :glean:assembleRelease
|
|
|
85
|
+gradle_flags="--no-daemon -PuniffiBindgenNoop=$builddir/uniffi-rs/uniffi-bindgen $deps_flags"
|
|
|
86
|
+gradle $gradle_flags :glean:assembleRelease [% IF c('var/generate_gradle_dependencies_list') %]2>&1 | tee -a $gradle_logs[% END %]
|
|
80
|
87
|
gradle $gradle_flags publish
|
|
81
|
88
|
mv build/maven $distdir
|
|
82
|
89
|
|
|
|
90
|
+[% IF c('var/generate_gradle_dependencies_list') -%]
|
|
|
91
|
+ cd $distdir
|
|
|
92
|
+ $rootdir/gen-gradle-deps-file.py $gradle_logs
|
|
|
93
|
+[% END -%]
|
|
|
94
|
+
|
|
83
|
95
|
cd /var/tmp/dist
|
|
84
|
96
|
[% c('tar', {
|
|
85
|
97
|
tar_src => [ project ],
|
projects/glean/config
| ... |
... |
@@ -6,11 +6,16 @@ filename: '[% project %]-[% c("version") %]-[% c("var/build_id") %].tar.[% c("co |
|
6
|
6
|
version: 66.1.1
|
|
7
|
7
|
container:
|
|
8
|
8
|
use_container: 1
|
|
|
9
|
+ disable_network:
|
|
|
10
|
+ build: '[% !c("var/generate_gradle_dependencies_list") %]'
|
|
9
|
11
|
|
|
10
|
12
|
var:
|
|
11
|
13
|
# This should be updated when the list of gradle dependencies is changed.
|
|
12
|
14
|
gradle_dependencies_version: 66.1.1-1
|
|
13
|
15
|
gradle_version: 8.14.3
|
|
|
16
|
+ # Uncomment this to run an online build to grab an updated
|
|
|
17
|
+ # gradle-dependencies-list.txt.
|
|
|
18
|
+ # generate_gradle_dependencies_list: 1
|
|
14
|
19
|
|
|
15
|
20
|
steps:
|
|
16
|
21
|
build:
|
| ... |
... |
@@ -36,6 +41,7 @@ steps: |
|
36
|
41
|
- URL: '[% pc("glean-parser", "var/glean_wheels_url") %]'
|
|
37
|
42
|
name: glean-wheels
|
|
38
|
43
|
sha256sum: '[% pc("glean-parser", "var/glean_wheels_sha256sum") %]'
|
|
|
44
|
+ enable: '[% !c("var/generate_gradle_dependencies_list") %]'
|
|
39
|
45
|
- name: cargo_vendor
|
|
40
|
46
|
project: glean
|
|
41
|
47
|
pkg_type: cargo_vendor
|
| ... |
... |
@@ -44,5 +50,8 @@ steps: |
|
44
|
50
|
- filename: 'gradle-dependencies-[% c("var/gradle_dependencies_version") %]'
|
|
45
|
51
|
name: gradle-dependencies
|
|
46
|
52
|
exec: '[% INCLUDE "fetch-gradle-dependencies" %]'
|
|
|
53
|
+ enable: '[% !c("var/generate_gradle_dependencies_list") %]'
|
|
|
54
|
+ - filename: gen-gradle-deps-file.py
|
|
|
55
|
+ enable: '[% c("var/generate_gradle_dependencies_list") %]'
|
|
47
|
56
|
- filename: use-custom-gradle-repo.diff
|
|
48
|
57
|
- filename: use-uniffi-noop.diff |
|