Hello, You can find at this URL a proposal to refactor the tor browser bundle build process, using an other tool to replace gitian: https://people.torproject.org/~boklm/automation/tor-automation-proposals.html#build-tool (also added as attached file to this email) I made a prototype to show how it would work: https://github.com/boklm/burps-tor The main improvements in this prototype from the current build process are: - all components are built separately, and include in their output file name the commit hash or version, architecture and OS (if architecture dependant). This allows us to keep previous builds if the commit/version/architecture/OS didn't change. So we can rebuild a bundle very quickly when the browser didn't change. - the gitian replacment has features to download tarballs and verify them with sha256sum or gpg signature, so this can replace the fetch-inputs.sh script. - we can remove the linux/windows/macosx descriptors duplication, and instead use template directives for the parts that differ between those builds (it's still possible to use separate files if they differ completly). - we can define variables based on selected OS. This allows for instance to build python 2.7 when building on Ubuntu Lucid, but avoid building it on other distros which already provide python 2.7. - we can define variables based on "targets" that are set on command line. For instance in the prototype using "--target enable_pt" instructs to include the portable transports (only pyptlib in this prototype) in the bundle. - we can easily switch from building in a VM to building locally - build descriptors can depend on the result of another build descriptor. This remove needs for scripts like mkbundle-*.sh. And I think those improvements should make it easier to rebuild a new bundle automatically when any of the components of the bundle receives a new commit, and then run tests on this bundle. Nicolas
Goals
~~~~~
It would be useful to have a tool that can be used for building Tor
Bundle Browser, and packages for any of the Tor components.
Tor Bundle Browser
^^^^^^^^^^^^^^^^^^
In the current build process for the Tor Bundle Browser, gitias is used
to :
- create and start an Ubuntu VM in KVM or LXC
- install build dependencies in the VM
- copy the sources files in the VM and start the build
The parts that are not done by gitian but by a shellscript based wrapper
around gitian:
- initial clone of the git repository containing the sources
(fetch-inputs.sh)
- git tag gpg signature check (verify-tags.sh)
- download tarballs from mirrors, and verify gpg signature or sha256sum
(fetch-inputs.sh)
Improvement that can be done from the current process:
- some informations are duplicated in different places, such as git URLs
(both in gitian descriptor and in fetch-inputs.sh)
- linux, mac and windows gitian descriptors are separate, so part of
them are duplicating some informations. It would be better to be able
to share the parts that are common on all OSs.
- build process is not very modular. It is not possible or simple to
build only some components.
- the build process can currently only be run from Ubuntu
- the current process requires using git tags if gpg signature
verification is wanted. It could be useful to be able to use gpg
signed commits instead.
To solve this, we can have a tool that handle both the tasks currently
done by gitian, and the tasks currently done by shellscript wrappers:
- store in a single descriptor the build instructions, build dependencies,
git URL, tarball URLs, gpg keyring file, sha256sum, etc ... and have
all the operations done by the tool. This should remove the need for
wrappers.
- create a separate descriptor for each software that we build, so that
they can be built separately. The tool should be able to manage
dependencies between separate builds to create a bundle of different
software.
- a templating system allows us to have some variables and use them at
different places. This allows us to have a common build descriptor
for all OSs, and use conditional templating instructions where the
process differ between the different OSs.
A prototype to show a possible build process is available, with more
details below.
Packages
^^^^^^^^
In addition to building the Tor Browser Bundle, we also need to build
some rpm or Debian packages for different components. Some of them are
also included in the Tor Browser Bundle.
To be able to do continuous packaging, we need a tool that would allow
us to produce an rpm or Debian package from a git commit. In the
jenkins/tools.git repository, some shell scripts are used to produce
Tor Debian packages from the latest commit.
To avoid duplicating build instructions, URLs, gpg keyring and other
infos, it would be useful if a single tool was able to produce the Tor
Browser Bundle and also individual packages for some of the components.
Prototype and current status
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
I have created recently a tool that could be used to build Bundle, and
Debian and rpm packages : http://burps.boklm.eu/[burps (build & upload
reproducible packaging system)].
It currently has the following features:
- build software from a git repository, with tag signature verification
and/or commit signature verification
- download files from an URL, and check gpg signature, or sha256sum
- use build result from the build of an other component
- build on a remote host, VM or chroot
- build rpm or Debian packages, or tarballs
- define variables in the descriptors, and use them anywhere using
template instructions
- override some variables on command line by selecting build targets.
It is currently missing the following features from gitian:
- creation and start/stop of the Ubuntu build VMs. We can keep the
gitian scripts for that, and improve them later.
- definition of the build dependencies in the descriptor and installation.
This should not be a lot of work to add this feature.
Prototype
^^^^^^^^^
A prototype for a new build process for the Tor Browser Bundle using
this tool is available at the following URL:
https://github.com/boklm/burps-tor
It currently includes the following components:
libevent, tor, tor-browser, torbutton, tor-launcher, https-everywhere,
all built separately and combined in the bundle.
It does not produce a functional build yet, but is enough to show what
the build process would look like.
Installation
^^^^^^^^^^^^
Debian packages for 'burps' should be available soon. Until they are
available you can do a manual install:
- clone the git repository in some directory
- install the following perl modules: YAML::XS, Getopt::Long, Template,
IO::CaptureOutput, File::Slurp, String::ShellQuote, Sort::Versions,
Data::Dump
On Debian this can be done with:
----
$ apt-get install libyaml-libyaml-perl libtemplate-perl \
libio-captureoutput-perl libfile-slurp-perl \
libstring-shellquote-perl libsort-versions-perl \
libdata-dump-perl
$ git clone https://github.com/boklm/burps ~/burps
----
In the commands from the following examples, replace 'burps' with the
path to the burps executable (for instance '~/burps/burps').
How it works
~~~~~~~~~~~~
To understand how it works, we can look at different files from the
prototype and see what they do.
Files and directories list
^^^^^^^^^^^^^^^^^^^^^^^^^^
The following files and directories are present in the repository:
----
.
├── burps.conf - The main configuration file
├── git_clones - directory containing git clones
├── keyring
│ ├── libevent.gpg - The gpg keyring for the libevent project
│ ├── tor.gpg - The gpg keyring for the tor project
│ └── ... - other keyring files
├── out - Output directory for build files
└── projects
├── libevent
│ ├── build - build instructions for libevent
│ └── config - configuration for libevent
├── tbb
│ ├── build - build instructions for the bundle
│ └── config - configuration for the bundle
├── tor
│ ├── build - build instructions for tor
│ └── config - configuration for tor
└── ... - other projects directories
----
burps.conf
^^^^^^^^^^
The 'burps.conf' file is used to indicate the root of the burps
repository. It also contains the configuration options shared by all
projects. All those options can be overriden by each project
configuration.
libevent
^^^^^^^^
The file 'projects/libevent/config' contains the configuration for the
libevent build:
----
git_url: https://github.com/libevent/libevent.git
version: '[% c("abbrev") %]'
var:
dest_filename: 'libevent-[% c("version") %]-[% c("lsb_release/id") %]-[% c("lsb_release/release") %]-[% c("arch") %].tar'
targets:
notarget: dev
stable:
git_hash: release-2.0.21-stable
tag_gpg_id: 1
dev:
git_hash: master
----
The following options are defined:
git_url:: This option contains the URL of the git repository that should
be cloned to create the sources tarball.
version:: This option contains the string that is used as version number
in the sources tarball. In this case, we're saying that we want to use
'abbrev' as version string, which is the abbreviated hash of the commit
that has been selected.
var/dest_filename:: This option is used in the 'build' file, and defines
the name of the output file. The name of this option is arbitrary, so
to avoid a conflict with an option used by a future version of burps,
we add the 'var' prefix. The reason to use a variable to define this
file name is to allow other projects which need this file (in this case,
this libevent build is used by tor), to know the name of the file that
is produced. In this case, we can see that the filename contains the
following informations: abbreviated commit hash, distribution name and
release, architecture.
targets:: We can see that different targets are defined. Targets allows
us to set different options depending on the target selected on command
line. In this case, we defined the 'stable' and 'dev' targets.
http://burps.boklm.eu/burps_targets.html[This page] has more details
about how the targets work.
git_hash:: This option is used to select the git commit from which to
create the sources tarball. This can be anything that git calls a
'tree-ish' (hashes, tags, branch names).
tag_gpg_id:: This option indicates that the gpg signature of the tag
pointed by 'git_hash' should be checked, using the keyring file
'libevent.gpg' (as defined by the 'keyring' option in 'burps.conf'). We
can only set this option for the stable target, because the dev target
does not use a tag. The libevent commits are not currently signed, but
if they were we could set the 'commit_gpg_id' option to check signature
of the commit in the master branch.
The file 'projects/libevent/config' contains the build instructions for
libevent:
----
[%- USE date -%]
#!/bin/sh
set -e
[% c('var/reproducible_init') %]
tar xvf [% project %]-[% c('version') %].tar.[% c('compress_tar') %]
cd [% project %]-[% c('version') %]
./autogen.sh
[% c('var/touch_directory', { directory => '.' }) %]
mkdir -p inst/libevent
./configure --disable-static --prefix="$(pwd)/inst/libevent"
make -j$(nproc)
make install
cd inst
[% c('var/touch_directory', { directory => '.' }) %]
[% c('tar', { tar_src => 'libevent', tar_args => '-cf ' _ dest_dir _ '/' _ c('var/dest_filename') }) %]
----
We can see that it uses the following options:
var/reproducible_init:: This option is defined in 'burps.conf', and
export the environment variables to enable 'libfaketime'. The time used
by libfaketime is defined by the option 'timestamp', which by default
is the time of the selected git commit.
var/touch_directory:: This option sets the modification time of all
files in the selected directory, using the time from 'timestamp' option.
tar:: This option is defined in burps (you can run 'burps showconf' to
see it), and is used to create a deterministic tar file. It is doing
something similar to the script gitian/build-helpers/dtar.sh from the
current tbb build process.
Following are some examples of commands that we can use to build libevent.
Creating a sources tarball for the 'stable' target. In this example we
can see that the gpg signature of the git tag is checked:
----
$ burps tar libevent --target stable
Tag release-2.0.21-stable is signed with key B35BF85BF19489D04E28C33C21194EBB165733EA
Created /home/boklm/work/burps-tor/out/libevent-64177777165d.tar.gz
----
We can do the same for the 'devel' target, and we will get the sources
from the master branch instead:
----
$ burps tar libevent --target dev
Created /home/boklm/work/burps-tor/out/libevent-4c8ebcd3598a.tar.gz
----
If what we want is a build, we use the 'build' command instead of 'tar'.
----
$ ~/work/mkpkg/burps build libevent --target dev
Created /home/boklm/tmp/burps-keaW4/libevent-4c8ebcd3598a.tar.gz
[some build logs]
$ ls -l out/libevent-4c8ebcd3598a-Mageia-3-x86_64.tar
-rw-r--r-- 1 boklm boklm 3430400 Jan 2 14:43 out/libevent-4c8ebcd3598a-Mageia-3-x86_64.tar
$ tar tvf out/libevent-4c8ebcd3598a-Mageia-3-x86_64.tar
drwx------ root/root 0 2013-12-24 21:02 libevent/
drwx------ root/root 0 2013-12-24 21:02 libevent/lib/
lrwxrwxrwx root/root 0 2013-12-24 21:02 libevent/lib/libevent_pthreads.so -> libevent_pthreads-2.1.so.3.0.0
lrwxrwxrwx root/root 0 2013-12-24 21:02 libevent/lib/libevent_openssl-2.1.so.3 -> libevent_openssl-2.1.so.3.0.0
lrwxrwxrwx root/root 0 2013-12-24 21:02 libevent/lib/libevent_openssl.so -> libevent_openssl-2.1.so.3.0.0
lrwxrwxrwx root/root 0 2013-12-24 21:02 libevent/lib/libevent-2.1.so.3 -> libevent-2.1.so.3.0.0
lrwxrwxrwx root/root 0 2013-12-24 21:02 libevent/lib/libevent_core.so -> libevent_core-2.1.so.3.0.0
lrwxrwxrwx root/root 0 2013-12-24 21:02 libevent/lib/libevent_core-2.1.so.3 -> libevent_core-2.1.so.3.0.0
-rwx------ root/root 117534 2013-12-24 21:02 libevent/lib/libevent_openssl-2.1.so.3.0.0
-rwx------ root/root 1018 2013-12-24 21:02 libevent/lib/libevent_core.la
-rwx------ root/root 1042 2013-12-24 21:02 libevent/lib/libevent_pthreads.la
-rwx------ root/root 25459 2013-12-24 21:02 libevent/lib/libevent_pthreads-2.1.so.3.0.0
lrwxrwxrwx root/root 0 2013-12-24 21:02 libevent/lib/libevent_pthreads-2.1.so.3 -> libevent_pthreads-2.1.so.3.0.0
-rwx------ root/root 861887 2013-12-24 21:02 libevent/lib/libevent_core-2.1.so.3.0.0
lrwxrwxrwx root/root 0 2013-12-24 21:02 libevent/lib/libevent_extra.so -> libevent_extra-2.1.so.3.0.0
-rwx------ root/root 1024 2013-12-24 21:02 libevent/lib/libevent_extra.la
-rwx------ root/root 1364542 2013-12-24 21:02 libevent/lib/libevent-2.1.so.3.0.0
[...]
----
We can do the same on the 'stable' target:
----
$ ~/work/mkpkg/burps build libevent --target stable
Tag release-2.0.21-stable is signed with key B35BF85BF19489D04E28C33C21194EBB165733EA
Created /home/boklm/tmp/burps-0d1Cy/libevent-64177777165d.tar.gz
[...]
autoreconf: running: aclocal --force -I m4
aclocal: warning: autoconf input should be named 'configure.ac', not 'configure.in'
configure.in:15: error: 'AM_CONFIG_HEADER': this macro is obsolete.
You should use the 'AC_CONFIG_HEADERS' macro instead.
/usr/share/aclocal-1.13/obsolete-err.m4:12: AM_CONFIG_HEADER is expanded from...
configure.in:15: the top level
autom4te: /usr/bin/m4 failed with exit status: 1
aclocal: error: echo failed with exit status: 1
autoreconf: aclocal failed with exit status: 1
Error: Error running build
----
In this example I am using Mageia, and the version of autoconf we have
there doesn't work with this version of libevent, so the build fails.
However we can run the build inside a Ubuntu Lucid chroot (more details
about running builds in a chroot or VM below):
----
$ burps build libevent --target chroot-ubuntu-lucid --target stable
Tag release-2.0.21-stable is signed with key B35BF85BF19489D04E28C33C21194EBB165733EA
Created /home/boklm/tmp/burps-6PNXA/libevent-64177777165d.tar.gz
[...]
$ ls out/libevent-64177777165d-Ubuntu-10.04-x86_64.tar
out/libevent-64177777165d-Ubuntu-10.04-x86_64.tar
$ tar tvf out/libevent-64177777165d-Ubuntu-10.04-x86_64.tar
drwx------ root/root 0 2012-11-18 07:39 libevent/
drwx------ root/root 0 2012-11-18 07:39 libevent/lib/
lrwxrwxrwx root/root 0 2012-11-18 07:39 libevent/lib/libevent_pthreads.so -> libevent_pthreads-2.0.so.5.1.9
lrwxrwxrwx root/root 0 2012-11-18 07:39 libevent/lib/libevent_openssl.so -> libevent_openssl-2.0.so.5.1.9
lrwxrwxrwx root/root 0 2012-11-18 07:39 libevent/lib/libevent_core.so -> libevent_core-2.0.so.5.1.9
-rwx------ root/root 942988 2012-11-18 07:39 libevent/lib/libevent-2.0.so.5.1.9
-rwx------ root/root 398207 2012-11-18 07:39 libevent/lib/libevent_extra-2.0.so.5.1.9
-rwx------ root/root 1032 2012-11-18 07:39 libevent/lib/libevent_core.la
-rwx------ root/root 1056 2012-11-18 07:39 libevent/lib/libevent_pthreads.la
-rwx------ root/root 569248 2012-11-18 07:39 libevent/lib/libevent_core-2.0.so.5.1.9
-rwx------ root/root 21158 2012-11-18 07:39 libevent/lib/libevent_pthreads-2.0.so.5.1.9
-rwx------ root/root 87749 2012-11-18 07:39 libevent/lib/libevent_openssl-2.0.so.5.1.9
lrwxrwxrwx root/root 0 2012-11-18 07:39 libevent/lib/libevent_extra.so -> libevent_extra-2.0.so.5.1.9
-rwx------ root/root 1038 2012-11-18 07:39 libevent/lib/libevent_extra.la
[...]
----
We can notice that the modification time of all files in the archive is
2012-11-18, which is the time of the commit '2.0.21-stable'.
libevent build is then used in the build of tor. We can se it in
input_file of the tor config:
----
input_files:
- filename: '[% pc("libevent", "var/dest_filename") %]'
name: libevent
project: libevent
pkg_type: build
----
The libevent build file name contains the abbreviated git hash,
distribution name, release and architecture, so if any of those
change, libevent will be rebuilt automatically. Otherwise, the existing
build will be used.
Python
^^^^^^
Python is an example of tool that we build using a tarball downloaded
from an URL.
The config file for python contains this:
----
version: 2.7.5
timestamp: 1
var:
dest_filename: 'python-[% c("version") %]-[% c("lsb_release/id") %]-[% c("lsb_release/release") %]-[% c("arch") %].tar.xz'
input_files:
- name: python
filename: 'Python-[% c("version") %].tar.bz2'
URL: 'http://www.python.org/ftp/python/[% c("version") %]/[% c("filename") %]'
sha256sum: 3b477554864e616a041ee4d7cef9849751770bc7c39adaf78a94ea145c488059
----
We can see that the filename includes the python version. If we update
the version number, then the new tarball will be downloaded automatically.
If the version number doesn't change, then the tarball will be downloaded
only once, and reused each time.
This Python build is used during the tor-browser build, only if we are
running a distribution that provides an older version of Python (Ubuntu
Lucid). We can see it in the tor-browser config file:
----
git_url: https://git.torproject.org/tor-browser.git
version: '[% c("abbrev") %]'
var:
dest_filename: 'tor-browser-[% c("abbrev") %]-[% c("lsb_release/id") %]-[% c("lsb_release/release") %]-[% c("arch") %].tar'
dest_filename_debug: 'tor-browser-[% c("abbrev") %]-debug-[% c("lsb_release/id") %]-[% c("lsb_release/release") %]-[% c("arch") %].tar'
targets:
dev:
git_hash: tor-browser-24.2.0esr-3.5.1-build1
tag_gpg_id: 1
stable:
git_hash: tor-browser-24.2.0esr-3.5.1-build1
tag_gpg_id: 1
input_files:
- filename: '[% pc("python", "var/dest_filename") %]'
name: python
project: python
pkg_type: build
enable: '[% c("build_python") %]'
distributions:
- lsb_release:
id: Ubuntu
codename: lucid
var:
build_python: 1
link_yasm: 1
----
In this file, we can see that the options 'var/build_python' and
'var/link_yasm' are set to '1' if the selected distribution is Ubuntu
with codename 'lucid' (more details about how to set distribution
specific options can be found on http://burps.boklm.eu/burps_distributions.html[this page]).
The option 'var/build_python' is used in the 'input_files' to enable or
disable the python build. It is also used in the tor-browser 'build'
file, to decide if a python tarball should be extracted:
----
[% IF c('var/build_python') %]
tar xvf [% c('input_files_by_name/python') %]
export PATH="$rootdir/python/bin:$PATH"
[% END %]
----
Pluggable transports
^^^^^^^^^^^^^^^^^^^^
This prototype does not include the pluggable transports, except 'pyptlib',
to show how they can be included optionally.
The 'burps.conf' file contains an 'enable_pt' target:
----
targets:
include_pt:
var:
include_pt: 1
----
The option 'var/include_pt' is then used in tbb include_files config,
to optionaly include pyptlib as input:
----
- filename: '[% pc("pyptlib", "var/dest_filename") %]'
name: pyptlib
project: pyptlib
pkg_type: build
enable: '[% c("var/include_pt") %]'
----
In the tbb build script, it is used to optionaly extract pyptlib in the
Tor directory:
----
[% IF c('var/include_pt') %]
pushd "$instdir/Tor"
tar xvf "$rootdir"/[% c('input_files_by_name/pyptlib') %]
popd
[% END %]
----
We can now use the 'include_pt' target to include pyptlib in the bundle:
----
$ burps build tbb --target dev --target include_pt
Created /home/boklm/tmp/burps-uLDRA/tbb-x.dev.tar.gz
Using file /home/boklm/work/burps-tor/projects/tbb/tor-573ee36eae63-Mageia-3-x86_64.tar
Using file /home/boklm/work/burps-tor/out/tor-browser-a451d459fa5e-Mageia-3-x86_64.tar
Using file /home/boklm/work/burps-tor/out/tor-launcher-1e34dd95dc7f.xpi
Using file /home/boklm/work/burps-tor/out/https-everywhere-4be8003e7a6a.xpi
Using file /home/boklm/work/burps-tor/projects/tbb/torbutton-2c825593cbcb.xpi
Using file /home/boklm/work/burps-tor/out/pyptlib-97be8ad7c74e.tar
[...]
$ tar tvf out/tbb-x.dev-Mageia-3-x86_64.tar.xz | grep pypt | wc -l
45
----
If we're not using the 'include_pt' target, pyptlib is not included:
----
$ ~/work/mkpkg/burps build tbb --target dev
Created /home/boklm/tmp/burps-Zvgm4/tbb-x.dev.tar.gz
Using file /home/boklm/work/burps-tor/projects/tbb/tor-573ee36eae63-Mageia-3-x86_64.tar
Using file /home/boklm/work/burps-tor/out/tor-browser-a451d459fa5e-Mageia-3-x86_64.tar
Using file /home/boklm/work/burps-tor/out/tor-launcher-1e34dd95dc7f.xpi
Using file /home/boklm/work/burps-tor/out/https-everywhere-4be8003e7a6a.xpi
Using file /home/boklm/work/burps-tor/projects/tbb/torbutton-2c825593cbcb.xpi
[...]
$ tar tvf out/tbb-x.dev-Mageia-3-x86_64.tar.xz | grep pypt | wc -l
0
----
Build in a chroot or VM
^^^^^^^^^^^^^^^^^^^^^^^
In order to produce reproducible builds, we need to build all software
in a fixed environment. To do that, the current build process is using
Ubuntu Lucid (for Linux builds) and Ubuntu Precise (for Windows and Mac
OS X builds) inside a VM or LXC container.
The creation of a VM image or chroot directory is not yet automated in
this prototype. However, there is support building in a chroot or in a
VM, if you create it manually.
http://burps.boklm.eu/burps_remote.html[This page] has details about
how to do that.
If you want to build a bundle in an Ubuntu lucid chroot, first create
the chroot in some directory, using 'debootstrap', and create in this
chroot the user that will be used to build the software.
You can then add the following configuration in your main configuration
file '/etc/burps.conf' (this would also work if it was added to the
repository configuration file, but as it contains configuration specific
to your local system, it's better to have it in your system's configuration
file):
----
targets:
chroot-ubuntu-lucid:
lsb_release:
id: Ubuntu
release: 10.04
codename: lucid
chroot_path: /home/chroots/ubuntu-lucid
chroot_user: build
remote:
build:
exec: '[% c("remote_chroot/exec") %]'
----
We are setting the following options:
lsb_release:: we are saying that the distribution on which we are
building is Ubuntu lucid. If you are not setting this, the output from
the 'lsb_release' command will be used.
chroot_path:: The path to the chroot you want to use. This is used by
the 'remote_chroot/exec' option.
chroot_user:: The user inside the chroot that should be used to build.
This is used by the 'remote_chroot/exec' option.
remote/build/exec:: This is setting the command that should be used to
execute something inside the chroot. We are using the 'remote_chroot'
option (you can see the content of this option with 'burps showconf').
This command is using 'sudo chroot', so you should have sudo access to
do that.
After adding those changes to the configuration, you should be able to
build any software inside this chroot by adding the arguments '--target
chroot-ubuntu-lucid' to your burps commands. For instance if you want
to build a bundle in the dev target, you can run:
----
$ burps build tbb --target chroot-ubuntu-lucid --target dev
----
Debugging tips
^^^^^^^^^^^^^^
When building some project, burps will create a temporary directory,
copy all the input files and a build script in that directory, then run
the build script. After the build finish (or fails), that temporary
directory is removed, which can be annoying if you want to understand
why the build failed.
To avoid this, you can enable the 'debug' option. Add the following
line to '/etc/burps.conf':
----
debug: 1
----
Or use the '--debug' command line argument. If the debug option is
enabled and a build fails, burps will open an interactive shell in the
temporary build directory, so you can inspect the build script and other
files, and maybe continue the build. When you exit that shell, the
temporary directory will be removed.
When you are writting burps project descriptors, you can use some
template directives, target specific options, distribution specific
options, etc ... It can sometime be difficult to understand what will
be the value of an option, or the content of a build script. To check
the value of an option, you can use the 'showconf' burps command. This
option takes a project's name, an option name, and the same options as
other build commands.
For instance if you want to check the abbreviated git commit in dev and
stable targets for libevent, you could run the following commands:
----
$ burps showconf libevent abbrev --target dev
4c8ebcd3598a
$ burps showconf libevent abbrev --target stable
64177777165d
----
If you want to compare the build script of tbb with and without the
'include_pt' target:
----
$ burps showconf tbb build --target dev --target include_pt
[...]
$ burps showconf tbb build --target dev
[...]
----
Pubishing builds
^^^^^^^^^^^^^^^^
burps can be used to publish the builds. The 'publish' command will
build the selected project, and if the build suceeds run the script
from the 'publish' option to publish the build result. The prototype
has an example publish script, in the file 'projects/common/publish':
----
#!/bin/bash
set -e
pubdir='[% c('var/publish_dir', {error_if_undef => 1}) %]/[% project %]/[% c('version') %]'
mkdir -p "$pubdir"
mv * "$pubdir"
rm -f "$pubdir/build"
----
We can see that it uses the option 'var/publish_dir' to select the
directory where the files should be put, so we can define it in
'/etc/burps.conf' like this:
----
var:
publish_dir: /tmp/publish
----
We can now run the following command if we want to publish a build of
libevent:
----
$ burps publish libevent --target dev
[...]
$ find /tmp/publish/libevent/
/tmp/publish/libevent/
/tmp/publish/libevent/4c8ebcd3598a
/tmp/publish/libevent/4c8ebcd3598a/libevent-4c8ebcd3598a-Mageia-3-x86_64.tar
----
'burps build' can be the command that jenkins use to build and publish
new builds.
This publish script could be extended to compress tarballs, send emails
or update the xref:builds-database[builds database].
Testing builds
^^^^^^^^^^^^^^
burps does not have yet a 'test' command, but it could be added, with a
script to be run after the build succesfully ended, and before the
publish script is run.
Using the xref:vm-tools[VM tools] described below, those test scripts
could create and start some VM or containers in which the project that
was built and useful test helpers are installed, and some tests are run.
This would be used by Jenkins to run the tests, but is also available for
developers who want to run the tests on their own computer.
Summary of improvements
^^^^^^^^^^^^^^^^^^^^^^^
The main improvements in this prototype from the current build process
are:
- all components are built separately, and include in their output file
name the commit hash or version, architecture and OS (if architecture
dependant). This allows us to keep previous builds if the
commit/version/architecture/OS didn't change. So we can rebuild a
bundle very quickly when the browser didn't change.
- the gitian replacment has features to download tarballs and verify them
with sha256sum or gpg signature, so this can replace the fetch-inputs.sh
script.
- we can remove the linux/windows/macosx descriptors duplication, and
instead use template directives for the parts that differ between
those builds (it's still possible to use separate files if they differ
completly).
- we can define variables based on selected OS. This allows for instance
to build python 2.7 when building on Ubuntu Lucid, but avoid building
it on other distros which already provide python 2.7.
- we can define variables based on "targets" that are set on command
line. For instance in the prototype using "--target enable_pt" instructs
to include the portable transports (only pyptlib in this prototype) in
the bundle.
- we can easily switch from building in a VM to building locally
- build descriptors can depend on the result of another build descriptor.
This remove needs for scripts like mkbundle-*.sh.
And I think those improvements should make it easier to rebuild a new
bundle automatically when any of the components of the bundle receives
a new commit, and then run tests on this bundle.
What is currently missing
^^^^^^^^^^^^^^^^^^^^^^^^^
The following features are not yet implemented in this prototype but
will be needed:
package dependencies installation:: We should be able to list the
required build dependencies, so that they are installed automatically
(and optionally removed at the end of the build). I have some ideas
about how to do that so that it can work on multiple distributions,
but have not implemented it yet.
VM or chroot creation:: While burps can be used to build inside a chroot,
or a VM with ssh, the current prototype does not handle the creation
of a VM image or chroot. So we need a tool to automate that part,
similar to the 'gitian/make-vms.sh' script.
Future work
^^^^^^^^^^^
Here are some ideas for future work to be done on this tool.
Integration of tests:: We can make it possible to run some tests from
this tool after a build. Those tests can instanciate some VMs to run
the tests inside those VMs.
Integration with builds database:: See the proposal about a builds database.
We can integrate burps so that builds can be automatically uploaded to
this database. We can also make it possible to download some input files
instead of build them, if the files are available in the build database
with enought signatures from trusted people.
Reproducible build testing:: We can add an option to build software
using a library that randomize readdir(3) and python dict.iterkeys to
make non-reproducibility issues more reproducible, as
https://lists.torproject.org/pipermail/tor-dev/2013-December/005925.html[discussed on tor-dev].
Attachment:
pgptbotP3sraZ.pgp
Description: PGP signature
_______________________________________________ tor-dev mailing list tor-dev@xxxxxxxxxxxxxxxxxxxx https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev