On Mon, 09 Dec 2013, Georg Koppen wrote: > Hi Nicolas, > > some remarks are below. Hi Georg, Thanks for your remarks. > > Nicolas Vigier: > > In order to help me doing that, I'm very interested to receive from > > developers of any tor components : > > > > - a description or ticket number of bugs that you wish could have been > > detected earlier with automated tests > > https://trac.torproject.org/projects/tor/ticket/8143 comes to mind here. Ok. We should soon be able to push tor browser to Mozilla's Try servers, which will run all the unit tests, so we should be able to detect bugs like #8143. https://wiki.mozilla.org/ReleaseEngineering/TryServer > > > - any wish or specific needs that you may have > > You write: "Tor Browser includes some patches to make the build > reproducible. We could have a test that check the reproducibility of the > build by building the browser twice." > > While this is indeed a good idea, it won't be enough as we had bugs in > the past that were only visible when builds on different machines got > compared. So, what I'd like to have (in addition to running browser > builds twice? on different machines?) are tests that cover specific bugs > we avoided (see: the "Remaining Build Reproducibility Issues" in Mike's > blog post covering the technical details of the Gitian build) or tracked > down. See: https://trac.torproject.org/projects/tor/ticket/10159 for an > example for the latter. (There, one could write a test that automates > the creation of the browser.manifest which would eventually (i.e. if run > a couple of times) show whether this bug still exists or whether not).) Ok, we can have a test rebuilding several times the files that are most likely to become non deterministic. However it would be better if we can find some way to trigger those non-deterministic builds with only two builds. Maybe we can try something like this : - a library that we put in LD_PRELOAD as a wrapper on readdir to return directory entries in random order instead of inodes number, so it's more likely to be different on each build. - a python library that we put in PYTHONPATH, to override dict.iterkeys to return keys in random order. If I understand ticket #10159 correctly, it was caused by iterkeys returning keys in the same order most of the time, but not all the times. > > > - anything else that you think might be useful for me to know > > You write: "We can produce some packages for Tor Browser, to make > testing of the browser easier." > > In which regard is it easier to test the browser if you have packages? Hmm yes, it does not really help in that regard, I've removed this. What can help is packages nightly builds, as it makes it easy for someone to update their tor browser every day to act as a beta tester. > And how should that look like outside of the Tor Browser Bundle? I fear > we create extra bugs if we move the browser outside the environment it > will be used in (i.e. the TBB) just for testing purposes. Even if we > won't create extra bugs this way we might miss some. To be clear, > running the tests you call "Usablility tests" and "Reproducible build > test" outside the bundle, seems fine to me, while my concerns apply to > the fingerprinting and privacy tests. What it would look like is similar to what they use in tails : https://tails.boum.org/contribute/release_process/iceweasel/ But you are right that we can miss some bugs if we only test it outside TBB, so we should also run the tests in TBB. > > The test helpers are a good idea. You might want to rename "Cookies > tool" to something like "Identifier tool" matching the Tor Browser > specification more closely (especially as you actually mean "Identifier > tool" as "Later versions could be extended to also use other techniques > for storing informations in the browser" indicates). Ok, updated. An other thing that can be done is keeping an archive of all builds for 3 months (or more depending on space available), so that git bisect can be done more quickly. Would that be helpful ?
Attachment:
pgpPuCZnuOafm.pgp
Description: PGP signature
_______________________________________________ tor-dev mailing list tor-dev@xxxxxxxxxxxxxxxxxxxx https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev