[Author Prev][Author Next][Thread Prev][Thread Next][Author Index][Thread Index]
Re: [tor-bugs] #29693 [Core Tor/Tor]: Decrease probability of stochastic failures in test-slow
#29693: Decrease probability of stochastic failures in test-slow
----------------------------------------+----------------------------------
Reporter: teor | Owner: asn
Type: defect | Status: assigned
Priority: High | Milestone: Tor:
| 0.4.0.x-final
Component: Core Tor/Tor | Version: Tor:
| 0.4.0.1-alpha
Severity: Normal | Resolution:
Keywords: tor-ci, tor-test, 040-must | Actual Points:
Parent ID: | Points: 0.5
Reviewer: | Sponsor: Sponsor2-must
----------------------------------------+----------------------------------
Comment (by teor):
Another failure at:
https://travis-ci.org/tlyu/tor/jobs/508233862#L4244
Here's what Riastradh said on IRC today:
> nickm: Hi! You are welcome to publish the IRC discussion we had earlier
about stochastic tests.
> (I don't remember which one, but you have my permission to publish all
of the discussions we've had about stochastic tests and the distribution
samplers since November or whenever this all started.)
> I saw that there was an issue about changing the false positive rate.
I'm low on energy right now, but here's the three things that I would
suggest doing, some of which I might do if I had more energy:
> 1. Write some tests of the tests -- that is, write a _buggy_ sampler for
a distribution, and apply a stochastic test to it, and confirm the
stochastic test _fails_.
> Examples:
https://github.com/brave/crypto/blob/master/test/randomTest.js,
https://github.com/probcomp/crosscat/blob/master/cpp_code/tests/test_random_number_generator.cpp
> You'll want to estimate the false positive rate of these test-tests
(i.e., the statistical power of the tests to detect the bugs) empirically,
since for most bugs there will be no neat analytic expression for it.
> 2. Tweak NTRIALS and NPASSES_MIN so that the false positive rates of the
usual tests _and_ of the test-tests are acceptable. The first one you can
compute analytically as I described in past conversations; the second will
necessarily be based on the empirical measurements in (1).
> 3. Teach the CI to report the alarm rates -- not just number of alarms,
but ratio of alarms to total tests run. And keep this state continuously
across CI jobs so it can be aggregated over time.
In 0.4.0, I think increasing NTRIALS is our best option. I don't have
access to the previous conversation about NTRIALS. If we can't find it,
let's ask Riastradh, or just double NTRIALS.
--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/29693#comment:4>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online
_______________________________________________
tor-bugs mailing list
tor-bugs@xxxxxxxxxxxxxxxxxxxx
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs