[Author Prev][Author Next][Thread Prev][Thread Next][Author Index][Thread Index]
Re: [tor-bugs] #29693 [Core Tor/Tor]: Decrease probability of stochastic failures in test-slow
#29693: Decrease probability of stochastic failures in test-slow
----------------------------------------+----------------------------------
Reporter: teor | Owner: asn
Type: defect | Status: needs_review
Priority: High | Milestone: Tor:
| 0.4.0.x-final
Component: Core Tor/Tor | Version: Tor:
| 0.4.0.1-alpha
Severity: Normal | Resolution:
Keywords: tor-ci, tor-test, 040-must | Actual Points: 0.5
Parent ID: | Points: 0.5
Reviewer: | Sponsor: Sponsor2-must
----------------------------------------+----------------------------------
Comment (by asn):
Replying to [comment:11 teor]:
> Here are our worst-case scenarios:
> * a merge forward from 0.4.0 (or earlier) to master:
> * 3 (maint-0.4.0, release-0.4.0, master) * 14 jobs (travis, appveyor)
> * With N_TRIALS=3 and alpha=1e-06, a travis build will fail with 50%
chance after 1651 builds.
> * pull requests to 0.4.0 and master:
> * 2 (branch, PR) * 2 (0.4.0, master) * 14 jobs (travis, appveyor)
> * With N_TRIALS=3 and alpha=1e-06, a travis build will fail with 50%
chance after 1238 builds.
>
> These error rates are acceptable, but only until we split off
maint-0.4.1. Then we will need to go to 4 trials, or find some other
solution.
ACK. I also pushed https://github.com/torproject/tor/pull/824 which bumps
it to 4. Let's minimize false positive interruptions for now and think of
smarter approaches in #29847.
For the record:
{{{
With N_TRIALS=4 and alpha=1e-08 we have failure_rate_for_test_suite
9.99999959506e-08
With N_TRIALS=4 and alpha=1e-08 we have travis build fail
1.39999903326e-06
With N_TRIALS=4 and alpha=1e-08, a travis build will fail with 50% chance
after 495106 builds.
}}}
--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/29693#comment:13>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online
_______________________________________________
tor-bugs mailing list
tor-bugs@xxxxxxxxxxxxxxxxxxxx
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs