[Author Prev][Author Next][Thread Prev][Thread Next][Author Index][Thread Index]
Re: [tor-bugs] #29693 [Core Tor/Tor]: Decrease probability of stochastic failures in test-slow
#29693: Decrease probability of stochastic failures in test-slow
----------------------------------------+----------------------------------
Reporter: teor | Owner: asn
Type: defect | Status: assigned
Priority: High | Milestone: Tor:
| 0.4.0.x-final
Component: Core Tor/Tor | Version: Tor:
| 0.4.0.1-alpha
Severity: Normal | Resolution:
Keywords: tor-ci, tor-test, 040-must | Actual Points:
Parent ID: | Points: 0.5
Reviewer: | Sponsor: Sponsor2-must
----------------------------------------+----------------------------------
Comment (by asn):
OK, my suggestion is to increase N_TRIALS from 2 to 3 for now, and open a
ticket for the future to do more advanced stuff like the suggestions from
comment:4.
In particular with N_TRIALS=2 now and 14 travis jobs per build, we have
1.3% probability of a travis build failing because of the stoch tests.
This means that one travis build will fail after 50 travis builds with 50%
chance.
If we bump N_TRIALS to 3, then we have 0.013% probability of a travis
build failing because of the stoch tests. This means that one travis
build will fail after 4952 travis builds with 50% chance. Not so annoying
anymore.
Here is a (potentially borken) python script I used to calculate the data
above, along with the documentation here
https://github.com/torproject/tor/blob/938d97cb0d4acfdd1ea57ec0a3094bcc2101f13d/src/test/test_prob_distr.c#L866
:
{{{
N_TRIALS = 3
# Probability of a stochastic test failing
alpha = pow(0.01, N_TRIALS)
# number of stochastic tests
n_stoch_tests = 10
# Probability of at least one stochastic test failing
failure_rate_for_test_suite = 1 - pow(1 - alpha, n_stoch_tests)
print("With N_TRIALS={} and alpha={} we have failure_rate_for_test_suite
{}".format(N_TRIALS, alpha, failure_rate_for_test_suite))
# Number of travis jobs per build
n_travis_jobs = 14
# Probability of at least one travis job failing
failure_rate_of_travis_build = 1 - pow(1 - failure_rate_for_test_suite,
n_travis_jobs)
print("With N_TRIALS={} and alpha={} we have travis build fail
{}".format(N_TRIALS,alpha, failure_rate_of_travis_build))
for n in xrange(5000):
# Probability of travis build failing after n builds
p = 1 - pow(1 - failure_rate_of_travis_build, n)
if p > 0.5:
print("With N_TRIALS={} and alpha={}, a travis build will fail
with 50% chance after {} builds.".format(N_TRIALS,alpha,n))
break
}}}
--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/29693#comment:10>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online
_______________________________________________
tor-bugs mailing list
tor-bugs@xxxxxxxxxxxxxxxxxxxx
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs