[Author Prev][Author Next][Thread Prev][Thread Next][Author Index][Thread Index]
Re: [tor-bugs] #4086 [Analysis]: Compare performance of TokenBucketRefillInterval params in simulated network
#4086: Compare performance of TokenBucketRefillInterval params in simulated
network
-------------------------------------+--------------------------------------
Reporter: arma | Owner:
Type: task | Status: new
Priority: normal | Milestone:
Component: Analysis | Version:
Keywords: performance flowcontrol | Parent: #4465
Points: | Actualpoints:
-------------------------------------+--------------------------------------
Comment(by robgjansen):
Replying to [comment:27 arma]:
> Replying to [comment:26 arma]:
>
>
> > Ok. I'm increasingly thinking that a "run" should be k runs in a row,
averaged.
> >
> >
>
> And if we want to get super fancy, we could draw bars on the data points
in each case, to give a sense of variance between runs. (Does that notion
even make sense for cdf graphs?)
Doesn't it make more sense to just show the cumulative results for all
experiments with the same configuration. I normally do this before
publishing results in papers, after I am confident I understand the code
changes and their effects well enough. I'm not sure we are there yet with
the work in this ticket.
Though, it would be nice to be able to determine how far one CDF varies
from another. I attempted to do something like this [comment:15 above in
comment 15], but I think in that case I wasn't comparing apples to apples.
I'd like to emphasize that Shadow is already cutting out as many random
variances as possible. In other words, if I run the same experiment twice
without changing anything, the results are exactly the same (except for
memory addresses and timestamps in log files :P). I've verified this
several times.
But, it is still the case that a given experiment ''could'' get unlucky
with its seed to the master PRNG, and a configuration change ''could''
change the randomness enough to avoid the "unlucky" behaviors. I have not
tested "run vanilla Tor with several seeds and analyze the variances"
recently, but its probably a good idea.
--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/4086#comment:29>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online
_______________________________________________
tor-bugs mailing list
tor-bugs@xxxxxxxxxxxxxxxxxxxx
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs