Re: [tor-bugs] #11573 [Onionoo]: Ponder using a database for Onionoo rather than keeping indexes in memory and contents on disk

["Tor Bug Tracker & Wiki" <blackhole@xxxxxxxxxxxxxx>]

#11573: Ponder using a database for Onionoo rather than keeping indexes in memory
and contents on disk
-----------------------------+-----------------
     Reporter:  karsten      |      Owner:
         Type:  enhancement  |     Status:  new
     Priority:  minor        |  Milestone:
    Component:  Onionoo      |    Version:
   Resolution:               |   Keywords:
Actual Points:               |  Parent ID:
       Points:               |
-----------------------------+-----------------

Comment (by karsten):

 Let's see.  We do have some performance statistics now which we didn't
 have when this ticket was created.

 {{{
 Request statistics (2014-08-19 18:04:00, 3600 s):
   Total processed requests: 11371
   Most frequently requested resource: details (8372), weights (1759),
 bandwidth (1120)
   Most frequently requested parameter combinations: [lookup] (5015),
 [fingerprint] (3749), [lookup, fields] (2371)
   Matching relays per request: .500<2, .900<2, .990<8, .999<16384
   Matching bridges per request: .500<1, .900<1, .990<2, .999<16384
   Written characters per response: .500<2048, .900<16384, .990<32768,
 .999<16777216
   Milliseconds to handle request: .500<8, .900<16, .990<64, .999<128
   Milliseconds to build response: .500<4, .900<16, .990<512, .999<16384
 }}}

 The last two lines are most relevant here.  "Handle request" means parsing
 a request with all parameters and looking up relays or bridges that should
 go into the response.  "Build response" means taking a list of relay or
 bridge fingerprints and writing all JSON files from disk to the response.

 `.999<128` in the last but one line means that 99.9% of requests are
 handled in under 128 milliseconds.  In fact, 128 is the next bigger power
 of two, it could also be less than 65 milliseconds.  But this seems like a
 fine value to me.

 However, `.990<512` and `.999<16384` in the last line worry me a bit.  It
 means that the 0.9% of requests between 99% and 99.9% can take up to 16
 seconds to build.  And it's unclear from these data how much longer the
 slowest 0.1% take.

 But before we replace the file system storage with a database, should we
 find out if it's really the file system that slows us down here?  It could
 as well be the network that doesn't let us write bytes as fast as they're
 coming from disk.  (And even if it's the file system, let's benchmark
 databases first before deciding on one.)

 Want to write a patch to ResourceServlet's doGet() to obtain more
 meaningful statistics that tell us if we really want to mess with
 databases?

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/11573#comment:3>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online
_______________________________________________
tor-bugs mailing list
tor-bugs@xxxxxxxxxxxxxxxxxxxx
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs