[Author Prev][Author Next][Thread Prev][Thread Next][Author Index][Thread Index]

Re: [tor-bugs] #25100 [Metrics/CollecTor]: Make CollecTor's webstats module use less RAM and wall time

#25100: Make CollecTor's webstats module use less RAM and wall time
-------------------------------+------------------------------
 Reporter:  karsten            |          Owner:  iwakeh
     Type:  enhancement        |         Status:  needs_review
 Priority:  High               |      Milestone:
Component:  Metrics/CollecTor  |        Version:
 Severity:  Normal             |     Resolution:
 Keywords:                     |  Actual Points:
Parent ID:                     |         Points:
 Reviewer:                     |        Sponsor:
-------------------------------+------------------------------

Comment (by iwakeh):

 I didn't find any more bugs processing the following benchmarks for
 determining how to approach the import using the two latest branches of
 this and the child ticket.

 The second quarter 2017 of logs from weschniakowii amounts to 32M
 (compressed) and can be processed in 36min using 8G.  The entire year
 won't work with just 8G.

 85min and 16G are needed for the entire available archives of meronense
 and weschniakowii together (compressed 59M).  The used heap median usage
 is 8.5G and the max 15.8G.

 So, depending on the hardware a conservative import strategy might be to
 import slices of quarters with subsequent CollecTor runs.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/25100#comment:18>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online

_______________________________________________
tor-bugs mailing list
tor-bugs@xxxxxxxxxxxxxxxxxxxx
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs