[Author Prev][Author Next][Thread Prev][Thread Next][Author Index][Thread Index]
Re: [tor-bugs] #19249 [Metrics/Onionoo]: Onionoo server runs out of memory when importing a full month of data
#19249: Onionoo server runs out of memory when importing a full month of data
-----------------------------+---------------------
Reporter: karsten | Owner:
Type: defect | Status: new
Priority: Medium | Milestone:
Component: Metrics/Onionoo | Version:
Severity: Normal | Resolution:
Keywords: | Actual Points:
Parent ID: | Points:
Reviewer: | Sponsor:
-----------------------------+---------------------
Comment (by iwakeh):
An initial analysis:
With enough memory provided the archive data import of a month doesn't
take that long.
First I altered the code to not run the download part in order to get a
clearer picture there, i.e.
the download descriptors part is omitted (hope, this isn't totally
unreasonable).
I ran the imports with the tars listed above and an otherwise empty
Onionoo environment.
Using 4g to 6g it needs less than two hours for the entire import:
{{{
2016-06-05 18:47:47,866 INFO o.t.o.cron.Main:191 Reading descriptors.
2016-06-05 18:47:47,867 INFO o.t.o.u.DescriptorSource:153 Reading archived
descriptors...
...
2016-06-05 20:23:10,832 INFO o.t.o.cron.Main:236 Cleaning up.
2016-06-05 20:23:10,833 INFO o.t.o.cron.Main:242 Done.
2016-06-05 20:23:10,833 INFO o.t.o.cron.Main:246 Releasing lock.
2016-06-05 20:23:10,865 INFO o.t.o.cron.Main:248 Released lock
}}}
Using way less causes the OOM, e.g. only 2g, which happens pretty quickly.
When 3g are provided the processing runs for a very, very long time as
visible in the log excerpt in the description. This happens b/c the jvm is
garbage collection most of the time to stay running, but there is hardly
any processing time left for the onionoo updater. With such a scarce
amount of memory it just hits oom anyway (as in the initial error
description).
I assume, that the full onionoo single run will need memory >= 6g to run
comfortably when importing a month worth of data.
The 4g are not sufficient and just keep the jvm busy collecting garbage
and the processing takes ages.
Hope, it is possible to increase the memory! As an immediate measure at
least.
Still other means of reducing the memory consumption should be
investigated.
Those should be new trac issues then.
PS: I also caught the nfe's (#19259).
--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/19249#comment:1>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online
_______________________________________________
tor-bugs mailing list
tor-bugs@xxxxxxxxxxxxxxxxxxxx
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs