[Author Prev][Author Next][Thread Prev][Thread Next][Author Index][Thread Index]
[tor-bugs] #3406 [Torctl]: High memory usage from parse_ns_body
#3406: High memory usage from parse_ns_body
--------------------+-------------------------------------------------------
Reporter: atagar | Owner:
Type: defect | Status: new
Priority: normal | Milestone:
Component: Torctl | Version:
Keywords: | Parent:
Points: | Actualpoints:
--------------------+-------------------------------------------------------
From earlier today...
09:16 < atagar> Figured out my memory issue. It's from calling TorCtl's
get_network_status function which consumes 3.5 MB of memory at once.
09:16 < atagar> I immediately discard those results and the memory is
freed, however because of how large objects are allocated in python 'ps'
is never aware that the memory's been released:
09:16 < atagar> http://effbot.org/pyfaq/why-doesnt-python-release-the-
memory-when-i-delete-a-large-object.htm
09:16 < atagar> Fix: either ignore it (it's misreported memory usage) or
implement my own less memory intensive version of get_network_status.
09:20 < mikeperry> atagar: does python *never* release said memory? in
otherwords, do you experience this as a leak, or just a consistent 3.5MB
of unneeded overhead?
09:20 < mikeperry> s/release/reuse
09:22 < atagar> I'm not quite sure how to tell the difference. If I call
get_network_status and discard the results (ie, don't get a reference to
the nslist object it returns) then ps reports that memory as being used
for the life of the python process.
09:23 < atagar> though I'd imagine this memory would be reused by python
for other activity when needed
09:25 < mikeperry> hopefully. there is definitely a leak in the bw
authorities that is proving ridiculously hard to track down. we're down to
it either being a python leak, an sqlite leak, or a python-sqlite bindings
leak
09:26 < atagar> side note: the memory all comes from the NetworkStatus
objects generated on line 1246 (ie, the problem is definitely because we
have a list with all that memory allocated at one time, rather than the
incremental results contributing to it)
09:27 < atagar> ie, if we were able to have an iterator pattern here then
there would be no issue
09:28 < mikeperry> if you want to write an intermediate funtion that only
gives you them one at a time, you can do that. just don't change the
external behavior of the current function.
09:28 < mikeperry> I'd merge that
09:29 < atagar> Will do. I'm a little worried about a future change
accidently making a get_network_status call, but I'll edit the comment to
warn of this.
09:29 < atagar> btw, do you have any idea when you'll be able to look at
the five torctl patches I sent on Monday?
09:30 * nickm is going to be semi- offline for most of today afternoon
doing errands writing reviews and hacking. Hoping this is a good idea. :)
09:34 < atagar> mikeperry: I'll also fix update_consensus so it doesn't
encounter this issue (no external change but will avoid new consensus
events from triggering huge memory usage)
--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/3406>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online
_______________________________________________
tor-bugs mailing list
tor-bugs@xxxxxxxxxxxxxxxxxxxx
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs