[Author Prev][Author Next][Thread Prev][Thread Next][Author Index][Thread Index]
Re: [tor-bugs] #20335 [Metrics/CollecTor]: ReferenceChecker causes OOM
#20335: ReferenceChecker causes OOM
-------------------------------+---------------------
Reporter: iwakeh | Owner:
Type: defect | Status: new
Priority: High | Milestone:
Component: Metrics/CollecTor | Version:
Severity: Normal | Resolution:
Keywords: | Actual Points:
Parent ID: | Points:
Reviewer: | Sponsor:
-------------------------------+---------------------
Changes (by karsten):
* priority: Medium => High
Comment:
I ''believe'' that's a smart thing to do. I'm not sure if that would
break something.
But I think it doesn't solve the issue here. Or, I ran out of memory with
a similar patch applied. And I think I know why:
{{{
-rw-r--r-- 1 karsten staff 2.1G Oct 18 11:52
/Users/karsten/src/collector/recent/relay-
descriptors/votes/2016-10-18-09-22-46-votes
}}}
With the recent change to append all votes to a single file, we're
creating a 2.1G large file when doing the first synchronization run.
However, metrics-lib wants to read that entire file to memory which
probably doubles in size when it's parsed. Ugh!
Note that the CollecTor's reference checker is just one application that
breaks when input files are too large. Other applications using metrics-
lib would similarly be affected.
A longer-term fix might be to change metrics-lib to read through large
files and pause when it has parsed enough descriptors that the application
still has to process.
A short-term fix would be to stick with the current naming scheme for
files in the `recent/` directory, as that would have avoided this issue.
Fun times!
--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/20335#comment:10>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online
_______________________________________________
tor-bugs mailing list
tor-bugs@xxxxxxxxxxxxxxxxxxxx
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs