[Author Prev][Author Next][Thread Prev][Thread Next][Author Index][Thread Index]
Re: [tor-bugs] #20395 [Metrics/Library]: Add capability to handle large descriptor files
#20395: Add capability to handle large descriptor files
-----------------------------+------------------------------
Reporter: iwakeh | Owner: karsten
Type: defect | Status: needs_review
Priority: Medium | Milestone:
Component: Metrics/Library | Version:
Severity: Normal | Resolution:
Keywords: | Actual Points:
Parent ID: | Points:
Reviewer: | Sponsor:
-----------------------------+------------------------------
Changes (by karsten):
* status: accepted => needs_review
Comment:
I started making some improvements here. Here's my train of thought:
1. Rather than reading the whole file to memory at the beginning, we
could read it in chunks and start parsing as soon as we have seen a full
descriptor. This sounds like a useful improvement, but it's actually very
limited, at least on its own. Reading the 70M descriptor file I used for
testing is actually done really fast. It's the parsing that takes long. As
long as we need the full descriptor file contents in memory, we don't have
to think about reading files in chunks. (See also 3. below.)
2. Rather than parsing all descriptors contained in a given file into a
list and then taking all parsed descriptors and throwing them into the
`BlockingIterator<Descriptor>`, we could just skip the list in the middle.
The effect is that the time to first descriptor is reduced by a huge
amount of time, whereas the time to last descriptor stays the same. I
prepared a patch for this. The commit message contains more details.
3. Rather than storing descriptor file contents in a `byte[]`, we could
go through the file, read descriptor by descriptor, and store a `File`
reference together with offset and length into the file. The effect would
be that we're avoiding to keep the raw descriptor file contents in memory
at all. We'd still keep parsed contents in memory. A possible downside is
that the file must not be deleted or moved away while the application
processes descriptors, which should be safe to require. Still, this is a
larger change than 2. And it requires 1. That's why I postponed this.
Please review [https://gitweb.torproject.org/user/karsten/metrics-
lib.git/commit/?h=task-20395&id=ef9406c148a477720cdca67c6a2891ecd850f912
commit ef9406c in my task-20395 branch].
--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/20395#comment:15>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online
_______________________________________________
tor-bugs mailing list
tor-bugs@xxxxxxxxxxxxxxxxxxxx
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs