[Author Prev][Author Next][Thread Prev][Thread Next][Author Index][Thread Index]
Re: [tor-bugs] #18910 [Metrics/CollecTor]: distributing descriptors accross CollecTor instances
#18910: distributing descriptors accross CollecTor instances
-------------------------------+---------------------------------
Reporter: iwakeh | Owner: iwakeh
Type: enhancement | Status: needs_review
Priority: High | Milestone: CollecTor 1.1.0
Component: Metrics/CollecTor | Version:
Severity: Normal | Resolution:
Keywords: ctip | Actual Points:
Parent ID: | Points:
Reviewer: | Sponsor:
-------------------------------+---------------------------------
Comment (by karsten):
Here's a fresh bug report: synchronizing relay descriptors takes a few
''hours'' on average on my laptop, and I think I know why. Whenever we
append to an existing file in `recent/`, we copy the entire file content
to a temporary file, append the new content there, and then rename. And
we do that for every single server descriptor or extra-info descriptor we
receive. That is bad for two reasons. First reason is that it obviously
doesn't scale, with some sync runs taking 4 hours or longer. Second
reason is that files in `recent/` shouldn't change after they're included
in `index.json` (with the inglorious exception of Torperf measurements),
which is why we should append all descriptors to a temporary file and
rename when we're sure we're done. See also
`ArchiveWriter.cleanUpRsyncDirectory()` where we're renaming temporary
files to destination files at the very end of the update run.
--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/18910#comment:88>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online
_______________________________________________
tor-bugs mailing list
tor-bugs@xxxxxxxxxxxxxxxxxxxx
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs