[Author Prev][Author Next][Thread Prev][Thread Next][Author Index][Thread Index]
Re: [tor-bugs] #15844 [Onionoo]: Develop database schema to support Onionoo's search parameter efficiently
#15844: Develop database schema to support Onionoo's search parameter efficiently
-----------------------------+-----------------
Reporter: karsten | Owner:
Type: enhancement | Status: new
Priority: normal | Milestone:
Component: Onionoo | Version:
Resolution: | Keywords:
Actual Points: | Parent ID:
Points: |
-----------------------------+-----------------
Comment (by karsten):
Ueland, it might be that search engines can help us here, but AFAIK,
nobody has investigated that yet. Note that both our data and possible
searches are highly structured, so that databases might be a better fit
than search engines. That's why I'm more optimistic about databases than
search engines. But I'm not ruling out anything here, and if you want to
look into search engines for this purpose, that might be quite
interesting. If you're interested, please take a look at the
[https://onionoo.torproject.org/protocol.html#methods protocol
specification] for the possible searches that the search engine would have
to support. Also note the possible simplifications to that protocol
mentioned on this ticket.
teor, leeroy, thanks a lot for your comments. I started writing a
response a couple of times now, but was overwhelmed by the amount of
detail in this thread and gave up. But I really want to respond, because
I feel we're getting somewhere. I hope the following response makes
sense, even though it doesn't address all your suggestions. But all the
good ideas remain in this thread, and whenever we start writing code, we
should revisit this entire thread.
We don't need to support `%foo%bar%`. We should check all user input and
avoid searches including `%` or other special characters before passing it
to the database.
If substring searches are just too hard to implement, let me suggest
another simplification: how about we sacrifice the `contact` parameter
together with substring searches in the `nickname` field? I think that
people are using the `contact` parameter to obtain all relays run by the
same entity, but they could as well use the `family` parameter for that.
If the choice is to either keep supporting that parameter or to offer
searches over the entire history, I'd prefer the latter. Again, not
something to decide quickly, but worth considering. I should ask people
on tor-dev@ before we make a final decision though.
At the same time, maybe we shouldn't restrict searches to 3+ characters
for the reasons you state above. Scratch that idea, it probably wouldn't
help that much anyway.
Example: a search for "u" would still return all relays called "Unnamed"
and "u", but not "default".
Regarding performance, I'm not concerned about mean time but much more
about variance. If a request takes 100 ms on average, that's fine by me,
but if some requests take 60 s, then something's very wrong. I also worry
about potential lack of scalability, either to thousands of concurrent
clients or to twice or three times as many entries in the database. I'd
prefer a solution that has reasonable performance for most use cases,
rather than one that is highly optimized for one use case and that might
perform badly for use cases we didn't think. Of course it's no hard
requirement that the database performs better than the current in-memory
search.
How do we proceed? leeroy, did you want to write some database code and
run some performance measurements? If so,
[https://people.torproject.org/~karsten/volatile/summary.xz here's some
sample data] (the same that I mentioned above).
Thanks everyone!
--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/15844#comment:10>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online
_______________________________________________
tor-bugs mailing list
tor-bugs@xxxxxxxxxxxxxxxxxxxx
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs