[Author Prev][Author Next][Thread Prev][Thread Next][Author Index][Thread Index]
[tor-bugs] #26585 [Metrics/Onionoo]: improve AS number and name coverage (switch maxmind to RIPE Stat)
#26585: improve AS number and name coverage (switch maxmind to RIPE Stat)
---------------------------------+--------------------------
Reporter: nusenu | Owner: metrics-team
Type: enhancement | Status: new
Priority: Medium | Milestone:
Component: Metrics/Onionoo | Version:
Severity: Normal | Keywords:
Actual Points: | Parent ID:
Points: | Reviewer:
Sponsor: |
---------------------------------+--------------------------
Onionoo currently uses maxmind for IP to as_number and as_name resolution.
This is fast as it is a local DB lookup but it is less up-to-date and has
less coverage than RIPEstat https://stat.ripe.net/
This is a problem for tools that depend on onionoo's as_name and as_number
fields like Relay Search, OrNetStats and OrNetRadar
(I don't know maybe there are also others that are affected?)
Currently it might takes weeks or months before new ASes get added to
maxmind so this information
is also missing when people lookup relays on Relay Search.
As of today onionoo is missing AS level data for about 100 relays,
but this value depends on how far we are away from the last maxmind
update.
How about we use RIPEstat API as a data source + local cache.
To minimize the amount of required online queries against the RIPEstat API
we can do the following
to create the IP to AS map initially (pseudocode):
if ip_prefix in cache
use cached entry
else
perform an online lookup (query RIPEstat API)
add new prefix entry to cache
expire cache entries after 15 days?
(it makes sense to log how many entries changed
after 15 days so we know whether this value is to large or to small)
This will significantly reduce the amount of required online API calls.
To give you an idea of the scale (based on onionoo data from a random
day in May 2018)
total relay records: 8116
unique IPv4 addresses: 7794
unique IPv4 BGP prefixes: 3884
each day about 50 new relays appear,
lets assume the worst case (every new relay is not in an
So on the estimated daily amount of queries you do would be around
4000/15+50 = ~320 requests/day = 1 req every ~4 minutes
which appears acceptable.
IP to prefixes (this can return multiple matches), as_number and as_name
lookup:
https://stat.ripe.net/data/related-
prefixes/data.json?resource=103.114.160.21
IP to prefix and as number (no as_name) lookup:
https://stat.ripe.net/data/network-info/data.json?resource=103.114.160.21
ASN to as_name lookup:
https://stat.ripe.net/data/as-overview/data.json?resource=AS40676
documentation:
https://stat.ripe.net/docs/data_api
--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/26585>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online
_______________________________________________
tor-bugs mailing list
tor-bugs@xxxxxxxxxxxxxxxxxxxx
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs