[Author Prev][Author Next][Thread Prev][Thread Next][Author Index][Thread Index]

Re: [tor-dev] Proposal 274: A Name System API for Tor Onion Services



Hi,

I think that this work is extremely important, and I applaud the
authors and contributors for their work.  It seems that it will also
be complicated, semantically.  What lies where, and which thing
does what?  The example recently about what is sent in the Host
header is a good example of this challenge.

I have below, in situ, made a few suggested corrections, and offered
my suggestions on a few topics.  I am not precious about my 
opinions and am happy to have them shot down by people with
a deeper understanding of the challenge at hand.

My overriding suggestion is to keep this as simple as possible in
its initial iteration.  It will almost certainly get more complex down
the line, and working out what the core principles are early on will
help guard against "feature creep".

Well done, and good luck,

  Hugo

NB: all comments prefixed with '> '
---

Filename: 274-naming-layer-api.txt
Title: A Name System API for Tor Onion Services
Author: George Kadianakis, Yawning Angel, David Goulet
Created: 04-Oct-2016
Status: Draft

  Table Of Contents:

                        1. Introduction
                                1.1. Motivation
                                1.2. Design overview and rationale
                        2. System Specification
                                2.1. System overview [SYSTEMOVERVIEW]
                                2.2. System Illustration
                                2.3. System configuration [TORRC]
                                        2.3.1. Tor name resolution logic
                                2.4. Name system initialization [INITPROTOCOL]
                                2.5. Name resolution using NS API
                                        2.5.1. Message format
                                        2.5.2. RESOLVED status codes
                                        2.5.3. Further name resolution behavior
                                2.6. Cancelling a name resolution request
                                2.7. Launching name plugins [INITENVVARS]
                                2.8. Name plugin workflow [NSBEHAVIOR]
                                        2.8.1. Name plugin shutdown [NSSHUTDOWN]
                                2.9. Further details of stdin/stdout communication
                                        2.9.1. Message Format
                        3. Discussion
                                3.1. Using second-level domains instead of tld
                                3.2. Name plugins handling all tlds '*'
                                3.3. Deployment strategy
                                3.4. Miscellaneous discussion topics
                        4. Acknowledgements
                        A.1: Example communication Tor <-> name plugin [PROTOEXAMPLE]
                        A.2: Example plugins [PLUGINEXAMPLES]

1. Introduction

   This proposal specifies a modular API for integrating name systems with Tor.

1.1. Motivation

   Tor onion service addresses are decentralized and self-authenticated but
   they are not human-memorable (e.g. 3g2upl4pq6kufc4m.onion). This is a source
   of poor usability, since Internet users are familiar with the convenient
   naming of DNS and are not used to addresses being random text.

   In particular, onion addresses are currently composed of 16 random base32
   characters, and they look like this:

                      3g2upl4pq6kufc4m.onion
                      vwakviie2ienjx7t.onion
                      idnxcnkne4qt76tg.onion
                      vwakviie2ienjx6t.onion

   When Proposal 224 get deployed, onion addresses will become even
   bigger: 53 base32 characters. That's:

        llamanymityx4fi3l6x2gyzmtmgxjyqyorj9qsb5r543izcwymle.onion
        lfels7g3rbceenuuqmpsz45z3lswakqf56n5i3bvqhc22d5rrsza.onion
        odmmeotgcfx65l5hn6ejkaruvai222vs7o7tmtllszqk5xbysola.onion
        qw3yvgovok3dsarhqephpu2pkiwzjdk2fgdfwwf3tb3vgzxr5kba.onion

   Over the years Tor users have come up with various ad-hoc ways of handling
   onion addresses. For example people memorize them, or use third-party
   centralized directories, or just google them everytime.

> Additionally, onion service providers have sometimes placed effort
> in creating more readable and memorable names, for example:
>
>		facebookcorewwwi.onion

   We believe that the UX problem of non-human-memorable problems is not
   actually solved with the above ad-hoc solutions and remains a critical
   usability barrier that prevents onion services from being used by a wider
   audience.

1.2. Design overview and rationale

   During the past years there has been lots of research on secure naming and
   various such systems have been proposed (e.g. GNS, Namecoin, etc.).

   Turns out securely naming things is a very hard research problem, and hence
> It turns out that ..
   none of the proposed systems is a clear winner: all of them carry various
   trade-offs. Furthermore, none of the proposed systems has seen widespread use
   so far, which makes it even harder to pick a single project.

   Given the plenitude of options, one approach to decide which system
> Given the plentitude ...
   is best is to make various decent name systems available and let the
   Tor community and the sands of time pick the winner. Also, it might
   be that there is no single winner, and perhaps different specialized
   name system should be used in different situations. We believe that
   by getting secure name systems actually get utilized by real users,
   the whole field will mature and existing systems will get battle-hardened.

   Hence, the contribution of this proposal is a modular Name System API
   (NSA) that allows developers to integrate their own name systems in
   Tor. The interface design is name-system-agnostic, and it's heavily
   based on the pluggable transports API (proposal 180). It should be
   flexible enough to accommodate all sorts of name systems (see [PLUGINEXAMPLES]).

2. System Specification

   A developer that wants to integrate a name system with Tor needs to first
   write a wrapper that understands the Tor Name System API (NS API). Using the
   Name System API, Tor asks the name system to perform name queries, and
   receives the query results. The NS API works using environment variables and
   stdin/stdout communication. It aims to be portable and easy to implement.

2.1. System overview [SYSTEMOVERVIEW]

   Here is an overview of the Tor name system:

   Alice, a Tor user, can activate various name systems by editing her
   torrc file and specifying which tld each name system is responsible
   for. For this section, let's consider a simple fictional name system,
   unicorn, which magically maps domains with the .corn tld to the
   correct onion address. Here it is:

       OnionNamePlugin 0 .corn   /usr/local/bin/unicorn

   After Alice enables the unicorn plugin, she attempts connecting to
   elephantforum.corn. Tor will intercept the SOCKS request, and use the
   executable at /usr/local/bin/unicorn to query the unicorn name system
   for elephantforum.corn. Tor communicates with the unicorn plugin
   using the Tor NS API through which name queries and their results can
   be transported using stdin/stdout.

   If elephantforum.corn corresponds to an onion address in the unicorn
   name system, unicorn should return the onion address to Tor using the
   Tor NS API. Tor must then internally rewrite the elephantforum.corn
   address to the actual onion address, and initiate a connection to it.

2.2. System Illustration

   Here is a diagram illustrating how the Tor Name System API works. The name
   system used in this example is GNS, but there is nothing GNS-specific here
   and GNS could be swapped for any other name system (like hosts files, or
   Namecoin).

   The example below illustrates how a user who types debian.zkey in their Tor
   browser gets redirected to sejnfjrq6szgca7v.onion after Tor consults the GNS
   network.

   Please revisit this illustration after reading the rest of the proposal.

       |                                    $~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~$
       |  1.                                $          4. GNS magic!!              $
       |  User: SOCKS CONNECT to            $ debian.zkey -> sejnfjrq6szgca7v.onion$
       |        http://debian.zkey/         $~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~$~~~~~~~$
       |                                                                   $
 +-----|-----------------------------------------+                         $
 |+----v-----+     2.                 +---------+|       3.                $
 ||Tor       |     debian.zkey        |Tor      ||       debian.zkey     +-$-------+
 ||Networking------------------------->Naming   ------------------------->         |
 ||Submodule |                        |Submodule||  Tor Name System API  |  GNS    |
 ||          <-------------------------         <-------------------------  wrapper|
 ||          | 6.                     |         ||5.                     |         |
 |+----|-----+ sejnfjrq6szgca7v.onion +---------+|sejnfjrq6szgca7v.onion +---------+
 +-----|-----------------------------------------+
       |  7.
       |  Tor: Connect to
       |       http://sejnfjrq6szgca7v.onion/
       v

> Points for the phrase "GNS magic!!"  (A little humor goes a long way in technical
> specification documents)

2.3. System configuration [TORRC]

   As demonstrated in [SYSTEMOVERVIEW], a Tor user who wants to use a name
   system has to edit their configuration file appropriately. Here is the torrc
   line format:

       OnionNamePlugin <priority> <tld> <path>

   where <priority> is a positive integer denoting the priority with which this
> where <priority> is a non-negative integer ...
   name plugin should be consulted. <tld> is a string which restricts the scope
   of this plugin to a particular tld.  Finally, <path> is a filesystem path to
   an executable that speaks the Tor Name System API and can act as an
   intermediary between Tor and the name system.

   For example here is a snippet from a torrc file:
       OnionNamePlugin 0 .hosts      /usr/local/bin/local-hosts-file
       OnionNamePlugin 1 .zkey       /usr/local/bin/gns-tor-wrapper
       OnionNamePlugin 2 .bit        /usr/local/bin/namecoin-tor-wrapper
       OnionNamePlugin 3 .scallion   /usr/local/bin/community-hosts-file

2.3.1. Tor name resolution logic

   When Tor receives a SOCKS request to an address that has a name
   plugin assigned to it, it needs to perform a query for that address
   using that name plugin.

   If there are multiple name plugins that correspond to the requested
   address, Tor queries all relevant plugins sorted by their priority
   value, until one of them returns a successful result. If two plugins
   have the same priority value, Tor MUST abort.

   If all plugins fail to successfuly perform the name resolution, Tor SHOULD
   default to using the exit node for name resolution.
   XXX or not?  because of leaks?

> I suggest that the decision of whether to fallback to "DNS at exit node" may
> be a domain specific choice.  i.e  Yes for .scallion and .hosts, but No for
> .zkey and .bit.  Thus, perhaps an additional, mandatory configuration for each
> OnionNamePlugin configuration line of Yes|No to indicate if fallback should
> occur.
>
> Thus:
>       OnionNamePlugin 0 .hosts    Yes  /usr/local/bin/local-hosts-file yes
>       OnionNamePlugin 1 .zkey     No   /usr/local/bin/gns-tor-wrapper no
>       OnionNamePlugin 2 .bit      No   /usr/local/bin/namecoin-tor-wrapper
>       OnionNamePlugin 3 .scallion Yes  /usr/local/bin/community-hosts-file
>
> Another way to handle this is for the NS API to provide "please use DNS at exit
> node" as a valid response type.  E.g Tor asks the Banana Name API to resolve
> orange.banana and the Banana Name API responses with an NS API status code which
> means "please use DNS at exit node".
>
> This second idea may be the best, as each Name system will have its own idea
> as to whether this is a good idea or not and thus Tor does not need to decide
> and no configuration is required.
>
> Additionally this makes writing the "No Op" name api as trivial as responding
> successfully to INIT and then just always returning the status code for "please
> use DNS at exit node".
>
> Furthermore, it allows a name system to be able to choose this behaviour based
> on context.  i.e the name system knows that the orange.banana domain/service
> is happy to have its name resolved with normal DNS but also knows that 
> pear.banana does not wish to have normal DNS ever used to resolve it.
>
> This distinction is something that Tor will never be able to make by itself.



2.4. Name system initialization [INITPROTOCOL]

   When Tor finds OnionNamePlugin lines in torrc, it launches and initializes
   their respective executables.

   When launching name plugins, Tor sets various environment variables to pass
   data to the name plugin (e.g. NS API version, state directory, etc.). More
   information on the environment variables at [INITENVVARS].

   After a name plugin initializes and parses all needed environment
   variables, it communicates with Tor using its stdin/stdout.

   The first line that a name plugin sends to stdout signifies that it's ready
   to receive name queries. This line looks like this:

      INIT <VERSION> <STATUS_CODE> [<STATUS_MSG>]

   where VERSION is the Tor NS API protocol version that the plugin supports,
> where VERSION is a comma-separated list of the Tor NS API protocol versions ...
   STATUS_CODE is an integer status code, and STATUS_MSG is an optional string
   error message. STATUS_CODE value 0 is reserved for "success", and all other
   integers are error codes.

   See [PROTOEXAMPLE] for an example of this protocol.

2.5. Name resolution using NS API

   Here is how actual name resolution requests are performed in NS API.

2.5.1. Message format

   When Tor receives a SOCKS request to an address with a tld that has a name
   plugin assigned to it, Tor performs an NS API name query for that address.

   Tor does this by printing lines on the name plugin stdout as follows:

      RESOLVE <QUERY_ID> <NAME_STRING>

   where QUERY_ID is a unique integer corresponding to this query, and
   NAME_STRING is the name to be queried.

   When the name plugin completes the name resolution, it prints the following
   line in its stdout:

      RESOLVED <QUERY_ID> <STATUS_CODE> <RESULT>

   where QUERY_ID is the corresponding query ID and STATUS_CODE is an integer
   status code. RESULT is the resolution result (an onion address) or an error
   message if the resolution was not succesful.

   See [PROTOEXAMPLE] for an example of this protocol.

   XXX Should <RESULT> be optional in the case of failure?

> Yes.  In the case of failure the name service will almost certainly not be
> able to provide a valid onion address.  Thus, it cannot provide this field.

2.5.2. RESOLVED status codes

   Name plugins can deliver the following status codes:

   0 -- The name resolution was successful.

   1 -- Name resolution generic failure.

   2 -- Name tld not recognized.

   3 -- Name not registered.

   4 -- Name resolution timeout exceeded.

   XXX add more status codes here as needed

> As mentioned above, I suggest that one of these RESOLVED status codes is
> 
> <some number> -- Name use DNS at the exit node
>
> I suggest that the number is 1.

2.5.3. Further name resolution behavior

   Tor and name plugins MAY cache name resolution results in memory as
   needed. Caching results on disk should be avoided.
> ... Caching results on disk SHOULD be avoided.
> perhaps even MUST be avoided.

   Tor SHOULD abort (or cancel) an ongoing name resolution request, if it takes
   more than NAME_RESOLUTION_TIMEOUT seconds.
   XXX NAME_RESOLUTION_TIMEOUT = ???

> 4 seconds ?

   Tor MUST validate that the resolution result is a valid .onion name.
   XXX should we also accept IPs and regular domain results???

> A very good question.  I suggest that the answer is No for both.
> The NS API is designed to allow people to map various domain systems
> into onion addresses, and thus, only onion addresses should be accepted.
>
> What is happening here is similar to the getent(1) and /etc/nsswitch.conf
> We dont want getent passwd foo to return data about groups, but only those
> about user accounts.

   XXX perhaps we should make sure that results are not names that need
       additional name resolution to avoid endless loops. e.g. imagine
       some sort of loop like this:
        debian.zkey -> debian-bla.zkey -> debian.zkey -> etc.

> Yes, as expressed above.

2.6. Cancelling a name resolution request

   Tor might need to cancel an ongoing name resolution request
   (e.g. because a timeout passed, or the client is not interested in
   that address anymore). In this case, Tor sends the following line to
   the plugin stdout as follows:

     CANCEL <QUERY_ID>

   to which the name plugin, after performing the cancellation, SHOULD
   answer with:

     CANCELED <QUERY_ID>

2.7. Launching name plugins [INITENVVARS]

   As described in [INITPROTOCOL], when Tor launches a name plugin, it sets
   certain environment variables. At a minimum, it sets (in addition to the
   normal environment variables inherited from Tor):

    "TOR_NS_STATE_LOCATION" -- A filesystem directory path where the
       plugin should store state if it wants to.  This directory is not
       required to exist, but the plugin SHOULD be able to create it if
       it doesn't.  The plugin MUST NOT store state elsewhere.
      Example: TOR_NS_STATE_LOCATION=/var/lib/tor/ns_state/

> Where does Tor get this information?  torrc I suppose.  Thus, you need to
> define the namespaces in torrc for setting these things.

    "TOR_NS_PROTO_VERSION" -- To tell the plugin which versions of this
       configuration protocol Tor supports. Future versions will give a
       comma-separated list.  Plugins MUST accept comma-separated lists
       containing any version that they recognize, and MUST work correctly even
       if some of the versions they don't recognize are non-numeric.  Valid
       version characters are non-space, non-comma printing ASCII characters.
      Example: TOR_NS_PROTO_VERSION=1,1a,2,4B

    "TOR_NS_PLUGIN_OPTIONS" -- Specifies configuration options for this
       name plugin as a semicolon-separated list of k=v strings with
       options that are to be passed to the plugin.

       Colons, semicolons, equal signs and backslashes MUST be escaped with a
       backslash.

       If there are no arguments that need to be passed to any of the
       plugins, "TOR_NS_PLUGIN_OPTIONS" MAY be omitted.

       For example consider the following options for the "banana" name plugin:

         TOR_NS_PLUGIN_OPTIONS=timeout=5;url=https://bananacake.com

         Will pass to banana the parameters 'timeout=5' and
         'url=https://bananacake.com'.

       XXX Do we like this option-passing interface? Do we have any lessons
           from our PT experiences?

   XXX Add ControlPort/SocksPort environment variables.

> Yes, this is almost certainly required.  A name service may wish to use Tor's
> network services to perform communication with its external name services.
> OnioNS is a good example of this.  It uses other onion services to do name services :)

   See [PROTOEXAMPLE] for an example of this environment

2.8. Name plugin workflow [NSBEHAVIOR]

   Name plugins follow the following workflow:

     1) Tor sets the required environment values and launches the name plugin
        as a sub-process (fork()/exec()). See [INITENVVARS].

     2) The name plugin checks its environment, and determines the supported NS
        API versions using the env variable TOR_NS_PROTO_VERSION.

        2.1) If there are no compatible versions, the name plugin writes
             an INIT message with a failure status code as in
             [INITPROTOCOL], and then shuts down.

     3) The name plugin parses and handles the rest of the environment values.

        3.1) If the environment variables are malformed, or otherwise
             invalid, the name plugin writes an INIT message with a
             failure status code as in [INITPROTOCOL], and then shuts
             down.

     4) After the name plugin completely initializes, it sends a successful
        INIT message to stdout as in [INITPROTOCOL]. Then it continues
        monitoring its stdin for incoming RESOLVE messages.

     6) When the name plugin receives a RESOLVE message, it performs the name
        resolution and replies with the appropriate RESOLVED message.

     7) Upon being signaled to terminate by the parent process [NSSHUTDOWN], the
        name plugin gracefully shuts down.

2.8.1. Name plugin shutdown [NSSHUTDOWN]

   To ensure clean shutdown of all name plugins when Tor shuts down, the
   following rules apply for name plugins:

   Name plugins MUST handle OS specific mechanisms to gracefully terminate
   (e.g. SIGTERM).

> I think that it would be valuable to be a little more specific here.

   Name plugins SHOULD monitor their stdin and exit gracefully when it is
   closed.

> A very important point.  This may obviate the need for "being more explicit"
> as called for above.

2.9. Further details of stdin/stdout communication

2.9.1. Message Format

   Tor communicates with its name plugins by writing NL-terminated lines to
   stdout.  The line metaformat is

      <Line> ::= <Keyword> <OptArgs> <NL>
      <Keyword> ::= <KeywordChar> | <Keyword> <KeywordChar>
      <KeyWordChar> ::= <any US-ASCII alphanumeric, dash, and underscore>
      <OptArgs> ::= <Args>*
      <Args> ::= <SP> <ArgChar> | <Args> <ArgChar>
      <ArgChar> ::= <any US-ASCII character but NUL or NL>
      <SP> ::= <US-ASCII whitespace symbol (32)>
      <NL> ::= <US-ASCII newline (line feed) character (10)>

   Tor MUST ignore lines with keywords that it doesn't recognize.

3. Discussion

3.1. Using second-level domains instead of tld

   People have suggested that users should try to connect to reddit.zkey.onion
   instead of reddit.zkey. That is, we should always preserve .onion as the
   tld, and only utilize second-level domains for naming.

   The argument for this is that this way users cannot accidentally leak
   addresses to DNS, as the .onion domain is reserved by RFC 7686.

   The counter-argument here is that this might be confusing to users since
   they are not used to the second-level domain being special (e.g. co.uk).
   Also, what happens when someone registers a 16-character name, that matches
   the length of a vanilla onion address?

   We should consider the concerns here and take the right decision.

> As mentioned several times above, my opinion is that the NS API is 
> a mechanism for mapping other name systems into onion addresses.
> If people want to do other "more interesting things" then they are
> welcome to do that with their own overlay networks (e.g p2p, gnunet etc.).
>
> i.e what NS API is doing is allowing them to utilise their name system 
> through Tor if they have set up an onion service within Tor.  If they have 
> just set up their service within their overlay network, then they can just 
> use its own infrastructure to assist people in reaching that service.
>
> I am rather confident that other people will have other opinions,
> and I am happy to be "wrong" about this.

3.2. Name plugins handling all tlds '*'

   In [TORRC], we assigned a single tld to each name plugin.  Should we also
   accept catch-all tlds using '*'? I'm afraid that this way a name system
   could try to resolve even normal domains like reddit.com .

   Perhaps we trust the name plugin itself, but maybe the name system
   network could exploit this? Also, the catch-all tld will probably
   cause some engineering complications in this proposal (as it did for PTs).

> The only existing name system that can "reliably" handle * is DNS.
> Thus, this is the 'no configuration required default' (DNS-at-exit-node).
> However, it may be some peoples desire to override this 'no configuration
> default'.  So, perhaps allow it to be specificed in torrc.


3.3. Deployment strategy

[snip ; no further comment by me].
_______________________________________________
tor-dev mailing list
tor-dev@xxxxxxxxxxxxxxxxxxxx
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev