On Thu, Sep 14, 2023 at 08:22:22PM +1000, Matt Jolly wrote:
I recently inadvertently opened a much larger can of worms than I'd
intended when fixing a bug reported downstream where cURL would,
when configured with certain DNS backends, fail to resolve .onion
addresses.
[...]
Hi Matt,
Thanks for the detailed problem description and references, and I'm
sorry to
hear about the trouble you're getting when trying to fix it.
I've asked in the Tor Specifications issue (inspired by Silvio's
suggestions), and in the cURL issue, but I seem to be getting nowhere
and the impacted users are clamouring for a quick band-aid solution,
which I feel will work out worse for everyone in the long run:
> How can client applications (safely):
>
> 1.discover that they're in a Tor-enabled environment
> 2.resolve onion services only via Tor in that circumstance
> 3.not leak .onion resolution attempts at all
>
> Right now, not making these requests in the first place is the
> safest (and correct) thing to do, however inconvenient it may be.
> Rather than immediately trying to come up with a band-aid approach
> to this problem, a sane mechanism needs to be implemented to:
>
> 1.prevent each application from coming up with their own solution
> 2.prevent inconsistency in .onion resolution (i.e. no "oh it only
> leaks if DO_ONION_RESOLUTION is set")
> 3.provide a standardised mechanism for applications that want to be Tor
> aware to discover that they're in a Tor-enabled environment.
I'm not particularly attached to that last point, but it's worth
discussing.
On a related note:
-is the use of a transparent proxy recommended?
I consider it a valid use case. The only concern I'm aware is about
proxy leaks. I don't see any reason not to recommend transparent Tor
proxies if using a configuration that avoids such leakage.
-is there a sane alternative that involves as minimal configuration
as possible for these users?
Can't think of anything right now besides what was already covered in
length in the threads you mentioned. Instead, I'll focus in the overall
problem.
I'm not sure what the best way forward is here, but I'm hoping that
actual Tor developers might have a useful opinion on the matter, or
at least be able to point me in the right direction.
The long analysis below does not necessarily represents an "official
Tor
position". It may be frustrating that there is no "official" position
right now, maybe because this still need further discussion, and also
because it may be a hard topic.
# Interpreting RFC 7686
Let's start with an hermeneutical exercise, by reading carefully what
RFC 7686 says:
2. Application Software: Applications (including proxies) that
implement the Tor protocol MUST recognize .onion names as special
by either accessing them directly or using a proxy (e.g., SOCKS
[RFC1928]) to do so. Applications that do not implement the Tor
protocol SHOULD generate an error upon the use of .onion and
SHOULD NOT perform a DNS lookup.
Applications MUST support .onion either directly or through a proxy.
The RFC
gives SOCKS as an example, but it could as well have mentioned
transparent
proxies. So applications could easily allow environment variables,
command line
flags, configuration parameters or "environment detection" to indicate
that
they support .onion domains.
I don't see why an upstream shouldn't allow users and operating systems
toggle the behavior regarding RFC 7686. Seems like the best default
approach so far is to generate an error, but nothing is blocking
applications to expose ways users can indicate that their systems acts
as proxies for .onion.
I suggest we take a corollary from this RFC portion: applications
implementing
the Tor protocol are those which either access .onion addresses
directly or use
a proxy that supports Tor. If the application does not support any way
to
properly resolve .onion addresses, then it must make sure to refuse any
attempts to resolve .onion addresses. Nothing is blocking an
application to
operate in two modes: rejecting .onion resolution attempts by default,
except
if it's told to do so.
I agree that "implement the Tor protocol" is rather vague. The "Tor
protocol"
might mean a lot of stuff, but I'm assuming here the _intention_ behind
this
statement is this: in order to build a Tor-enabled application, one has
to
interface with Tor somehow (being a Tor client or using a proxy).
As for curl, it seems to me that upstream wants uniform behavior with
all DNS
backend libraries. And c-ares is rejecting .onion resolution, so for
the sake
of uniformity curl is just rejecting it for all backends.
But the RFC explicitly mentions "Applications" and not "Libraries".
Shall we
assume then that the RFC is referring to curl, but not to c-ares?
Would c-ares maintainers be free do follow whatever they prefer?
Or this is not a concern for c-ares, since it's a library and not an
application? In this case they don't need to comply with the RFC, and
could just rollback this change, so curl maintainers could be more open
to accept toggling the RFC behavior.
We could say that users interact with applications, and these rely on
libraries. By the RFC interpretation above, it's in the application
level that
.onion resolution should be allowed or not. It just don't make any
statement
about what libraries SHOULD or SHOULD NOT do, or even how a system
resolver
should behave.
I don't really think curl maintainers should accept a toggle only if
c-ares is
rolled back, but maybe this could help change their minds.
Alternatively, c-ares could provide some optional initialization
parameter indicating it should actually try to resolve an .onion
address, if that's feasible.
Also note the use of SHOULD in the RFC. As of BCP 14 / RFC 8174:
3. SHOULD This word, or the adjective "RECOMMENDED", mean that there
may exist valid reasons in particular circumstances to ignore a
particular item, but the full implications must be understood and
carefully weighed before choosing a different course.
4. SHOULD NOT This phrase, or the phrase "NOT RECOMMENDED" mean that
there may exist valid reasons in particular circumstances when the
particular behavior is acceptable or even useful, but the full
implications should be understood and the case carefully weighed
before implementing any behavior described with this label.
From the discussion so far, there are enough "valid reasons in
particular
circumstances" when things can be done differently. I haven't followed
RFC
7686's process, but maybe the authors and reviewers had carefully
chosen the
terminology to acknowledge that there might be special cases -- such as
transparent proxies -- where the RFC requirements can be relaxed, given
that
implementors know what they're doing.
Hope this gives enough arguments to argue for implementing a tunable to
account
for the transparent proxy use case in curl and in other software.
# Standardization
What seems more difficult is to standardize a way that any application
from any
operating system could follow some setting, or auto-detect it's behind
an
.onion-enabled proxy.
I don't see an easy way to solve this that would make everyone happy in
the
short term.
But it would be possible to split proposals by timeframe (short, mid
and long
term) and scope (specifications, libraries, applications, operating
systems).
Short timescales and "band-aid" fixes usually happen within libraries
and
applications.
Operating system changes are somehow mid-term.
Tor specs ranges from quick fixes to mid or long term.
IETF standards are long term. Could take years to have a RFC amendment,
and years more to implementors follow the it. RFC 7686 could have
improved wording and be more detailed or nuanced in the possibilities.
Right now we don't have any concrete plans or people engaged in
amending
the RFC, but here and there people are discussing this. Example:
https://gitlab.torproject.org/tpo/onion-services/onion-support/-/issues/85
As for the quick fixes, I don't see nothing beyond advocating patches
implementing flags, environment variables, parameters etc for a RFC
7686-compliant tunable. Unfortunately this have to be done in a
case-by-case
basis, but having RFC 7686 may actually be an advantage when requesting
such
changes. I imagine that not having the RFC would make way harder to
propose
anything related.
# Approaches for transproxy auto-discovery
Now let's talk a bit about long-term, idealized fixes.
What follows is just brainstorming.
It could be considered some method for transproxy auto-discovery, be it
some widespread standard or only followed by a subset of related
applications and operating systems.
An .onion auto-detection feature could be implemented on applications
by trying to detect some special-use domain like "proxy.onion"
in two ways:
1. By supporting a local hostname similar to "localhost" in many
systems. If a
name "proxy.onion" resolves locally (say to 127.0.0.1), then an
application (or library) could assume it's behind a proxy, be it
SOCKS5h or
in a system with Tor transparent proxy.
2. By having an .onion-aware DNS resolver return a valid entry for
"proxy.onion":
* That won't leak the address, but may leak to the DNS that someone
is using a given library, which may be harmless if it's widespread.
* The query could be for a TXT record, which could give more space
for
custom replies, but could be incompatible with the option 1 above.
Upon initialization, an application could do this test to determine
whether it should try to resolve .onion addresses.
The Tor client would have to change as well, in order to support the
special
"proxy.onion" address instead of triggering a malformed .onion address
error.
Thanks for your time,
Cheers,
Matt
Thanks for all your effort and dedication put into this issue :)