[Author Prev][Author Next][Thread Prev][Thread Next][Author Index][Thread Index]
[or-cvs] r16857: {} Rename spec to be useful ; remove redundant copy of matt's s (updater/trunk/specs)
Author: nickm
Date: 2008-09-11 15:58:23 -0400 (Thu, 11 Sep 2008)
New Revision: 16857
Added:
updater/trunk/specs/glider-spec.txt
Removed:
updater/trunk/specs/U1-automatic-updates.txt
updater/trunk/specs/U2-formats.txt
Log:
Rename spec to be useful ; remove redundant copy of matt's spec draft.
Deleted: updater/trunk/specs/U1-automatic-updates.txt
===================================================================
--- updater/trunk/specs/U1-automatic-updates.txt 2008-09-11 19:57:46 UTC (rev 16856)
+++ updater/trunk/specs/U1-automatic-updates.txt 2008-09-11 19:58:23 UTC (rev 16857)
@@ -1,377 +0,0 @@
-
-Filename: xxx-automatic-updates.txt
-Title: Automatic Software Update Protocol
-Version: $Revision$
-Last-Modified: $Date$
-Author: Matt Edman
-Created: 30-July-2008
-Status: Open
-
-
-Scope
-
- This proposal specifies the method by which an automatic update client can
- determine the most recent recommended Tor installation package for the
- user's platform, download the package, and then verify that the package was
- downloaded successfully. While this proposal focuses on only the Tor
- software, the protocol defined is sufficiently extensible such that other
- components of the Tor bundles, like Vidalia, Polipo, and Torbutton, can be
- managed and updated by the automatic update client as well.
-
- The initial target platform for the automatic update framework is Windows,
- given that's the platform used by a majority of our users and that it lacks
- a sane package management system that many Linux distributions already have.
- Our second target platform will be Mac OS X, and so the protocol will be
- designed with this near-future direction in mind.
-
- Other client-side aspects of the automatic update process, such as user
- interaction, the interface presented, and actual package installation
- procedure, are outside the scope of this proposal.
-
-
-Motivation
-
- Tor releases new versions frequently, often with important security,
- anonymity, and stability fixes. Thus, it is important for users to be able
- to promptly recognize when new versions are available and to easily
- download, authenticate, and install updated Tor and Tor-related software
- packages.
-
- Tor's control protocol [2] provides a method by which controllers can
- identify when the user's Tor software is obsolete or otherwise no longer
- recommended. Currently, however, no mechanism exists for clients to
- automatically download and install updated Tor and Tor-related software for
- the user.
-
-
-Design Overview
-
- The core of the automatic update framework is a well-defined file called a
- "recommended-packages" file. The recommended-packages file is accessible via
- HTTP[S] at one or more well-defined URLs. An example recommended-packages
- URL may be:
-
- https://updates.torproject.org/recommended-packages
-
- The recommended-packages document is formatted according to Section 1.2
- below and specifies the most recent recommended installation package
- versions for Tor or Tor-related software, as well as URLs at which the
- packages and their signatures can be downloaded.
-
- An automatic update client process runs on the Tor user's computer and
- periodically retrieves the recommended-packages file according to the method
- described in Section 2.0. As described further in Section 1.2, the
- recommended-packages file is signed and can be verified by the automatic
- update client with one or more public keys included in the client software.
- Since it is signed, the recommended-packages file can be mirrored by
- multiple hosts (e.g., Tor directory authorities), whose URLs are included in
- the automatic update client's configuration.
-
- After retrieving and verifying the recommended-packages file, the automatic
- update client compares the versions of the recommended software packages
- listed in the file with those currently installed on the end-user's
- computer. If one or more of the installed packages is determined to be out
- of date, an updated package and its signature will be downloaded from one of
- the package URLs listed in the recommended-packages file as described in
- Section 2.2.
-
- The automatic update system uses a multilevel signing key scheme for package
- signatures. There are a small number of entities we call "packaging
- authorities" that each have their own signing key. A packaging authority is
- responsible for signing and publishing the recommended-packages file.
- Additionally, each individual packager responsible for producing an
- installation package for one or more platforms has their own signing key.
- Every packager's signing key must be signed by at least one of the packaging
- authority keys.
-
-
-Specification
-
- 1. recommended-packages Specification
-
- In this section we formally specify the format of the published
- recommended-packages file.
-
- 1.1. Document Meta-format
-
- The recommended-packages document follows the lightweight extensible
- information format defined in Tor's directory protocol specification [1]. In
- the interest of self-containment, we have reproduced the relevant portions
- of that format's specification in this Section. (Credits to Nick Mathewson
- for much of the original format definition language.)
-
- The highest level object is a Document, which consists of one or more
- Items. Every Item begins with a KeywordLine, followed by zero or more
- Objects. A KeywordLine begins with a Keyword, optionally followed by
- whitespace and more non-newline characters, and ends with a newline. A
- Keyword is a sequence of one or more characters in the set [A-Za-z0-9-].
- An Object is a block of encoded data in pseudo-Open-PGP-style
- armor. (cf. RFC 2440)
-
- More formally:
-
- Document ::= (Item | NL)+
- Item ::= KeywordLine Object*
- KeywordLine ::= Keyword NL | Keyword WS ArgumentChar+ NL
- Keyword ::= KeywordChar+
- KeywordChar ::= 'A' ... 'Z' | 'a' ... 'z' | '0' ... '9' | '-'
- ArgumentChar ::= any printing ASCII character except NL.
- WS ::= (SP | TAB)+
- Object ::= BeginLine Base-64-encoded-data EndLine
- BeginLine ::= "-----BEGIN " Keyword "-----" NL
- EndLine ::= "-----END " Keyword "-----" NL
-
- The BeginLine and EndLine of an Object must use the same keyword.
-
- In our Document description below, we also tag Items with a multiplicity in
- brackets. Possible tags are:
-
- "At start, exactly once": These items MUST occur in every instance of the
- document type, and MUST appear exactly once, and MUST be the first item in
- their documents.
-
- "Exactly once": These items MUST occur exactly one time in every
- instance of the document type.
-
- "Once or more": These items MUST occur at least once in any instance
- of the document type, and MAY occur more than once.
-
- "At end, exactly once": These items MUST occur in every instance of
- the document type, and MUST appear exactly once, and MUST be the
- last item in their documents.
-
- 1.2. recommended-packages Document Format
-
- When interpreting a recommended-packages Document, software MUST ignore
- any KeywordLine that starts with a keyword it doesn't recognize; future
- implementations MUST NOT require current automatic update clients to
- understand any KeywordLine not currently described.
-
- In lines that take multiple arguments, extra arguments SHOULD be
- accepted and ignored.
-
- The currently defined Items contained in a recommended-packages document
- are:
-
- "recommended-packages-format" SP number NL
-
- [Exactly once]
-
- This Item specifies the version of the recommended-packages format that
- is contained in the subsequent document. The version defined in this
- proposal is version "1". Subsequent iterations of this protocol MUST
- increment this value if they introduce incompatible changes to the
- document format and MAY increment this value if they only introduce
- additional Keywords.
-
- "published" SP YYYY-MM-DD SP HH:MM:SS NL
-
- [Exactly once]
-
- The time, in GMT, when this recommended-packages document was generated.
- Automatic update clients SHOULD ignore Documents over 60 days old.
-
- "tor-stable-win32-version" SP TorVersion NL
-
- [Exactly once]
-
- This keyword specifies the latest recommended release of Tor's "stable"
- branch for the Windows platform that has an installation package
- available. Note that this version does not necessarily correspond to the
- most recently tagged stable Tor version, since that version may not yet
- have an installer package available, or may have known issues on
- Windows.
-
- The TorVersion field is formatted according to Section 2 of Tor's
- version specification [3].
-
- "tor-stable-win32-package" SP Url NL
-
- [Once or more]
-
- This Item specifies the location from which the most recent
- recommended Windows installation package for Tor's stable branch can be
- downloaded.
-
- When this Item appears multiple times within the Document, automatic
- update clients SHOULD select randomly from the available package
- mirrors.
-
- "tor-dev-win32-version" SP TorVersion NL
-
- [Exactly once]
-
- This Item specifies the latest recommended release of Tor's
- "development" branch for the Windows platform that has an installation
- package available. The same caveats from the description of
- "tor-stable-win32-version" also apply to this keyword.
-
- The TorVersion field is formatted according to Section 2 of Tor's
- version specification [3].
-
- "tor-dev-win32-package" SP Url NL
-
- [Once or more]
-
- This Item specifies the location from which the most recent recommended
- Windows installation package and its signature for Tor's development
- branch can be downloaded.
-
- When this Keyword appears multiple times within the Document, automatic
- update clients SHOULD select randomly from the available package
- mirrors.
-
- "signature" NL SIGNATURE NL
-
- [At end, exactly once]
-
- The "SIGNATURE" Object contains a PGP signature (using a packaging
- authority signing key) of the entire document, taken from the beginning
- of the "recommended-packages-format" keyword, through the newline after
- the "signature" Keyword.
-
-
- 2. Automatic Update Client Behavior
-
- The client-side component of the automatic update framework is an
- application that runs on the end-user's machine. It is responsible for
- fetching and verifying a recommended-packages document, as well as
- downloading, verifying, and subsequently installing any necessary updated
- software packages.
-
- 2.1. Download and verify a recommended-packages document
-
- The first step in the automatic update process is for the client to download
- a copy of the recommended-packages file. The automatic update client
- contains a (hardcoded and/or user-configurable) list of URLs from which it
- will attempt to retrieve a recommended-packages file.
-
- Connections to each of the recommended-packages URLs SHOULD be attempted in
- the following order:
-
- 1) HTTPS over Tor
- 2) HTTP over Tor
- 3) Direct HTTPS
- 4) Direct HTTP
-
- If the client fails to retrieve a recommended-packages document via any of
- the above connection methods from any of the configured URLs, the client
- SHOULD retry its download attempts following an exponential back-off
- algorithm. After the first failed attempt, the client SHOULD delay one hour
- before attempting again, up to a maximum of 24 hours delay between retry
- attempts.
-
- After successfully downloading a recommended-packages file, the automatic
- update client will verify the signature using one of the public keys
- distributed with the client software. If more than one recommended-packages
- file is downloaded and verified, the file with the most recent "published"
- date that is verified will be retained and the rest discarded.
-
- 2.2. Download and verify the updated packages
-
- The automatic update client next compares the latest recommended package
- version from the recommended-packages document with the currently installed
- Tor version. If the user currently has installed a Tor version from Tor's
- "development" branch, then the version specified in "tor-dev-*-version" Item
- is used for comparison. Similarly, if the user currently has installed a Tor
- version from Tor's "stable" branch, then the version specified in the
- "tor-stable-*version" Item is used for comparison. Version comparisons are
- done according to Tor's version specification [3].
-
- If the automatic update client determines an installation package newer than
- the user's currently installed version is available, it will attempt to
- download a package appropriate for the user's platform and Tor branch from a
- URL specified by a "tor-[branch]-[platform]-package" Item. If more than one
- mirror for the selected package is available, a mirror will be chosen at
- random from all those available.
-
- The automatic update client must also download a ".asc" signature file for
- the retrieved package. The URL for the package signature is the same as that
- for the package itself, except with the extension ".asc" appended to the
- package URL.
-
- Connections to download the updated package and its signature SHOULD be
- attempted in the same order described in Section 2.1.
-
- After completing the steps described in Sections 2.1 and 2.2, the automatic
- update client will have downloaded and verified a copy of the latest Tor
- installation package. It can then take whatever subsequent platform-specific
- steps are necessary to install the downloaded software updates.
-
- 2.3. Periodic checking for updates
-
- The automatic update client SHOULD maintain a local state file in which it
- records (at a minimum) the timestamp at which it last retrieved a
- recommended-packages file and the timestamp at which the client last
- successfully downloaded and installed a software update.
-
- Automatic update clients SHOULD check for an updated recommended-packages
- document at most once per day but at least once every 30 days.
-
-
- 3. Future Extensions
-
- There are several possible areas for future extensions of this framework.
- The extensions below are merely suggestions and should be the subject of
- their own proposal before being implemented.
-
- 3.1. Additional Software Updates
-
- There are several software packages often included in Tor bundles besides
- Tor, such as Vidalia, Privoxy or Polipo, and Torbutton. The versions and
- download locations of updated installation packages for these bundle
- components can be easily added to the recommended-packages document
- specification above.
-
- 3.2. Including ChangeLog Information
-
- It may be useful for automatic update clients to be able to display for
- users a summary of the changes made in the latest Tor or Tor-related
- software release, before the user chooses to install the update. In the
- future, we can add keywords to the specification in Section 1.2 that specify
- the location of a ChangeLog file for the latest recommended package
- versions. It may also be desirable to allow localized ChangeLog information,
- so that the automatic update client can fetch release notes in the
- end-user's preferred language.
-
- 3.3. Weighted Package Mirror Selection
-
- We defined in Section 1.2 a method by which automatic update clients can
- select from multiple available package mirrors. We may want to add a Weight
- argument to the "*-package" Items that allows the recommended-packages file
- to suggest to clients the probability with which a package mirror should be
- chosen. This will allow clients to more appropriately distribute package
- downloads across available mirrors proportional to their approximate
- bandwidth.
-
-
-Implementation
-
- Implementation of this proposal will consist of two separate components.
-
- The first component is a small "au-publish" tool that takes as input a
- configuration file specifying the information described in Section 1.2 and a
- private key. The tool is run by a "packaging authority" (someone responsible
- for publishing updated installation packages), who will be prompted to enter
- the passphrase for the private key used to sign the recommended-packages
- document. The output of the tool is a document formatted according to
- Section 1.2, with a signature appended at the end. The resulting document
- can then be published to any of the update mirrors.
-
- The second component is an "au-client" tool that is run on the end-user's
- machine. It periodically checks for updated installation packages according
- to Section 2 and fetches the packages if necessary. The public keys used
- to sign the recommended-packages file and any of the published packages are
- included in the "au-client" tool.
-
-
-References
-
- [1] Tor directory protocol (version 3),
- https://tor-svn.freehaven.net/svn/tor/trunk/doc/spec/dir-spec.txt
-
- [2] Tor control protocol (version 2),
- https://tor-svn.freehaven.net/svn/tor/trunk/doc/spec/control-spec.txt
-
- [3] Tor version specification,
- https://tor-svn.freehaven.net/svn/tor/trunk/doc/spec/version-spec.txt
Deleted: updater/trunk/specs/U2-formats.txt
===================================================================
--- updater/trunk/specs/U2-formats.txt 2008-09-11 19:57:46 UTC (rev 16856)
+++ updater/trunk/specs/U2-formats.txt 2008-09-11 19:58:23 UTC (rev 16857)
@@ -1,627 +0,0 @@
-
-0. Preliminaries
-
-0.0. Scope
-
- This document describes a system for distributing Tor bundle updates.
-
-0.1. Proposed code name
-
- Since "auto-update" is so generic, I've been thinking about going with
- "glider", based on the sugar glider you get when you search for "handy
- pocket creature". I haven't yet done a search to find out whether
- somebody else is using the name, so we shouldn't get too attached to it
- before we see if it's taken.
-
-0.2. Goals
-
- Once Tor was a single executable that you could just run. Then it
- required Privoxy. Now, thanks to the Tor Browser Bundle and related
- projects, a full installation can contain Tor, Privoxy, Torbutton,
- Firefox, and more.
-
- We need to keep this software updated. When we make security fixes,
- quick uptake helps narrow the window in which attackers can exploit
- them.
-
- We need updates to be easy. Each additional step a user must take to
- get updated means that more users will stay with older insecure
- versions.
-
- We need updates to be secure. We're supposed to be good at crypto;
- let's act like it. There is no good reason in this day and age to
- subject users to rollback attacks or unsigned packages or whatever.
-
- We need administration to be simple. Tor doesn't have a release
- engineering team, so we can't add too many hard steps to putting out
- a new release.
-
- The system should be easy to implement; we may need to do multiple
- implementations on the client side at least.
-
-0.2.1. Goals for package formats and PKIs
-
- It should be possible to mirror a repository using only rsync and
- cron.
-
- Separate keys should be used for different people and different
- roles.
-
- Only a minimal set of keys should have to be kept online to keep
- the system running.
-
- The system should handle any single computer or system or person
- being unavailable.
-
- The formats and protocols should be pretty future-proof.
-
-0.3. Non-goals
-
- This is not a general-purpose package manager like yum or apt: it
- assumes that users will want to have one or more of a set of
- "bundles", not an arbitrary selection of packages dependant on one
- another. (Rationale: these systems do what they do pretty well.)
-
- This is also not a general-purpose package format. It assumes the
- existence of an external package format that can handle install,
- update, remove, and version query. (Rationale:
-
-1. System overview
-
- The basic unit of updatability is a "bundle". A bundle is a set of
- software components, or "packages", plus some rules about installing
- them. Example bundles could be "Tor Browser, stable series" or
- "Basic Tor, development series".
-
- When Glider has responsibility for keeping a bundle up to date, we
- say that a user has "subscribed" to that bundle.
-
- Conceptually, there are four parts to keeping a bundle up to date:
-
- Polling:
- - Periodically, Glider asks a mirror whether there is a newer
- version of some bundle that a user has subscribed to. If so,
- Glider determines what's in the bundle.
-
- Fetching:
- - If the bundle contains packages that Glider hasn't installed
- or hasn't cached, it needs to download them from a mirror.
- This can happen over any protocol; v1 should support at least
- http and https-over-Tor. V1 should also support resuming
- partial downloads, since many users have unreliable
- connections.
-
- Later versions could support Bittorrent, or whatever.
-
- Validation:
- - Throughout the process, Glider must ensure that all the
- bundles are signed correctly, all the packages are signed
- correctly, and everything is up-to-date.
-
- We want to specify this so that users can't be tricked about
- the contents of a bundle, can't install a malicious package,
- and can't be fooled into believing that an old bundle is
- actually the latest.
-
- Installation:
- - Now Glider has a set of packages to install. The format of
- these packages will be platform-dependent: they could be pkg
- files on OSX, MSI files on Win32, RPMs or DEBs on Linux, and
- so on. Glider should query the user for permission to start,
- then install the packages.
-
-1.1. The repository
-
- Each Glider instance knows about one or more "repositories". A
- repository is a filesystem somewhere that contains the packages in a
- set of bundles, and some associated metadata. A repository must
- exist at one or more canonical hosts, and may have a number of full
- or partial mirrors.
-
- In v1, each Glider instance will know about only one repository.
-
-1.2. The PKI
-
- The trust root for the whole system is, necessarily, whatever users
- download when they first download a copy of Glider. We need to make
- sure that the first download happens from a site we trust, using
- HTTPS.
-
- Glider ships with root keys, which in turn are used to verify the
- keys for all the other roles. There are a few root keys, operated by
- trusted admins for the system. If root keys ever need to be changed,
- we can just ship an update of Glider: it's supposed to be
- self-updating anyway.
-
- The root keys are only used to sign a 'key list' of all the other
- keys and their roles. A key list is valid if it has been signed by a
- threshold of root keys.
-
- Each package is signed with the key of its authorized builder. For
- example, one volunteer may be authorized to build the mac versions of
- several packages, and another may be authorized to build the windows
- version of just one.
-
- Each bundle is signed with the key of its maintainer. It's assumed
- that the bundle maintainer might be the package maintainer for some
- but not all of the packages.
-
- The list of mirrors is also signed. If the mirror list is
- automatically updated, this key must be kept online; otherwise, it
- can be offline.
-
- To prevent an adversary from replaying an out-of-date signed
- document, an automated process periodically signs a timestamped
- statement containing the hashes of the mirror list, the latest
- bundles, and the key list, using yet another special-purpose key.
- This key must be kept online.
-
-1.3. Threat Model And Analysis
-
- We assume an adversary who can operate compromised mirrors, and who
- can possibly compromise the main repository. At worst, such an
- adversary can DOS users in a way that they can detect.
-
- We're assuming for the moment an OSX/Win32-like execution model,
- where all packages will run equal privilege, but occasionally
- installation will require higher privilege. This means that once a
- hostile package is installed, it can basically do whatever it
- wants. As rootkit writers demonstrate, compromise is really
- tenuous: any attacker who can induce a user to install a hostile
- piece of code has, in effect, permanently compromised that user
- until they reinstall.
-
- Thus, if an adversary compromises enough keys to sign a compromised
- package, or tricks a packager into signing a compromised package,
- and manages to get that package into a signed bundle, the best we
- can do is to limit the number of users who are affected. We do
- this by compartmentalizing signing keys so that only the package
- and bundle in question are at risk.
-
- (If we had replicated build processes and a bit-by-bit reliable
- build process, we could have multiple packagers test that a binary
- was built properly, and multiply sign it. This would be effective
- against an adversary compromising a single packaging key, but not
- against one compromising a source repository.)
-
-2. The repository layout
-
- The filesystem layout in the repository is used for two purposes:
- - To give mirrors an easy way to mirror only some of the repository.
- - To specify which parts of the repository a given key has the
- authority to sign.
-
- The following files exist in all repositories and mirrors:
-
- /meta/keys.txt
-
- Signed by the root keys; indicates keys and roles.
- [???? I'm using the txt extension here. Is that smart?]
-
- /meta/mirrors.txt
-
- Signed by the mirror key; indicates which parts of the
- repository are mirrored at what mirrors.
-
- /meta/timestamp.txt
-
- Signed by the timestamp key; indicates hashes and timestamps
- for the latest versions of keys.txt and mirrors.txt. Also
- indicates the latest version of each bundle for each os/arch.
-
- This is the only file that needs to be downloaded for polling.
-
- /bundleinfo/bundlename/os-arch/bundlename-os-arch-bundleversion.txt
-
- Signed by the appropriate bundle key. Describes what
- packages make up a bundle, and what order to install,
- uninstall, and upgrade them in.
-
- /pkginfo/packagename/os-arch/version/packagename-os-arch-packageversion.txt
-
- Signed by the appropriate package key. Tells the name of the
- file that makes up a package, its hash, and what procedure
- is used to install it.
-
- /packages/packagename/os-arch/version/(some filename)
-
- The actual package file. Its naming convention will depend
- on the underlying packaging system.
-
-3. Document formats
-
-3.1. Metaformat
-
- All documents use Rivest's SEXP meta-format as documented at
- http://people.csail.mit.edu/rivest/sexp.html
- with the restriction that no "display hint" fields are to be used,
- and the base64 transit encoding isn't used either.
-
- (We use SEXP because it's really easy to parse, really portable,
- and unlike most other tagged data formats, has a
- trivially-specified canonical format suitable for hashing.)
-
- In descriptions of syntax below, we use regex-style qualifiers, so
- that in
- (sofa slipcover? occupant* leg+)
- the sofa will have an optional slipcover, zero or more occupants,
- and one or more legs. This pattern matches (sofa leg) and (sofa
- slipcover occupant occupant leg leg leg leg) but not (sofa leg
- slipcover).
-
- We also use a braces notation to indicate elements that can occur
- in any order. For example,
- (bread {flour+ eggs? yeast})
- matches a list starting with "bread", and then containing one or
- more of flours, zero or one occurrences of eggs, and one
- occurrence of yeast, in any order. This pattern matches (bread eggs
- yeast flour) but not (bread yeast) or (bread flour eggs yeast
- macadamias).
-
-3.2. File formats: general principles
-
- We use tagged lists (lists whose first element is a string) to
- indicate typed objects. Tags are generally lower-case, with
- hyphens used for separation. Think Lispy.
-
- We use attrlists [lists of (key value) lists] to indicate a
- multimap from keys to values. Clients MUST accept unrecognized
- keys in these attrlists. The syntax for an attrlist with two
- recognized and required keys is typically given as ({(key1 val1)
- (key2 val2) (ATTR VAL)*}), indicating that the keys can occur in
- any order, intermixed with other attributes.
-
- Timestamp files will be downloaded very frequently; all other files
- will be much smaller in size than package files. Thus,
- size-optimization for timestamp files makes sense and most other
- other space optimizations don't.
-
- Versions are represented as lists of the form (v I1 I2 I3 I4 ...)
- where each item is a number or alphanumeric version component. For
- example, the version "0.2.1.5-alpha" is represented as (v 0 2 1 5
- alpha).
-
- All signed files are of the format:
-
- (signed
- X
- (signature ({(keyid K) (method M) (ATTR VAL)*}) SIG)+
- )
-
- where: X is a list whose first element describes the signed object.
- K is the identifier of a key signing the document
- M is the method to be used to make the signature
- (ATTR VAL) is an arbitrary list whose first element is a
- string.
- SIG is a signature of the canonical encoding of X using the
- identified key.
-
- We define two signing methods at present:
- sha256-oaep : A signature of the SHA256 hash of the canonical
- encoding of X, using OAEP+ padding. [XXXX say more about mgf]
-
- All times are given as strings of the format "YYYY-MM-DD HH:MM:SS",
- in UTC.
-
- All keys are of the format:
- (pubkey ({(type TYPE) (ATTR VAL)*}) KEYVAL)
- where TYPE is a string describing the type of the key and how it's
- used to sign documents. The type determines the interpretation of
- KEYVAL.
-
- The ID of a key is the type field concatenated with the SHA-256
- hash of the canonical encoding of the KEYVAL field.
-
- We define one keytype at present: 'rsa'. The KEYVAL in this case
- is a 2-element list of (e n), with both values given in big-endian
- binary format. [This makes keys 45-60% more compact than using
- decimal integers.]
-
- All RSA keys must be at least 2048 bits long.
-
-
- Every role in the system is associated with a key. Replacing
- anything but a root key is supposed to be relatively easy.
-
- Root-keys sign other keys, and certify them as belonging to roles.
- Clients are configured to know the root keys.
-
- Bundle keys certify the contents of a bundle.
-
- Package keys certify packages for a given program or set of
- programs.
-
- Mirror keys certify a list of mirrors. We expect this to be an
- automated process.
-
- Timestamp keys certify that given versions of other metadata
- documents are up-to-date. They are the only keys that absolutely
- need to be kept online. (If they are not, timestamps won't be
- generated.)
-
-3.3. File formats: key list
-
- The key list file is signed by multiple root keys. It indicates
- which keys are authorized to sign which parts of the repository.
-
- (keylist
- (ts TIME)
- (keys
- ((key ({(roles (ROLE PATH)+) (ATTR VAL)*}) KEY)*)
- ...
- )
-
- The "ts" line describes when the keys file was updated. Clients
- MUST NOT replace a file with an older one, and SHOULD NOT accept a
- file too far in the future.
-
- A ROLE is one of "timestamp" "mirrors" "bundle" or "package".
-
- PATH is a path relative to the top of the directory hierarchy. It
- may contain "*" elements to indicate "any file", and may end with a
- "/**" element to indicate all files under a given point.
-
-3.4. File formats: mirror list
-
- The mirror list is signed by a mirror key. It indicates which
- mirrors are active and believed to be mirroring which parts of the
- repository.
-
- (mirrorlist
- (ts TIME)
- (mirrors
- ( (mirror ({(name N) (urlbase U) (contents PATH+) (weight W)
- (official)? (ATTR VAL)})) * )
- ...
- )
-
- Every mirror is a copy of some or all of the directory hierarchy
- containing at least the /meta, /bundles/, and /pkginfo directories.
-
- N is a descriptive name for the mirror; U is the URL of the mirror's
- base (i.e., the parent of the "meta" directory); and the PATH
- elements are the components describing how much of the packages
- directory is mirrored. Their format is as in the keylist file.
-
- W is an integer used to weight mirrors when picking at random;
- mirrors with more bandwidth should have higher weigths. The
- "official" element should only be present if the mirror is (one of
- the) official repositories operated by the Tor Project.
-
-3.5. File formats: timestamp files
-
- The timestamp file is signed by a timestamp key. It indicates the
- latest versions of other files, and contains a regularly updated
- timestamp to prevent rollback attacks.
-
- (ts
- ({(at TIME)
- (m TIME MIRRORLISTHASH)
- (k TIME KEYLISTHASH)
- (b NAME VERSION TIME PATH HASH)*})
- )
-
- TIME is when the timestamp was signed. MIRRORLISTHASH is the digest
- of the mirror-list file; KEYLISTHASH is the digest of the key list
- file; and the 'b' entries are a list of the latest version of each
- bundles and their locations and hashes.
-
-3.6. File formats: bundle files
-
- (bundle
- (at TIME)
- (os OS)
- [(arch ARCH)]
- (version V)
- (packages
- (NAME VERSION PATH HASH ({(order INST UPDATE REMOVE)
- (optional)?
- (gloss LANG TEXT)*
- (longloss LANG TEXT)*
- (ATTR VAL)*})? )* )
- )
-
- Most elements are self-explanatory; the INST, UPDATE, and REMOVE
- elements of the order element are numbers defining the order in
- which the packages are installed, updated, and removed respectively.
- The "optional" element is present if the package is optional.
- "Gloss" is a short utf-8 human-readable string explaining what the
- package provides for the bundle; "longloss" is a longer such
- utf-8 string.
-
- (Note that the gloss strings are meant, not to describe the package,
- but to describe what the package provides for the bundle. For
- example, "The Anonymous Email Bundle needs the Python Runtime to run
- Mixminion.")
-
- Multiple gloss strings are allowed; each should have a different
- language. The UI should display the must appropriate language to the
- user.
-
-3.7. File formats: package files
-
- (package
- ({(name NAME)
- (version VERSION)
- (format FMT ((ATTR VAL)*)? )
- (path PATH)
- (ts TIME)
- (digest HASH)
- (shortdesc LANG TEXT)*
- (longdesc LANG TEXT)*
- (ATTR VAL)* })
- )
-
- Most elements are self-explanatory. The "FMT" element describes the
- file format of the package, which should give enough information
- about how to install it.
-
- No two package files in the same repository should have the same
- name and version. If a package needs to be changed, the version
- MUST be incremented.
-
- Descriptions are tagged with languages in the same way as glosses.
-
-4. Detailed Workflows
-
-4.1. The client application
-
- Periodically, the client updater fetches a timestamp file from a
- mirror. If the timestamp in the file is up-to-date, the client
- first checks to see whether the keys file listed is one that the
- client has. If not, the client fetches it, makes sure the hash of
- the keys file matches the hash in the timestamp file, makes sure its
- date is more recent than any keys file they have but not too far in
- the future, and that it is signed by enough root keys that the
- client recognizes.
-
- [If the timestamp file is not up-to-date, the client tries a
- few mirrors until it finds one with a good timestamp.]
-
- [If the keys file from a mirror does not match the timestamp
- file, the client tries a new mirror for both.]
-
- [If the keys file is not signed by enough root keys, the client
- warns the user and tries another mirror for both the timestamp
- file and the keys file.]
-
- Once the client has an up-to-date keys file, the client checks the
- signature on the timestamp file. Assuming it checks out, the client
- refreshes the mirror list as needed, and refreshes any bundle files
- to which the user is subscribed if the client does not have
- the latest version of those files. The client checks signatures on
- these files, and fetches package metadata for any packages listed in
- the bundle file that the client does not have, checks signatures on
- these, and fetches binaries for packages that might need to be
- installed or updated. As the packages arrive, clients check their
- hashes.
-
- Once the client has gotten enough packages, it informs the user that
- new packages have arrived, and asks them if they want to update.
-
- Clients SHOULD cache at least the latest versions they have received
- of all files.
-
-4.1.1. Download preferences
-
- Users should be able to specify that packages must be only
- downloaded over Tor, or must only be downloaded over encrypted
- protocols, or both. Users should also be able to force
-
-4.2. Mirrors
-
- Periodically, mirrors do an rsync or equivalent to fetch the latest
- version of whatever parts of the repository have changed since the
- version they currently hold. Mirrors SHOULD replace older versions
- of the repository idempotently, so that clients are less likely to
- see inconsistent state. Mirrors SHOULD validate the information
- they receive, and not serve partial or inconsistent files.
-
-4.3. Workflow: Packagers
-
- When a new binary package is done, the person making the package
- runs a tool to generate and sign a package file, and sends both the
- package and the package file to a repository admin. Typically, the
- base package file will be generated by inserting a version into a
- template.
-
- Packages MAY have as part of their build process a script to
- generate the appropriately versioned package file. This script
- should at a minimum demand a build version, or use a timestamp in
- place of a build version, to prevent two packages with the same
- version from being created.
-
-4.4. Workflow: bundlers
-
- When the packages in a bundle are done, the bundler runs a tool on
- the package files to generate and sign a bundle file. Typically,
- this tool uses a template bundle file.
-
-4.5. Workflow: repository administrators
-
- Repository administrators use a tool to validate signed files into the
- repository. The repository should not be altered manually.
-
- This tool acts as follows:
- - Package files may be added, but never replaced.
- - Bundle files may be added, but never replaced.
- - No file may be added unless it is syntactically valid and
- signed by a key in the keys file authorized to sign files of
- this type in this file's location location.
-
- - A package file may not be added unless all of its binary
- packages match their hashes.
-
- - A bundle file may not be added unless all of its package files
- are present and match their hashes.
-
- - When adding a new keylist, bundle, or mirrors list, the
- timestamp file must be regenerated immediately.
-
-5. Parameter setting and corner cases.
-
-5.1. Timing:
-
- The timestamp file SHOULD be regenerated every 15 minutes. Mirrors
- SHOULD attempt to update every hour. Clients SHOULD accept a
- timestamp file up to 6 hours old.
-
-5.2. Format versioning and forward-compatibility:
-
- All of the above formats include the ability to add more
- attribute-value fields for backwards-compatible format changes. If
- we need to make a backwards incompatible format change, we create a
- new filename for the new format.
-
-5.3. Key management and migration:
-
- Root keys should be kept offline. All keys except timestamp and
- mirror keys should be stored encrypted.
-
- All the formats above allow for multiple keys to sign a single
- document. To replace a compromised root key, it suffices to sign
- keylist documents with both the compromised key and its replacement
- until all clients have updated to a new version of the autoupdater.
-
- To replace another key, it suffices to authorize the new key in the
- keylist. Note that a new package or bundle key must re-sign and
- issue new versions of all packages or bundles it has generated.
-
-
-
-F. Future directions and open questions
-
-F.1. Package decomposition
-
- It would be neat to decouple existing packages. Right now, we'd
- never want a windows user to have to fetch an openssl dll and Tor
- separately. But if they're using an auto-update tool, it'd be
- pretty keen to have them not need to fetch a new openssl every time
- Tor has a bugfix.
-
-F.2. Caching at Tor servers.
-
- See Tor Proposal number 127.
-
-F.3. Support for more download methods
-
- Ozymandns, chunked downloads, and bittorrent would all be neat
- ideas.
-
-
-R. Ideas I'm rejecting for the moment
-
-R.1. Considering recommended versions from Tor consensus directory documents
-
- This requires a working Tor to update Tor; that's not necessarily a
- great idea.
-
-R.2. Integration with existing GPG signatures
-
- The OpenPGP signature and key format is so complicated that you'd
- have to be mad to touch it.
-
-
-
-
-
Copied: updater/trunk/specs/glider-spec.txt (from rev 16854, updater/trunk/specs/U2-formats.txt)
===================================================================
--- updater/trunk/specs/glider-spec.txt (rev 0)
+++ updater/trunk/specs/glider-spec.txt 2008-09-11 19:58:23 UTC (rev 16857)
@@ -0,0 +1,627 @@
+
+0. Preliminaries
+
+0.0. Scope
+
+ This document describes a system for distributing Tor bundle updates.
+
+0.1. Proposed code name
+
+ Since "auto-update" is so generic, I've been thinking about going with
+ "glider", based on the sugar glider you get when you search for "handy
+ pocket creature". I haven't yet done a search to find out whether
+ somebody else is using the name, so we shouldn't get too attached to it
+ before we see if it's taken.
+
+0.2. Goals
+
+ Once Tor was a single executable that you could just run. Then it
+ required Privoxy. Now, thanks to the Tor Browser Bundle and related
+ projects, a full installation can contain Tor, Privoxy, Torbutton,
+ Firefox, and more.
+
+ We need to keep this software updated. When we make security fixes,
+ quick uptake helps narrow the window in which attackers can exploit
+ them.
+
+ We need updates to be easy. Each additional step a user must take to
+ get updated means that more users will stay with older insecure
+ versions.
+
+ We need updates to be secure. We're supposed to be good at crypto;
+ let's act like it. There is no good reason in this day and age to
+ subject users to rollback attacks or unsigned packages or whatever.
+
+ We need administration to be simple. Tor doesn't have a release
+ engineering team, so we can't add too many hard steps to putting out
+ a new release.
+
+ The system should be easy to implement; we may need to do multiple
+ implementations on the client side at least.
+
+0.2.1. Goals for package formats and PKIs
+
+ It should be possible to mirror a repository using only rsync and
+ cron.
+
+ Separate keys should be used for different people and different
+ roles.
+
+ Only a minimal set of keys should have to be kept online to keep
+ the system running.
+
+ The system should handle any single computer or system or person
+ being unavailable.
+
+ The formats and protocols should be pretty future-proof.
+
+0.3. Non-goals
+
+ This is not a general-purpose package manager like yum or apt: it
+ assumes that users will want to have one or more of a set of
+ "bundles", not an arbitrary selection of packages dependant on one
+ another. (Rationale: these systems do what they do pretty well.)
+
+ This is also not a general-purpose package format. It assumes the
+ existence of an external package format that can handle install,
+ update, remove, and version query. (Rationale:
+
+1. System overview
+
+ The basic unit of updatability is a "bundle". A bundle is a set of
+ software components, or "packages", plus some rules about installing
+ them. Example bundles could be "Tor Browser, stable series" or
+ "Basic Tor, development series".
+
+ When Glider has responsibility for keeping a bundle up to date, we
+ say that a user has "subscribed" to that bundle.
+
+ Conceptually, there are four parts to keeping a bundle up to date:
+
+ Polling:
+ - Periodically, Glider asks a mirror whether there is a newer
+ version of some bundle that a user has subscribed to. If so,
+ Glider determines what's in the bundle.
+
+ Fetching:
+ - If the bundle contains packages that Glider hasn't installed
+ or hasn't cached, it needs to download them from a mirror.
+ This can happen over any protocol; v1 should support at least
+ http and https-over-Tor. V1 should also support resuming
+ partial downloads, since many users have unreliable
+ connections.
+
+ Later versions could support Bittorrent, or whatever.
+
+ Validation:
+ - Throughout the process, Glider must ensure that all the
+ bundles are signed correctly, all the packages are signed
+ correctly, and everything is up-to-date.
+
+ We want to specify this so that users can't be tricked about
+ the contents of a bundle, can't install a malicious package,
+ and can't be fooled into believing that an old bundle is
+ actually the latest.
+
+ Installation:
+ - Now Glider has a set of packages to install. The format of
+ these packages will be platform-dependent: they could be pkg
+ files on OSX, MSI files on Win32, RPMs or DEBs on Linux, and
+ so on. Glider should query the user for permission to start,
+ then install the packages.
+
+1.1. The repository
+
+ Each Glider instance knows about one or more "repositories". A
+ repository is a filesystem somewhere that contains the packages in a
+ set of bundles, and some associated metadata. A repository must
+ exist at one or more canonical hosts, and may have a number of full
+ or partial mirrors.
+
+ In v1, each Glider instance will know about only one repository.
+
+1.2. The PKI
+
+ The trust root for the whole system is, necessarily, whatever users
+ download when they first download a copy of Glider. We need to make
+ sure that the first download happens from a site we trust, using
+ HTTPS.
+
+ Glider ships with root keys, which in turn are used to verify the
+ keys for all the other roles. There are a few root keys, operated by
+ trusted admins for the system. If root keys ever need to be changed,
+ we can just ship an update of Glider: it's supposed to be
+ self-updating anyway.
+
+ The root keys are only used to sign a 'key list' of all the other
+ keys and their roles. A key list is valid if it has been signed by a
+ threshold of root keys.
+
+ Each package is signed with the key of its authorized builder. For
+ example, one volunteer may be authorized to build the mac versions of
+ several packages, and another may be authorized to build the windows
+ version of just one.
+
+ Each bundle is signed with the key of its maintainer. It's assumed
+ that the bundle maintainer might be the package maintainer for some
+ but not all of the packages.
+
+ The list of mirrors is also signed. If the mirror list is
+ automatically updated, this key must be kept online; otherwise, it
+ can be offline.
+
+ To prevent an adversary from replaying an out-of-date signed
+ document, an automated process periodically signs a timestamped
+ statement containing the hashes of the mirror list, the latest
+ bundles, and the key list, using yet another special-purpose key.
+ This key must be kept online.
+
+1.3. Threat Model And Analysis
+
+ We assume an adversary who can operate compromised mirrors, and who
+ can possibly compromise the main repository. At worst, such an
+ adversary can DOS users in a way that they can detect.
+
+ We're assuming for the moment an OSX/Win32-like execution model,
+ where all packages will run equal privilege, but occasionally
+ installation will require higher privilege. This means that once a
+ hostile package is installed, it can basically do whatever it
+ wants. As rootkit writers demonstrate, compromise is really
+ tenuous: any attacker who can induce a user to install a hostile
+ piece of code has, in effect, permanently compromised that user
+ until they reinstall.
+
+ Thus, if an adversary compromises enough keys to sign a compromised
+ package, or tricks a packager into signing a compromised package,
+ and manages to get that package into a signed bundle, the best we
+ can do is to limit the number of users who are affected. We do
+ this by compartmentalizing signing keys so that only the package
+ and bundle in question are at risk.
+
+ (If we had replicated build processes and a bit-by-bit reliable
+ build process, we could have multiple packagers test that a binary
+ was built properly, and multiply sign it. This would be effective
+ against an adversary compromising a single packaging key, but not
+ against one compromising a source repository.)
+
+2. The repository layout
+
+ The filesystem layout in the repository is used for two purposes:
+ - To give mirrors an easy way to mirror only some of the repository.
+ - To specify which parts of the repository a given key has the
+ authority to sign.
+
+ The following files exist in all repositories and mirrors:
+
+ /meta/keys.txt
+
+ Signed by the root keys; indicates keys and roles.
+ [???? I'm using the txt extension here. Is that smart?]
+
+ /meta/mirrors.txt
+
+ Signed by the mirror key; indicates which parts of the
+ repository are mirrored at what mirrors.
+
+ /meta/timestamp.txt
+
+ Signed by the timestamp key; indicates hashes and timestamps
+ for the latest versions of keys.txt and mirrors.txt. Also
+ indicates the latest version of each bundle for each os/arch.
+
+ This is the only file that needs to be downloaded for polling.
+
+ /bundleinfo/bundlename/os-arch/bundlename-os-arch-bundleversion.txt
+
+ Signed by the appropriate bundle key. Describes what
+ packages make up a bundle, and what order to install,
+ uninstall, and upgrade them in.
+
+ /pkginfo/packagename/os-arch/version/packagename-os-arch-packageversion.txt
+
+ Signed by the appropriate package key. Tells the name of the
+ file that makes up a package, its hash, and what procedure
+ is used to install it.
+
+ /packages/packagename/os-arch/version/(some filename)
+
+ The actual package file. Its naming convention will depend
+ on the underlying packaging system.
+
+3. Document formats
+
+3.1. Metaformat
+
+ All documents use Rivest's SEXP meta-format as documented at
+ http://people.csail.mit.edu/rivest/sexp.html
+ with the restriction that no "display hint" fields are to be used,
+ and the base64 transit encoding isn't used either.
+
+ (We use SEXP because it's really easy to parse, really portable,
+ and unlike most other tagged data formats, has a
+ trivially-specified canonical format suitable for hashing.)
+
+ In descriptions of syntax below, we use regex-style qualifiers, so
+ that in
+ (sofa slipcover? occupant* leg+)
+ the sofa will have an optional slipcover, zero or more occupants,
+ and one or more legs. This pattern matches (sofa leg) and (sofa
+ slipcover occupant occupant leg leg leg leg) but not (sofa leg
+ slipcover).
+
+ We also use a braces notation to indicate elements that can occur
+ in any order. For example,
+ (bread {flour+ eggs? yeast})
+ matches a list starting with "bread", and then containing one or
+ more of flours, zero or one occurrences of eggs, and one
+ occurrence of yeast, in any order. This pattern matches (bread eggs
+ yeast flour) but not (bread yeast) or (bread flour eggs yeast
+ macadamias).
+
+3.2. File formats: general principles
+
+ We use tagged lists (lists whose first element is a string) to
+ indicate typed objects. Tags are generally lower-case, with
+ hyphens used for separation. Think Lispy.
+
+ We use attrlists [lists of (key value) lists] to indicate a
+ multimap from keys to values. Clients MUST accept unrecognized
+ keys in these attrlists. The syntax for an attrlist with two
+ recognized and required keys is typically given as ({(key1 val1)
+ (key2 val2) (ATTR VAL)*}), indicating that the keys can occur in
+ any order, intermixed with other attributes.
+
+ Timestamp files will be downloaded very frequently; all other files
+ will be much smaller in size than package files. Thus,
+ size-optimization for timestamp files makes sense and most other
+ other space optimizations don't.
+
+ Versions are represented as lists of the form (v I1 I2 I3 I4 ...)
+ where each item is a number or alphanumeric version component. For
+ example, the version "0.2.1.5-alpha" is represented as (v 0 2 1 5
+ alpha).
+
+ All signed files are of the format:
+
+ (signed
+ X
+ (signature ({(keyid K) (method M) (ATTR VAL)*}) SIG)+
+ )
+
+ where: X is a list whose first element describes the signed object.
+ K is the identifier of a key signing the document
+ M is the method to be used to make the signature
+ (ATTR VAL) is an arbitrary list whose first element is a
+ string.
+ SIG is a signature of the canonical encoding of X using the
+ identified key.
+
+ We define two signing methods at present:
+ sha256-oaep : A signature of the SHA256 hash of the canonical
+ encoding of X, using OAEP+ padding. [XXXX say more about mgf]
+
+ All times are given as strings of the format "YYYY-MM-DD HH:MM:SS",
+ in UTC.
+
+ All keys are of the format:
+ (pubkey ({(type TYPE) (ATTR VAL)*}) KEYVAL)
+ where TYPE is a string describing the type of the key and how it's
+ used to sign documents. The type determines the interpretation of
+ KEYVAL.
+
+ The ID of a key is the type field concatenated with the SHA-256
+ hash of the canonical encoding of the KEYVAL field.
+
+ We define one keytype at present: 'rsa'. The KEYVAL in this case
+ is a 2-element list of (e n), with both values given in big-endian
+ binary format. [This makes keys 45-60% more compact than using
+ decimal integers.]
+
+ All RSA keys must be at least 2048 bits long.
+
+
+ Every role in the system is associated with a key. Replacing
+ anything but a root key is supposed to be relatively easy.
+
+ Root-keys sign other keys, and certify them as belonging to roles.
+ Clients are configured to know the root keys.
+
+ Bundle keys certify the contents of a bundle.
+
+ Package keys certify packages for a given program or set of
+ programs.
+
+ Mirror keys certify a list of mirrors. We expect this to be an
+ automated process.
+
+ Timestamp keys certify that given versions of other metadata
+ documents are up-to-date. They are the only keys that absolutely
+ need to be kept online. (If they are not, timestamps won't be
+ generated.)
+
+3.3. File formats: key list
+
+ The key list file is signed by multiple root keys. It indicates
+ which keys are authorized to sign which parts of the repository.
+
+ (keylist
+ (ts TIME)
+ (keys
+ ((key ({(roles (ROLE PATH)+) (ATTR VAL)*}) KEY)*)
+ ...
+ )
+
+ The "ts" line describes when the keys file was updated. Clients
+ MUST NOT replace a file with an older one, and SHOULD NOT accept a
+ file too far in the future.
+
+ A ROLE is one of "timestamp" "mirrors" "bundle" or "package".
+
+ PATH is a path relative to the top of the directory hierarchy. It
+ may contain "*" elements to indicate "any file", and may end with a
+ "/**" element to indicate all files under a given point.
+
+3.4. File formats: mirror list
+
+ The mirror list is signed by a mirror key. It indicates which
+ mirrors are active and believed to be mirroring which parts of the
+ repository.
+
+ (mirrorlist
+ (ts TIME)
+ (mirrors
+ ( (mirror ({(name N) (urlbase U) (contents PATH+) (weight W)
+ (official)? (ATTR VAL)})) * )
+ ...
+ )
+
+ Every mirror is a copy of some or all of the directory hierarchy
+ containing at least the /meta, /bundles/, and /pkginfo directories.
+
+ N is a descriptive name for the mirror; U is the URL of the mirror's
+ base (i.e., the parent of the "meta" directory); and the PATH
+ elements are the components describing how much of the packages
+ directory is mirrored. Their format is as in the keylist file.
+
+ W is an integer used to weight mirrors when picking at random;
+ mirrors with more bandwidth should have higher weigths. The
+ "official" element should only be present if the mirror is (one of
+ the) official repositories operated by the Tor Project.
+
+3.5. File formats: timestamp files
+
+ The timestamp file is signed by a timestamp key. It indicates the
+ latest versions of other files, and contains a regularly updated
+ timestamp to prevent rollback attacks.
+
+ (ts
+ ({(at TIME)
+ (m TIME MIRRORLISTHASH)
+ (k TIME KEYLISTHASH)
+ (b NAME VERSION TIME PATH HASH)*})
+ )
+
+ TIME is when the timestamp was signed. MIRRORLISTHASH is the digest
+ of the mirror-list file; KEYLISTHASH is the digest of the key list
+ file; and the 'b' entries are a list of the latest version of each
+ bundles and their locations and hashes.
+
+3.6. File formats: bundle files
+
+ (bundle
+ (at TIME)
+ (os OS)
+ [(arch ARCH)]
+ (version V)
+ (packages
+ (NAME VERSION PATH HASH ({(order INST UPDATE REMOVE)
+ (optional)?
+ (gloss LANG TEXT)*
+ (longloss LANG TEXT)*
+ (ATTR VAL)*})? )* )
+ )
+
+ Most elements are self-explanatory; the INST, UPDATE, and REMOVE
+ elements of the order element are numbers defining the order in
+ which the packages are installed, updated, and removed respectively.
+ The "optional" element is present if the package is optional.
+ "Gloss" is a short utf-8 human-readable string explaining what the
+ package provides for the bundle; "longloss" is a longer such
+ utf-8 string.
+
+ (Note that the gloss strings are meant, not to describe the package,
+ but to describe what the package provides for the bundle. For
+ example, "The Anonymous Email Bundle needs the Python Runtime to run
+ Mixminion.")
+
+ Multiple gloss strings are allowed; each should have a different
+ language. The UI should display the must appropriate language to the
+ user.
+
+3.7. File formats: package files
+
+ (package
+ ({(name NAME)
+ (version VERSION)
+ (format FMT ((ATTR VAL)*)? )
+ (path PATH)
+ (ts TIME)
+ (digest HASH)
+ (shortdesc LANG TEXT)*
+ (longdesc LANG TEXT)*
+ (ATTR VAL)* })
+ )
+
+ Most elements are self-explanatory. The "FMT" element describes the
+ file format of the package, which should give enough information
+ about how to install it.
+
+ No two package files in the same repository should have the same
+ name and version. If a package needs to be changed, the version
+ MUST be incremented.
+
+ Descriptions are tagged with languages in the same way as glosses.
+
+4. Detailed Workflows
+
+4.1. The client application
+
+ Periodically, the client updater fetches a timestamp file from a
+ mirror. If the timestamp in the file is up-to-date, the client
+ first checks to see whether the keys file listed is one that the
+ client has. If not, the client fetches it, makes sure the hash of
+ the keys file matches the hash in the timestamp file, makes sure its
+ date is more recent than any keys file they have but not too far in
+ the future, and that it is signed by enough root keys that the
+ client recognizes.
+
+ [If the timestamp file is not up-to-date, the client tries a
+ few mirrors until it finds one with a good timestamp.]
+
+ [If the keys file from a mirror does not match the timestamp
+ file, the client tries a new mirror for both.]
+
+ [If the keys file is not signed by enough root keys, the client
+ warns the user and tries another mirror for both the timestamp
+ file and the keys file.]
+
+ Once the client has an up-to-date keys file, the client checks the
+ signature on the timestamp file. Assuming it checks out, the client
+ refreshes the mirror list as needed, and refreshes any bundle files
+ to which the user is subscribed if the client does not have
+ the latest version of those files. The client checks signatures on
+ these files, and fetches package metadata for any packages listed in
+ the bundle file that the client does not have, checks signatures on
+ these, and fetches binaries for packages that might need to be
+ installed or updated. As the packages arrive, clients check their
+ hashes.
+
+ Once the client has gotten enough packages, it informs the user that
+ new packages have arrived, and asks them if they want to update.
+
+ Clients SHOULD cache at least the latest versions they have received
+ of all files.
+
+4.1.1. Download preferences
+
+ Users should be able to specify that packages must be only
+ downloaded over Tor, or must only be downloaded over encrypted
+ protocols, or both. Users should also be able to force
+
+4.2. Mirrors
+
+ Periodically, mirrors do an rsync or equivalent to fetch the latest
+ version of whatever parts of the repository have changed since the
+ version they currently hold. Mirrors SHOULD replace older versions
+ of the repository idempotently, so that clients are less likely to
+ see inconsistent state. Mirrors SHOULD validate the information
+ they receive, and not serve partial or inconsistent files.
+
+4.3. Workflow: Packagers
+
+ When a new binary package is done, the person making the package
+ runs a tool to generate and sign a package file, and sends both the
+ package and the package file to a repository admin. Typically, the
+ base package file will be generated by inserting a version into a
+ template.
+
+ Packages MAY have as part of their build process a script to
+ generate the appropriately versioned package file. This script
+ should at a minimum demand a build version, or use a timestamp in
+ place of a build version, to prevent two packages with the same
+ version from being created.
+
+4.4. Workflow: bundlers
+
+ When the packages in a bundle are done, the bundler runs a tool on
+ the package files to generate and sign a bundle file. Typically,
+ this tool uses a template bundle file.
+
+4.5. Workflow: repository administrators
+
+ Repository administrators use a tool to validate signed files into the
+ repository. The repository should not be altered manually.
+
+ This tool acts as follows:
+ - Package files may be added, but never replaced.
+ - Bundle files may be added, but never replaced.
+ - No file may be added unless it is syntactically valid and
+ signed by a key in the keys file authorized to sign files of
+ this type in this file's location location.
+
+ - A package file may not be added unless all of its binary
+ packages match their hashes.
+
+ - A bundle file may not be added unless all of its package files
+ are present and match their hashes.
+
+ - When adding a new keylist, bundle, or mirrors list, the
+ timestamp file must be regenerated immediately.
+
+5. Parameter setting and corner cases.
+
+5.1. Timing:
+
+ The timestamp file SHOULD be regenerated every 15 minutes. Mirrors
+ SHOULD attempt to update every hour. Clients SHOULD accept a
+ timestamp file up to 6 hours old.
+
+5.2. Format versioning and forward-compatibility:
+
+ All of the above formats include the ability to add more
+ attribute-value fields for backwards-compatible format changes. If
+ we need to make a backwards incompatible format change, we create a
+ new filename for the new format.
+
+5.3. Key management and migration:
+
+ Root keys should be kept offline. All keys except timestamp and
+ mirror keys should be stored encrypted.
+
+ All the formats above allow for multiple keys to sign a single
+ document. To replace a compromised root key, it suffices to sign
+ keylist documents with both the compromised key and its replacement
+ until all clients have updated to a new version of the autoupdater.
+
+ To replace another key, it suffices to authorize the new key in the
+ keylist. Note that a new package or bundle key must re-sign and
+ issue new versions of all packages or bundles it has generated.
+
+
+
+F. Future directions and open questions
+
+F.1. Package decomposition
+
+ It would be neat to decouple existing packages. Right now, we'd
+ never want a windows user to have to fetch an openssl dll and Tor
+ separately. But if they're using an auto-update tool, it'd be
+ pretty keen to have them not need to fetch a new openssl every time
+ Tor has a bugfix.
+
+F.2. Caching at Tor servers.
+
+ See Tor Proposal number 127.
+
+F.3. Support for more download methods
+
+ Ozymandns, chunked downloads, and bittorrent would all be neat
+ ideas.
+
+
+R. Ideas I'm rejecting for the moment
+
+R.1. Considering recommended versions from Tor consensus directory documents
+
+ This requires a working Tor to update Tor; that's not necessarily a
+ great idea.
+
+R.2. Integration with existing GPG signatures
+
+ The OpenPGP signature and key format is so complicated that you'd
+ have to be mad to touch it.
+
+
+
+
+