[Author Prev][Author Next][Thread Prev][Thread Next][Author Index][Thread Index]

[tor-commits] [gettor/master] Added specs done at the beginning of gsoc. Not updated to the current status of development, but still useful.



commit 2003220ef45fad665d7c688049266118e9ab9a4f
Author: ilv <ilv@xxxxxxxxxxxxxxxxxxxxxxxx>
Date:   Wed Nov 5 20:00:51 2014 -0300

    Added specs done at the beginning of gsoc. Not updated to the current status of development, but still useful.
---
 spec/design/blacklist.txt |   74 ++++++++++++++++++++++
 spec/design/core.txt      |  155 +++++++++++++++++++++++++++++++++++++++++++++
 spec/design/smtp.txt      |  118 ++++++++++++++++++++++++++++++++++
 spec/design/twitter.txt   |   99 +++++++++++++++++++++++++++++
 spec/overview.txt         |   94 +++++++++++++++++++++++++++
 5 files changed, 540 insertions(+)

diff --git a/spec/design/blacklist.txt b/spec/design/blacklist.txt
new file mode 100644
index 0000000..34ce40e
--- /dev/null
+++ b/spec/design/blacklist.txt
@@ -0,0 +1,74 @@
+   Google Summer of Code 2014 GetTor Revamp - Blacklist module
+   Author: Israel Leiva - <israel.leiva@xxxxxxxx>
+   Last update: 2014-05-16
+   Version: 0.01
+   Changes: First version
+ 
+1. Preface
+ 
+   Since GetTor was created it has been a collection of functions and
+   classes separated in various modules. As its main purpose was
+   to serve files over SMTP, almost all current files have SMTP-related
+   procedures, including black and white lists. The proposed design for 
+   the blacklist module intends to separate GetTor services from the 
+   blacklist procedures.
+   
+2. Blacklist module
+
+   The main functionalities the blacklist module should provide are:
+   
+      * Check if a given entry is blacklisted for a given service
+      * Add/update/remove entries.
+      * Provide a standard critera to prevent flood.
+        
+3. Design
+
+   The new design should consist of the following files, directories and 
+   functions:
+   
+   * lib/gettor/blacklist/: Directory for storing blacklisted users of 
+     different services.
+     
+        ----- service1.blacklist: users blocked for service1
+        ----- service2.blacklist: users blocked for service2
+   
+   * lib/gettor/blacklist.py: Blacklist module of GetTor.
+         
+      isBlacklisted(address, provider)
+         Check if the given address is in the blacklist of services and
+         if it acomplish certain restrictions. For example, it could
+         check when was the last time it was updated on the blacklist.
+            
+   
+4. Roadmap
+
+   A possible example of how the blacklist module could work:
+   
+   a. A service receives a request and call the blacklist module. 
+   b. The blacklist module check the blacklist for that service.
+   c. If the user (hashed) is present in the blacklist, it checks when 
+      was the last time it was updated. If this date is more than X days 
+      ago, updates the entry with the current date and returns false. If 
+      not, returns true. If the user is not present in the file, it adds 
+      the user with the current date and returns false.
+   
+   
+5. Discussion
+
+5.1 Mistakes
+
+   Mistakes concerning the package requested should be consider. If I 
+   send a message asking for a Linux 'es' request but I write 'en' instead 
+   of 'es', should I be able to ask again? or wait until I'm no longer 
+   blacklisted?
+
+5.2 Users
+
+   The method presented above (Roadmap) should consider weekly or monthly
+   clean-up of the list.
+
+5.3 SorryMessage
+
+   May be when a user tries to send too many requests we could send him 
+   a message saying that he won't be able to ask for packages in the next
+   X days.
diff --git a/spec/design/core.txt b/spec/design/core.txt
new file mode 100644
index 0000000..ce07238
--- /dev/null
+++ b/spec/design/core.txt
@@ -0,0 +1,155 @@
+   Google Summer of Code 2014 GetTor Revamp - Core module
+   Author: Israel Leiva - <israel.leiva@xxxxxxxx, ilv@xxxxxxxxxx>
+   Last update: 2014-07-15
+   Version: 0.05
+   Changes: 
+            [0.05]
+            Added fingerprint support
+            [0.04]
+            Changed log format in 'Design'
+            Added section 'Features'
+            Deleted stuff from 'Discussion' and added some others
+            [0.03]
+            Changed proposed format for link files to RFC 882 (ConfigParser).
+            Read configuration from file with ConfigParser.
+            [0.02]
+            Combine official mirrors with providers links (as another provider).
+            Eliminated on demand link generation. Now it reads from files.
+            Modified description according to PEP-8.
+            [0.01]
+            First version.
+            
+ 
+1. Preface
+ 
+   Since GetTor was created it has been a collection of functions and
+   classes separated in various modules. As its main purpose was
+   to serve files over SMTP, almost all current files have SMTP-related
+   procedures, including address normalization, message composition, etc.
+   The proposed design for the core module intends to separate GetTor 
+   main functionalities which are independent of the service that 
+   transports the bundles.
+   
+2. Core module
+
+   The main functionalities the core module should provide are:
+   
+      * Receive a request with OS version, bundle language, and respond 
+        with the respective links.
+      * Read links from providers files.
+      * Log anonymous transactions.
+        
+3. Design
+
+   The new design should consist of the following files, directories and 
+   methods:
+   
+      * gettor.cfg: Configuration values, e.g. base directory.
+      
+      * providers/: Directory for generated links. Should be specified on
+                    gettor.cfg.
+   
+         ----- provider1.links: links from provider1.
+         ----- provider2.links: links from provider2.
+         ----- mirrors.links: links of official mirrors.
+      
+         All this data is generated automatically.
+   
+      * logs/: Directory for logs. Should be specified on gettor.cfg
+   
+         ----- all.log
+         ----- info.log
+         ----- debug.log
+         ----- warn.log
+         ----- error.log
+   
+      * Core module of GetTor.
+   
+         __init(config_file)__
+            Creates a new Core object. It reads its configuration from
+            the config_file using ConfigParser.
+            
+         get_links(operating_system, locale)
+            Public method to obtain the links. Returns links for 
+            operating_system in locale language. It checks if the operating
+            system and locale are supported and then calls the private 
+            method _get_links()
+            
+         
+            Example: get_links('linux', 'en')
+         
+         _get_links(operating_system, locale)
+            Gets the links for a specific operating system and locale 
+            according to the options received. It reads all the .links 
+            files inside the providers directory. Each one of these files 
+            should follow the ConfigParser's format. There should be a 
+            section [provider] with the option 'name' for the provider's 
+            name (e.g. Dropbox), and a section [key] with the option
+            'fingerprint' for the key's fingerprint that signed the
+            uploaded packages.
+
+            Following sections should specify the operating system and 
+            its options should be the locale. When more than one link is 
+            available per operating system and locale (always) the links 
+            should be specified as a multiline value. Each link has the 
+            format:
+
+            link link_signature
+            
+            Example:
+
+            [provider]
+            name: Dropbox
+
+            [key]
+            fingerprint: 123A 456B 789C 012D 345E 678F 901G 234H 567I 890J
+            
+            [linux]
+            en: https://foo.bar https://foo.bar.asc,
+                https://foo.bar https://foo.bar.asc
+
+            es: https://bar.baz https://bar.baz.asc
+            
+            PROVIDER NAME
+            operating_system locale link package_signature
+            
+            NOTE: For now, the official mirrors are considered just as 
+            another provider.
+     
+          
+         _log_request(operating_system, locale)
+            Log information about the request for future stats (e.g. which
+            OS from which service is the most required). All logging 
+            should be done with the logging module.
+ 
+      * Providers: There should be one module/script per provider in charge
+        of generating the .links file with the proper format, including
+        the official mirrors.
+
+   
+4. Roadmap
+
+   An example of how the core module work:
+   
+   a. The SMTP service receives a request. 
+   b. The SMTP calls core.get_links() with the options sent by the user.
+   c. get_links() calls to _log_request().
+   d. get_links() calls to _get_links().
+   e. get_links() constructs a message with the information obtained.
+   f. get_links() returns the message previously constructed.
+   g. The SMTP service creates a message with the links obtained and 
+      send it to the user.
+
+5. Discussion
+
+5.1 Daily logs
+
+   Currently, logs are separated by level of information (debug, info, error).
+   Is it necessary to do this by days/weeks too?
+   
+6. Features
+
+   Possible features to be added in the future (open to discussion)
+   
+   a. Send HTTP links (currently some official mirrors are HTTP only). 
+
diff --git a/spec/design/smtp.txt b/spec/design/smtp.txt
new file mode 100644
index 0000000..e86e864
--- /dev/null
+++ b/spec/design/smtp.txt
@@ -0,0 +1,118 @@
+   Google Summer of Code 2014 GetTor Revamp - SMTP module
+   Author: Israel Leiva - <israel.leiva@xxxxxxxx>
+   Last update: 2014-05-16
+   Version: 0.01
+   Changes: First version
+ 
+1. Preface
+ 
+   Since GetTor was created it has been a collection of functions and
+   classes separated in various modules. As its main purpose was
+   to serve files over SMTP, almost all current files have SMTP-related
+   procedures, including address normalization, message composition, etc.
+   The proposed design for the SMTP module intends to separate GetTor 
+   main functionalities from the services, in this case, SMTP.
+   
+2. SMTP module
+
+   The main functionalities the SMTP module should provide are:
+   
+      * Receive requests via mail. 
+      * Identify user instructions, such as ask for help or for a specific
+        bundle (OS version, language).
+      * Get the required links from the core module.
+      * Send different types of answers to the user.
+      * Manage black lists to avoid flood.
+      * Log requests for stats (anonymous).
+        
+3. Design
+
+   The new design should consist of the following files, directories and 
+   functions:
+   
+   * lib/gettor/services/smtp.py: SMTP module of GetTor.
+
+      __init__(configuration path)
+         Creates an object according to the configuration values.
+         
+      processEmail(email object)
+         Process emails received (by forwarding). 
+    
+      __parseEmail(email object)
+         Parse the raw email sent by processMail(). Check for multi-part
+         emails and then parse the text part. It also tries to get the
+         locale information from the user's address.
+         
+      __parseText(email object)
+         Parse the text part of an email looking for the package requested.
+         
+      __getFrom(email object)
+         Returns the from address of an email object.
+      
+      __getLocale(address)
+         Tries to get the locale information from an email address.
+         
+      __checkBlacklist(address)
+         Check if the given address is blacklisted by comparing the
+         hashed address. If address is not present, it's added. If present,
+         check for the date when it was added. Yet to define how many
+         days will be considered for blacklisting or if another method
+         will be used. For this it uses the blacklist module.
+         
+      __sendReply(address)
+         Sends a reply to the user with the links required. It asks for
+         the links to the core module.
+      
+      __sendDelayAlert(address)
+         If enabled (on configuration), sends a delay message to the user
+         letting him know that the links are on their way.
+         
+      __sendHelp(address)
+         Sends a message to the user with help instructions.
+         
+      __createEmail(from, to, subject, body)
+         Creates an email object.
+         
+      __logRequest(options)
+         Log information about the request for future stats (e.g. which
+         OS and language are the most required). If this is called
+         after a failure, a copy of the email should be stored.
+         
+   * BASE_DIR/logs/: Directory for logs. The BASE_DIR should be in the
+     configuration file.
+   
+         ----- yyyy-mm-dd.log: daily log of requests.
+            
+   
+4. Roadmap
+
+   An example of how the SMTP module should work:
+   
+   a. The SMTP service receives a request (via forwarding). 
+   b. The email sender is checked for blacklisting (comparing hashes).
+   c. The email is parsed, obtaining the package requested and the 
+      locale information.
+   d. If the email was asking for help, a help reply is sent.
+   e. If the email was invalid, the process break. This fail is logged
+      and the email that triggered it, too.
+   f. If the email was valid and the delay alert is set, then a reply 
+      informing the links are on their way is sent.
+   g. If the email was valid, the SMTP service asks for the links to the 
+      code module.
+   h. After that, a reply is sent to the user.
+   i. In all cases the request is logged (with no user information).
+   
+   
+5. Discussion
+   
+5.1 Email forwarding
+
+   Are we going to support forwarding emails as ForwardPackage() did in 
+   the old GetTor?
+
+5.2 Blacklist sublists
+
+   Now with less types of request (two if no forwarding is added), creating
+   sublists for different types of requests necessary to blacklist and 
+   email address? Or should we blacklist it if it floods anything?
+
diff --git a/spec/design/twitter.txt b/spec/design/twitter.txt
new file mode 100644
index 0000000..46a178a
--- /dev/null
+++ b/spec/design/twitter.txt
@@ -0,0 +1,99 @@
+   Google Summer of Code 2014 GetTor Revamp - Twitter module
+   Author: Israel Leiva - <israel.leiva@xxxxxxxx>
+   Last update: 2014-05-16
+   Version: 0.01
+   Changes: First version
+ 
+1. Preface
+ 
+   Since GetTor was created it has been a collection of functions and
+   classes separated in various modules. As its main purpose was
+   to serve files over SMTP, almost all current files have SMTP-related
+   procedures, including address normalization, message composition, etc.
+   The proposed design for the Twitter module intends to separate GetTor 
+   main functionalities from the services, in this case, Twitter.
+   
+2. Twitter module
+
+   The main functionalities the Twitter module should provide are:
+   
+      * Receive requests via direct messages. 
+      * Identify user instructions, such as ask for help or for a specific
+        bundle (OS version, language).
+      * Get the required links from the core module.
+      * Send different types of answers to the user.
+      * Split answers to fit Twitter's format.
+      * Manage black lists to avoid flood.
+      * Log requests for stats (anonymous).
+        
+3. Design
+
+   The new design should consist of the following files, directories and 
+   functions:
+   
+   * lib/gettor/services/Twitter.py: Twitter module of GetTor.
+
+      __init__(configuration path)
+         Creates an object according to the configuration values.
+         
+      processDM(message)
+         Process direct messages received. 
+    
+      __parseDM(message)
+         Parse the direct message received. Check for the package requested
+         and the locale information.
+         
+      __getUser(message)
+         Gets the user from the message sent. 
+         
+      __checkBlacklist(user)
+         Check if the given user is blacklisted by comparing the
+         hashed user. Yet to define how many days will be considered for 
+         blacklisting or if another method will be used. For this it uses 
+         the blacklist module.
+         
+      __sendReply(user)
+         Sends a reply to the user with the links required. It asks for
+         the links to the core module and then split them.
+         
+      __sendHelp(user)
+         Sends a message to the user with help instructions.
+      
+      __splitMessage(message)
+         Receives the links message and split it according to Twitter's
+         format.
+         
+      __CheckNewFollowers()
+         In order to ask for links the user has to follow the GetTor 
+         account. The Twitter module will be constantly checking for
+         new followers and follow them back.
+         
+      __FollowUser(user)
+         Follow the given user.
+         
+      __logRequest(options)
+         Log information about the request for future stats (e.g. which
+         OS and language are the most required). If this is called
+         after a failure, a copy of the DM should be stored.
+         
+   * BASE_DIR/logs/: Directory for logs. BASE_DIR should be in the 
+     configuration file.
+   
+         ----- yyyy-mm-dd.log: daily log of requests.
+            
+   
+4. Roadmap
+
+   An example of how the Twitter module should work:
+   
+   a. The Twitter account receives a DM.
+   b. The Twitter service check if is a valid message and if the user is
+      in the blacklist, and then tries to obtain the package requested and 
+      the locale information.
+   c. The Twitter service asks for the links to the core module, then it
+      splits the message received to adopt Twitter's format.
+   d. One or more DMs are sent back to the user.
+   e. For all this, the user must follow the GetTor account. The Twitter
+      service will be constantly checking for new followers and following
+      them back.
+   
diff --git a/spec/overview.txt b/spec/overview.txt
new file mode 100644
index 0000000..435d256
--- /dev/null
+++ b/spec/overview.txt
@@ -0,0 +1,94 @@
+   Google Summer of Code 2014 GetTor Revamp - Overview
+   Author: Israel Leiva - <israel.leiva@xxxxxxxx>
+   Last update: 2014-05-16
+   Version: 0.01
+   Changes: First version
+
+1. Background
+
+   GetTor was created as a program for serving Tor and related files over 
+   SMTP, thus avoiding direct and indirect censorship of Tor's software, 
+   in particular, the Tor Browser Bundle (TBB). Users interact with GetTor 
+   by sending emails to a specific email address. In the past, after the 
+   user specified his OS and language, GetTor would send him an attachment 
+   with the required package. This worked until the bundles were too large 
+   to be sent as attachments in most email providers. In order to fix this 
+   GetTor started to send links instead.
+
+2. Current status
+
+   The GetTor status can be summarized in the following points:
+
+      * Emails are sent to gettor@xxxxxxxxxxxxxx
+      * The GetTor reply contains: TBB links, signatures (with text guides
+        for verification), mirrors, support instructions in six languages.
+      * Dropbox links are sent to download the TBB and signatures.
+      * Users can not ask for packages in their language.
+      * English-only replies are sent.
+      * Any email directed to GetTor is replied with the same information, 
+        there is no recognition of instructions.
+      * Links generation is not fully automated.
+      * All code is written in Python. Various parts are not currently used.
+      * Current repositories are [0] and [1].
+   
+3. Proposal
+
+   The accepted proposal [2] for Google Summer of Code (GSoC) 2014 proposes 
+   rewriting the current GetTor using a modular design, with a core module 
+   that handles the main GetTor functionalities, and several other modules, 
+   one for each service (e.g. SMTP), which can interact with the core and 
+   send replies to the users. Three modules will be developed for the 
+   purposes of GSoC: SMTP, Twitter, Skype|XMPP. 
+   
+3.1. Goals
+
+   The main goals of this proposal are the following:
+
+      * Provide old GetTor functionalites, such as replies in several
+        languages and recognize user instructions.
+      * Send fewer information in each reply.
+      * Support more providers for uploading the TBB packages.
+      * Automate links generation.
+      * Clearer, modular and well-documented code.
+      * Possibilty to create new modules for other common services.
+   
+3.2. Design
+
+   Preliminar designs for the core module and the services can be found 
+   in the design/ folder. All services consider creating a python script
+   to add the logic for using them. For example, there should be a script 
+   that receives the emails and uses the SMTP module. For simplicity,
+   I've tried to specify mostly the main functions of every module; there 
+   are some functions, like opening and writing files, that were not 
+   considered in this preliminar phase.
+     
+4. Discussion
+
+4.1. Skype
+
+   My co-mentor for GSoC, Nima, has publicly rejected the idea of creating
+   a module for Skype and proposed to implement one for XMPP instead.
+   I've chosen Skype for its popularity, but I have no other main reason 
+   to maintain this option, so it's probable that XMPP transport will be 
+   implemented.
+
+4.2. Storing links
+
+   My original proposal considered the fact that links could be stored 
+   somewhere with restricted access, ideally a git repository. Nima 
+   mentioned that ideally the links shouldn't be stored. May be this idea 
+   could be used only to the mirrors and providers configuration (see
+   core module design).
+
+4.3. Generating unique URLs
+
+   Nima mentioned that unique URLs could be generated for each request, 
+   and in case the user don't have access to SSL, these links could be 
+   served and later deleted or recycled. I like this idea.
+
+4. References
+
+   [0] https://gitweb.torproject.org/gettor.git
+   [1] https://gitweb.torproject.org/user/sukhbir/gettor.git
+   [2] https://ileiva.github.io/gettor_proposal.html
+   



_______________________________________________
tor-commits mailing list
tor-commits@xxxxxxxxxxxxxxxxxxxx
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits