[or-cvs] r22654: {projects} Add documentation on how GetTor currently blacklists users. (in projects/gettor: . lib/gettor)

[Christian Fromme <kaner@xxxxxxxxxx>]

Author: kaner
Date: 2010-07-19 10:37:09 +0000 (Mon, 19 Jul 2010)
New Revision: 22654

Modified:
   projects/gettor/README
   projects/gettor/TODO
   projects/gettor/lib/gettor/packages.py
   projects/gettor/lib/gettor/utils.py
   projects/gettor/setup.cfg
Log:
Add documentation on how GetTor currently blacklists users. Also do some cosmetic cleanups


Modified: projects/gettor/README
===================================================================
--- projects/gettor/README	2010-07-19 09:54:28 UTC (rev 22653)
+++ projects/gettor/README	2010-07-19 10:37:09 UTC (rev 22654)
@@ -213,3 +213,50 @@
 	http://mail.google.com/support/bin/answer.py?hl=en&answer=8770
 
 Other providers are not as transparent and this may cause failures.
+
+SPAM / ABUSE PREVENTION
+------------------------------
+
+GetTor does some spam/abuse prevention. This is primarily because we don't want 
+to hammer innocent inboxes with huge amounts of tor packages.
+
+First of all, there is a DKIM checker (see "WHAT'S DKIM / WHY DKIM" above) that
+prevents GetTor from answering useless spam emails. On top of that, there are 
+some hard-coded checks being performed on the sender address. Only mails coming
+from yahoo.com.cn, yahoo.com and gmail.com are allowed. However, DKIM checking 
+and hard-coded domain checks are currently disabled in GetTor. Mails from all
+addresses are being passed to GetTor processing.
+
+On top of that, there is a blacklisting mechanism. It works as follows: Each 
+unique email address is allowed to mail GetTor the same type of mail once each
+7 days. Mail types are sendDelayAlert, sendPackage, sendPackageHelp, 
+sendSplitPackage. Blacklists are stored in the form of hashed email files under
+directories. In reality, that looks as follows:
+
+    gettor@moria:~/opt/gettor$ ls -d /home/gettor/gettor/bl/*
+        /home/gettor/gettor/bl/general
+        /home/gettor/gettor/bl/sendPackage
+        /home/gettor/gettor/bl/sendSplitPackage
+        /home/gettor/gettor/bl/sendDelayAlert
+        /home/gettor/gettor/bl/sendPackageHelp
+
+    gettor@moria:~/opt/gettor$ ls /home/gettor/gettor/bl/sendDelayAlert \
+                                                                    | head -n 2
+     0154d8584c0afa6290e21098e7ab4cc635b7d50a
+     02a33e16feece8671f1274de62de32068a67cf20
+
+In addition to this automatic whitelisting, there is the possibility to add
+blacklist entries by hand as follows:
+
+    gettor@moria:~/opt/gettor$ ./GetTor.py -b someone@xxxxxxxx
+
+Email addresses that are added this way, go to the general/ directory in the 
+blacklist directory.
+
+Besides the blacklisting mechanism, there is a whitelisting mechanism. It works
+analogous to the manual blacklisting mechanism:
+
+    gettor@moria:~/opt/gettor$ ./GetTor.py -w someone@xxxxxxxx
+
+Whitelisting wins over blacklisting. If a user if blacklisted for X, but also
+whitelisted, he will be allowed to do X.

Modified: projects/gettor/TODO
===================================================================
--- projects/gettor/TODO	2010-07-19 09:54:28 UTC (rev 22653)
+++ projects/gettor/TODO	2010-07-19 10:37:09 UTC (rev 22654)
@@ -1,7 +1,5 @@
 These are planned changes to the gettor system.
 
-- Write up how and when and for how long we block requests by users on 
-  blacklists, e.g. write up what our spam prevention technology is
 - Split (at least) tiger bundle into several smaller archives to avoid ~20MB
 - React in some way if the user requested a split download but no split 
   downloads are available for that specific package (maybe send out the non-

Modified: projects/gettor/lib/gettor/packages.py
===================================================================
--- projects/gettor/lib/gettor/packages.py	2010-07-19 09:54:28 UTC (rev 22653)
+++ projects/gettor/lib/gettor/packages.py	2010-07-19 10:37:09 UTC (rev 22654)
@@ -172,7 +172,6 @@
             return False
 
     def buildSplitFiles(self, pack, dirname, filename):
-        log.info("Building split files..")
         packSplitDir = None
         try:
             splitpack = pack + ".split"
@@ -226,7 +225,6 @@
                 else:
                     log.error("Uhm, expected signature file for %s to be: %s" % (file, ascfile))
 
-        log.info("Done.")
         return True
 
     def initRsync(self, mirror="rsync.torproject.org", silent=False):

Modified: projects/gettor/lib/gettor/utils.py
===================================================================
--- projects/gettor/lib/gettor/utils.py	2010-07-19 09:54:28 UTC (rev 22653)
+++ projects/gettor/lib/gettor/utils.py	2010-07-19 10:37:09 UTC (rev 22654)
@@ -134,6 +134,8 @@
     except IOError:
         log.error("Error initiating package list.")
         return False
+    # Currently not needed. Keep it here in case someone decides to change the
+    # directory structure back
     #packs.preparePackages()
     if not packs.buildPackages():
         log.error("Building packages failed.")
@@ -279,7 +281,6 @@
         return False
 
 def renameExe(filename, renameFile=True):
-    log.info("Renaming exe..")
     if renameFile and not os.access(filename, os.R_OK):
         log.error("Could not access file %s" % filename)
         raise OSError

Modified: projects/gettor/setup.cfg
===================================================================
--- projects/gettor/setup.cfg	2010-07-19 09:54:28 UTC (rev 22653)
+++ projects/gettor/setup.cfg	2010-07-19 10:37:09 UTC (rev 22654)
@@ -1,4 +1,4 @@
 [install]
-install-purelib=~/opt/gettor
-install-scripts=~/opt/gettor
-install-data=~/opt/gettor
+install-purelib=$HOME/opt/gettor
+install-scripts=$HOME/opt/gettor
+install-data=$HOME/opt/gettor