[Author Prev][Author Next][Thread Prev][Thread Next][Author Index][Thread Index]

[or-cvs] r23335: {arm} added: hiding duplicate log entries (feature request by asn) (in arm/trunk: . src/interface src/interface/graphing src/util)



Author: atagar
Date: 2010-09-29 16:30:48 +0000 (Wed, 29 Sep 2010)
New Revision: 23335

Modified:
   arm/trunk/TODO
   arm/trunk/armrc.sample
   arm/trunk/src/interface/controller.py
   arm/trunk/src/interface/graphing/bandwidthStats.py
   arm/trunk/src/interface/graphing/graphPanel.py
   arm/trunk/src/interface/graphing/psStats.py
   arm/trunk/src/interface/headerPanel.py
   arm/trunk/src/interface/logPanel.py
   arm/trunk/src/util/conf.py
   arm/trunk/src/util/connections.py
   arm/trunk/src/util/hostnames.py
   arm/trunk/src/util/log.py
   arm/trunk/src/util/panel.py
   arm/trunk/src/util/sysTools.py
   arm/trunk/src/util/torTools.py
   arm/trunk/src/util/uiTools.py
Log:
added: hiding duplicate log entries (feature request by asn)
change: making the number of lines an entry displays customizable
change: improved performance and capabilities of the cropStr function and dropped splitLine (no longer needed)
change: caching daybreak and deduplication results for the latest events listing
change: using previously drawn content as an estimate for the content size rather than estimating beforehand (simpler, better performance, and much less of a headache)
fix: dumping a stacktrace to /tmp and exiting immediately if exceptions are raised while redrawing
fix: providing ellipse when truncating the fingerprint in the header panel



Modified: arm/trunk/TODO
===================================================================
--- arm/trunk/TODO	2010-09-29 10:57:17 UTC (rev 23334)
+++ arm/trunk/TODO	2010-09-29 16:30:48 UTC (rev 23335)
@@ -13,9 +13,7 @@
         - log to file, allowing non-runlevel events to be saved (provide both
             a continuous option and snapshots taking into account the current
             filter)
-        - make the maximum line count for entries configurable
         - log cropping based on time (idea by voidzero)
-        - drop duplicate or overly verbose messages (feature request by asn)
       [ ] conf panel
         - move torrc validation into util
         - fetch text via getinfo rather than reading directly?
@@ -46,6 +44,9 @@
         - provide bridge / client country statistics
             Include bridge related data via GETINFO option (feature request
             by waltman and ioerror).
+        - pick apart applications like iftop and pktstat to see how they get
+            per-connection bandwidth usage. Forum thread discussing it:
+            https://bbs.archlinux.org/viewtopic.php?pid=715906
       [ ] controller and popup panels
         - country data for client connections (requested by ioerror)
         - allow arm to resume after restarting tor
@@ -106,6 +107,11 @@
     * connections aren't cleared when control port closes
 
 - Features
+  * general purpose method of erroring nicely
+    Some errors cause portions of the display to die, but curses limps along
+    and overwrites the stacktrace. This has been mostly solved, but all errors
+    should result in a clean death, with the stacktrace saved and a nice
+    message for the user.
   * client mode use cases
     * not sure what sort of information would be useful in the header (to
       replace the orport, fingerprint, flags, etc)

Modified: arm/trunk/armrc.sample
===================================================================
--- arm/trunk/armrc.sample	2010-09-29 10:57:17 UTC (rev 23334)
+++ arm/trunk/armrc.sample	2010-09-29 16:30:48 UTC (rev 23335)
@@ -8,6 +8,8 @@
 features.colorInterface true
 
 # log panel parameters
+# showDateDividers: show borders with dates for entries from previous days
+# maxLinesPerEntry: max number of lines to display for a single log entry
 # prepopulate: attempts to read past events from the log file if true
 # prepopulateReadLimit: maximum entries read from the log file
 # maxRefreshRate: rate limiting (in milliseconds) for drawing the log if
@@ -17,10 +19,11 @@
 # instance, if arm's only listening for ERR entries but the log has all
 # runlevels then this will stop reading after <prepopulateReadLimit> lines.
 
+features.log.showDateDividers true
+features.log.maxLinesPerEntry 4
 features.log.prepopulate true
 features.log.prepopulateReadLimit 5000
 features.log.maxRefreshRate 300
-features.log.showDateDividers true
 
 # general graph parameters
 # height:   height of graphed stats

Modified: arm/trunk/src/interface/controller.py
===================================================================
--- arm/trunk/src/interface/controller.py	2010-09-29 10:57:17 UTC (rev 23334)
+++ arm/trunk/src/interface/controller.py	2010-09-29 16:30:48 UTC (rev 23335)
@@ -42,7 +42,12 @@
   ["torrc"]]
 PAUSEABLE = ["header", "graph", "log", "conn"]
 
-CONFIG = {"logging.rate.refreshRate": 5, "features.graph.type": 1, "log.torEventTypeUnrecognized": log.NOTICE, "features.graph.bw.prepopulate": True, "log.refreshRate": log.DEBUG, "log.configEntryUndefined": log.NOTICE}
+CONFIG = {"logging.rate.refreshRate": 5,
+          "features.graph.type": 1,
+          "log.torEventTypeUnrecognized": log.NOTICE,
+          "features.graph.bw.prepopulate": True,
+          "log.refreshRate": log.DEBUG,
+          "log.configEntryUndefined": log.NOTICE}
 
 class ControlPanel(panel.Panel):
   """ Draws single line label for interface controls. """
@@ -450,6 +455,9 @@
   
   # TODO: popups need to force the panels it covers to redraw (or better, have
   # a global refresh function for after changing pages, popups, etc)
+  
+  # TODO: come up with a nice, clean method for other threads to immediately
+  # terminate the draw loop and provide a stacktrace
   while True:
     # tried only refreshing when the screen was resized but it caused a
     # noticeable lag when resizing and didn't have an appreciable effect
@@ -651,8 +659,11 @@
           
           regexLabel = "enabled" if panels["log"].regexFilter else "disabled"
           popup.addfstr(5, 41, "<b>f</b>: log regex filter (<b>%s</b>)" % regexLabel)
-          popup.addfstr(6, 2, "<b>x</b>: clear event log")
           
+          hiddenEntryLabel = "hidden" if panels["log"].isDuplicatesHidden else "visible"
+          popup.addfstr(6, 2, "<b>u</b>: duplicate log entries (<b>%s</b>)" % hiddenEntryLabel)
+          popup.addfstr(6, 41, "<b>x</b>: clear event log")
+          
           pageOverrideKeys = (ord('m'), ord('n'), ord('s'), ord('i'), ord('d'), ord('e'), ord('r'), ord('f'), ord('x'))
         if page == 1:
           popup.addfstr(1, 2, "<b>up arrow</b>: scroll up a line")

Modified: arm/trunk/src/interface/graphing/bandwidthStats.py
===================================================================
--- arm/trunk/src/interface/graphing/bandwidthStats.py	2010-09-29 10:57:17 UTC (rev 23334)
+++ arm/trunk/src/interface/graphing/bandwidthStats.py	2010-09-29 16:30:48 UTC (rev 23335)
@@ -20,7 +20,12 @@
 PREPOPULATE_SUCCESS_MSG = "Read the last day of bandwidth history from the state file"
 PREPOPULATE_FAILURE_MSG = "Unable to prepopulate bandwidth information (%s)"
 
-DEFAULT_CONFIG = {"features.graph.bw.transferInBytes": False, "features.graph.bw.accounting.show": True, "features.graph.bw.accounting.rate": 10, "features.graph.bw.accounting.isTimeLong": False, "log.graph.bw.prepopulateSuccess": log.NOTICE, "log.graph.bw.prepopulateFailure": log.NOTICE}
+DEFAULT_CONFIG = {"features.graph.bw.transferInBytes": False,
+                  "features.graph.bw.accounting.show": True,
+                  "features.graph.bw.accounting.rate": 10,
+                  "features.graph.bw.accounting.isTimeLong": False,
+                  "log.graph.bw.prepopulateSuccess": log.NOTICE,
+                  "log.graph.bw.prepopulateFailure": log.NOTICE}
 
 class BandwidthStats(graphPanel.GraphStats):
   """

Modified: arm/trunk/src/interface/graphing/graphPanel.py
===================================================================
--- arm/trunk/src/interface/graphing/graphPanel.py	2010-09-29 10:57:17 UTC (rev 23334)
+++ arm/trunk/src/interface/graphing/graphPanel.py	2010-09-29 16:30:48 UTC (rev 23335)
@@ -41,7 +41,11 @@
 WIDE_LABELING_GRAPH_COL = 50  # minimum graph columns to use wide spacing for x-axis labels
 
 # used for setting defaults when initializing GraphStats and GraphPanel instances
-CONFIG = {"features.graph.height": 7, "features.graph.interval": 0, "features.graph.bound": 1, "features.graph.maxWidth": 150, "features.graph.showIntermediateBounds": True}
+CONFIG = {"features.graph.height": 7,
+          "features.graph.interval": 0,
+          "features.graph.bound": 1,
+          "features.graph.maxWidth": 150,
+          "features.graph.showIntermediateBounds": True}
 
 def loadConfig(config):
   config.update(CONFIG)

Modified: arm/trunk/src/interface/graphing/psStats.py
===================================================================
--- arm/trunk/src/interface/graphing/psStats.py	2010-09-29 10:57:17 UTC (rev 23334)
+++ arm/trunk/src/interface/graphing/psStats.py	2010-09-29 16:30:48 UTC (rev 23335)
@@ -12,7 +12,11 @@
 # attempts to use cached results from the header panel's ps calls
 HEADER_PS_PARAM = ["%cpu", "rss", "%mem", "etime"]
 
-DEFAULT_CONFIG = {"features.graph.ps.primaryStat": "%cpu", "features.graph.ps.secondaryStat": "rss", "features.graph.ps.cachedOnly": True, "log.graph.ps.invalidStat": log.WARN, "log.graph.ps.abandon": log.WARN}
+DEFAULT_CONFIG = {"features.graph.ps.primaryStat": "%cpu",
+                  "features.graph.ps.secondaryStat": "rss",
+                  "features.graph.ps.cachedOnly": True,
+                  "log.graph.ps.invalidStat": log.WARN,
+                  "log.graph.ps.abandon": log.WARN}
 
 class PsStats(graphPanel.GraphStats):
   """

Modified: arm/trunk/src/interface/headerPanel.py
===================================================================
--- arm/trunk/src/interface/headerPanel.py	2010-09-29 10:57:17 UTC (rev 23334)
+++ arm/trunk/src/interface/headerPanel.py	2010-09-29 16:30:48 UTC (rev 23335)
@@ -154,7 +154,8 @@
     if self.vals["tor/orPort"]:
       # Line 4 / Line 2 Right (fingerprint)
       y, x = (1, leftWidth) if isWide else (3, 0)
-      self.addstr(y, x, "fingerprint: %s" % self.vals["tor/fingerprint"])
+      fingerprintLabel = uiTools.cropStr("fingerprint: %s" % self.vals["tor/fingerprint"], width)
+      self.addstr(y, x, fingerprintLabel)
       
       # Line 5 / Line 3 Left (flags)
       if self._isTorConnected:

Modified: arm/trunk/src/interface/logPanel.py
===================================================================
--- arm/trunk/src/interface/logPanel.py	2010-09-29 10:57:17 UTC (rev 23334)
+++ arm/trunk/src/interface/logPanel.py	2010-09-29 16:30:48 UTC (rev 23335)
@@ -35,8 +35,38 @@
 RUNLEVEL_EVENT_COLOR = {"DEBUG": "magenta", "INFO": "blue", "NOTICE": "green", "WARN": "yellow", "ERR": "red"}
 DAYBREAK_EVENT = "DAYBREAK" # special event for marking when the date changes
 
-DEFAULT_CONFIG = {"features.log.prepopulate": True, "features.log.prepopulateReadLimit": 5000, "features.log.maxRefreshRate": 300, "features.log.showDateDividers": True, "cache.logPanel.size": 1000, "log.logPanel.prepopulateSuccess": log.INFO, "log.logPanel.prepopulateFailed": log.WARN}
+ENTRY_INDENT = 2 # spaces an entry's message is indented after the first line
+DEFAULT_CONFIG = {"features.log.showDateDividers": True,
+                  "features.log.maxLinesPerEntry": 4,
+                  "features.log.prepopulate": True,
+                  "features.log.prepopulateReadLimit": 5000,
+                  "features.log.maxRefreshRate": 300,
+                  "cache.logPanel.size": 1000,
+                  "log.logPanel.prepopulateSuccess": log.INFO,
+                  "log.logPanel.prepopulateFailed": log.WARN}
 
+DUPLICATE_MSG = " [%i duplicate%s hidden]"
+
+# static starting portion of common log entries, used to deduplicate entries
+# that have dynamic content: 
+# [NOTICE] We stalled too much while trying to write 125 bytes to address [scrubbed]...
+# [NOTICE] Attempt by %s to open a stream from unknown relay. Closing.
+# [WARN] You specified a server "Amunet8" by name, but this name is not registered
+COMMON_LOG_MESSAGES = ["We stalled too much while trying to write",
+                       "Attempt by ",
+                       "You specified a server "]
+
+# messages with a dynamic beginning (searches the whole string instead)
+# [WARN] 4 unknown, 1 missing key, 3 good, 0 bad, 1 no signature, 4 required
+COMMON_LOG_MESSAGES_INTERNAL = ["missing key, "] 
+
+# cached values and the arguments that generated it for the getDaybreaks and
+# getDuplicates functions
+CACHED_DAYBREAKS_ARGUMENTS = (None, None) # events, current day
+CACHED_DAYBREAKS_RESULT = None
+CACHED_DUPLICATES_ARGUMENTS = None # events
+CACHED_DUPLICATES_RESULT = None
+
 def expandEvents(eventAbbr):
   """
   Expands event abbreviations to their full names. Beside mappings provided in
@@ -215,7 +245,7 @@
   log.log(DEFAULT_CONFIG["log.logPanel.prepopulateSuccess"], msg)
   return loggedEvents
 
-def getDaybreaks(events):
+def getDaybreaks(events, ignoreTimeForCache = False):
   """
   Provides the input events back with special 'DAYBREAK_EVENT' markers inserted
   whenever the date changed between log entries (or since the most recent
@@ -223,15 +253,23 @@
   entry.
   
   Arguments:
-    events - chronologically ordered listing of events
+    events             - chronologically ordered listing of events
+    ignoreTimeForCache - skips taking the day into consideration for providing
+                         cached results if true
   """
   
+  global CACHED_DAYBREAKS_ARGUMENTS, CACHED_DAYBREAKS_RESULT
   if not events: return []
   
   newListing = []
   timezoneOffset = time.altzone if time.localtime()[8] else time.timezone
-  lastDay = int((time.time() - timezoneOffset) / 86400)
+  currentDay = int((time.time() - timezoneOffset) / 86400)
+  lastDay = currentDay
   
+  if CACHED_DAYBREAKS_ARGUMENTS[0] == events and \
+    (ignoreTimeForCache or CACHED_DAYBREAKS_ARGUMENTS[1] == currentDay):
+    return list(CACHED_DAYBREAKS_RESULT)
+  
   for entry in events:
     eventDay = int((entry.timestamp - timezoneOffset) / 86400) # days since epoch
     if eventDay != lastDay:
@@ -241,8 +279,67 @@
     newListing.append(entry)
     lastDay = eventDay
   
+  CACHED_DAYBREAKS_ARGUMENTS = (list(events), currentDay)
+  CACHED_DAYBREAKS_RESULT = list(newListing)
+  
   return newListing
 
+def getDuplicates(events):
+  """
+  Deduplicates a list of log entries, providing back a tuple listing with the
+  log entry and count of duplicates following it. Entries in different days are
+  not considered to be duplicates.
+  
+  Arguments:
+    events - chronologically ordered listing of events
+  """
+  
+  global CACHED_DUPLICATES_ARGUMENTS, CACHED_DUPLICATES_RESULT
+  if CACHED_DUPLICATES_ARGUMENTS == events:
+    return list(CACHED_DUPLICATES_RESULT)
+  
+  eventsRemaining = list(events)
+  returnEvents = []
+  
+  while eventsRemaining:
+    entry = eventsRemaining.pop(0)
+    duplicateIndices = []
+    
+    for i in range(len(eventsRemaining)):
+      forwardEntry = eventsRemaining[i]
+      
+      # if showing dates then do duplicate detection for each day, rather
+      # than globally
+      if forwardEntry.type == DAYBREAK_EVENT: break
+      
+      if entry.type == forwardEntry.type:
+        if entry.msg == forwardEntry.msg: isDuplicate = True
+        else:
+          isDuplicate = False
+          for commonMsg in COMMON_LOG_MESSAGES:
+            if entry.msg.startswith(commonMsg) and forwardEntry.msg.startswith(commonMsg):
+              isDuplicate = True
+              break
+          
+          if not isDuplicate:
+            for commonMsg in COMMON_LOG_MESSAGES_INTERNAL:
+              if commonMsg in entry.msg and commonMsg in forwardEntry.msg:
+                isDuplicate = True
+                break
+        
+        if isDuplicate: duplicateIndices.append(i)
+    
+    # drops duplicate entries
+    duplicateIndices.reverse()
+    for i in duplicateIndices: del eventsRemaining[i]
+    
+    returnEvents.append((entry, len(duplicateIndices)))
+  
+  CACHED_DUPLICATES_ARGUMENTS = list(events)
+  CACHED_DUPLICATES_RESULT = list(returnEvents)
+  
+  return returnEvents
+
 class LogEntry():
   """
   Individual log file entry, having the following attributes:
@@ -360,13 +457,16 @@
       config.update(self._config)
       
       # ensures prepopulation and cache sizes are sane
+      self._config["features.log.maxLinesPerEntry"] = max(self._config["features.log.maxLinesPerEntry"], 1)
       self._config["features.log.prepopulateReadLimit"] = max(self._config["features.log.prepopulateReadLimit"], 0)
       self._config["features.log.maxRefreshRate"] = max(self._config["features.log.maxRefreshRate"], 10)
       self._config["cache.logPanel.size"] = max(self._config["cache.logPanel.size"], 50)
     
+    self.isDuplicatesHidden = True      # collapses duplicate log entries, only showing the most recent
     self.msgLog = []                    # log entries, sorted by the timestamp
     self.loggedEvents = loggedEvents    # events we're listening to
     self.regexFilter = None             # filter for presented log events (no filtering if None)
+    self.lastContentHeight = 0          # height of the rendered content when last drawn
     self.scroll = 0
     self._isPaused = False
     self._pauseBuffer = []              # location where messages are buffered if paused
@@ -387,10 +487,6 @@
     self._titleCache = None
     self._titleArgs = (None, None, None)
     
-    # _getContentLength (args: msgLog, regexFilter pattern, height, width, day)
-    self._contentLengthCache = None
-    self._contentLengthArgs = (None, None, None, None, None)
-    
     # fetches past tor events from log file, if available
     torEventBacklog = []
     if self._config["features.log.prepopulate"]:
@@ -430,6 +526,9 @@
     finally:
       log.LOG_LOCK.release()
     
+    # leaving lastContentHeight as being too low causes initialization problems
+    self.lastContentHeight = len(self.msgLog)
+    
     # adds listeners for tor and torctl events
     conn = torTools.getConn()
     conn.addEventListener(TorEventObserver(self.registerEvent))
@@ -517,23 +616,24 @@
   def handleKey(self, key):
     if uiTools.isScrollKey(key):
       pageHeight = self.getPreferredSize()[0] - 1
-      contentHeight = self._getContentLength()
-      newScroll = uiTools.getScrollPosition(key, self.scroll, pageHeight, contentHeight)
+      newScroll = uiTools.getScrollPosition(key, self.scroll, pageHeight, self.lastContentHeight)
       
       if self.scroll != newScroll:
         self.valsLock.acquire()
         self.scroll = newScroll
         self.redraw(True)
         self.valsLock.release()
+    elif key in (ord('u'), ord('U')):
+      self.valsLock.acquire()
+      self.isDuplicatesHidden = not self.isDuplicatesHidden
+      self.redraw(True)
+      self.valsLock.release()
   
   def setPaused(self, isPause):
     """
     If true, prevents message log from being updated with new events.
     """
     
-    # TODO: minor bug - if the date changes and the panel is redrawn then the
-    # new date marker is shown
-    
     if isPause == self._isPaused: return
     
     self._isPaused = isPause
@@ -557,21 +657,29 @@
     self.addstr(0, 0, self._getTitle(width), curses.A_STANDOUT)
     
     # restricts scroll location to valid bounds
-    contentHeight = self._getContentLength()
-    self.scroll = max(0, min(self.scroll, contentHeight - height + 1))
+    self.scroll = max(0, min(self.scroll, self.lastContentHeight - height + 1))
     
     # draws left-hand scroll bar if content's longer than the height
-    xOffset = 0 # offset for scroll bar
-    if contentHeight > height - 1:
-      xOffset = 3
-      self.addScrollBar(self.scroll, self.scroll + height - 1, contentHeight, 1)
+    msgIndent, dividerIndent = 0, 0 # offsets for scroll bar
+    if self.lastContentHeight > height - 1:
+      msgIndent, dividerIndent = 3, 2
+      self.addScrollBar(self.scroll, self.scroll + height - 1, self.lastContentHeight, 1)
     
     # draws log entries
     lineCount = 1 - self.scroll
-    eventLog = getDaybreaks(self.msgLog) if self._config["features.log.showDateDividers"] else self.msgLog
-    seenFirstDateDivider, dividerAttr = False, curses.A_BOLD | uiTools.getColor("yellow")
-    for i in range(len(eventLog)):
-      entry = eventLog[i]
+    seenFirstDateDivider = False
+    dividerAttr, duplicateAttr = curses.A_BOLD | uiTools.getColor("yellow"), curses.A_BOLD | uiTools.getColor("green")
+    
+    isDatesShown = self.regexFilter == None and self._config["features.log.showDateDividers"]
+    eventLog = getDaybreaks(self.msgLog, self._isPaused) if isDatesShown else list(self.msgLog)
+    if self.isDuplicatesHidden: deduplicatedLog = getDuplicates(eventLog)
+    else: deduplicatedLog = [(entry, 0) for entry in eventLog]
+    
+    # determines if we have the minimum width to show date dividers
+    showDaybreaks = width - dividerIndent >= 3
+    
+    while deduplicatedLog:
+      entry, duplicateCount = deduplicatedLog.pop(0)
       
       if self.regexFilter and not self.regexFilter.search(entry.getDisplayMessage()):
         continue  # filter doesn't match log message - skip
@@ -580,63 +688,94 @@
       if entry.type == DAYBREAK_EVENT:
         # bottom of the divider
         if seenFirstDateDivider:
-          if lineCount >= 1:
-            self.win.vline(lineCount, xOffset - 1, curses.ACS_LLCORNER | dividerAttr, 1)
-            self.win.hline(lineCount, xOffset, curses.ACS_HLINE | dividerAttr, width - xOffset)
+          if lineCount >= 1 and lineCount < height and showDaybreaks:
+            self.win.vline(lineCount, dividerIndent, curses.ACS_LLCORNER | dividerAttr, 1)
+            self.win.hline(lineCount, dividerIndent + 1, curses.ACS_HLINE | dividerAttr, width - dividerIndent - 1)
             self.win.vline(lineCount, width, curses.ACS_LRCORNER | dividerAttr, 1)
           
           lineCount += 1
         
         # top of the divider
-        if lineCount >= 1 and lineCount < height:
+        if lineCount >= 1 and lineCount < height and showDaybreaks:
           timeLabel = time.strftime(" %B %d, %Y ", time.localtime(entry.timestamp))
-          self.win.vline(lineCount, xOffset - 1, curses.ACS_ULCORNER | dividerAttr, 1)
-          self.win.hline(lineCount, xOffset, curses.ACS_HLINE | dividerAttr, 1)
-          self.addstr(lineCount, xOffset + 1, timeLabel, curses.A_BOLD | dividerAttr)
+          self.win.vline(lineCount, dividerIndent, curses.ACS_ULCORNER | dividerAttr, 1)
+          self.win.hline(lineCount, dividerIndent + 1, curses.ACS_HLINE | dividerAttr, 1)
+          self.addstr(lineCount, dividerIndent + 2, timeLabel, curses.A_BOLD | dividerAttr)
           
-          lineLength = width - xOffset - len(timeLabel) - 1
-          self.win.hline(lineCount, xOffset + len(timeLabel) + 1, curses.ACS_HLINE | dividerAttr, lineLength)
-          self.win.vline(lineCount, xOffset + len(timeLabel) + 1 + lineLength, curses.ACS_URCORNER | dividerAttr, 1)
+          if dividerIndent + len(timeLabel) + 2 <= width:
+            lineLength = width - dividerIndent - len(timeLabel) - 2
+            self.win.hline(lineCount, dividerIndent + len(timeLabel) + 2, curses.ACS_HLINE | dividerAttr, lineLength)
+            self.win.vline(lineCount, dividerIndent + len(timeLabel) + 2 + lineLength, curses.ACS_URCORNER | dividerAttr, 1)
         
         seenFirstDateDivider = True
         lineCount += 1
       else:
-        for line in entry.getDisplayMessage().split("\n"):
-          # splits over too lines if too long
-          if len(line) < width:
-            if lineCount >= 1:
-              if seenFirstDateDivider:
-                self.win.vline(lineCount, xOffset - 1, curses.ACS_VLINE | dividerAttr, 1)
-                self.win.vline(lineCount, width, curses.ACS_VLINE | dividerAttr, 1)
-              
-              self.addstr(lineCount, xOffset, line, uiTools.getColor(entry.color))
-            lineCount += 1
-          else:
-            (line1, line2) = uiTools.splitLine(line, width - xOffset)
-            if lineCount >= 1:
-              if seenFirstDateDivider:
-                self.win.vline(lineCount, xOffset - 1, curses.ACS_VLINE | dividerAttr, 1)
-                self.win.vline(lineCount, width, curses.ACS_VLINE | dividerAttr, 1)
-              
-              self.addstr(lineCount, xOffset, line1, uiTools.getColor(entry.color))
-            if lineCount >= 0 and lineCount + 1 < height:
-              if seenFirstDateDivider:
-                self.win.vline(lineCount + 1, xOffset - 1, curses.ACS_VLINE | dividerAttr, 1)
-                self.win.vline(lineCount + 1, width, curses.ACS_VLINE | dividerAttr, 1)
-              
-              self.addstr(lineCount + 1, xOffset, line2, uiTools.getColor(entry.color))
-            lineCount += 2
+        # entry contents to be displayed, tuples of the form:
+        # (msg, formatting, includeLinebreak)
+        displayQueue = []
+        
+        msgComp = entry.getDisplayMessage().split("\n")
+        for i in range(len(msgComp)):
+          displayQueue.append((msgComp[i].strip(), uiTools.getColor(entry.color), i != len(msgComp) - 1))
+        
+        if duplicateCount:
+          pluralLabel = "s" if duplicateCount > 1 else ""
+          duplicateMsg = DUPLICATE_MSG % (duplicateCount, pluralLabel)
+          displayQueue.append((duplicateMsg, duplicateAttr, False))
+        
+        cursorLoc, lineOffset = msgIndent, 0
+        maxEntriesPerLine = self._config["features.log.maxLinesPerEntry"]
+        while displayQueue:
+          msg, format, includeBreak = displayQueue.pop(0)
+          drawLine = lineCount + lineOffset
+          if lineOffset == maxEntriesPerLine: break
+          
+          maxMsgSize = width - cursorLoc
+          if len(msg) >= maxMsgSize:
+            # message is too long - break it up
+            if lineOffset == maxEntriesPerLine - 1:
+              msg = uiTools.cropStr(msg, maxMsgSize)
+            else:
+              msg, remainder = uiTools.cropStr(msg, maxMsgSize, 4, 4, uiTools.END_WITH_HYPHEN, True)
+              displayQueue.insert(0, (remainder.strip(), format, includeBreak))
+            
+            includeBreak = True
+          
+          if drawLine < height and drawLine >= 1:
+            if seenFirstDateDivider and width - dividerIndent >= 3 and showDaybreaks:
+              self.win.vline(drawLine, dividerIndent, curses.ACS_VLINE | dividerAttr, 1)
+              self.win.vline(drawLine, width, curses.ACS_VLINE | dividerAttr, 1)
+            
+            self.addstr(drawLine, cursorLoc, msg, format)
+          
+          cursorLoc += len(msg)
+          
+          if includeBreak or not displayQueue:
+            lineOffset += 1
+            cursorLoc = msgIndent + ENTRY_INDENT
+        
+        lineCount += lineOffset
       
       # if this is the last line and there's room, then draw the bottom of the divider
-      isLastLine = i == len(eventLog) - 1
-      if isLastLine and seenFirstDateDivider and lineCount < height:
-        self.win.vline(lineCount, xOffset - 1, curses.ACS_LLCORNER | dividerAttr, 1)
-        self.win.hline(lineCount, xOffset, curses.ACS_HLINE | dividerAttr, width - xOffset)
-        self.win.vline(lineCount, width, curses.ACS_LRCORNER | dividerAttr, 1)
+      if not deduplicatedLog and seenFirstDateDivider:
+        if lineCount < height and showDaybreaks:
+          # when resizing with a small width the following entries can be
+          # problematc (though I'm not sure why)
+          try:
+            self.win.vline(lineCount, dividerIndent, curses.ACS_LLCORNER | dividerAttr, 1)
+            self.win.hline(lineCount, dividerIndent + 1, curses.ACS_HLINE | dividerAttr, width - dividerIndent - 1)
+            self.win.vline(lineCount, width, curses.ACS_LRCORNER | dividerAttr, 1)
+          except: pass
+        
         lineCount += 1
-      
-      if lineCount >= height: break # further log messages wouldn't fit
     
+    self.lastContentHeight = lineCount + self.scroll - 1
+    
+    # if we're off the bottom of the page then redraw the content with the
+    # corrected lastContentHeight
+    if self.lastContentHeight > height and self.scroll + height - 1 > self.lastContentHeight:
+      self.draw(subwindow, width, height)
+    
     self.valsLock.release()
   
   def redraw(self, forceRedraw=False, block=False):
@@ -783,50 +922,3 @@
     self.valsLock.release()
     return panelLabel
   
-  def _getContentLength(self):
-    """
-    Provides the number of lines the log's contents would currently occupy,
-    taking into account filtered/wrapped lines, the scroll bar, etc.
-    """
-    
-    if self._config["features.log.showDateDividers"]:
-      timezoneOffset = time.altzone if time.localtime()[8] else time.timezone
-      currentDay = int((time.time() - timezoneOffset) / 86400)
-    else: currentDay = 0
-    
-    # if the arguments haven't changed then we can use cached results
-    self.valsLock.acquire()
-    height, width = self.getPreferredSize()
-    currentPattern = self.regexFilter.pattern if self.regexFilter else None
-    isUnchanged = self._contentLengthArgs[0] == self.msgLog
-    isUnchanged &= self._contentLengthArgs[1] == currentPattern
-    isUnchanged &= self._contentLengthArgs[2] == height
-    isUnchanged &= self._contentLengthArgs[3] == width
-    isUnchanged &= self._contentLengthArgs[4] == currentDay
-    if isUnchanged:
-      self.valsLock.release()
-      return self._contentLengthCache
-    
-    contentLengths = [0, 0] # length of the content without and with a scroll bar
-    eventLog = getDaybreaks(self.msgLog) if self._config["features.log.showDateDividers"] else self.msgLog
-    for entry in eventLog:
-      if not self.regexFilter or self.regexFilter.search(entry.getDisplayMessage()):
-        if entry.type == DAYBREAK_EVENT:
-          contentLengths[0] += 2
-          contentLengths[1] += 2
-        else:
-          for line in entry.getDisplayMessage().split("\n"):
-            if len(line) >= width: contentLengths[0] += 2
-            else: contentLengths[0] += 1
-            
-            if len(line) >= width - 3: contentLengths[1] += 2
-            else: contentLengths[1] += 1
-    
-    # checks if the scroll bar would be displayed to determine the actual length
-    actualLength = contentLengths[0] if contentLengths[0] <= height - 1 else contentLengths[1]
-    
-    self._contentLengthCache = actualLength
-    self._contentLengthArgs = (list(self.msgLog), currentPattern, height, width, currentDay)
-    self.valsLock.release()
-    return actualLength
-

Modified: arm/trunk/src/util/conf.py
===================================================================
--- arm/trunk/src/util/conf.py	2010-09-29 10:57:17 UTC (rev 23334)
+++ arm/trunk/src/util/conf.py	2010-09-29 16:30:48 UTC (rev 23335)
@@ -20,7 +20,8 @@
 import log
 
 CONFS = {}  # mapping of identifier to singleton instances of configs
-CONFIG = {"log.configEntryNotFound": None, "log.configEntryTypeError": log.INFO}
+CONFIG = {"log.configEntryNotFound": None,
+          "log.configEntryTypeError": log.INFO}
 
 def loadConfig(config):
   config.update(CONFIG)

Modified: arm/trunk/src/util/connections.py
===================================================================
--- arm/trunk/src/util/connections.py	2010-09-29 10:57:17 UTC (rev 23334)
+++ arm/trunk/src/util/connections.py	2010-09-29 16:30:48 UTC (rev 23335)
@@ -51,7 +51,11 @@
 RESOLVER_FAILURE_TOLERANCE = 3      # number of subsequent failures before moving on to another resolver
 RESOLVER_SERIAL_FAILURE_MSG = "Querying connections with %s failed, trying %s"
 RESOLVER_FINAL_FAILURE_MSG = "All connection resolvers failed"
-CONFIG = {"queries.connections.minRate": 5, "log.connLookupFailed": log.INFO, "log.connLookupFailover": log.NOTICE, "log.connLookupAbandon": log.WARN, "log.connLookupRateGrowing": None}
+CONFIG = {"queries.connections.minRate": 5,
+          "log.connLookupFailed": log.INFO,
+          "log.connLookupFailover": log.NOTICE,
+          "log.connLookupAbandon": log.WARN,
+          "log.connLookupRateGrowing": None}
 
 def loadConfig(config):
   config.update(CONFIG)

Modified: arm/trunk/src/util/hostnames.py
===================================================================
--- arm/trunk/src/util/hostnames.py	2010-09-29 10:57:17 UTC (rev 23334)
+++ arm/trunk/src/util/hostnames.py	2010-09-29 16:30:48 UTC (rev 23335)
@@ -41,7 +41,11 @@
 DNS_ERROR_CODES = ("1(FORMERR)", "2(SERVFAIL)", "3(NXDOMAIN)", "4(NOTIMP)", "5(REFUSED)", "6(YXDOMAIN)",
                    "7(YXRRSET)", "8(NXRRSET)", "9(NOTAUTH)", "10(NOTZONE)", "16(BADVERS)")
 
-CONFIG = {"queries.hostnames.poolSize": 5, "queries.hostnames.useSocketModule": False, "cache.hostnames.size": 700000, "cache.hostnames.trimSize": 200000, "log.hostnameCacheTrimmed": log.INFO}
+CONFIG = {"queries.hostnames.poolSize": 5,
+          "queries.hostnames.useSocketModule": False,
+          "cache.hostnames.size": 700000,
+          "cache.hostnames.trimSize": 200000,
+          "log.hostnameCacheTrimmed": log.INFO}
 
 def loadConfig(config):
   config.update(CONFIG)

Modified: arm/trunk/src/util/log.py
===================================================================
--- arm/trunk/src/util/log.py	2010-09-29 10:57:17 UTC (rev 23334)
+++ arm/trunk/src/util/log.py	2010-09-29 16:30:48 UTC (rev 23335)
@@ -24,7 +24,8 @@
 # mapping of runlevels to the listeners interested in receiving events from it
 _listeners = dict([(level, []) for level in range(1, 6)])
 
-CONFIG = {"cache.armLog.size": 1000, "cache.armLog.trimSize": 200}
+CONFIG = {"cache.armLog.size": 1000,
+          "cache.armLog.trimSize": 200}
 
 def loadConfig(config):
   config.update(CONFIG)

Modified: arm/trunk/src/util/panel.py
===================================================================
--- arm/trunk/src/util/panel.py	2010-09-29 10:57:17 UTC (rev 23334)
+++ arm/trunk/src/util/panel.py	2010-09-29 16:30:48 UTC (rev 23335)
@@ -2,6 +2,8 @@
 Wrapper for safely working with curses subwindows.
 """
 
+import sys
+import traceback
 import curses
 from threading import RLock
 
@@ -222,6 +224,15 @@
         self.win.erase() # clears any old contents
         self.draw(self.win, self.maxX - 1, self.maxY)
       self.win.refresh()
+    except:
+      # without terminating curses continues in a zombie state (requiring a
+      # kill signal to quit, and screwing up the terminal)
+      # TODO: provide a nicer, general purpose handler for unexpected exceptions
+      try:
+        tracebackFile = open("/tmp/armTraceback", "w")
+        traceback.print_exc(file=tracebackFile)
+      finally:
+        sys.exit(1)
     finally:
       CURSES_LOCK.release()
   

Modified: arm/trunk/src/util/sysTools.py
===================================================================
--- arm/trunk/src/util/sysTools.py	2010-09-29 10:57:17 UTC (rev 23334)
+++ arm/trunk/src/util/sysTools.py	2010-09-29 16:30:48 UTC (rev 23335)
@@ -16,7 +16,11 @@
 IS_FAILURES_CACHED = True           # caches both successful and failed results if true
 CALL_CACHE_LOCK = threading.RLock() # governs concurrent modifications of CALL_CACHE
 
-CONFIG = {"cache.sysCalls.size": 600, "log.sysCallMade": log.DEBUG, "log.sysCallCached": None, "log.sysCallFailed": log.INFO, "log.sysCallCacheGrowing": log.INFO}
+CONFIG = {"cache.sysCalls.size": 600,
+          "log.sysCallMade": log.DEBUG,
+          "log.sysCallCached": None,
+          "log.sysCallFailed": log.INFO,
+          "log.sysCallCacheGrowing": log.INFO}
 
 def loadConfig(config):
   config.update(CONFIG)

Modified: arm/trunk/src/util/torTools.py
===================================================================
--- arm/trunk/src/util/torTools.py	2010-09-29 10:57:17 UTC (rev 23334)
+++ arm/trunk/src/util/torTools.py	2010-09-29 16:30:48 UTC (rev 23335)
@@ -47,7 +47,9 @@
 
 TOR_CTL_CLOSE_MSG = "Tor closed control connection. Exiting event thread."
 UNKNOWN = "UNKNOWN" # value used by cached information if undefined
-CONFIG = {"log.torCtlPortClosed": log.NOTICE, "log.torGetInfo": log.DEBUG, "log.torGetConf": log.DEBUG}
+CONFIG = {"log.torCtlPortClosed": log.NOTICE,
+          "log.torGetInfo": log.DEBUG,
+          "log.torGetConf": log.DEBUG}
 
 # events used for controller functionality:
 # NOTICE - used to detect when tor is shut down

Modified: arm/trunk/src/util/uiTools.py
===================================================================
--- arm/trunk/src/util/uiTools.py	2010-09-29 10:57:17 UTC (rev 23334)
+++ arm/trunk/src/util/uiTools.py	2010-09-29 16:30:48 UTC (rev 23335)
@@ -31,8 +31,10 @@
 TIME_UNITS = [(86400.0, "d", " day"),                   (3600.0, "h", " hour"),
               (60.0, "m", " minute"),                   (1.0, "s", " second")]
 
+END_WITH_ELLIPSE, END_WITH_HYPHEN = range(1, 3)
 SCROLL_KEYS = (curses.KEY_UP, curses.KEY_DOWN, curses.KEY_PPAGE, curses.KEY_NPAGE, curses.KEY_HOME, curses.KEY_END)
-CONFIG = {"features.colorInterface": True, "log.cursesColorSupport": log.INFO}
+CONFIG = {"features.colorInterface": True,
+          "log.cursesColorSupport": log.INFO}
 
 def loadConfig(config):
   config.update(CONFIG)
@@ -54,7 +56,7 @@
   if not COLOR_ATTR_INITIALIZED: _initColors()
   return COLOR_ATTR[color]
 
-def cropStr(msg, size, minWordLen = 4, addEllipse = True):
+def cropStr(msg, size, minWordLen = 4, minCrop = 0, endType = END_WITH_ELLIPSE, getRemainder = False):
   """
   Provides the msg constrained to the given length, truncating on word breaks.
   If the last words is long this truncates mid-word with an ellipse. If there
@@ -71,74 +73,59 @@
   ""
   
   Arguments:
-    msg        - source text
-    size       - room available for text
-    minWordLen - minimum characters before which a word is dropped, requires
-                 whole word if -1
-    addEllipse - includes an ellipse when truncating if true (dropped if size
-                 size is 
+    msg          - source text
+    size         - room available for text
+    minWordLen   - minimum characters before which a word is dropped, requires
+                   whole word if None
+    minCrop      - minimum characters that must be dropped if a word's cropped
+    endType      - type of ending used when truncating:
+                   None - blank ending
+                   END_WITH_ELLIPSE - includes an ellipse
+                   END_WITH_HYPHEN - adds hyphen when breaking words
+    getRemainder - returns a tuple instead, with the second part being the
+                   cropped portion of the message
   """
   
-  if minWordLen < 0: minWordLen = sys.maxint
+  if minWordLen == None: minWordLen = sys.maxint
+  minWordLen = max(0, minWordLen)
+  minCrop = max(0, minCrop)
   
-  if len(msg) <= size: return msg
-  else:
-    msgWords = msg.split(" ")
-    msgWords.reverse()
-    
-    returnWords = []
-    sizeLeft = size - 3 if addEllipse else size
-    
-    # checks that there's room for at least one word
-    if min(minWordLen, len(msgWords[-1])) > sizeLeft: return ""
-    
-    while sizeLeft > 0:
-      nextWord = msgWords.pop()
-      
-      if len(nextWord) <= sizeLeft:
-        returnWords.append(nextWord)
-        sizeLeft -= (len(nextWord) + 1)
-      elif minWordLen <= sizeLeft:
-        returnWords.append(nextWord[:sizeLeft])
-        sizeLeft = 0
-      else: sizeLeft = 0
-    
-    returnMsg = " ".join(returnWords)
-    if addEllipse: returnMsg += "..."
-    return returnMsg
-
-def splitLine(message, width, indent = "  "):
-  """
-  Divides message into two lines, attempting to do it on a wordbreak. This
-  adds an ellipse if the second line is too long.
+  # checks if there's room for the whole message
+  if len(msg) <= size:
+    if getRemainder: return (msg, "")
+    else: return msg
   
-  Arguments:
-    message - string being divided
-    width   - maximum width constraint for the split
-    indent  - addition made to the start of the second line
-  """
+  # since we're cropping, the effective space available is less with an
+  # ellipse, and cropping words requires an extra space for hyphens
+  if endType == END_WITH_ELLIPSE: size -= 3
+  elif endType == END_WITH_HYPHEN: minWordLen += 1
   
-  if len(message) < width: return (message, "")
+  # checks if there isn't the minimum space needed to include anything
+  if size <= minWordLen:
+    if getRemainder: return ("", msg)
+    else: return ""
   
-  lastWordbreak = message[:width].rfind(" ")
-  if width - lastWordbreak < 10:
-    line1 = message[:lastWordbreak]
-    line2 = "%s%s" % (indent, message[lastWordbreak:].strip())
-  else:
-    # over ten characters until the last word - dividing
-    line1 = "%s-" % message[:width - 2]
-    line2 = "%s%s" % (indent, message[width - 2:].strip())
+  lastWordbreak = msg.rfind(" ", 0, size + 1)
+  includeCrop = size - lastWordbreak - 1 >= minWordLen
   
-  # ends line with ellipsis if too long
-  if len(line2) > width:
-    lastWordbreak = line2[:width - 4].rfind(" ")
-    
-    # doesn't use wordbreak if it's a long word or the whole line is one 
-    # word (picking up on two space indent to have index 1)
-    if width - lastWordbreak > 10 or lastWordbreak == 1: lastWordbreak = width - 4
-    line2 = "%s..." % line2[:lastWordbreak]
+  # if there's a max crop size then make sure we're cropping at least that many characters
+  if includeCrop and minCrop:
+    nextWordbreak = msg.find(" ", size)
+    if nextWordbreak == -1: nextWordbreak = len(msg)
+    includeCrop = nextWordbreak - size + 1 >= minCrop
   
-  return (line1, line2)
+  if includeCrop:
+    returnMsg, remainder = msg[:size], msg[size:]
+    if endType == END_WITH_HYPHEN: returnMsg = returnMsg[:-1] + "-"
+  else: returnMsg, remainder = msg[:lastWordbreak], msg[lastWordbreak:]
+  
+  # if this is ending with a comma or period then strip it off
+  if returnMsg[-1] in (",", "."): returnMsg = returnMsg[:-1]
+  
+  if endType == END_WITH_ELLIPSE: returnMsg += "..."
+  
+  if getRemainder: return (returnMsg, remainder)
+  else: return returnMsg
 
 def isScrollKey(key):
   """