[Author Prev][Author Next][Thread Prev][Thread Next][Author Index][Thread Index]
[or-cvs] r20678: {arm} More issues discussed on irc. fix: made netstat lookups a be (in arm/trunk: . interface)
Author: atagar
Date: 2009-09-28 02:56:21 -0400 (Mon, 28 Sep 2009)
New Revision: 20678
Added:
arm/trunk/interface/connResolver.py
Modified:
arm/trunk/ChangeLog
arm/trunk/TODO
arm/trunk/arm.py
arm/trunk/interface/bandwidthMonitor.py
arm/trunk/interface/connCountMonitor.py
arm/trunk/interface/connPanel.py
arm/trunk/interface/controller.py
arm/trunk/interface/cpuMemMonitor.py
arm/trunk/interface/logPanel.py
Log:
More issues discussed on irc.
fix: made netstat lookups a best-effort service, separate from draw thread (caught by arma and StrangeCharm)
fix: using ps as final fallback if otherwise unable to determine pid (suggested by Sebastian)
fix: appends tor's pwd if torrc path is relative (caught by arma)
Modified: arm/trunk/ChangeLog
===================================================================
--- arm/trunk/ChangeLog 2009-09-28 00:51:28 UTC (rev 20677)
+++ arm/trunk/ChangeLog 2009-09-28 06:56:21 UTC (rev 20678)
@@ -1,6 +1,13 @@
CHANGE LOG
-9/27/09 - version 1.1.2
+9/28/09 - version 1.1.3
+More issues discussed on irc.
+
+ * fix: made netstat lookups a best-effort service, separate from draw thread (caught by arma and StrangeCharm)
+ * fix: using ps as final fallback if otherwise unable to determine pid (suggested by Sebastian)
+ * fix: appends tor's pwd if torrc path is relative (caught by arma)
+
+9/27/09 - version 1.1.2 (r20674)
Few issues discussed on irc.
* added: changelog and cleaned up todo documents (requested by arma)
Modified: arm/trunk/TODO
===================================================================
--- arm/trunk/TODO 2009-09-28 00:51:28 UTC (rev 20677)
+++ arm/trunk/TODO 2009-09-28 06:56:21 UTC (rev 20678)
@@ -1,11 +1,7 @@
TODO
- Bugs
- * make netstat lookups a best-effort service (separate from draw thread)
- Call appears to be heavier than expected and causing display to be
- unusable on especially active relays (like directory servers).
- caught by arma and StrangeCharm, notify coderman for testing
- * Mac OSX and BSD may have issues with netstat options
+ * Mac OSX and BSD have issues with netstat options
Reported that they aren't cross platform. Possibly use lsof as a
fallback if an issue's detected.
caught by Christopher Davis
@@ -13,14 +9,8 @@
Not sure how to address this - problem is that the calls to 'host' can
take a while to time out. Might need another thread to kill the calls?
Or forcefully terminate thread if it's taking too long (might be noisy)?
- * connection details covers right side
* version labels provided on Debian are longer than expected
caught by hexa
- * unable to load torrc if it was loaded via a relative path
- When tor's started via "tor -f <relative path>" we don't know what it's
- relative of - check to see if there's a way of finding the pwd of
- another process.
- caught by arma
* new connections don't have uptime tracked when not visible
Previous fix attempted to resolve, but evidently didn't work.
Modified: arm/trunk/arm.py
===================================================================
--- arm/trunk/arm.py 2009-09-28 00:51:28 UTC (rev 20677)
+++ arm/trunk/arm.py 2009-09-28 06:56:21 UTC (rev 20678)
@@ -19,8 +19,8 @@
from interface import controller
from interface import logPanel
-VERSION = "1.1.2"
-LAST_MODIFIED = "Sep 27, 2009"
+VERSION = "1.1.3"
+LAST_MODIFIED = "Sep 28, 2009"
DEFAULT_CONTROL_ADDR = "127.0.0.1"
DEFAULT_CONTROL_PORT = 9051
Modified: arm/trunk/interface/bandwidthMonitor.py
===================================================================
--- arm/trunk/interface/bandwidthMonitor.py 2009-09-28 00:51:28 UTC (rev 20677)
+++ arm/trunk/interface/bandwidthMonitor.py 2009-09-28 06:56:21 UTC (rev 20678)
@@ -99,7 +99,6 @@
def getHeaderLabel(self, width, isPrimary):
graphType = "Downloaded" if isPrimary else "Uploaded"
- total = self.primaryTotal if isPrimary else self.secondaryTotal
stats = [""]
# conditional is to avoid flickering as stats change size for tty terminals
Modified: arm/trunk/interface/connCountMonitor.py
===================================================================
--- arm/trunk/interface/connCountMonitor.py 2009-09-28 00:51:28 UTC (rev 20677)
+++ arm/trunk/interface/connCountMonitor.py 2009-09-28 06:56:21 UTC (rev 20678)
@@ -2,55 +2,42 @@
# connCountMonitor.py -- Tracks the number of connections made by Tor.
# Released under the GPL v3 (http://www.gnu.org/licenses/gpl.html)
-import os
-import time
from TorCtl import TorCtl
-import connPanel
import graphPanel
class ConnCountMonitor(graphPanel.GraphStats, TorCtl.PostEventListener):
"""
- Tracks number of connections, using cached values in connPanel if recent
- enough (otherwise retrieved independently). Client connections are counted
- as outbound.
+ Tracks number of connections, counting client and directory connections as
+ outbound.
"""
- def __init__(self, connectionPanel):
+ def __init__(self, conn, connResolver):
graphPanel.GraphStats.__init__(self)
TorCtl.PostEventListener.__init__(self)
graphPanel.GraphStats.initialize(self, "green", "cyan", 10)
- self.connectionPanel = connectionPanel # connection panel, used to limit netstat calls
+ self.connResolver = connResolver # thread performing netstat queries
+ self.orPort = conn.get_option("ORPort")[0][1]
+ self.dirPort = conn.get_option("DirPort")[0][1]
+ self.controlPort = conn.get_option("ControlPort")[0][1]
def bandwidth_event(self, event):
# doesn't use events but this keeps it in sync with the bandwidth panel
# (and so it stops if Tor stops - used to use a separate thread but this
# is better)
- if self.connectionPanel.lastUpdate + 1 >= time.time():
- # reuses netstat results if recent enough
- counts = self.connectionPanel.connectionCount
- self._processEvent(counts[0], counts[1] + counts[2] + counts[3])
- else:
- # cached results stale - requery netstat
- inbound, outbound, control = 0, 0, 0
- netstatCall = os.popen("netstat -npt 2> /dev/null | grep %s/tor 2> /dev/null" % self.connectionPanel.pid)
- try:
- results = netstatCall.readlines()
-
- for line in results:
- if not line.startswith("tcp"): continue
- param = line.split()
- localPort = param[3][param[3].find(":") + 1:]
-
- if localPort in (self.connectionPanel.orPort, self.connectionPanel.dirPort): inbound += 1
- elif localPort == self.connectionPanel.controlPort: control += 1
- else: outbound += 1
- except IOError:
- # netstat call failed
- self.connectionPanel.monitor_event("WARN", "Unable to query netstat for connection counts")
+ inbound, outbound, control = 0, 0, 0
+ results = self.connResolver.getConnections()
+
+ for line in results:
+ if not line.startswith("tcp"): continue
+ param = line.split()
+ localPort = param[3][param[3].find(":") + 1:]
- netstatCall.close()
- self._processEvent(inbound, outbound)
+ if localPort in (self.orPort, self.dirPort): inbound += 1
+ elif localPort == self.controlPort: control += 1
+ else: outbound += 1
+
+ self._processEvent(inbound, outbound)
def getTitle(self, width):
return "Connection Count:"
Modified: arm/trunk/interface/connPanel.py
===================================================================
--- arm/trunk/interface/connPanel.py 2009-09-28 00:51:28 UTC (rev 20677)
+++ arm/trunk/interface/connPanel.py 2009-09-28 06:56:21 UTC (rev 20678)
@@ -2,7 +2,6 @@
# connPanel.py -- Lists network connections used by tor.
# Released under the GPL v3 (http://www.gnu.org/licenses/gpl.html)
-import os
import time
import socket
import curses
@@ -98,19 +97,19 @@
Lists netstat provided network data of tor.
"""
- def __init__(self, lock, conn, torPid, logger):
+ def __init__(self, lock, conn, connResolver, logger):
TorCtl.PostEventListener.__init__(self)
util.Panel.__init__(self, lock, -1)
self.scroll = 0
- self.conn = conn # tor connection for querrying country codes
- self.logger = logger # notified in case of problems
- self.pid = torPid # tor process ID to make sure we've got the right instance
- self.listingType = LIST_IP # information used in listing entries
- self.allowDNS = True # permits hostname resolutions if true
- self.showLabel = True # shows top label if true, hides otherwise
- self.showingDetails = False # augments display to accomidate details window if true
- self.lastUpdate = -1 # time last stats was retrived
- self.localhostEntry = None # special connection - tuple with (entry for this node, fingerprint)
+ self.conn = conn # tor connection for querrying country codes
+ self.connResolver = connResolver # thread performing netstat queries
+ self.logger = logger # notified in case of problems
+ self.listingType = LIST_IP # information used in listing entries
+ self.allowDNS = True # permits hostname resolutions if true
+ self.showLabel = True # shows top label if true, hides otherwise
+ self.showingDetails = False # augments display to accomidate details window if true
+ self.lastUpdate = -1 # time last stats was retrived
+ self.localhostEntry = None # special connection - tuple with (entry for this node, fingerprint)
self.sortOrdering = [ORD_TYPE, ORD_FOREIGN_LISTING, ORD_FOREIGN_PORT]
self.resolver = hostnameResolver.HostnameResolver()
self.fingerprintLookupCache = {} # chache of (ip, port) -> fingerprint
@@ -123,6 +122,7 @@
self.clientConnectionCache = None # listing of nicknames for our client connections
self.clientConnectionLock = RLock() # lock for clientConnectionCache
self.isDisabled = False # prevent panel from updating entirely
+ self.lastNetstatResults = None # used to check if raw netstat results have changed
self.isCursorEnabled = True
self.cursorSelection = None
@@ -206,7 +206,6 @@
Reloads netstat results.
"""
- if not self.pid or self.isDisabled: return
self.connectionsLock.acquire()
self.clientConnectionLock.acquire()
@@ -223,63 +222,55 @@
for entry in (self.connections if not self.isPaused else self.connectionsBuffer):
connTimes[(entry[CONN_F_IP], entry[CONN_F_PORT])] = entry[CONN_TIME]
- # looks at netstat for tor with stderr redirected to /dev/null, options are:
- # n = prevents dns lookups, p = include process (say if it's tor), t = tcp only
- netstatCall = os.popen("netstat -npt 2> /dev/null | grep %s/tor 2> /dev/null" % self.pid)
+ results = self.connResolver.getConnections()
+ if results == self.lastNetstatResults: return # contents haven't changed
- try:
- results = netstatCall.readlines()
+ for line in results:
+ if not line.startswith("tcp"): continue
+ param = line.split()
+ local, foreign = param[3], param[4]
+ localIP, foreignIP = local[:local.find(":")], foreign[:foreign.find(":")]
+ localPort, foreignPort = local[len(localIP) + 1:], foreign[len(foreignIP) + 1:]
- for line in results:
- if not line.startswith("tcp"): continue
- param = line.split()
- local, foreign = param[3], param[4]
- localIP, foreignIP = local[:local.find(":")], foreign[:foreign.find(":")]
- localPort, foreignPort = local[len(localIP) + 1:], foreign[len(foreignIP) + 1:]
+ if localPort in (self.orPort, self.dirPort):
+ type = "inbound"
+ connectionCountTmp[0] += 1
+ elif localPort == self.controlPort:
+ type = "control"
+ connectionCountTmp[4] += 1
+ else:
+ fingerprint = self.getFingerprint(foreignIP, foreignPort)
+ nickname = self.getNickname(foreignIP, foreignPort)
- if localPort in (self.orPort, self.dirPort):
- type = "inbound"
- connectionCountTmp[0] += 1
- elif localPort == self.controlPort:
- type = "control"
- connectionCountTmp[4] += 1
- else:
- fingerprint = self.getFingerprint(foreignIP, foreignPort)
- nickname = self.getNickname(foreignIP, foreignPort)
-
- isClient = False
- for clientName in self.clientConnectionCache:
- if nickname == clientName or (len(clientName) > 1 and clientName[0] == "$" and fingerprint == clientName[1:]):
- isClient = True
- break
-
- if isClient:
- type = "client"
- connectionCountTmp[2] += 1
- elif (foreignIP, foreignPort) in DIR_SERVERS:
- type = "directory"
- connectionCountTmp[3] += 1
- else:
- type = "outbound"
- connectionCountTmp[1] += 1
+ isClient = False
+ for clientName in self.clientConnectionCache:
+ if nickname == clientName or (len(clientName) > 1 and clientName[0] == "$" and fingerprint == clientName[1:]):
+ isClient = True
+ break
- try:
- countryCodeQuery = "ip-to-country/%s" % foreign[:foreign.find(":")]
- countryCode = self.conn.get_info(countryCodeQuery)[countryCodeQuery]
- except (socket.error, TorCtl.ErrorReply):
- countryCode = "??"
- if not self.providedGeoipWarning:
- self.logger.monitor_event("WARN", "Tor geoip database is unavailable.")
- self.providedGeoipWarning = True
-
- if (foreignIP, foreignPort) in connTimes: connTime = connTimes[(foreignIP, foreignPort)]
- else: connTime = time.time()
-
- connectionsTmp.append((type, localIP, localPort, foreignIP, foreignPort, countryCode, connTime))
- except IOError:
- # netstat call failed
- self.logger.monitor_event("WARN", "Unable to query netstat for new connections")
- return
+ if isClient:
+ type = "client"
+ connectionCountTmp[2] += 1
+ elif (foreignIP, foreignPort) in DIR_SERVERS:
+ type = "directory"
+ connectionCountTmp[3] += 1
+ else:
+ type = "outbound"
+ connectionCountTmp[1] += 1
+
+ try:
+ countryCodeQuery = "ip-to-country/%s" % foreign[:foreign.find(":")]
+ countryCode = self.conn.get_info(countryCodeQuery)[countryCodeQuery]
+ except (socket.error, TorCtl.ErrorReply):
+ countryCode = "??"
+ if not self.providedGeoipWarning:
+ self.logger.monitor_event("WARN", "Tor geoip database is unavailable.")
+ self.providedGeoipWarning = True
+
+ if (foreignIP, foreignPort) in connTimes: connTime = connTimes[(foreignIP, foreignPort)]
+ else: connTime = time.time()
+
+ connectionsTmp.append((type, localIP, localPort, foreignIP, foreignPort, countryCode, connTime))
# appends localhost connection to allow user to look up their own consensus entry
selfAddress, selfPort, selfFingerprint = None, None, None
@@ -304,7 +295,6 @@
else:
self.localhostEntry = None
- netstatCall.close()
self.lastUpdate = time.time()
# assigns results
@@ -316,6 +306,7 @@
# hostnames are sorted at redraw - otherwise now's a good time
if self.listingType != LIST_HOSTNAME: self.sortConnections()
+ self.lastNetstatResults = results
finally:
self.connectionsLock.release()
self.clientConnectionLock.release()
Added: arm/trunk/interface/connResolver.py
===================================================================
--- arm/trunk/interface/connResolver.py (rev 0)
+++ arm/trunk/interface/connResolver.py 2009-09-28 06:56:21 UTC (rev 20678)
@@ -0,0 +1,90 @@
+#!/usr/bin/env python
+# connResolver.py -- Background thread for retrieving tor's connections.
+# Released under the GPL v3 (http://www.gnu.org/licenses/gpl.html)
+
+import os
+import time
+from threading import Thread
+from threading import RLock
+
+MIN_LOOKUP_WAIT = 5 # minimum seconds between lookups
+SLEEP_INTERVAL = 1 # period to sleep when not making a netstat call
+FAILURE_TOLERANCE = 3 # number of subsiquent failures tolerated before pausing thread
+FAILURE_MSG = "Unable to query netstat for new connections"
+SERIAL_FAILURE_MSG = "Failing to query netstat (connection related portions of the monitor won't function)"
+
+class ConnResolver(Thread):
+ """
+ Service that periodically queries Tor's current connections to allow for a
+ best effort, non-blocking lookup. This is currently implemented via netstat.
+ In case of failure this gives an INFO level warning and provides the last
+ known results. This process provides an WARN level warning and pauses itself
+ if there's several subsiquent failures (probably indicating that netstat
+ isn't available).
+ """
+
+ def __init__(self, pid, logPanel):
+ Thread.__init__(self)
+ self.pid = pid # tor process ID to make sure we've got the right instance
+ self.logger = logPanel # used to notify of lookup failures
+
+ self.connections = [] # unprocessed lines from netstat results
+ self.connectionsLock = RLock() # limits concurrent access to connections
+ self.isPaused = False
+ self.halt = False # terminates thread if true
+ self.lastLookup = -1 # time of last lookup (reguardless of success)
+ self.subsiquentFailures = 0 # number of failed netstat calls in a row
+ self.setDaemon(True)
+
+ def getConnections(self):
+ """
+ Provides the last querried connection results.
+ """
+
+ connectionsTmp = None
+
+ self.connectionsLock.acquire()
+ try: connectionsTmp = list(self.connections)
+ finally: self.connectionsLock.release()
+
+ return connectionsTmp
+
+ def run(self):
+ if not self.pid: return
+
+ while not self.halt:
+ if self.isPaused or time.time() - MIN_LOOKUP_WAIT < self.lastLookup: time.sleep(SLEEP_INTERVAL)
+ else:
+ try:
+ # looks at netstat for tor with stderr redirected to /dev/null, options are:
+ # n = prevents dns lookups, p = include process (say if it's tor), t = tcp only
+ netstatCall = os.popen("netstat -npt 2> /dev/null | grep %s/tor 2> /dev/null" % self.pid)
+ results = netstatCall.readlines()
+ if not results: raise IOError
+
+ # assign obtained results
+ self.connectionsLock.acquire()
+ try: self.connections = results
+ finally: self.connectionsLock.release()
+
+ self.subsiquentFailures = 0
+ except IOError:
+ # netstat call failed
+ self.subsiquentFailures += 1
+ self.logger.monitor_event("INFO", "%s (%i)" % (FAILURE_MSG, self.subsiquentFailures))
+
+ if self.subsiquentFailures >= FAILURE_TOLERANCE:
+ self.logger.monitor_event("WARN", SERIAL_FAILURE_MSG)
+ self.setPaused(True)
+ finally:
+ self.lastLookup = time.time()
+ netstatCall.close()
+
+ def setPaused(self, isPause):
+ """
+ If true, prevents further netstat lookups.
+ """
+
+ if isPause == self.isPaused: return
+ self.isPaused = isPause
+
Modified: arm/trunk/interface/controller.py
===================================================================
--- arm/trunk/interface/controller.py 2009-09-28 00:51:28 UTC (rev 20677)
+++ arm/trunk/interface/controller.py 2009-09-28 06:56:21 UTC (rev 20678)
@@ -10,6 +10,7 @@
import os
import time
import curses
+import socket
from threading import RLock
from TorCtl import TorCtl
@@ -21,6 +22,7 @@
import descriptorPopup
import util
+import connResolver
import bandwidthMonitor
import cpuMemMonitor
import connCountMonitor
@@ -262,13 +264,37 @@
except IOError: pass # netstat call failed
netstatCall.close()
+ if not torPid:
+ try:
+ # third try, use ps if there's only one possability
+ psCall = os.popen("ps -o pid -C tor 2> /dev/null")
+ results = psCall.readlines()
+ if len(results) == 2 and len(results[0].split()) == 1: torPid = results[1].strip()
+ except IOError: pass # ps call failed
+ psCall.close()
+
+ confLocation = conn.get_info("config-file")["config-file"]
+ if confLocation[0] != "/":
+ # relative path - attempt to add process pwd
+ try:
+ pwdxCall = os.popen("pwdx %s 2> /dev/null" % torPid)
+ results = pwdxCall.readlines()
+ if len(results) == 1 and len(results[0].split()) == 2: confLocation = "%s/%s" % (results[0].split()[1], confLocation)
+ except IOError: pass # pwdx call failed
+ pwdxCall.close()
+
panels = {
"header": headerPanel.HeaderPanel(cursesLock, conn, torPid),
"popup": util.Panel(cursesLock, 9),
"graph": graphPanel.GraphPanel(cursesLock),
"log": logPanel.LogMonitor(cursesLock, loggedEvents),
- "torrc": confPanel.ConfPanel(cursesLock, conn.get_info("config-file")["config-file"])}
- panels["conn"] = connPanel.ConnPanel(cursesLock, conn, torPid, panels["log"])
+ "torrc": confPanel.ConfPanel(cursesLock, confLocation)}
+
+ # starts thread for processing netstat queries
+ connResolutionThread = connResolver.ConnResolver(torPid, panels["log"])
+ connResolutionThread.start()
+
+ panels["conn"] = connPanel.ConnPanel(cursesLock, conn, connResolutionThread, panels["log"])
panels["control"] = ControlPanel(cursesLock, panels["conn"].resolver)
# prevents netstat calls by connPanel if not being used
@@ -280,7 +306,7 @@
# statistical monitors for graph
panels["graph"].addStats("bandwidth", bandwidthMonitor.BandwidthMonitor(conn))
panels["graph"].addStats("cpu / memory", cpuMemMonitor.CpuMemMonitor(panels["header"]))
- panels["graph"].addStats("connection count", connCountMonitor.ConnCountMonitor(panels["conn"]))
+ panels["graph"].addStats("connection count", connCountMonitor.ConnCountMonitor(conn, connResolutionThread))
panels["graph"].setStats("bandwidth")
# listeners that update bandwidth and log panels with Tor status
@@ -299,7 +325,6 @@
isUnresponsive = False # true if it's been over ten seconds since the last BW event (probably due to Tor closing)
isPaused = False # if true updates are frozen
page = 0
- netstatRefresh = time.time() # time of last netstat refresh
regexFilters = [] # previously used log regex filters
while True:
@@ -309,8 +334,6 @@
cursesLock.acquire()
try:
- y, x = stdscr.getmaxyx()
-
# if sighup received then reload related information
if sighupTracker.isReset:
panels["header"]._updateParams(True)
@@ -341,6 +364,7 @@
panels[panelKey].recreate(stdscr, tmpStartY)
tmpStartY += panels[panelKey].height
+
# if it's been at least ten seconds since the last BW event Tor's probably done
if not isUnresponsive and panels["log"].getHeartbeat() >= 10:
isUnresponsive = True
@@ -350,11 +374,7 @@
isUnresponsive = False
panels["log"].monitor_event("NOTICE", "Relay resumed")
- # if it's been at least five seconds since the last refresh of connection listing, update
- currentTime = time.time()
- if not panels["conn"].isPaused and (currentTime - netstatRefresh >= 5):
- panels["conn"].reset()
- netstatRefresh = currentTime
+ if not panels["conn"].isPaused: panels["conn"].reset()
# I haven't the foggiest why, but doesn't work if redrawn out of order...
for panelKey in (PAGE_S + PAGES[page]): panels[panelKey].redraw()
Modified: arm/trunk/interface/cpuMemMonitor.py
===================================================================
--- arm/trunk/interface/cpuMemMonitor.py 2009-09-28 00:51:28 UTC (rev 20677)
+++ arm/trunk/interface/cpuMemMonitor.py 2009-09-28 06:56:21 UTC (rev 20678)
@@ -29,7 +29,6 @@
self._processEvent(float(self.headerPanel.vals["%cpu"]), float(self.headerPanel.vals["rss"]) / 1024.0)
else:
# cached results stale - requery ps
- inbound, outbound, control = 0, 0, 0
psCall = os.popen('ps -p %s -o %s 2> /dev/null' % (self.headerPanel.vals["pid"], "%cpu,rss"))
try:
sampling = psCall.read().strip().split()[2:]
Modified: arm/trunk/interface/logPanel.py
===================================================================
--- arm/trunk/interface/logPanel.py 2009-09-28 00:51:28 UTC (rev 20677)
+++ arm/trunk/interface/logPanel.py 2009-09-28 06:56:21 UTC (rev 20678)
@@ -2,7 +2,6 @@
# logPanel.py -- Resources related to Tor event monitoring.
# Released under the GPL v3 (http://www.gnu.org/licenses/gpl.html)
-import re
import time
import curses
from curses.ascii import isprint
@@ -166,7 +165,7 @@
def monitor_event(self, level, msg):
# events provided by the arm monitor - types use the same as runlevel
- self.registerEvent("ARM-%s" % level, msg, RUNLEVEL_EVENT_COLOR[level])
+ if level in self.loggedEvents: self.registerEvent("ARM-%s" % level, msg, RUNLEVEL_EVENT_COLOR[level])
def registerEvent(self, type, msg, color):
"""