Re: Log analysis requirements

[John Lange <lists@darkcore.net>]

> Thanks for the information on this tool.  However, I want to ask,
> how would one go about handling multiple IPs on a single box?
> 
> I may not have made it clear, but I want to use IPTraf to track the
> bandwidth usage on a *single* machine with multiple IP addresses.
> My script summarizes the logs and dumps out the bandwidth usage.

When I said you install it on as many machine as you want, I did not mean 
to imply that it was limited to a single IP or machine.

For example, In my setup, I have a machine which I dedicate to monitoring 
our link to the internet (for bandwidth summaries). The machine is on our 
internal network with a second NIC connected to a mirror port on our 
switch which is set to mirror our outbound link.

The ruleset determines what is monitored and how it is summarized. The 
rules could be individual IPs, groups of IPs, whatever you like. 
(actually, it is not limited to IPs, it will do IPX, DecNet, ICMP, most 
everything).

> I understand that IPTraf was originally written as a real-time
> monitor, but I don't see any reason that it cannot also be a
> metering tool.

Not saying it couldn't be and I'm sure it suits this purpose just fine, 
just trying to point out that it may not be the best tool for the job. For 
example, as was pointed out in an earlier post, iptraf is strongly tied to 
ncurses due to its history as a non-daemon, real-time tool.

For most metering iptraf is probably a good choice. However, in my limited 
experience with NeTraMet, it is a much more powerful tool because of its 
complex rulesets. However, that also makes it quite a bit harder to setup. 
Its not trivial. Its designed more for high-level network admins who have 
to monitor a large network.

That being said, It also has some real limitations. My chief issue with it 
is that it re-uses flow identifiers. Each connection is assigned a "flow" 
and given an index. By default there are only 10,000 flow indexes and then 
it wraps around and re-uses them. There is no option to time stamp them or 
in any way uniquely identify them so when you are parsing log files that 
can cause problems. It should at least "time-stamp" the flows to make them 
unique.

My work-around is to up the limit to something like 500,000 and then 
rotate the logs daily keeping my fingers crossed that I don't get more 
than 500,000 flows in any given day.

Given the seeming lack of suitable tools designed for bandwidth metering, 
I was thinking about taking the source for NeTraMet and improving it (it 
hasn't been worked on much since the mid-90s). Then packaging that up with 
some add-ons for sumorizing the logs that I've written in perl and 
re-releasing it as a new package. Ahhh if only I had the time....

John

On Thursday 16 May 2002 11:49 am, you wrote:
> --- John Lange <lists@darkcore.net> wrote:
> >
> > You install the Meters on as many machines as you like, then you
> > upload your rule-sets using an admin tool (NeMaC). Then you pull down
> > the  statistics from all your meters at whatever interval you like
> 
> John,
> 
> Thanks for the information on this tool.  However, I want to ask,
> how would one go about handling multiple IPs on a single box?
> 
> I may not have made it clear, but I want to use IPTraf to track the
> bandwidth usage on a *single* machine with multiple IP addresses.
> My script summarizes the logs and dumps out the bandwidth usage.
> 
> Also, the way I've written my script, it would be trivial to add
> a list of IP addresses that I want to track of traffic that comes
> through the box running IPTraf.
> 
> I understand that IPTraf was originally written as a real-time
> monitor, but I don't see any reason that it cannot also be a
> metering tool.
> 
> > The primary problem being that iptraf likes to group all its stats by
> > pairs of IP addresses (both source and destination) which doesn't
> give
> > you a good idea of which machine IP on your network is taking the
> most
> > total traffic in a given time frame (bandwidth per month for
> example).
> 
> Maybe I'm missing something here, but part of the analysis that my
> script does is to pull out data from the log on a per-IP basis.  The
> fact that source and dest are combined in the log really isn't that
> big an issue, is it?
> 
> greg_fenton.
> 
> 
> 
> =====
> Greg Fenton
> greg_fenton@yahoo.com
> 
> __________________________________________________
> Do You Yahoo!?
> LAUNCH - Your Yahoo! Music Experience
> http://launch.yahoo.com
> 

-- 
John Lange