Simple Network Management with Syslog

How do you easily track thousands of network elements and separate truly critical problems from ordinary network events? No easy task, I know. What approach do you use?

I’m a big fan of using syslog to automatically collect event messages from network devices, rather than monitor devices individually. Over the years, the syslog protocol has become an industry standard for creating, sending and collecting event messages.


A standard syslog message includes the following information:
- Facility type (OS, app or service) that originated the message
- Severity level associated with the message
- Date and time the message was sent
- Hostname or IP address of the sending server or network device
- Message text containing the event description

OS’s, applications and services continuously send these messages to centralized syslog servers. Depending on the capabilities of a particular syslog server, syslog messages can be sorted and viewed according to criteria such as message source, event severity or key words in the message text. Isn’t that convenient?

Also, the syslog server may filter syslog messages and raise alarms based on severity level. These alarms let you deal with unstable network elements first and the respective syslog messages give you the details to understand the source of the problem.

Here’s a simple table showing syslog severity levels:

Seems simple right? One challenge is that sifting through a ton of syslogs to troubleshoot a single problem can take hours. You need a way to consolidate, analyze and visualize the information to reduce system downtime and increase network performance. We have a great way for you to see your network status at a glance, check out dVUE6, a cool desktop gadget (for XP and Vista) that monitors the availability and alarm status of your five most alarmed devices in your network. For more information, and to download the gadget, follow this link.

Here is a graphical view of dVUE6

What is Fueling Future Network Growth? Your Thoughts...

What do you think is driving future network growth?According to a new Cisco study, many residential, business, and mobile IP networking trends are being driven largely by a combination of video, social networking and advanced collaboration applications, termed “visual” networking traffic.

Service provider networks are carrying a significant amount of visual networking traffic, with more than one-third of the average global broadband connection supporting video, social networking and collaboration applications each month. Maybe social networking isn’t a fad.                                                                             

Cisco VNI (Visual Networking Index) Usage Highlights:

Aggregate Broadband Findings (Q3CY09):                               

- Globally, the average broadband connection (primarily residential subscribers and some business users) generates approximately 11.4 gigabytes of Internet traffic per month.

Per connection per day, this amount is roughly equivalent to downloading 3,000 text emails, 100 MP3 music files or 360 text-only e-books.)

- Globally, the average broadband connection consumes about 4.3 gigabytes of visual networking applications advanced services such as video, social networking and collaboration) traffic per month.

Per connection per day, this amount is roughly the equivalent of approximately 20.5 short-form Internet videos or approximately 1.1 hours of Internet video, whether streamed on its own, embedded in a Web page, or viewed as part of video communications.)

- Top 1 percent of global subscribers generated more than 20 percent of all traffic.

- Top 10 percent of global subscribers generated more than 60 percent of all traffic.

Peak Broadband Usage During Internet Prime Time:

- In an average day over the reported quarter, Internet "prime time" spans from approximately 9 p.m to 1 a.m. around the world. This contrasts with broadcast TV prime time, which is generally from 7 p.m. to 11 p.m. across most global markets.

- 25 percent (or 93.3 megabytes per day per connection) of global Internet traffic is generated during the Internet "prime time" period.

- A peak Internet hour has 20 percent more traffic than a nonpeak Internet hour. The peak Internet hour averages 18 megabytes of traffic per connection (per hour), while nonpeak Internet hours average 15 megabytes of traffic per connection (per hour).

The peak Internet visual networking hour has almost 25 percent more traffic than average hourly Internet traffic.

It’s one thing to imagine the amount of traffic passing through networks every day, but it’s another to actually see the numbers. My favorite statistic is the number of GBs used per connection per day compared to text messages, e-books and music. The numbers are huge! I can’t imagine sending 3,000 text messages per day, can you?

What do you think about these findings? What are the future implications? I’m interested in hearing your thoughts.

Network Disaster Preparedness Tips

With the winter season approaching, big storms bringing everything from heavy rain and lightning to snow and wind will be a constant threat to network operations. When was the last time your local IT team reviewed disaster preparedness procedures? Now is a great time to start. If any form of a disaster hits, do you or your team know your capabilities and how to react? Here are some important questions that your IT team should be able to answer and use to improve your disaster procedures:

  

1. Are you aware of your power situation?
a. What happens when a power outage occurs?
b. What is the operational status of the UPS system?
c. How long will the UPS backup systems sustain key functions?
d. What do we do if the outage is longer?

2. What if the building becomes unavailable? (fire or water damage)
a. Are the offsite backups current?
b. If a network device or server is ruined, what is the procedure to replace it?
c. Does everyone know the primary and secondary facility contacts to use should an after-hours emergency occur?

3. What if access to the building is limited? (snow, tornado warnings, etc)
a. Is VPN access updated for all employees that may need to work from home?
b. Can all of the required maintenance procedures be done remotely or skipped for several days?

4. What if the phone and/or Internet connection is lost?

5. What is the customer impact when any of these conditions occur?

Advance planning is the best approach. A good network design can minimize the impact of storm and disaster related problems. Having redundant phone and data lines from different carriers minimizes the inbound/outbound traffic risk. Using an adequate number of UPS devices mitigates all but very lengthy power outages and network routing protocols like HSRP reduce the risk of single device point of failures.

Even monitoring your network with disaster prevention in mind can be helpful in avoiding unnecessary failures. These tips are a great starting point:

1. Enable redundant polling of critical devices
2. Map out HSRP primary and secondary links
3. Know the status of the UPS systems
4. Make sure you have 24x7 access to your management system client

 If you have tips for network disaster prepardeness, please share them with us.

Resolve Network Problems Faster with Wireshark

I’m a big fan of Wireshark, an open source tool to capture and analyze network traffic. Wireshark can be very helpful in terms of network troubleshooting and analysis. I recently came across a great tip from Chris Greer, a Senior Network Analyst on how to add useful columns, as well as recommendations for which columns are most commonly used (detailed below). If you have any tips that you would like to share to help network managers, please send them along.

What are the best columns to display?
Wireshark allows you to easily display specific packet data in the summary view. Depending on the problem you are looking for, it may be useful to add or remove custom columns so you only see information that will really help you solve the problem. For example, it’s not always useful to display the IP identification number for every packet in the trace file as a column on the top of the screen. But, one thing that is quite useful to display for most problems is the size of each packet.

How to add and remove columns.
Open up any trace file in Wireshark. At first, in the top screen, you will see a frame number, a time column (usually elapsed time from the start of the capture), source and destination addresses, what protocol is used, and finally a summary of what the packet contains. While this data is great to start out with, it won’t take long before adding a column of specific data will help.

To add a column, select Edit Preferences. This will bring up the Preferences window where Columns can be selected from the left. The default columns will be displayed here. To add a column, click the Add button and select the desired information to add from the format menu. Name the column and then you are done.

For example, to add the packet length, select add, name the column Length, then select Packet Length from the Format Bar. By default, it will be added at the bottom of the list, which will make it appear on the far right of the summary view. To move it up, just drag and drop it up where it can be more easily seen, such as between the destination and protocol fields.


After adding the column, click the apply button and the length of the packet will now be displayed in the summary view of Wireshark.


What are the most useful columns?
This of course depends on what problem you are trying to resolve. Here are the ones I use with a description of why:


Delta Time - Handy to have in just about any situation. It is needed for measuring server response time, network roundtrip time, and other delays. Delta time can be displayed by modifying the already present time column in the View menu, or by adding delta time as its own column.


Cumulative Bytes – When moving large blocks of data, this column displays how much data has been sent. You can then divide by the time it took to send it and figure out the throughput used by the application.

In this data copy, after filtering on one direction of traffic and resetting the timers, we find that in .022 seconds, there was 193,220 bytes sent. (193,200 x 8)/.022 = Bitrate. This transfer took up 70Mbps of my line. Very useful when looking at backups!


TCP Window Size – Useful when TCP Window size is an issue, usually in larger file transfers. It can be added by selecting Custom from the column menu, then entering tcp.window_size in the field name. This column will show when and how often the TCP window drops, without needing to dig for this value in the packet details.

IP TOS (DiffSrv) – When monitoring traffic using the TOS field, such as VoIP, this column is helpful in viewing what bits are set and if QoS is configured for this value. It can be added selecting the IP DSCP Value option.

Using these columns makes it easier to spot problems in the summary view. They save you from needing to dig deep into every packet for a specific field, which saves a ton of time when troubleshooting!

Announcing the release of dopplerVUE 2.0

dopplerVUE 2.0 is now available for download with exciting new features including discovering and mapping HSRP primary and secondary links, a new interface with graphics for better visibility, improved SNMP table polling and more! The 30-day trial is free, just like it was for the pre-release.

OPSCenter Screenshot

Netflow Screenshot















For those of you who missed the pre-release announcement, here’s the new feature list:

• New interface with new graphics for better readability
• Manage ANY metric on ANY device with improved SNMP table polling
• Discover and map HSRP primary and secondary links
• New WMI wizard
• Easily create personal workspaces with the new dVUE templates
• Distributed architecture to extend the number of managed devices
• Improved database and architecture for increased performance

Plus - for distributed enterprise networks dopplerVUE connects with NeuralStar to create a two-tiered, centrally monitored, fully replicated enterprise network management system that provides:

• Increased fault tolerance
• Improved performance through load balancing
• Powerful disaster recovery capabilities
• Enhanced scalability

I’d love to hear any dopplerVUE or network-related comments or questions you have!

IT Hiring Improving?

Here are some interesting data points about IT hiring from an industry article I just read. As usual there are some mixed indicators, but hopefully the market continues to improve.

• IT unemployment rate (5.5%) is only about half the overall market (9.8%)

• IT budgets are expected to grow a bit in 2010 (IDC predicts 2.9% growth and Gartner estimates a 1.53% increase)

• Hiring is increasing in areas including New York, Washington, D.C., Chicago, Los Angeles, Dallas and Houston (according to Robert Half Technology)

• Robert Half reports that health care organizations plan to increase IT hiring by 5 percent in the fourth quarter, while overall IT hiring is expected to remain flat

• Network security, especially securing government data, is one of the most desired skills (according to Dice.com)

• Many projects with higher paying skills have been put off, but the associated skills remain in demand : .Net, ERP development, SharePoint development, Virtualization (according to Dice.com and Robert Half Technology)

• Much of the hiring is occurring on the desktop side, desktop support analysts, PC techs, as well as program analysts and web development (according to Robert Half Technology)

If you’re looking for open positions, check out the careers section of our website. We currently have a web developer position open as well as many other IT positions in various geographic locations.

Before Adding More Bandwidth…

Does your network reach capacity at similar times every day? If so, before you decide to add more bandwidth at additional cost, think about adjusting traffic patterns at no cost.

As a first step, consider using a network monitoring tool to baseline the most common time periods when your network is at capacity. With this critical information, you can take some pro-active steps to optimize your network traffic.

Move maintenance tasks such as back-ups, updates, source control synchronization and large file transfers to off peak periods to decrease the load on the network.

If you can’t alter the timeline for maintenance tasks, encourage users to schedule bandwidth intensive tasks (video teleconferencing and webcasts) that are not time sensitive to business operations to off peak periods. Most users want the best possible viewing experience and will appreciate being able to schedule online events to take advantage of improved network availability.

Of course, if you don’t have flexibility to adjust traffic patterns to optimize the network, it may be time to add more bandwidth.