LandWarNet 2009 in Review

I spent the week of August 17th in Ft. Lauderdale, FL attending the LandWarNet Conference. This was my first time at the conference, so I figured I would share a bit of my experience with you.

This year’s theme was, “A Global Network Enterprise Enabling Full Spectrum Operations for the Joint Warfighter”, and the exhibiting companies came from a range of technology industries including government consulting, military equipment to software.

While the show is predominately Army-focused, there were some smaller mixed groups in attendance as well. Attendance was strong, considering the state of the economy and the majority of the show sessions were full. The show floor was lively the first two days, but emptied out on day three. According to many LandWarNet veterans, the show is a magnet for hurricanes and has been closed early numerous times, so I’m not surprised people didn’t stick around the entire time.

Here are a few pictures from LandWarNet:

Using SNMP for Network Troubleshooting

Here is a network troubleshooting tip to consider. Simple Network Management Protocol (SNMP) is used for getting notifications and performance data about networking devices. It can also be useful in a non-traditional way as well.

When most networking devices are very busy they stop responding to management requests and focus on core routing or traffic-level tasks. As a result, if you see increasing SNMP response times or worse complete SNMP response failure, it’s likely a leading indicator that a device’s performance is nearing the user impact stage. The device may not be down yet but, it very well could be shortly. Use this indicator to your advantage and start monitoring SNMP response time and failure rates. Be sure to keep a close eye on these devices and you’ll know about problems before it's too late.

Why use SNMP v3?
Need stronger security than community strings and access lists can provide? Consider moving to SNMPv3 if you haven't done so already. SNMP v1 and v2 are sufficient for many networks. When management traffic must pass over the public Internet, however, SNMP v3 provides improved message security that can be important.

Enhanced security features in SNMPv3 include:

- Message integrity to ensure a packet has not been tampered with in transit.
- Authentication and verification that the message is from a valid source
- Encryption of packets to prevent snooping by an unauthorized source

Whirlwind Tradeshow Tour...

This week I'm in Ft. Lauderdale at the LandWarNet show and next week I'm off to AFITC in Montgomery, AL. The whirlwind tour is off to a good start. The skies are bright and no hurricanes are on the forecast. For those that won't make it to LandWarNet this year, I'll be updating the blog with some information about the show and the seminar tracks.

It's always interesting to see what technologies industry is providing the military to improve their information advantage in the battlespace at these shows. If you're interested in learning about some solutions Kratos Defense will be presenting drop by booth #1513 at LandWarNet and booth #542 at AFITC. We are going to be demonstrating some interesting solutions including example workspaces created to provide Program Managers with a centralized view of all project related information, business intelligence dashboards displaying operational metrics, network management solutions for strategic and tactical networks and learning solutions to optimize workforce development.

Hope to see you soon.

Preview the next set of Free Network Utilities…

I just finished previewing three free network utilities that Kratos Defense & Security Solutions has developed. These have NOT officially launched yet, but I wanted to give blog readers the first crack at downloading and checking them out. You can choose from Army or Air Force skins (or both) and use these utilities from your desktop to:

1. Test download speeds (nice to know if your T1 is doing well)
2. Monitor any SNMP OID metric (Poll a CPU Utilization from a router etc…)
3. Monitor Syslog events (I sent a few firewall activities at it)

These will officially be released at upcoming Army and Air Force events in late August (LandWarNet & AFITC).

The skins make for a very cool display, everybody that came by my desk today asked about them. Just don’t tell anybody where you found the links or, I might not be given the preview next time :)

Free Network Utilities Download Links:

AirForce

Army

Five Ways to Get IT Training Cost-Effectively

In these tough economic times, keeping up-to-date and sharp with the latest high-tech skills can be very beneficial for the health of your career. Here are five creative ways to get training cost-effectively according to Network World.

1. Share the cost with your employer
One option is to offer to split the cost of training with your employer. It would cost the company more to bring in a new person than to train you in the skills they need. The solution serves both parties well.

2. Train yourself
You can gain a wealth of knowledge from self-study from online courses, books, videos and webcasts. Another resource for IT pros looking to learn is CBT Nuggets, which offers fee-based and free training products online.

3. Build your own learning environment
Building a home lab can help progress your career and be done cost-effectively (in some cases). Finding used parts and complimentary software can help with Microsoft and Cisco self-training. Check Microsoft for free trials of their products online and open source tools. Two open source projects worth reviewing include Dynamips, for simulating Cisco networks, and Olive for simulating Juniper networks.

4. Practice your skills
Use your knowledge and skills to help friends, family, charities or small businesses at low or no-cost to gain valuable experience. The effort can pay dividends on your resume and help you learn about new technologies without investing in training. This can also serve as the start of your own business.

5. Look for training discounts
Many training vendors are offering significant discounts because of the recession. It may be worth the time to do a little research to find some cost-effective training. Many vendors offer educational content at events and online as part of their marketing efforts. Keep an eye out for specific opportunities at your favorite vendors’ websites and for upcoming low cost or complimentary technology tradeshows.

Maintaining Network Awareness with Twitter…

Wanted to try a fun experiment and feed my network management alerts to Twitter instead of my email. Now let me explain, I’m using this for the alerts that are between super critical that need my instant attention (and are emailed to me) and those that can simply sit and wait all weekend or more for my attention (i.e. when I feel like looking at the NMS screen). By sending some events to Twitter (its on my phone and TweetDeck is on my pc) I can maintain “network awareness” while tweeting J or on the go.

Using the dopplerVUE network management tool, all alerts can be sent to a .Net assembly. Since Twitter has an API this part was simple – email me and I’ll send you the necessary files so you can create updates with your own Twitter account information.

Here are the steps to implement a twitter feed (takes about 5 minutes).

The ABCs of adding a twitter feed to dopplerVUE
A. Update your system with the new twitter.dll and EventService.exe.config files
B. Create a Twitter action that can be re-used.
C. Define a rule that will use the Twitter action

A – Updating the dopplerVUE system
1. Add the twitter.dll to the //systechnologies/dopplerVUE/Server folder
2. Back up your existing //systechnologies/dopplerVUE/EventService.exe.config file and replace with the new one provided.
3. Edit the EventService.exe.config file user and password lines with your actual Twitter account username and password.
4. Preference>dopplerVUE System>dopplerVUE Services Then restart the dvEventService and dvDataService

Now the Twitter action is ready to use.

B - Create a reusable Twitter action
1. Expand the NetOps> Action tree
2. Right Click on Actions and select Create Action
3. Enter an Action Name
4. Select Action Type: Custom DLL
5. Enter the assembly value of: Twitter.dll
6. Select Create.

C- Define a rule that will use the Twitter action
1. Expand the NetOps>Metric>Rules tree
2. Right click on an existing rule and select edit
3. At step 5, expand the Custom DLL folder and select the Tweet This! Action
4. Select Finish.

Getting A Handle On Dropped Packets – 4 Key Troubleshooting Tips

In today’s net-centric environment, organizations often depend on the network for voice over IP, video conferencing and webcasts. Network problems caused by packet loss can create noticeable performance issues.

Is packet loss a problem your organization is experiencing? Could it be contributing to larger problems? One thing’s for sure, almost every network experiences packet loss to one degree or another. Dropped packets can originate in almost any part of the network path, from bad cables to flakey applications. Here are a few common causes and what you can do to fix them:

1 . Find the source. First, understand where the packet loss is occurring. The command line tracer tool helps determine the exact location. Then, learn the extent of the problem by using the “Netstat –s –p tcp” command. This will display the total segments sent and total segments retransmitted. (Check out this site for more information on how to use command line tools)

2. It's probably the cables. More often than not, it’s this simple. Check that nothing has been placed on top of the cable and that the connections are tight. Test by replacing the potentially bad cable with a known good one. Frequently, very long cables will only show excessive problems when they are processing a heavy load.

3. Check for duplex mismatches. Many systems auto negotiate the duplex speed, however errors do occur. Consider manually setting it to a known level and see if the problem goes away.

4. Unleash the routers! If your routers are overwhelmed, they will drop packets. Check for excessive utilization across each link and make sure the system overall is not saturated. Not all routers drop packets at the same traffic level. Some Cisco routers can begin dropping packets at a CPU load of 50%; on other models this may not occur until 95% or more.

Tip for dopplerVUE users: The Locator view lets you sort all interfaces by packet loss so you can isolate the location of dropped packets instantly.

Network Troubleshooting: IP SLA+ WMI = Better Web Services

Why is the network so slow? I’m sure you’ve never heard this complaint before :) Diagnosing the problem isn’t always easy with so many possible culprits. You can start by running down the network troubleshooting checklist:
The DNS service?
The web server?
The WAN link?

IP SLA and WMI information is critical to diagnosing potential network problems. For most Cisco devices, IP SLA can give you performance information for the connectivity layers of a net-centric service like a web application or VoIP. In Microsoft environments, WMI can do the same for the application/server/desktop layer. Combining WMI with IP SLA provides performance information about both layers and gives an end-to-end view of your web application or other net-centered service to most efficiently troubleshoot any issues.

Using IP SLA to Access the User Experience
IP SLA (Internet Protocol Service Level Agreements) is embedded in the Cisco IOS (Internet Operating System) for most Cisco routers and switches. IP SLA operations can measure delay (round trip time), jitter, packet loss, connectivity, voice quality scores, and many other key metrics for monitoring and troubleshooting network elements.

Additionally, threshold levels can be set for most metrics. When a metric crosses a threshold level, IP SLA sends an SNMP trap to the specified IP addresses.

You can configure an IP SLA HTTP operation to monitor the overall user experience for the “connectivity layer” of a web application (or any other net-centered application such as email, VoIP or videoconferencing). This operation uses a synthetic web transaction to measure the total round trip time (RTT) to perform a DNS query, establish a TCP connection to the HTTP service, and retrieve the web site’s home page. By configuring the HTTP operation on the LAN switch closest to users, the total RTT (or latency) is an accurate measure of the users’ experience (as opposed to measuring RTT from a central network management server).

Next, configure an IP SLA ICMP Echo operation to monitor RTT between the switch on the user LAN and the switch to which the web server is connected. This way, if the HTTP operation indicates the web transaction is slow or unresponsive, you can check the WAN RTT between the switches to see whether the problem is related to the WAN link or something on the web server.

Watching the Applications and Servers: Adding WMI
WMI (Windows Management Instrumentation) is an instrumentation tool similar to IP SLA that Microsoft has created for its products. WMI provides thousands of performance metrics for applications such as MS Exchange and MS SQL Server, as well as for server hardware and operating system components.

Microsoft has a built-in performance administration tool for monitoring WMI data for applications and servers. Using the tool you can view each server’s CPU utilization, physical memory and free disk space. Each of these sub systems is critical to the server’s performance regardless of the application running. Lack of memory, CPU cycles and low disk space are common causes of slowdowns on a server. You’ll have to go into each server to view the individual performance counters or you can use network management software to simplify the process by collecting any of the thousands of available WMI counters from across multiple servers.

Getting the End-To-End View
An end-to-end view of the network will really help you troubleshoot network problems much faster and avoid the common complaints you often hear. To get an end-to-end view consider network management software such as dopplerVUE that integrates fault and performance data from a variety of sources, including SNMP, syslog, WMI and IP SLA, you can integrate metrics from both layers of a web service into a single end-to-end dashboard view. Using dopplerVUE’s drag-and-drop interface, you can quickly create an integrated view of both layers of the service without having to shift between tools or viewers (screenshot below).

Monitoring Bandwidth Part 2: Examining SNMP Traffic Data

Let’s start by discussing what we really want to know about bandwidth:

1. How much is moving across any given interface?
2. Is the interface maxed out?
3. Is the device or devices beyond this one slow (or down)?

SNMP MIB-II enabled devices provide the following key metrics that will be used to derive answers to 1 & 2.

ifSpeed - The interfaces current bandwidth in bits per second
ifInOctets - The total number of octets received on the interface
ifOutOctets - The total number of octets transmitted out on the interface
Source: RFC 1213

The octet metrics are simple counters that grow as traffic is passed on an interface. Using these metrics we can poll devices two times and use some “simple” math to determine the delta between the polling jobs. This will give us the amount of traffic that has passed in the interval. You can divide this by the amount of time to get an average bit per second rate. Or you could simply use a tool like dopplerVUE that does the math for you (screenshot below).

* Important Tip - The measurement for the size of a file and the speed that an interface passes traffic is not the same. Despite looking and sounding similar each measurement is calculated in a different way. This is a common error. For example, network speeds are notated in bits per second. Files are normally referred to in bytes. There are 8 bits in a byte, then you need to factor in that file notation grows by 1024 not simple 1000s.

Notation examples:
Network Speed
1 Kbps = 1,000 bits per second
1 Mbps = 1,000,000 bits per second
1 Gbps = 1,000,000,000 bits per second

Data file size
1 KB = 1,024 Bytes
1 MB = 1,024 KB
1 GB = 1,024 MB

Now that the amount of traffic is known you can compare this information to the ifSpeed metric to determine the percentage of the pipe that is full. You can figure out the math or let the tools do it for you (dopplerVUE screenshot below).

To answer the final question about if the traffic is causing a slowdown on the network, check the ping response time to the device and devices beyond (if router or switch).

There are many other items we can look at regarding traffic that indicate problems in the network. You can look for packet loss, discards and errors that are occurring (dopplerVUE screenshot below). We’ll explain why these issues occur and how to correct them in a different posting, but you should consider checking these metrics as well.

Email Failure – Not Again…

A common complaint I hear and a pain I’ve experienced myself (too often) is email going down. Email is now the most common form of business communication and its use is only growing. According to Osterman Research, email traffic between 2008 and 2012 is projected to grow by 68%.

In any given 12-month time period, there is a 72% likelihood of an unplanned email outage and a 24% likelihood of a planned email outage in any given company according to a Dell survey.

One major cause of email downtime is network connectivity failures. Connectivity loss can be caused by anything from a hub, switch or router failure to a broken or damaged cable or fiber. Not to mention a whole host of other complicated issues.

To reduce the chance of an outage, consider proactively monitoring and managing your network to avoid a failure or respond as early as possible to a problem – saving you the headache of a bunch of complaints.

The network management software I recommend and work with everyday has the ability to generate alarms based on both network faults and network performance. This integration allows alarming on multiple issues to best anticipate and accurately pinpoint network problems and avoid system downtime. Below is a screenshot showing this functionality.

Is SOA dead? – Not at All…According to Forrester Research

According to a recent Forrester Research survey, SOA is alive and well! Here are a few of the highlights:
- Only 1% of current SOA adopters have received little or no benefit from the methodology
- 60% of IT executives have seen some benefits
- 75% of IT executives and technology decision-makers said they will be using SOA by the end of 2009
- 60% of current SOA users are expanding their use of the methodology
- 30% of respondents appear to see SOA as a potential vehicle for strategic business transformation

This is great news from my perspective, since the network is such a critical success factor for successful SOA implementations.

If you’d like to read further, check out Joe McKendrick’s guest post on TechRepublic’s blog.

Interop Las Vegas in Review

I spent last week in Las Vegas at Interop and thought I’d share my experience with you. Attendance was definitely down this year, but exhibitor attendance was about the same and booth extravagance seemed to be at an all time high. The exhibitor floor was much smaller and orange seemed to be the color of choice. However, it was easier to spend quality time with attendees that stopped by the booth. I had many great conversations with folks about their network management challenges and the need for enhanced visualization, alarm and bandwidth management and the benefits of all-in-one tools. The conference sessions were worth attending and covered the hottest topics including cloud computing, virtualization, green IT, SOA and web 2.0. I was tweeting live from some of the conference sessions – take a look at the trail of tweets, if you’re interested in some details about each session. All-in-all it was a great trip – got some good leads from the show and came home without losing too much face at the blackjack table.

A Shot of Caffeine While on the Go…

I found a great cure to satisfy my caffeine addiction while on the go! I was hunting down some registration information on the Interop site for my annual trip when I came across the Handpresso - a small gadget that delivers an espresso wherever you go, whenever you want. Sounds like heaven doesn’t it? On the last day of the show when everyone is starting to drag - an energy boost could be a life saver and make this gadget worth the $100 price tag. Something to think about…

Welcome to the Network Zone…Let’s Talk About Network Management Needs

Thanks for coming by to read this blog. I look forward to hearing your views and comments on this post and future ones. I’ll keep the posts fun and helpful.

I read a great article on SearchNetworking by Shamus McGillicuddy. It discussed the state of the market for network management products and more importantly, some of the top issues network management staff are facing. Here are some of my take aways from reading it...

There is a clear need for network management that lets engineers move away from spreadsheets, manual processes and tools that simply require too much overhead to use. The proof is in the IDC research that shows the market is growing in these tough times, and it looks like more specialty vendors are becoming both competitors and complementary players in the space.

The article references the network management space as still being part of the Wild West. The growth in new products and modules is staggering. One vendor actually boasts their core product has over 45 utilities (I can only handle about 5-6 tools then its overload).

Often so many tools are being used that it creates a new problem...

"Too many tools can lead to a lack of integration," said Steven Guthrie, director of product marketing at CA. "The success of low-end niche tools in the enterprise tells us that these point products come in for good reasons. But then they outlive their usefulness, and they end up stalling mean-time-to-repair. That data source [in the niche tool] may not sync up with other data sources you may have."

As network engineers, we’re bombarded with new information all the time and simply don't want or have the time to keep learning and finding the right application when a crisis occurs. A central point of management is necessary. Shamus was right on when he wrote...

"But, clearly, IT pros would love to consolidate the number of management tools they use on a daily basis, if for no other reason than to save money and make their operations more efficient."

Tools that have a single database, can consolidate multiple sources of information and have a low maintenance overhead will ultimately win out. In the long run they make your life simpler.

To read the full article on SearchNetworking.com http://searchnetworking.techtarget.com/news/article/0,289142,sid7_gci1355151,00.html