Continue reading »]]> There’s a little known tool that’s provided as part of a Windows installation, it’s certainly available under XP and Vista which seems to be overlooked in Network troubleshooting.

Called pathping, it’s a command-line utility that will help you to troubleshoot intermediate hops between your source and a destination host. Something of a combination of ping and tracert (or traceroute for the *Nix users out there).

Tracert will show the intermediate hops between you and a destination, together with the link latency, and packet loss rate. In other words it will very likely show you where a problem lies between two nodes on the network.

The command itself takes some minutes to run, dependant of course on the number of hops between you and a destination host. The below example takes for me 475 seconds to run because Google is 10 hops away, via my corporate internet connection. Sample output is shown below (but of course the IP Addresses have been changed to protect the innocent!)

Usage is simply pathping [destination host or IP]

C:\>pathping www.google.com
Tracing route to www.l.google.com [64.233.183.104] over a maximum of 30 hops:
0  mydesktoppc [192.168.1.10]
1  mydefaultgw [192.168.1.254]
2  internal-local-ce-router-01 [10.0.0.1]
3  service-provider-pe-router-01 [172.16.1.10]
4  internal-remote-ce-router-01 [172.17.24.1]
5  remote-core-switch-01 [192.168.254.254]
6  remote-inner-perimiter-firewall [192.168.0.100]
7  rate-shaping-switch-perimiter [192.168.74.12]
8  remote-outer-perimiter-firewall [192.168.75.6]
9  isp-router
10  unspecified-00.ukcore.bt.net [1.2.3.4]
11  unspecified-01.ukcore.bt.net [2.3.4.5]
12  unspecified-02.ukcore.bt.net [3.4.5.6]
13  195.99.125.82
14  209.85.255.175
15  66.249.95.130
16  64.233.175.246
17  72.14.233.79
18  216.239.43.34
19  nf-in-f104.google.com [64.233.183.104]

Computing statistics for 475 seconds...
Source to Here   This Node/Link
Hop  RTT    Lost/Sent = Pct  Lost/Sent = Pct  Address
0                                           mydesktoppc [192.168.1.10]
0/ 100 =  0%   |
1    0ms     0/ 100 =  0%     0/ 100 =  0%  mydefaultgw [192.168.1.254]
0/ 100 =  0%   |
2    0ms     0/ 100 =  0%     0/ 100 =  0%  internal-local-ce-router-01 [10.0.0.1]
0/ 100 =  0%   |
3    6ms     2/ 100 =  2%     2/ 100 =  2%  service-provider-pe-router-01 [172.16.1.10]
0/ 100 =  0%   |
4   11ms     0/ 100 =  0%     0/ 100 =  0%  internal-remote-ce-router [172.17.24.1]
0/ 100 =  0%   |
5   12ms     0/ 100 =  0%     0/ 100 =  0%  remote-core-switch-01 [192.168.254.254]
0/ 100 =  0%   |
6   12ms     0/ 100 =  0%     0/ 100 =  0%  remote-inner-perimiter-firewall [192.168.0.100]
0/ 100 =  0%   |
7   12ms     0/ 100 =  0%     0/ 100 =  0%  rate-shaping-switch-perimiter [192.168.74.12]
0/ 100 =  0%   |
8   13ms     0/ 100 =  0%     0/ 100 =  0%  remote-outer-perimiter-firewall [192.168.75.6]
0/ 100 =  0%   |
9   22ms     0/ 100 =  0%     0/ 100 =  0%  isp-router
0/ 100 =  0%   |
10   21ms     0/ 100 =  0%     0/ 100 =  0%  unspecified-00.ukcore.bt.net [1.2.3.4]
0/ 100 =  0%   |
11   21ms     0/ 100 =  0%     0/ 100 =  0%  unspecified-01.ukcore.bt.net [2.3.4.5]
0/ 100 =  0%   |
12   22ms     0/ 100 =  0%     0/ 100 =  0%  unspecified-02.ukcore.bt.net [3.4.5.6]
0/ 100 =  0%   |
13   29ms     0/ 100 =  0%     0/ 100 =  0%  195.99.125.82
0/ 100 =  0%   |
14   26ms     0/ 100 =  0%     0/ 100 =  0%  209.85.255.175
0/ 100 =  0%   |
15   26ms     3/ 100 =  3%     3/ 100 =  3%  66.249.95.130
0/ 100 =  0%   |
16   37ms     0/ 100 =  0%     0/ 100 =  0%  64.233.175.246
0/ 100 =  0%   |
17   34ms     0/ 100 =  0%     0/ 100 =  0%  72.14.233.79
0/ 100 =  0%   |
18   39ms     0/ 100 =  0%     0/ 100 =  0%  216.239.43.34
1/ 100 =  1%   |
19   35ms     1/ 100 =  1%     0/ 100 =  0%  nf-in-f104.google.com [64.233.183.104]
Trace complete.

So what is the output showing?

In this case I’m loosing 2% of packets getting to my service-provider-pe-router-01 and a further 3% at one of the later hops (probably a network interconnect) later down the chain.

When PathPing is executed, first section shows the route for the traffic, as would be shown by Tracert.

PathPing then displays a busy message which will vary based on 25 seconds per hop to the destination, during which time it will gather information from all the routers previously listed and from the links between them. At the end of this period, it displays the test results.

The two rightmost columns — “This Node/Link Lost/Sent=%” and “Address” — contain the most useful information.

The loss rates displayed for the links (marked as a “|” in the rightmost column) indicate losses of packets being forwarded along the path. This loss indicates link congestion. The loss rates displayed for routers (indicated by their IP addresses in the rightmost column) indicate that those routers’ CPUs or packet buffers might be overloaded. These congested routers might also be a factor in end-to-end problems, especially if packets are forwarded by software routers.
Loss Calculation

The raw data that PathPing obtains describes how many ICMP Echo Requests are lost between the source and an intermediate router. The diagram below shows how PathPing estimates the per-hop loss statistics. While at first this calculation might seem trivial, it is complicated by differences between the forwarding code path and the code path taken in responding to ping packets (ICMP Echo Requests/Replies).

The horizontal lines indicate the “fast path” of a router, which is taken by packets that are not sent to or from the local computer. That is, the fast path is the code path taken by transit packets that require no special processing other than forwarding, and is highly optimized for such packets.

In the diagram, the vertical lines indicate the extra processing taken when an ICMP Echo Request is sent to the local computer. This kicks it out of the fast path and delivers it to an ICMP module (often using separate queues and processors). Assuming no packets are dropped due to queue overflows, the ICMP module then generates an ICMP Echo Reply, which is forwarded back to the original sender.

Since packet loss can occur in the path indicated by the vertical lines (but such loss does not necessarily imply loss on the horizontal forwarding path itself), the raw numbers obtained from pings do not by themselves determine end-to-end packet loss. For example, pinging an intermediate router might create a 10 percent loss even though no end-to-end packet loss is occurring. PathPing’s algorithm uses the change in values from hop-to-hop to estimate actual per hop loss rather than losses in the higher-level router components. This actual per hop loss is the result provided in the “This Node/Link” column of the final PathPing report.

Continue reading »]]> Prompted by some fellow Networksy types, (Jeff and Jeremy), who’s recent entries on the contents of their toolbags I found extremely interesting;  I thought I’d go through mine as I have a few unusual items!

Starting with the fairly standard stuff:

Wire Strippers

Good old fashioned and simple wire strippers.  These ones have an adjustable lock to stop you stripping too much cable unintentionally.

RJ45 Crimping Tool

Then of course the RJ45 Crimps.  Nothing too fancy, just cheap and practical.  I have never needed to crimp an RJ-11, so all I need are RJ-45 ones.

RJ45-DB9 and RJ45-DB25 Console Cables

A standard set of Cisco Console cables;  I tend to have a couple of brand new ones as well as tried and tested ones in the box, as I often end up leaving them behind attached to “core” devices, so there’s at least one on any site that we touch.

A Krone Tool, or Punchdown Tool

A good old Krone punchdown tool.  My old and trusty one is starting to be a bit tempramental now when it comes to cutting the wires, so I’ve recently added a new one which has yet to be christened.  Fortunately I don’t need these THAT often!

Cage Nut Tools

The quite rare and exceedingly valuable lesser spotted Cage-Nut Tool. I try to keep hold of these as they have a tendancy to vanish from my toolbox when other engineers realise what they are and how useful they can be! Most decent brands of Rack seem to come with one, so I’m not sure quite how they are so rare. Maybe they live under datacentre floors along with the cage nut eating mice?

Cage Nuts and Velcro Ties

Some Cage Nuts/Bolts, and some Velcro ties, have to try and keep things nice and neat now!

Multi Tool

This one is a relatively new2 addition, a freebie I picked up at Infosec this year, but it replaces a rather worn and similar one.

RJ45/Cat5e Couplers

I usually keep a handful of these Couplers at the ready.  I’ve used them for all sorts of strange purposes over the years.  Everything from extending Cisco Console Cables for those REALLY hard to reach devices, to temporarily connecting cables in lieu of an IP Phone with a built-in Switch.

Anti Static Wrist Strap

I probably don’t need to explain why these are here, but it’s a good “just in case”.

Baby Cat5e Cables

Always a good standby, but I keep some of these 6″ Cat5e cables, they seem to come in handy all over the place!

A collection of Serial Adapters, Converters and Loops

A mixture of Adapters, Gender Benders, Modem Eliminators, RJ45-RS232 adapters, and Loop Tools.

Now on to the slightly more interesting stuff!

Not sure what this is called!

I’m not sure what this is called; I managed to pick this up several years ago when doing a project involving opening and unboxing over 1200 new IP Phones which were boxed in boxes of 4.  The sheer quantity of cardboard and the number of boxes to be opened quickly caused fingers to be cut to shreds and fingernails to be damaged, but this really made life so much easier.

USB Memory Stick

I’m always equipped with a USB Memory Stick, simply because my lanyard with my Swipe Card on it has one built in.  This was a Promotional item from a supplier a couple of years ago, and has 1Gb storage, which is plenty for most things I’m likely to need.

Tone Generator and Probe

This one is so valuable when trying to trace cables and find those frustratingly misnumbered !  I stick with this slightly older model because it has both an RJ45 connection, and a set of Croc-Clips for wiring only checks.

A Butt

Not sure if I’ve spelt that one correctly!   The Butt is becoming less useful these days in the “modern” world of IP Telephony, but it’s still useful to have for tracing Wires and Analogue lines.

My current Cable Tester

A fairly standard Cat5 cable tester.  I haven’t had cause to use the older BNC style cable tester for some time, but this one is a two-part unit for testing local cables or entire cable runs, and will identify Straight Through or Crossover cables.

Short Wave Radio's

I added these some years ago after myself and a colleague were working in a building, one of us in the Patch Panel / Comms Room, and another person at the other end of a cable run.  I actually have a set of four but keep two in my Toolbox and two at home.

Star Drive

An all-in-one unit for most sizes of star type screw, this one is a really useful one to keep handy.

Digital Multimeter

Again an all-purpose tool which has a multiplicity of uses!  I tend to use the conductivity test and Voltage readings most of the time, but it has saved me from a nasty shock on at least one occasion!

Cable Snake Head

This one is in the spirit of keeping things tidy!  I sometimes have cause to use the plastic surround which wraps a bundle of cables in a protective sheath.  This is the special head which is used to apply the covering.  It opens up and is clipped around the cables to be protected, and the sheath slides over the protruding knob.  You then slide the head along the cable while pushing the protective sheath on to the head and hey presto, the cable is all nice and neat and protected.

MiFi and USB Extension Cable

The MiFi is used to get me Internet connectivity in places where corporate network connections are not available, or too slow.  You never know when you’re going to need to download a different IOS image, and if the Network is broken – well you simply have to have an alternative.  It’s often also faster than Hotel Wifi as I can get up to 8Mb all to myself on this, depending on how good the 3G coverage is at the time.   The USB extension cable is used to get to those really hard to reach USB ports from time to time.

Head Light!

Interestingly I note some of my peers keep a Flashlight in their tools;  this is my equivalent.  With an elasticated head fitting, it can go over my head to give me hands free illumination while working.

Cable Clips

I’m not sure where I picked these up, or what their proper purchase is, but they are ideal for temporarily holding some cables “up” together in the back of a rack somewhere.  The metal clips will hook on to all sorts of things, and the Velcro at the bottom keeps the cables secure.

CompactFlash Adapter

Used for connecting directly to the Laptop and transferring stuff straight to/from a CF card before putting it in a Router or Switch.

Special Tool to Factory Reset a certain type of UTM Appliance

This is a specific tool for a certain brand of UTM appliance.  It’s credit card sized but fairly thick with a push-out USB cable head.  Plugging it in to the USB port on an appliance will cause the appliance to reset to it’s factory default, so the ultimate in password reset tools I guess!

Miscellaneous Widgets

And lastly a collection of Miscellaneous Widgets including some Permanent Marker Pens, a collection of Batteries, a standard USB Memory Stick, a Wireless USB Network Adapter, and a GLC-T GBIC.

That’s all.  I’ve not shown the “boring” screwdrivers, or must have label printer etc which are probably an ever present feature of most Network Guys’ toolkits.  My only problem at the moment is that I don’t have a proper “box” to put all the tools in;  I’d like one similar to this Stanley one, but currently this is all in a plastic crate at work which will slide in to the back of the car nice and easily if needed.

So, what’s in your Toolkit?

Continue reading »]]> Good old Ivan Pepelnjak has collated an MPLS Cheet Sheet over on the IOS Hints & Tricks Blog, which may well come in very handy!

Or go straight for the PDF Download here: http://wiki.nil.com/wk/images/b/b1/MPLS_VPN_cheatsheet.pdf

I could think of one or two engineers at a certain large Telco that might find that quite handy…