As a network administrator, you’ll become very familiar with the command line. In this video, you’ll learn about some of the most popular command line tools.
<< Previous Video: Software Tools Next: Wired Network Troubleshooting >>
As a network professional, you’ll do a lot of trouble shooting at the command line. In this video, we’ll show you some of the most popular command line tools. If I had to pick the tool that you would use the most for your network troubleshooting, I would probably choose Ping.
Ping is a way to test the reachability from your machine to another device somewhere on the network. Not only does it tell you if that device is responding to you, it also gives you an idea of the round-trip time between you and this other device. Ping uses ICMP or the Internet Control Message Protocol to be able to send these messages across the network.
Ping is usually the first thing you try to make sure that you can access a device across the network. And it’s usually the last thing that you try after troubleshooting to make sure that the device is now available. The Ping utility was written by Mike Muuss in 1983. And although some people have said this is a backronym called Packet Internet Groper, Mike said he really just built the utility with a name that would sound like the sonar ping that’s used with submarines.
On my Windows workstation, we’ll use Ping by typing ping, and then we’ll choose an address to ping. And in my example, I’ll choose Quad9. Windows will send four separate ping attempts, and it will tell you if you received a reply for each of those attempts. For example, we received a reply from 9.9.9.9. 32 bytes was the size of the ping that I sent, and it was the size that I had returned to me. It took 14 milliseconds. And the time to live for this TCP packet is 57 hops.
And you can see I performed four separate pings with this single command, and I have the round-trip times that are pretty close to each other– 14 milliseconds, 18 milliseconds, 17 milliseconds and 13 milliseconds. Overall, I sent four packets and received four packets for a 0% loss to Quad9. And you can see the round-trip time minimums, maximums, and averages for these four pings.
By pinging Quad9, we saw that the device is available on the network. And based on the time to live, we know there were a number of hops between our device and the Quad9 device. By using the traceroute command, we can map the route that’s used between our device and that remote workstation.
If you’re using Windows, you can run the tracert command. Or if you’re using Linux or Mac OS X, you can use traceroute. Traceroute uses the ICMP time to live exceeded message to be able to calculate what hop is the first hop, the second hop, the third hop, and so on.
Your device not only needs to be able to receive these ICMP time to live exceeded messages, the routers that these packets are going through have to send you the ICMP time to live exceeded message. And not all routers will send that message back to you. Sometimes they are disabling that feature, or ICMP is filtered on that network.
Different operating systems use different methods to be able to perform a traceroute. For example, Windows sends an ICMP echo request and looks for the ICMP echo reply from the final device. This is commonly what Ping uses to send its messages. Of course, we’re looking to receive an ICMP time to live exceeded message. And you also have to make sure that none of those protocols are going to be filtered between you and that final device.
Some operating systems give you more control over the traceroute. For example, Linux, Unix, and Mac OS allow you to change what protocol you’d like to use during the traceroute. And if you’re using a mobile device that runs iOS, it’s going to send UDP datagrams over port 33434. You may be able to change this port number with some of the extended options available in iOS.
Regardless of the operating system being used, the basic operation of traceroute is going to be the same across all of them. Let’s say that we’re going to traceroute between Sam’s workstation and Jack’s workstation. We’ll perform a traceroute to 10.10.30.10– that’s Jack’s workstation– over a maximum of 30 hops. That’s the default for the traceroute utility.
The first thing that will happen is Sam will send out a packet that has the traceroute TTL set to 1. That means when this packet hits the first router, the router will decrease the time to live by one. And when the time to live hits zero, which it will on this first router, a time to live exceeded is now sent back to Sam.
Sam recognizes that that message came from Router 1 and says for this first connection, we received a time to live exceeded from 10.10.10.1 Now Sam’s going to create another packet and this one, the time to live will be equal to 2. That means it will make it to Router 1 which will subtract one from the time to live number and send that packet on to the next route.
This router will see time to live equal to 1, subtract 1 and see that the time to live has been exceeded. And that message will be sent back to Sam informing Sam that the time to live was exceeded. Sam notes that this came from 172.16.1.2, so that must be the next router along the way.
The process repeats, except the time to live is now increased to 3. So it makes it to the third router along the way until the time to live is exceeded and a message is sent back to Sam with the IP address of the device that had the time to live exceeded message. That information is made available in the traceroute.
And in the last packet, where time to live is equal to 4, it will make it through the first router, will make it through Router 3, will make it through Router 4, and finally, to Jack’s workstation where the ICMP will be replied, and Sam will make a note of that device’s IP address as the last route in the traceroute.
Let’s perform a traceroute on my Windows workstation. We’ll perform tracert to Quad9. And we’ll see that it will hop through all of the different routers on its way between my device and the Quad9 DNS server. You can see there were eight different hops to complete this route. We started with 10.1.10.1, which is my local router, and then all of the other routers in between until we finally hit dns.quad9.net.
Now that we know the route that it takes to go between our workstation and Quad9, we can reference that later. If there’s a problem between any of these particular links, we can run this traceroute again and see exactly where along the way we’re having problems. There’s a saying among network administrators that the problem is always somehow related to DNS. And because DNS is such an important part of our network communication, it’s certainly important that you know how to perform troubleshooting with your DNS server.
Two commands that can help you do this troubleshooting our nslookup and dig. Using either of these command line tools, we can query a DNS server and receive information back such as canonical name lookups, IP addresses, cache timers, and much more information that’s contained on that DNS server. Nslookup is the utility you’ll find in Windows, Mac OS X, Linux, and almost any other operating system.
It can look up names and IP addresses from a DNS server. And although it’s available on many different operating systems, the use of nslookup is now deprecated, and it’s preferable that you use the dig utility instead. Dig is the Domain Information Groper, and it provides similar information to what you would see in nslookup.
The dig utility is not native to any of the Windows distributions, but you can download a version of dig for Windows from isc.org/downloads/bind. On my Windows workstation, we’ll run nslookup and we’ll look up www.professormesser.com. Response will be that we are accessing a DNS server on Comcast.net. And there’s the IPv6 address that I’m using to access that DNS server.
And that server provides me with a non-authoritative answer that www.professormesser.com. is located at 104.20.215.41 and 104.20.214.1. In my case, I have multiple IP addresses configured in DNS so that if one IP address isn’t available, you’ll use the other. On my Linux workstation, let’s use the dig command. And, again, we’ll choose www.professormesser.com. The format is a little bit different, but the information is the same.
Dig provides us with a breakdown of what our query happened to be. And in my case, I was questioning what the address is for www.professormesser.com. And you can see, the answer section shows me a little more information than nslookup did. It shows that www.professormesser.com has a time to live of 300 seconds before you would need to re-query that particular server.
This allows me to change the IP addresses behind the scenes, and everyone will be caught up in 300 seconds. And it also shows me the two IP addresses. And those are the same as the IP addresses we saw earlier. And then there’s a summary of the query and exactly when the query was performed and how long it took.
When you first sit down to a device and start troubleshooting, one of the first things you will do is begin pinging other devices on the network to see just how many devices are reachable from your workstation. But you won’t know what to ping unless you know what the local address of that device happens to be. One way you can determine the IP address of a Windows workstation is to use the ipconfig command. And on a Linux or OS X device, you would use an ifconfig command.
These commands will tell you much more than just the IP address of this device. You’ll get subnet mask information. You’ll know what the default gateway happens to be. You’ll learn about DNS settings and any other configurations that are specific to the IP stack. These two commands are very similar. There’s only one letter difference between the two. If you’re on a Windows workstation, use IP config. And on a Linux device use ifconfig.
I’m running Mac OS, so let’s run the ifconfig command. And I’m going to specify EN0 which is the ethernet adapter on my computer. And you can see, it does show me hardware information about this device along with the IPv6 and the IPv4 addresses that are defined on this computer.
If we were running Windows, and we ran ipconfig, then we’d be able to see similar information. Here’s the MAC address information on this Windows device. This device was not able to get a DHCP address, so we can see an APIPA address on this device. And you can see if there are any IPv6 or other DNS server configurations all from the ipconfig command.
If you’re a Windows user, you’re probably familiar with the Windows Firewall. If you’re a Linux user, then you have many options for firewall in the Linux operating system, and one of the more popular versions is iptables. This allows you to provide stateful filtering in the kernel of the Linux operating system. It’s common to see on workstations, servers, and any other device where you want to control the inbound and outbound network traffic.
Some organizations will use iptables as the primary firewall between the inside of their network and the internet. And they can configure iptables to filter by IP address, port, application, and other criteria. Some Linux distributions use firewalld instead of using iptables, but there’s many options available for controlling network traffic on a Linux device.
Another utility that’s available on many different operating systems is netstat. You can find netstat on Linux, on Unix, on Windows, and many others. Netstat stands for Network Statistics, and it provides you with many different views of what the statistics are for network communications on that particular device.
So for example, if you wanted to see all of the active connections that were on an individual machine, you could run netstat-a. On a Windows machine, you could combine the netstat-a with the netstat-b, which would not only show you the number of active connections, it would tell you exactly the Windows binary that was used to create that connection across the network.
And if you wanted to get rid of the resolved names and just get IP addresses with netstat, you can run netstat-n. Let’s run a netstat on my computer. I’m just going to run netstat with no other parameters, and it’s going to show me all of the network communications that I have made. You can see a number of them out to professormesser.com and to other web services. And you can see, as it’s working through all of the active connections, it begins updating those on the screen.
In many network troubleshooting situations you’re going to need a packet capture. Gathering packets from the network is going to provide you with a lot of detail of exactly what’s happening across the network. One very common way to capture packets at the command line of a device is to use tcpdump. Tcpdump is usually included with many Linux distributions and OS X. And for Windows, you can download a Windows version of this called WinDump. |
Tcpdump has a number of different options that you can include at the command line. You can apply filters and you can watch the packets go by in real time. You can also choose to save all of the packets you’ve collected into a pcap-formatted file. This is a file format that’s easily readable by many different protocol analyzers such as Wireshark.
The amount of information that you’re gathering, and the detail included in these packets can be overwhelming. But if you spend a bit of time to learn the formats and be able to filter out what you don’t need, you’ll find there’s a lot of valuable information stored in those packets. Here’s some data I captured on my system using tcpdump. You can see it’s a combination of IPv6 and IPv4 data. There’s a lot of different kinds of data going back and forth. And this can be a little difficult to read if you’re not familiar with exactly the format.
This becomes a little bit easier over time. But you can see why it’s useful sometimes to capture large amounts of data and then import that data later on into a protocol analyzer. Already in this video, we’ve used both ping and traceroute. Windows has a utility that combines these together and adds some additional functionality, and that utility is called pathping. This is available in all recent versions of Microsoft Windows.
There are two phases to pathping. The first phase creates the path that runs a traceroute to build a map between you and another device. And in the second phase, it begins to measure round trip time on every link along the way. And you can see not only round trip time but also packet loss. This takes a little bit of time to run, but the results provide you with a nice view of how traffic may be flowing between each individual path along the way.
Let’s run pathping on my Windows device. And, again, I’m going to choose Quad9. The first step will be a traceroute. We’ve seen this before when we ran the traceroute by itself. But after that finishes, you see that it will begin computing statistics for the next 200 seconds. Let’s let pathping compute these statistics and see what the results are going to be.
When pathping finishes its analysis, it will show you, again, the eight hops– the round trip time it takes to get to each hop. It will show you the amount of packets loss between the source device and this particular hop along the way. And then it also shows you how many frames were lost in between each one of these hops. Using pathping, you can start to see where problems may be occurring between different routers and be able to focus your troubleshooting efforts there.
Nmap stands for Network Mapper. It’s a way to search for other devices and learn what services are running on your network. Nmap is probably best known as a port scanner. It will find open ports on a particular device, and tell you what those ports may be doing. It can also scan the operating systems of those devices without having to log into the device. Nmap will send very specific queries to a device. And depending on the response from those queries, Nmap will determine the operating system running on that computer.
Nmap is also able to determine the type of services and the versions of services that may be running on devices in a very similar way. It doesn’t have to log in or use that service, it simply sends some very specific queries and examines the responses to those queries. And Nmap includes its own scripting language that allows you to extend the capabilities of this very popular port scanner. This is NSE or the Nmap Scripting Engine. And there are a number of scripts that are included with Nmap that greatly extend its functionality.
Let’s run a very simple Nmap scan. I’m going to choose one of the devices on my network. And it tells me very quickly that there were 991 ports that were closed. But it did show that port 22 was open, which is normally the SSH service. Port 80 was open for HTTP. Port 443 was open using HTTPS, and there’s others here as well.
We can also have Nmap perform an operating system scan. In this particular case, Nmap requires that I run it as root, so going to perform a sudo, and I’m going to choose to run Nmap with a dash O option to that same IP address to see if we can figure out what operating system is running on that device. It asked for my password, since I asked to run as root.
And as it goes out and performs the scan, it will show me information about those open ports. But notice, it also shows me the MAC address of this device which is a Synology device. And it is indeed running Linux. And it was able to tell me that all based from this Nmap scan.
Another important piece of information when you’re troubleshooting is knowing where traffic will be routed. In Windows, we can view this with the route print command. From my Windows workstation, I’ll type route print, and we’ll get a breakdown of both IPv4 and IPv6 routing tables.
At the top of the route print command, it tells us that there are four interfaces in this device. Two of them are physical network interfaces, one is the loopback interface, and the other is a tunneling interface used by Teredo. If we look at our IPv4 routing table, we can start to see the differences on where traffic might go.
For example, anything that would go to a destination 10.1.10.0 with a netmask of 255.255.255.0, would go to 10.1.10.17 that has a metric of 281. But if traffic was being sent to 172.16.231.0, again with a 24-bit subnet mask, that traffic would be routed out 172.16.231.129. So by looking at these routing tables, we can tell where traffic may be going based on what that destination IP address is.
Address Resolution Protocol, or ARP, allows us to find the MAC address of a device when all we have is the IP address. You can see this ARP table in your Windows device by typing arp-a. On my Windows device I’ll type arp-a, and we’ll see the ARP cache entries that are available for both of the interfaces that are in this particular computer.
There’s a 10.1.10.17 that has an ARP cache. And I have another network interface card on this device of 172.16.231.129, and it has its own separate ARP cache. Previously, my device has communicated to 10.1.10.1 and 10.1.10.2. And when it performed an ARP for those IP addresses, it received MAC addresses associated with those IPs and stored them in this ARP cache. If I need to communicate with those devices again, I don’t need to perform another ARP across the network, I’ll simply use what’s stored in the cache.
Notice on my 10.1.10.17, I don’t have an entry for 10.1.10.14. So let’s ping that address– 10.1.10.14– and watch some traffic go back and forth to that device. Then I’ll perform an arp-a again, and you’ll notice that a new entry has been made in my ARP cache for 10.1.10.14, and you can see the MAC address that’s associated with that device.