Log files can provide a comprehensive record of data flows, firewall dispositions, and many other important data points. In this video, you’ll learn about logs from firewalls, applications, endpoints, operating systems, and more.
We store a massive amount of security related information in log files contained in servers, devices, and other components on our network. Those log files contain information such as the traffic flows that were blocked and the traffic flows that were allowed. We can get a list of exploit attempts, especially from intrusion prevention devices. We might want to see what categories of URLs may be blocked on a particular user’s workstation. And we can see DNS sinkhole traffic, which could point to some type of malicious process occurring within our own network.
This allows us to document every traffic flow on the network, where we can provide information on what attacks may be occurring, and we can correlate that information with other logs contained in other devices. One of those devices that contains an amazing amount of detail about the traffic on our network is the firewall log. Our firewalls are often monitoring all traffic that goes from the inside to the outside of our network and vice versa.
And from there, we can get information about the source and destination IP addresses associated with those traffic flows. We can see what port numbers are being used. And we can see what the firewall does with those traffic flows. Does the firewall allow the traffic flow to proceed? Or is that traffic flow blocked on the firewall? If you’re using a Next Generation Firewall, or NGFW, you can also get information about the applications that are in use.
These next generation firewalls are also very good at providing feedback on URLs or URL categories that are being used. And they may also be able to point us to suspicious data or any anomalies with the information within these traffic flows. Here’s a view of firewall logs from a next generation firewall. Each one of these lines is a separate flow of traffic traversing the network. Each one of these traffic flows contains the time and date of the traffic flow, the source IP address, and in the case of this firewall, the MAC address that received that data.
You can also see the destination IP address and the application that is used for this traffic flow. And then you finally get to see the disposition, or result of this traffic flow whether it was accepted through the firewall or whether it was blocked. The applications that we use can also create log files that may be very useful when analyzing security events. This might include information from the Windows Event Viewer log, specifically the application log section of the Event Viewer.
If you’re using Linux or Mac OS, you can look in the /var/log directory. And all of this information would probably be rolled up into one single security information and event manager, or SIEM. And all of this information can then be filtered out or viewed in different ways inside of the SIEM itself. You may not realize it, but the endpoint devices you’re using also contain a great deal of log detail. If you’re using a laptop, a desktop, a phone, a tablet, or any other endpoint device, there is an extensive amount of log information available.
For example, you may be able to view information on log in and log off events. Maybe there’s information on system events or processes running on that endpoint. And there might be details about the management of the device, such as password changes or lock outs. And you can also view information on any directory services. All of these endpoint logs can also be rolled up to a SIEM so that you can then parse out that data and view different correlations between what’s happening on the endpoint, what’s happening on the network, and any other devices that you may be monitoring.
Once you have all of this endpoint log information in the SIEM, you can now start comparing and correlating that data against log file information from other devices. So you can track each step of the way for a particular traffic flow or a potential security event inside of the SIEM, all by consolidating these log files together into one single source. There’s also an extensive amount of security information stored in the operating systems themselves. Many operating systems keep a log file associated with security events.
So you can monitor individual applications. You can see if there are brute force attacks or any changes to critical system files, and anything relating to authentication is usually also stored in these security log files. This log file information inside the operating system may be able to provide you with a heads up of any type of security events. For example, this log file might show that a particular service was disabled. And that service is not one that would normally be manually disabled by the administrator.
That single log file event may cause a security alert to be generated, and you may be able to stop a particular event from occurring just by monitoring this log data. As you can imagine, we are collecting a very large amount of data across all of our systems. And so we may not want to send all of our log information into a SIEM, but instead, only send the information that’s important for us to be able to make security decisions.
Another great place to gather information is from IPS or IDS events. This would be associated with intrusion prevention systems or intrusion detection systems. These days, we don’t tend to use standalone IPS or IDS systems. Instead, that functionality is often built into a next generation firewall. An IPS log is going to provide information about known vulnerabilities or known types of attacks. So it might look something like the log file I’ve taken from an open source IPS called Snort.
In this IPS, we have a timestamp. It shows us the class of the alert, and it tells us that it’s a possible denial of service attack, specifically a SYN flood attack. It has a priority inside the IPS of two. It gives us a source IP address, a source port number, and a destination IP address and destination port number. We can now take all of these individual events and also roll those up into our SIEM so that we can now start extracting and correlating this data with all of the other devices on our network.
There’s also a great deal of log information we can gather from our network infrastructure devices. This might include our switches, our routers, our wireless access points, or even VPN concentrators. These log files can identify any changes that might occur to any of our routing tables. We might be able to identify authentication errors that are occurring to someone trying to gain access to a switch or to a router. And we might also be able to identify other types of attacks that are occurring on the network.
For example, in this log, we can see there is an informational entry that shows a TCP SYN attack was identified on port gigabit eight. And we can see that the TCP SYN traffic destined to the local system has been automatically blocked for 60 seconds. Sometimes, you can gather important information that is contained within the documents that were transferring over the network. Stored inside of the documents that we are often creating in our word processors, our spreadsheets, our graphics programs, and others is information that describes that particular file.
For example, if you’re reading an email in your email client, you don’t often see the large amount of metadata that’s stored within the header of that email. But if you ask your email client to show you the entire email document, you can often see information that’s hidden within the email headers, things such as the servers that sent the email or the addresses that were specified as the destination.
You can also see this if you look at the description of pictures taken with a mobile device. You may be able to tell what mobile device was used and even information about the GPS coordinates that were associated with this location of the picture. There’s even metadata in the browsers that we’re using, things like the operating system that we’re using, the browser type, or the IP address that you’re using.
And if you look at the metadata that’s inside of a word processing document or a spreadsheet, you may find information on the person that created that document, their address, their phone number, and perhaps their title. To give you an idea of what some of this metadata might look like, let’s look at the header of an email message. You can see there is extensive information in this email header including the IP addresses that this message was received by.
We can see SPF information, other details about signatures and information that can help you determine where this email message originated, and the process that it used to be transferred into your inbox. If you’re performing vulnerability scans on your network, then you’re creating an extensive amount of log information. This is going to give you details about what this vulnerability scam was able to identify.
For example, it may identify devices on your network that don’t have a firewall configured. There may be no antivirus on that device or anti-spyware. And we’re able to identify that in the logs of our vulnerability scans. We might also be able to identify devices that are misconfigured. For example, there may be shares that are available that you can access without using any type of username or password. Or perhaps there’s an operating system that has the guest access turned on when the best practice for your organization might be to completely disable all guest accounts.
And of course, once you update the signatures inside of your vulnerability scanner, it can identify any operating systems or applications which may have known vulnerabilities that need to be patched. Here’s a summary of the results from a vulnerability scan. This log information shows that there are certain operating systems that were running on our network that are currently unsupported. And there might even be NFS shares that are on our network that are readable by anyone in the world.
As you’ve probably seen already, there is an extensive amount of log information that you would need to go through to be able to find details hidden within all of this data. Fortunately, most SIEMs have the ability to create a set of automated reports. This may be a feature that is built into the SIEM itself, or you may be able to use a third party report generator to simply access the information that is currently stored in the SIEM.
Of course, these reports aren’t very valuable unless someone actually reads them. And one stumbling block that many organizations will find is they will create these automated reports, but then simply ignore them when they arrive in their inbox. This might also take a bit of finesse to be able to find the right mix between the type of report that you need and the amount of time that it takes to create that report. When you have a SIEM that may contain terabytes and terabytes of data, it may take an extensive amount of processing power just to create a single report.
So you may need to be very specific about the reports that you’d like to generate so that you can create them as efficiently as possible. Instead of waiting for a day or two to receive reports that are generated by the SIEM, it might be useful to have a summary of information that you could view at a glance. This is often available through a dashboard. Many SIEM and other security devices allow you to customize a screen, containing information that will be important to review at a glance.
These dashboards may be customizable, or there might be predefined dashboards built into the SIEM itself. This is often the type of data that’s useful to see at a glance or on the main screen of your security operations center. This commonly doesn’t show long-term information primarily because it takes such a long amount of time to be able to compile all of that data together. The dashboard is designed to give you something that you can view instantly and get an understanding of how the current status might be.
For example, you can see a breakdown of the system itself, any active firewall rules, you can see warnings that have come through, and information about users and devices that might be on the network. One of the best places to gather data on your network is from the network itself. Being able to analyze the packets going over the network can give you a great deal of insight into the operations of the networking equipment, the applications, and any security issues.
This may require that you use a third party utility, such as the Wireshark utility that we see here. This may be able to provide you with the ability to capture data that’s running across the wired network and the wireless network. And in some cases, devices like switches, routers, or firewalls may have the ability to capture packets inside of those devices themselves. These captures give us detailed information about traffic flows at the packet level. Everything going across the network is captured, and that allows us to analyze every bit and byte that’s transferred over the network.
The Wireshark summary view here in the top pane shows a packet by packet breakdown of everything that’s being sent over the network. You can see the highlighted frame is one that’s being sent as HTTP traffic. And you can see the GET command is written inside of the packet capture itself. You can then see all of the following packets that describe the transfer of this file that was requested with the GET command.
The bottom half of the screen is the detail pane. This provides us with a breakdown of everything highlighted in that single frame at the top of the page. In this case, we can see the ethernet data. We can see what’s involved in the IPv4 header. We have details about the TCP header. And then we have the HTTP data itself being shown all in that detail view.