Where to Start

When analyzing logs, there has to be a start point, but sometimes finding this start point is not so easy. Especially, when you introduce several sources of logs, there can be an overwhelming amount of information that may be daunting to start analyzing. I talked about log volume in another post, but it's an issue that we see pop back up in event analysis.

What am I even looking for?

With log analysis, it is important to be able to look at any number of logs and correlate them together into events that occurred. Often times, there is more than one log that generates from a users activity, and there will likely also be a nice trail of the actions they took on a system. However, it is another matter to actually be able to identify and tie these events together. There are logs that don't hold much information and really aren't that helpful, but an analyst needs to be able to pick these out to paint the full picture.

Event Correlation

This was another issue that I talked about earlier that we see once again. It is up to the analyst to identify what logs were made as parts of the same event. Logs act as the breadcrumb trail that we have to follow to find the full scope of what actually happened. However, you need the ability to follow that breadcrumb trail and weed out what isn't actually important or relevant. Similarly, you also need to be able to see early indicators of attack, to stop them before they happen, and identify what may just be a false alarm.

Are logs really boring?

I think its a common thought that log analysis is just boring, and I would agree that it can be. However, I stand by the idea that logs are absolutely fascinating and that they can be fun. Being able to stop an attack or even getting to the bottom of one that occurred is not only satisfying, but fulfilling. To me, this justifies the boring stretches of looking through logs. On top of that, just the things you can learn about a system or network from logs is invaluable information and can act as a health check. However, these boring moments can also be a good sign, because it means that there isn't an attack happening.

I always stand by the fact that logs are great and I probably always will, given that you know how to utilize them. Log analysis will always take some amount of time, but with the right tools and skills, it can be a rewarding and important process. But, at the end of the day, the struggle is still real.

The Event Record ID

The Event Record ID should not be confused with the Event ID. Event IDs are the identifying numbers that we talk about on W2L and categorizes each log into the event that occurred. Event Record IDs are a sequential number that is applied to each log to keep track of which order they were generated in. They are applied to every single log that is generated by the Windows The counting starts at 1 and goes sequentially up from there, and is tracked individually in each log file. This means that the System log and Security log may both have an event that has the event record ID of 5, but they will be different events.

Why use it?

Event record ID's can server as a sort of backup identifier to determine the true order of events. In cases where the event log has been modified and the dates or times of a log may have been changed, you can still see what logs happened in order. This is particularly helpful for event correlation, as you can see the events in a sequence even if they have been tampered with.

Hidden Information

I mentioned earlier that this was a hidden feature in the event log. When looking at each event, I expected to see this number readily displayed, but it wasn't. Even if you go into the Details tab (in "Friendly View"), this isn't displayed. It's not until you view the log in XML view that you can actually see this tag. The same is true of trying to view this log in the command line via wevtutil. If you view the log with the "/f:text" modifier, the information is not shown there, but if that modifier is removed, then you can find the event record ID in the XML data (although it is a little harder to find, as the data in this format is less formatted and more of a chunk of text). I can't help but wonder why this tag is not shown natively, as it can be helpful, especially as a backup.

In Conclusion

I think this tag stands to show that there is a wealth of further information that we can garner from event logs if we look at them a bit deeper. For example, in the screenshot above, there is a Correlation Activity ID. This is another piece of information that is hidden in the log XML data, but could serve to guide an audit or investigation. I also thinks this shows how cool logs are, as these are often generated so fast, but hold so much information and can be so incredibly helpful and there's sometimes more information than meets the eye.

What2Log's Role in Volume

You have probably noticed, if you use the site, that we have logging levels in our sidebar on the log pages, broken down into minimum, ideal and extreme.

This is because logs take up space and it really isn't practical for anyone to be capturing and analyzing every single log all the time. Using industry standards and some of our own intuition, we are compiling what we believe are the most important and relevant logs on a system to save you the time of doing so. Minimum level are all things we think you should be logging no matter what, these are the bare minimum logs you should have. Ideal is events that will likely generate more logs, but are still important and should be added. Extreme is events that will generate a lot of events (looking at you application opened and closed) but are still hugely helpful when analyzing events.

Too Many Logs

Quite frankly, there are logs that exist that will never be used and plenty more that will be seldom used. While the quiet assurance that your machine is logging everything (and I mean everything- almost) is comforting, there is an argument to be made that these logs should be culled at times. This is not to say you should throw out all your logs out after a week to make room, and you really shouldn't do that, but you should be mindful that you can delete logs.

Many approaches to logging include "we need everything" and this isn't always beneficial. Storage is expensive, especially when you're looking at needing to store the logs from a large network, and if you are capturing every single event. Eventually, you're going to be drowning in hard drives. I personally use Windows 10 and my own (and relatively new) computer has 29k logs in the Security log alone. Apply this to a network scale, and you're looking at hundreds of thousands, if not millions, of logs.

Compliances

There are compliances that require you to keep your logs for a certain amount of time. PCI DSS comes to mind as it requires logs to be kept for at least a year. Similarly, a company may have their own preferences for how long to keep logs. These of course, should be followed, but it does not mean that every event has to be logged.

What is the Best Approach?

Tackling volume can come in choosing to only log certain events up front or by optimizing a log aggregator on a network, but it is something that should be done. There is no one best approach, unfortunately, computers are funny that way. What2Log hopes to help by showing that you really don't need to log everything and there is time and money to be saved by not doing so. Are we the final voice or the only voice you should listen to? Absolutely not. What2Log is still missing a lot of information; we're constantly researching and adding new logs that we think belong on here.  After all, the struggle is real.

Sawmill

The biggest new feature is Sawmill. This was something that we have been working on for months and had always intended to have early on in W2L's life, so to see it in production is refreshing. Sawmill is a "shopping cart" for commands so you can build an easy to run script for logging. Just select what you want and we do the heavy lifting behind the scenes and you get a script for just what you need. We plan to expand this feature further to make it work a bit more seamlessly and add new features in the future.

Quality of Life

I'll admit, I'm not perfect. This update includes a slew of formatting and spelling fixes across the site. One honorable mention in this category is parentheses, which are now no longer formatted and should appear as " instead of ". This was particularly important as they appear frequently in our commands. Hopefully things make a bit more sense now!

Splitting Logs

We also split a few logs up so that there is now only one log per page. This in particular applies to Group Membership change (now Member Added to Group and Member Removed from Group)  and WiFi Connection (now WiFi Connection and WiFi Disconnection).

WiFi Connection Logs

We have figured out a way to make this log command line compatible! We now have a way to view the log of a WiFi connection or disconnection from the command line.

Logging Levels

We changed a few logs around, too. Failed user login is now a Minimum level log as we think it is a really important event to be on the lookout for. We also bumped Software Updated down to Ideal level, as this is less likely to be an indicator of attack, but is still a useful log.

We hope you enjoy these new changes to What2Log as much as we do!

To put it simply, this log is generated whenever someone with administrator privileges logs on to a machine. These events are seen usually in close proximity with Event ID 4624, which is a User Logon. In fact, if an administrator logs on, you will see both of these events generated, with 4624 occurring first.

Admin Rights

Because these events are generated when someone with admin privileges logs on, they can be used to see if a user has admin privileges. You expect to see this event after a log on if this is true. Similarly, you will be able to see if a user has unexpectedly gained admin privileges. This event can be used to monitor the usage of administrator privileges on a system and verify who has them. While not indicative if they are being used, it can still be an indicator that they are being misused or mistakenly assigned. After all, there is a big difference in what a normal user can do versus what a user with administrative powers can do.

More than One Log

Logs can help in more than just the cleanup and analysis of an attack; they are a very useful monitoring tool. Event ID 4672 highlights this principal, although it can certainly be used in both instances. I think this event also helps show the number of things that actually happen when a user logs in to a computer. Someone may try and cover their tracks by deleting a 4624 event, but there may still be a 4672 event showing that they logged on and actions were performed. Using multiple logs from different sources can paint a better and more full picture of what happened at any particular time on a machine or network.

Logs can paint a picture of what happened or even what may happen and that is why they are so important. By using multiple logs, you can understand every step that was taken (and potentially block it.) Using Event ID 4672, especially in combination with other logs, can show when an administrator was on a network (or if someone has gained admin powers was on).

It is important to note that they may not all be sequential, a port scan from last week may tie into an attack that occurred today. Similarly, event correlation goes beyond simply tying in the logs from different sources and seeing what occurred at the same time. While it is important to use the different sources to create a full picture of what happened on the network, it is also important to look at the history in the logs and see what happened in the lead up to the incident.

There is, of course, something out there to make life easier when it comes to event correlation. Certain software allows for automatic event correlation, sometimes in combination with data aggregation. This makes the process of correlating logs to certain events a lot easier and can save a lot of time. It is possible that events may be missed that would have been caught through manual parsing, but the same is true in reverse.

Event correlation can even be used to help block an attack before it begins. Similar patterns that were seen in previous attacks or known patterns may show up and allow for better preparations to be made.

However, event correlation is really a chance to learn. By understanding exactly what events occurred in an attack, one can better prepare themselves or their network for the future. There are logs that may seem inconsequential, but when correlated to an event, paint a bigger picture that it really was an early warning sign. Obviously there is no one solution, as there may be false positives, but it is certainly something that can help better prepare a network for attacks or unusual activity. At the end of the day, the struggle is still real, but there event correlation can help make it a little less so.

Log aggregation is the practice of consolidating  all of a networks logs into one "container" to make them more easily searchable. It is important to note that this is not a guide to log aggregation tools, as plenty of those exist already. This is simply a discussion of both sides of log aggregation and the pros and cons that it presents. Centralized log aggregation seems like the smartest thing to do, especially with multiple log sources that may be on a network. By centralizing the logs, you are making your life so much easier by combining all of your information in one, easy to access location. However, is log aggregation all that it's cracked up to be?

Pros

Aggregation certainly has its pros, otherwise it would not be as popular as it is. Log aggregation means that you don't have to go to each individual log source to analyze them, but you can instead just look in one place and see all the logs in one place. This, obviously, can save a lot of time and frustration of trying to analyze each machine that generates logs. There is also the ability that is then presented to quickly search through or analyze all of the logs from a network, not just the ones unique to a system. Some aggregation software also allows for real time monitoring, creating a dashboard of all the combined logs as they happen.

The biggest draw for log aggregation has to be the time saved, as it really does make your life easier. In a larger network, where you may be seeing logs from half a dozen or more sources, such as firewalls and cloud services. If they are in one place, they can be looked through all at once and may help in data correlation. When all the logs are in front of you, it is much easier to identify patterns and see the signs of an attack.

Cons

As mentioned, log aggregation definitely has its cons. If you don't have a good aggregation software, you may end up with a confusing mess of different log types all combined into one. This could arguably make analysis more difficult as you have to take the extra step to make sure you catch the differences in each log file. This sounds harmless, and usually is, but a date format difference may cause an issue with data correlation. If the day/month/year format is used, then 1/4/2021 means the 4th of January, but if it is month/day/year, this would be the 1st of April. If you are not aware of these changes, or your software hasn't caught it, then you may severely skew the events of what happened on a particular day.

Another important point is that a log aggregation server or storage device would likely become a prime target during an attack. You absolutely need a backup of any log aggregation you have to ensure that you are not completely dead in the water in case of an attack.

Conclusion

In the end, it is easy to see why log aggregation is such a popular choice in log management. As log as one is aware of the potential challenges they may face, it seems that the pros outweigh the cons. The amount of time that stands to be saved is a benefit that anyone would be hard pressed to argue is a bad thing. Log aggregation is a time saving utility that makes the job of log analysis easier, which I firmly believe is highly important. As long as one is mindful and does their research when selecting a log aggregation tool (or even making your own solution), most everyone will see improvement from log aggregation. However, at the end of the day, there is no one definite answer and the struggle is still real.

-Log collection
-Centralized log aggregation
-Long-term storage and retention
-Log rotation
-Log analysis
-Log searching and reporting

By breaking this field down, we can begin to understand the array of work that is required when we start looking at logs and why to manage them.

You might wonder why there is just not one way this is done, a single framework that we all can follow to ensure we are the most secure and prepared when it comes to logs. The short answer is there is no one answer when it comes to logging. Logging (and in turn, log management) is an individualized process that needs to be evaluated by companies and individuals on a case-by-case basis. What works for a multinational corporation probably won't be practical for a small home network.

Logging is not without its problems. We have issues of volume, especially when we introduce firewall and other network device logs. If we run out of space where we store our logs, we may have to delete certain logs or simply delete our current ones. Neither of these are good solutions, but they are sometimes seen as the only way to get around such a problem. Another problem is that analysis can take too long and may be unwieldy. Especially in a situation where you are seeing logs from multiple sources in different formats, trying to understand all of the information that is presented to you is almost an art. And, it is important to note, that if you do miss something, it could lead to big trouble in your network. We also see issues with data correlation, tying events together. It's easy to miss a single event that ties together two big events, but this is a key piece of information that we want to know when analyzing logs.

This "series" hopes to talk about some of these issues a little more in depth and help get you thinking about what can be done to help fight these issues. I know that we will not have the holy grail of answers as to how to deal with logs, but hopefully starting the conversation is enough. After all, the struggle is real.

What2Log started out as a passion project when we noticed a pretty major gap in information surrounding computer logging. We all know that logging is important for a variety of reasons and not just in the wake of an attack, so this lack of information was pretty eye-opening. We decided to change this and started compiling all that we could on computer logs and thus What2Log (or W2L) was born. What2Log is an ongoing project of InfoSec Innovations.

About The Logs

What2Log is a resource of OS's, their built in tools, and their built in logs simplified so any user can understand how and why to view them. Logs are often seen as a chore, but we wanted to get rid of that idea. Logs can give a user a lot of really important information and we wanted to highlight that.

The Future

We know it's incomplete, but that is where we are turning to community suggestions and advice. We want this to be a reference that people can give feedback on so that we can help grow this. We have pretty ambitious plans with some future expansions and tools in the works, but we want to hear from you.
With future plans to include more Operating Systems and a scripting system, W2L is an ambitious project.

We'd love to see you get involved! We are on Reddit and Twitter, so please come join in the conversation.

-Thanks!