Trawling event logs, man I’m excited. This is the kind of stuff I dream about and it’s what gets me going in the morning. I can’t wait to get stuck into millions of lines of events with the same error innocuous occurring every hundreds of times a second. With that one critical alert buried in all the noise.
I love it as much as I love grating my knuckles on a cheese grater.. Wait a minute, that sounds awful, hang on reviewing event logs is equally awful!
Now that I’ve wasted your time with my pointless preamble, here is how I’ve made things better. In my years reviewing event logs, I’ve taken a few learnings.
- It’s tedious
- It’s really tedious
- I want the information summarized
- I want to know if the error is new, or re-occuring.
Yes I can do all the above stuff manually, but why on earth would anyone want to! So I used powershell to improve things.
Let me set the scene. You’ve just patched a critical server and it’s been rebooted. Some of the first things you should review (yes really you should) is the windows system and application event log. So we RDP, open the event log, clikety clickety click and all is well. Sweet, move on to another 50 servers… Well that sucks. So here is a better way.
I produced the following script to do the following.
- Get all System and Application log events that are not Informational, so thats the Critical, Error, and Warning events
- Group them. If a certain event has repeated itself 1000 times, just tell me once and let me know how many times it appears
- Based on item 2, let me know the last time it appeared
- Was this event present before I rebooted the server (let’s look back 7 days as a line in the sand)
- Who on earth wants to log on to a server, let it take a ComputerName parameter instead
- Alternate credentials need to be specified as well as hey you’re running a best practice shop right 😉
Ok enough talk here is the script, hopefully it makes some sort of sense:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 |
function Get-EventLogStats ($ComputerName = $env:COMPUTERNAME, $Credential) { #First up get the Date and Time of last boot write-host "Getting $ComputerName Last Boot Time..." $query = "Select LastBootUpTime From Win32_OperatingSystem" $LastBootUpTime = (Get-WmiObject -Class Win32_OperatingSystem -Property LastBootUpTime ` -ComputerName $ComputerName -Credential $Credential).LastBootUpTime #Then, get all Non informational events between now and last boot from the System and Application event logs write-host "Getting $ComputerName Event Logs since last boot..." $query = "Select * From Win32_NTLogEvent Where (TimeWritten>='$LastBootUpTime') AND (Logfile='System' Or Logfile='Application') AND (Type<>'Information')" $RecentEvents = Get-WmiObject -Query $query -ComputerName $ComputerName -Credential $Credential #If there logs are found continue.. if ($RecentEvents) { $daysBack = 7 # look back 7 days from the last boot for historical analysis write-host "Getting $ComputerName Event Logs $daysBack days prior to last boot.." #Jiggery Pokery stuff - The timestamp is in file format, convert to date to subtrack 7 days the reformat it. $priorDate = (([wmi]("")).ConvertToDateTime($LastBootUpTime)).AddDays(-$daysBack) $priorDate = ([wmi]("")).ConvertFromDateTime($priorDate) #Get all events between the last logon date and 7 days ($daysback) prior to that $query = "Select * From Win32_NTLogEvent Where (TimeWritten>='$priorDate' AND TimeWritten<='$LastBootUpTime') AND (Logfile='System' Or Logfile='Application') AND (Type<>'Information')" $HistoricEvents = Get-WmiObject -Query $query -ComputerName $ComputerName -Credential $Credential #lets start comparing! write-host "Anaylizing Events..." #First up group the recent events by Event Type, event source, and event ID (Code) $grouped = $RecentEvents | Group-Object -Property Type,SourceName, EventCode #loop through each group $results = ForEach ($group in $grouped) { # $details is a 3 dimensional array of Event Type, Source, and ID $details = $group.Name.Split(",") $EventType = $details[0].Trim() $Source = $details[1].Trim() $EventID = $details[2].Trim() #Grab the most recent event in the group. $lastestEvent = $group.Group | Sort-Object TimeGenerated | Select -Last 1 #Assume event is unique by default $preExisting = $false #Check Historical events to see if there is a match based on Event Type, Source and Event ID $matches = $HistoricEvents | Where-Object {$lastestEvent.EventType -eq $_.EventType -and ` $lastestEvent.SourceName -eq $_.SourceName -and $lastestEvent.EventCode -eq $_.EventCode} #if we get a match note the event is pre existing if ($matches) { $preExisting = $true } # Create the event object to return (stored as an array in $results) $props=@{ LogName = $lastestEvent.LogFile Count = $group.Count EventType = $EventType Source = $Source EventID = $EventID LatestMessage = $lastestEvent.Message LatestTimeGenerated = ([wmi]("")).ConvertToDateTime($lastestEvent.TimeGenerated) PreExistingEvent = $preExisting } New-Object PsObject -Property $props } #display results $results } # If no events are found, awesome! Or more likely, whats wrong?? else { Write-Host "No events found, you fortunate bugger!" } } |
And the result?
And there you go, lives saved! (from boredom that is, I’m not some kind of hero.. or am I??)
Hopefully that is of use to somebody, in a later blog I will talk about how you can display stuff like this in fancy HTML pages styled with bootstrap. I find that stuff so much cooler than grating your knuckles.
Cheers!