Splunk And Your Logs - Who Is That IP?
If you’re using Splunk already, it’s a fair assumption that you’re currently investigating some form of network access logs. Whether that’s from your edge firewalls or some sort of IDS or IPS, email or web filtering gateway, a proxy, a wireless access point or any other form of Internet facing system. Somewhere within your Splunk instance you will be seeing public IP addresses of locations that someone or something within your network is trying to get out.
The question that I hear often asked is: “Who are we speaking to?” If you have IDS or IPS, often the scarier version of this question is “Who is trying to speak to us?”
We all know within Splunk we can run a simple search; take that as a baseline and then create a dashboard using that base search in just a few clicks. Metrics are one of the things that Splunk excels at. Transforming numbers into real world information is another area that we can use Splunk and its extensions and add-ons to great effect. Often when you’re tasked with reading through some firewall logs you already have some parts of the puzzle. You likely know the source internally, and quite probably the user who was browsing at the time of the incident. Knowing where they were trying to get to is the unknown piece in most cases.
Let’s look at some data first to get the ball rolling. If you’re familiar with firewall traffic syslogs, then the below will make sense. I’ve taken the traffic from the last 15 minutes from one of our more dodgy users and built a table of all the allowed destination IP’s he’s successfully communicated with (I’ve also included the “Apps” for a bit more immediate context):
From here you can click on any of the IP addresses to gather a subset of the results:
Once the events load, you’re back to the standard Splunk search, tailored down to the IPs that you were looking at earlier. Not a huge help so far…
Ordinarily here you would need to take each of those IPs that were deemed suspicious and pop them into Virustotal or “A N Other” DNS/threatintelligence lookup site or application you may already have. A lot of the time the first piece of information you requires is: Who owns this IP? So let’s answer that question, or better yet, get Splunk to fill in the blanks for you.
While logged into Splunk as a Power User, clicks Settings > Fields > Workflow Actions, select an appropriate App Context (e.g. if you already have an app for your firewall) and then click “Add New”. Complete the following information:
- Label - Give the workflow a name
- Apply only to the following fields – insert the field(s) that you require this action to work from, in my example: dest_ip
- Apply only to the following event types – This can be any event type you want to limit this action to
- Show action in – Choice of fields, events or both. I have chosen both, but it depends where you want it to show up
- Action type – Link or Search. Link will run an external query, Search will search again with Splunk, so I have chosen Link
- URI – This shows when you have chosen “Link” in the option above. For our example we will use: http://www.dnsstuff.com/tools/whois.ch?ip=$dest_ip$ (the $dest_ip$ passes the value in the dest_ip field as the content
- Open link in – choice of New Window or Current Window. As we don’t want to lose our Splunk search, we choose New window
- Link Method – either Get or Post, depending on the external search, ours is a get
Save the above, and return to your search…
Expand the event itself (click on the arrow to the left of the event next to the time stamp) and then either click on Event Actions and see your new workflow at the bottom of the list) or click on the downward facing arrow to the right of the field you chose (in my case, dest_ip)
I can now automate the lookup procedure for any IP that occurs in that field, in my example, the IP is owned by Google.
These workflow actions are not just used for looking up IPs, you can also call CMDBs for example for asset information, any externally facing API, other apps and APIs in Splunk Apps and lots more.