Blog Archives

Technology: SPLUNK! Take the SH out of IT….

Yes, you read it correctly!That’s one of many clever tag lines used by Splunk’s Marketers!  And it’s not half as clever by far as the Application!

It’s another free product!  Amazingly, the basic version of Splunk is gratis!  and can be downloaded from

It’s at it’s most basic level a log crunching tool, but it can do so very much more!  It’s one of those concepts that seems really simple, but the mind boggles at the complexity behind it.

I’ve been in a situation where I’ve had to wade through hundreds of Mb’s of SYSLOG’s grepping (looking!) for a particular IP address or two, and thought to myself “there has to be an easier way than this!”  and there is!

I’ll have to explain a little about what I did to make it work, but after no more than half a day’s tinkering, I was up and running with an amazingly powerful tool.  And the half day included messing about setting up VMWare ESXi Server and a Virtual Machine running Red Hat Enterprise Linux 5 to host it on!

Anyway, I had to do a bit of messing about from the base RHEL5 Webserver build to get it up and running, including tweaks to SELinux for the Security (yes, I ended up turning it OFF!) and installing a few PERL packages that I needed for some of the plugins, and installing the SYSLOG-NG package so I could do some clever filtering (More on that another time!)

Anyway, I set up SYSLOG-NG to filter different types of device identified by IP Address to different SYSLOG files.   I identified 2-3 specific categories of device that I wanted to filter;

  • Cisco PIX Firewalls
  • Nokia IP Security Appliances
  • Windows Boxes (using the SNARE Agent to convert Windows Event Logs to SYSLOG!)
  • Cisco Routers & Switches

I started with these for reasons that should become apparent as I explain further.

I then used good old “mkfifo” to create fifo handles for these Syslogs.

Then having installed Splunk, which was as simple as downloading and installing the RPM, and tweaking the RC files to ensure SYSLOG-NG and Splunk started automagically, I was in business.

Connecting to the server using a Web Browser on Port 8000, and added each of my FIFO’s as an Input Source and I was ready to go!  Splunk has a number of free packages that you can install that can influence it’s understanding of various different types of log, (this is why I separated the types of log to different pipes), so by appying these filters to the different input sources from SYSLOG meant that Splunk understood much more about the data it was actually receiving.

Then I simply swapped the Splunk Box for the IP Address of a UNIX box that sits there gathering SYSLOG’s all day, rather than revisit hundreds of Network Devices and tell them to send their SYSLOG’s somewhere else!

Bearing in mind you can send it just about any sort of textual log file, and can use Netcat, Nessus Scans, even TCP Dumps, or LEA Exports from your Checkpoint Firewalls (more on that in another post!), and the data actually starts to accumulate and flow.

This is where the sheer power of Splunk comes in to it’s own.  The best thing I can really do is point you at one of the Live Demo’s so you can play for yourself… so here:

Anyway, one day, a little tinkering and one “Enterprise Evaluation License” later, I’m wanting to reach for the proverbial Cheque book and buy licenses!  My colleagues have actually been using the tool to troubleshoot and fix real live problems on the Network, and they all love it too.

The three problems I have right now tho are that:

  1. It’s installed on a Virtual Machine, so the CPU and Disk Use are a concern!
  2. The Free Enterprise Eval License won’t allow me to index more than 5Gb of logs in a day
  3. I want it NOW NOW NOW!