How to Build Logging Pipelines That Outlast Your Security Vendor

I have always said that security at its most elegant is simplicity. Do the basics first and after you build that foundation begin to add your walls, doors, windows and flooring. Yet time and again I meet with customers who want to pick out roof shingles on a house that has no foundation. They want automation, they want machine learning, and yet in many cases they do not have a complete logging architecture. Actually in more cases than I can remember I have seen where they have no logs at all—and they want to use the vendor’s POCs to help them build an architecture that should already have been built. Building out a world-class security program requires a mastery of the basics where logging is a fundamental basis of a security program. 

When we look at elite athletes we marvel at their abilities. It’s because what we are seeing is someone who has built their foundation on the basics years ago. Yes, any world-class athlete has mastered the basics first.  Similarly, to build a world-class security program it makes sense to master the basics before going for the advanced features. In this article I’m going to help you build your logging pipelines so you can build a foundation for the future. 

I can already hear the excuses, see the tears, and hear the complaints about cost, time and quite frankly in many cases, politics. But remember, if we want to build a world-class program we have to push those excuses aside.  

Building your logging architecture with Syslog

Syslog has been around for decades. You can follow numerous architectures to build out a resilient and fault-tolerant syslog architecture. Depending on your compliance level there will be some cost in storing syslog. What is great is that the function of syslog is already built into most of your devices.  This article shows how to build a TCP/UDP syslog architecture using NGINX where there is no cost other than hardware.

Building your logging architecture with Windows logs

Windows logs can be a bit trickier but fortunately Microsoft solved this problem long ago. In 2008 they released Windows Event forwarding. With a few simple commands and a virtual machine acting as a Windows Event Collector you can have all of your windows logs centralized into one location…yes, a Windows Event logging pipeline!  There are numerous articles on how to build a WEF solution and Microsoft has guidelines on sizing based on the number of machines but again the only cost is a virtual machine. See these articles “Use Windows Event Forwarding to help with intrusion detection” and “Setting up a Source Initiated Subscription on an Event Collector Computer” for more on the topic.

Using Supercharger

There is a tool that can be used to help set up, configure and monitor an entire environment, and that tool is called supercharger.  It was created by Randy Franklin Smith of the Ultimate Windows Security fame and it takes a lot of the challenges of the environment away. I have used it to set up a WEF environment in as little as 15 minutes with over 2,000 machines. It is worth your time and energy to investigate using such a handy tool. 

I have also found that I have fewer objections  from Windows admins because they appreciate the fact that I don’t have to deploy agents on their domain controllers. It also helps that they know the reputation of that website as they have used it to troubleshoot Windows Event codes for years.  You can get a 60-day licensed copy of Supercharger here. 

There’s a real benefit to building a strong security foundation starting with your logging pipelines. It is worth your upfront effort and is quite simple. The three options for building your logging pipeline I’ve shared are easy to set up and will have you up and running in no time. It will also give you the data sources on which to build a solid security incident and event management system.