Escaping Dante’s SOC Inferno: The Anger of Shattered Dreams  

What the…Hell? (An Open Letter)

Cutting straight to the chase: I’m angry. And here’s the story about why I’m feeling this way. Is this your story too?

For years I’ve wanted to get into InfoSec. Ever since I was a teenager hacking away at my parent’s laptop, I knew I wanted to get into the field. So, I went to university, majoring in Computer Science with a concentration in Cybersecurity. I graduated with a 3.8 GPA and academic honors. But then… crickets. 

Dozens and dozens of interviews over the next few months following graduation. Zero job offers. 

I began to question if I had made a serious mistake. Was I not cut out for this kind of thing? Did I waste four years of my life in college and put myself into financial debt for no good reason?

So, I accepted my plight and took a bulls**t job completely unrelated to my degree, grinding it out for the next several years. Yet the hunger to get into InfoSec never subsided. I had this constant yearning inside of me. I just knew I would be great at it… 

If someone would just give me a chance! 

As the weeks turned into months, and months turned into years, I had just about given up all hope. Then, it happened! An old college friend had just become the director of security for a Fortune 500 oilfield services company based in Houston, Texas. She rang me up to see if I was interested in interviewing for a role as a SOC analyst on her team. I jumped at the chance to explore the opportunity. 

Thinking that my inside connection would give me a leg up, the interview process was grueling! Three phone interviews (one with HR and two technical screens), followed by a full day of back-to-back interviews with potential team members and key stakeholders. 

Replaying the events of the day, I sat in my hotel room that evening thinking to myself that despite my connections there’s no way I was going to get an offer. The next day, I packed my bag and hopped my flight back to Boston — feeling completely defeated.

As I landed at Logan Airport that afternoon, I turned my phone back on and was shocked to see a text message from my friend:

Congratulations! HR will be sending you an offer letter tomorrow. Can’t wait to have you on the team!

My dream of finally being in InfoSec was about to come true! (or so I thought)

Getting back to the anger portion of our program… 

What my friend failed to mention before, during or after the interview process is that oilfield services companies seemingly have a very limited budget for staffing in security operations (IKR?!).  

So, here I was, 16 months into my “dream job,” and I absolutely hated it! I shouldn’t say that. I didn’t really hate the job, per se. Just the part about being forced to do it with one hand tied behind my back, whilst having my mission defined by inappropriate metrics (measuring how fast I can close tickets does not equal security!), and subpar tools that were a pain in the ass to get data from. Apart from that, things were okay, kinda. My team members were all super smart and fun to work with, but we were all suffering together.   

Over 100,000 endpoints and 10,000+ servers of various flavors, all generating a ton of alerts. Not to mention all the IoT in the mix. 

So, my average day was spent filtering through hundreds and hundreds of security events and alerts, looking for the needle in the haystack because most were false positives, and those that weren’t were missing any useful context. Endless rabbit holes that wasted my time and effort.

Get alert. Go down the rabbit hole. Tune alert. Lather, rinse, repeat. Argh!

So frustrating!

Just look at the math: 

If just one out of every 20 servers throws up an alert during the typical eight-hour shift, we’re talking about ~500 alerts during any given day. That’s about one alert per minute that must be investigated! It’s an impossible task for anyone to accomplish. At least by triaging them manually. 

We’re always going to be a step behind our adversaries and there’s little that can be done to change that. Zero-days will always exist. Malware will constantly morph itself to avoid detection. And the three-letter agencies will continue to have WikiLeaks. Let’s face it, threat actors only have to get it right once; we have to get it right every single damn time! 

So, I’m stuck in a living hell trying to keep up with it all without any relief in sight. It’s not what I imagined my career in InfoSec would be like. Again, I felt utterly defeated.

But, after my one-year work anniversary, I finally felt like I had enough time with the company that I could propose a solution to address the problem and began researching SIEM platforms to replace our forked GitHub project used for alert triage.

After extensive research putting together a feature comparison matrix and identifying my top five products, I put together a proposal afor my boss. Of all the tools I looked at, I liked what I saw from Exabeam the most. Their platform looked like it ticked all the right boxes – automation of investigations, reduction in false positives, and the elusive context I was so desperate for.    

Everything rides on this

Presentation day wasn’t quite what I hoped for. My boss was clearly supportive of my efforts, but I could tell something wasn’t right. I had known her throughout college and something was off. I could just see it in her eyes. “Budgets are really tight at the moment,” she concluded with a sigh. I countered with “let’s get them in and see what they have to say.”  

Ultimately, she agreed to have the Exabeam folks come in and do a proof-of-concept (POC) for us that blew away anyone’s expectations. The data architecture, analytics and orchestration were phenomenal. We could potentially reduce our alert volume from one per minute to perhaps as few as a dozen an hour, while still introducing user and entity behavior analytics into our monitoring (something we were lacking). We loved the automated timelines, that we knew would save us hours of manually digging around, and the triage capabilities which would allow us to focus on the more complex and interesting work. And we would finally be able to detect compromised credentials, which our current system was totally incapable of doing. I was so excited about the difference this would make to my day-to-day efforts and knew it would really bolster our security posture.  

So, we concluded our POC but weirdly there were crickets from the management team. Meanwhile, our SOC team sat here trying and failing to process 500 alerts, day in and day out awaiting the thumbs up. Then two weeks ago, my director called me into her office to explain to me that while she and the senior management are very appreciative of my effort to create such a compelling case for Exabeam, in the current climate they’re “deferring spend until things settle down.” I just don’t get it. 

Lesson learned the hard way

While I like my co-workers and management team (even though I don’t always agree with their decisions), I can’t keep carrying on. I feel myself getting burned out already. And I’m not encouraged by the stats I see regarding the longevity of SOC analysts. I can’t keep at this forever, but I also realize that it’s going to look bad if I leave without putting in at least a couple of years here. 

One thing I will make certain of when interviewing for my next role: ask about the tools they use! 

I didn’t know enough when I interviewed here to think about the impact that it would have on my daily life working with tools held together with duct tape and bubble gum. 

It’s exhaustingly frustrating.

Don’t be like me. Do your homework. And should you propose a solution like Exabeam, don’t forget to talk about risks and return on investment. These are the language of the decision makers and budget holders…

Signed,

A. Burned-out SOC Analyst

This person doesn’t exist, but you may recognize their story

Okay, this person doesn’t really exist. But true stories just like theirs absolutely do. All too often, we hear the horror stories of the hell that SOC analysts go through trying to perform their jobs and lacking the tools needed to achieve the results that management expects.

We’re here to help. In fact, we’ve put together a special resource if you’re looking to accelerate your SOC operations:

Evaluating and Selecting SIEM Tools: A Buyer’s Guide

This guide will help you to explore the needs, use cases and required capabilities of SIEM products and we think you’ll find it very useful as you explore next-generation SIEMs. And if you already have a SIEM, we can add analytics and intelligence to improve your detection, investigation, and response capabilities, and create a much better environment where you can shine.  

A dream job should never be hell.  

Want to follow this blog series on escaping Dante’s SOC Inferno? Sign up to be notified when new posts are published. You can also join us on November 19th for our first webinar diving deeper into the theme of Dante’s SOC Inferno and how to escape it. Register now. 

Be sure to check out our other articles in this series:

• Escaping Dante’s SOC Inferno: The Heresy of Orthodox SecOps
• Escaping Dante’s SOC Inferno: The Violence of Destructive Metrics
• Escaping Dante’s SOC Inferno: Fraud and the Truth About Normal
• Escaping Dante’s SOC Inferno: The Treachery of SIEM Lies