HOW TO BEST PREPARE FOR A ROLE IN THE SECURITY OPERATIONS CENTER (SOC)

Kip Boyle:
Hi, I’m Kip Boyle. I’m here with my co-host, Jason Dion. Today’s episode is going to be all about preparing you for a role in the security operation center. If that’s where you are trying to get to next, and it’s a common place, the security operation center is a great first cybersecurity job. It’s a wonderful place for you to learn basically how everything really works, and you also get to talk to a lot of people. As you’re thinking about, “Okay, I got on the SOC, where am I going to go next? What’s my next promotion? What job do I want?”

You’re going to get exposure to a ton of different people. You’re going to be able to see all kinds of different types of jobs that you might want to go to next, try it before you [inaudible] situation. Working in the security operation center is really a wonderful place for you to build a foundation in cybersecurity. Now, working in the security operation center may require some shift work, right? It’s possibly not ideal for every thing, but it’s a great place to start.

Now, if you’re going to give this a shot, what we’re going to talk about today is what skills do you really need in order to be able to land that job? Because remember, you want to make yourself irresistible to hiring managers. Part of doing that is showing up with the ability to demonstrate that you can solve problems, that you can use common tools. That’s what we’re going to talk about today. Right, Jason?

Jason Dion:
Yeah, exactly. Before we start diving into all that are different tools, we’re going to talk about five general areas or skills that people need to have in their hard skill bucket. I want to bring out one point really when we talk about SOCs. Not all SOCs are made the same. Every SOC is just a little bit different depending on what industry you’re in, who built the SOC, what part of the world you’re in and all those kind of things. As we-

Kip Boyle:
Size of the company?

Jason Dion:
Yes. Size of the company’s a big one, right? Whether you’re doing it on internal SOC for your own organization, or you’re a managed service provider providing SOC services for other companies. All of those things make SOCs look and feel a little bit different. We are going to talk more in generalities today, but these are the five tools that I see that are just really things you have to have in your tool belt if you’re going to be a SOC analyst or work in a larger SOC environment.

Now, when we talk about a SOC, that is a security operation center. Usually, these are going to be 24/7, 365, which means they operate 24 hours a day, seven days a week, 365 days a year. I’ve worked at a couple of SOCs in my time, and I can tell you I’ve been in the SOC on New Year’s Eve, I’ve been in the SOC on Christmas Day, I’ve been on the SOC on my birthday because when you’re working shift work, you just get stuck with whatever shift you happen to have.

One year, the shift schedule we were running, we ended up getting Thanksgiving, New Year’s, and Christmas, all of them, which really sucked for us. The other team was off [crosstalk] for their break of four to six days off during those three holiday periods and they loved it. It just does depend where you get sometimes the luck of the draw. I want to point that out as we talk about the SOCs as well. Now-

Kip Boyle:
Yeah. I also want to add one upfront comment too, Jason, which is when you want to staff a security operations center from a high manager’s point of view, that is a massive commitment. Because in order to do 24/7 operations every day, you count on your fingers like, “Okay, I need three people every day, one to take each shift.” But then you have to stop and think, “Oh, wait a minute. Well, what if one of those people gets sick? What if one of those people wants a vacation? What if one of those people needs to go to training?”

You start thinking about, “Okay, well now I need to have backup for all those people. Oh, wow. Okay. Now if I want to do three shifts, now I’m talking about not three people, but five people, or maybe even seven people, depending on the skill sets and all this stuff.” For a medium sized organization in the private sector, it may be unaffordable to actually build their own security operations center, and so that’s why there’s this prevalence of all of these security operations center vendors that you can contract with so that you can share a single SOC team with multiple other companies, and that’s what makes it affordable for you. If you’ve ever been wondering, “Why are all these companies out there selling SOC services?” I just thought I would give you a little bit of a peak behind the curtains as to why, what are the economics of that?

Jason Dion:
Yeah, that’s very true. Right? If I have one position, 24 hours a day, 365 days a year, that one position, I actually need to have five qualified people to do that, which means I have five qualified people who are on some sort of a rotating shift, and those are only going to be 12-hour shifts. You’re going to work three days on, four days off, four days on three days off, an average of 42 hours a week, every week of the month. Then you’re going to have to have people who are in training. That’s another one or two people.

Then you also have some backup folks. What I’ve traditionally seen is, like Kip said, somewhere around seven to eight people for every one position. As opposed to if I do 9:00 to 5:00, Monday through Friday, I only need one person for that one position or maybe two people for that one position. It is a big undertaking, and as Kip said, for a smaller or medium sized organization, you probably don’t have the need for a 24/7 SOC. You’re just going to outsource it because if I take 10 of those companies, I can then manage that as a SOC for all 10 of those companies, then it starts making that economy [crosstalk] work for us.

Kip Boyle:
Let me give you a hint, job seekers. When you decide, “Hey, I’m going to see if I can get a job on a security operation center,” ask about the staffing when you go into it, because it’s entirely possible that a hiring manager is trying to do it on the cheap and they’ve only got three people, your life is going to be miserable if you end up on a 24/7, 365 SOC, there’s only three of you or four of you. It’s really not enough. We don’t want you to burn out, so do your homework there. Okay. Are we ready to talk about the five skill areas and the tools, Jason?

Jason Dion:
Definitely. Yeah. The first one we’re going to talk about is log analysis. Now, this is probably the primary function that most people are going to do on a SOC, especially if you’re a level one SOC analyst. If you’re a level one cyber security analyst, you’re going to spend a lot of time looking at the glass, looking at your laptop or your desktop, and looking through the logs to try to identify things that look suspicious or malicious.

Once you do that, then you’re going to either log that incident or you’re going to analyze that incident depending on how your organization works. In a lot of organizations, they’re just going to have the level one people find things that look suspicious, create tickets and push those up to the next tier for resolution. But it really does depend again how your organization is set up and how you’re going to be working it.

But again, that big piece of that is going to be log analysis. Now, what log analysis tools are you going to use? Well, that’s going to depend on your SOC. You may be using something like Splunk, which is very popular. You might be using something open source, like Security Onion or Kibana. There’s lots of different tools out there. I’m not really focused necessarily on the tools themself in this particular podcast, but more about the fact that you need to understand how to do log analysis. Now-

Kip Boyle:
Can I just take a moment to say that log analysis is potentially one of the most boring things, the most tedious things you’ll ever be asked to do in your entire life?

Jason Dion:
It most certainly is. You are correct. It’s one of those things that you’re looking for a needle in a haystack, right? You’re seeing all the IPs. Yeah.

Kip Boyle:
Or the other thing I’ve heard lately is looking for a needle in a needle stack.

Jason Dion:
Yeah, you’re right. Because as people have gotten better, it’s harder to identify what is really malicious, right? A lot of things are malicious, but you can’t see it because it looks like normal user traffic. When you’re doing this log analysis, right? You’re going to use some sort of a SIEM, you’re going to use some sort of a CIS log server that’s going to collect all the logs from your network devices, your routers, your switches, your firewalls, your endpoints, your servers, all that stuff.

It’s going to get all consolidated, and they you’re going to be able to start doing things like trend analysis. You’re going to be able to start doing things like where are they coming from and where are they going to, and be able to identify those different patterns and being able to see what looks like it’s out of place? Now, when you’re dealing with log analysis, as I said, this specific tool doesn’t matter as much.

It’s really going to depend on your organization, and whether they want to spend a big licensing fee to use something like Splunk, or they want to do it on the cheap and use something like Kibana, which again is a perfectly good tool. It’s just open source so it just works a little differently. These are things you have to learn. Now, with all these different tools we’re going to talk about, one of the big reasons I wanted to talk about these different tools and types of tools is that a lot of people go and they get certified in some sort of thing, like Security+ or CySA+ or PenTest+ or CEH or whatever it is.

Now, these certifications do a great job of giving you a good overview of the environment. Especially if you’re looking at CySA+, you’re going to spend some time learning log analysis, and you’re going to learn some time doing some of the other things we’re going to talk about. But just passing your CySA+ exam doesn’t tell me you’re an expert in log analysis. It means you can tell me about log analysis, you might be able to read a couple of lines of code, but that’s about it. It’s a very surface level when you’re doing these things in a certification.

What you need to do, if you want to go work in a SOC, again, remember what Kip said earlier, you need to be able to be hired and solve problems immediately. Just telling me your CySA+ certified is not enough. I need to know that you’ve done some log analysis. Now, how do you do that if you’ve never had a job before? Well, there’s a couple of ways to do this. One is you could set up your own systems on your own internal home network, and then go through that log analysis and look at that yourself.

For example, here at my home, I have what’s called a unified dream station, which is a unified threat management tool. It’s basically a little cylinder, and my internet plugs into that, and then from that, it goes out to my Wi-Fi access points and all my wired switches. That is a full firewall and threat detection system in a box. It has an IDS in it, it has an IPS in it, it has the firewalls in it, and I can go through and look at those logs and do analysis on it. You can buy something like that for about $100 to $200 and plug it into your network.

Because I guarantee if your network is on the public internet, which all of us are, especially if you’re listening to this episode, you’re going to be getting hit with traffic all the time that it’s going to block, and you can identify those things and see what it looks like. Another thing you can do is go through log analysis challenges. There’s a lot of challenges online you can find from previous Capture-the-Flag competitions and things like that. In those, you’ll have a thing where you’ll have a download of a log analysis.

It might be 10 gigabytes worth of log files. Now, you’re going to have to find the needle in the haystack, as we talked about earlier, to find what was malicious inside those 10 gigs, and then there’s going to be a solution and you can check your answers against the solution. That’s another great way to do it. Or you can compete in some of these live Capture-the-Flags as part of the defense team and work as the log analysis person. Lots of different ways to do this. Another thing is you can go to places that have practical skills certification training, places like Antisyphon.

We talked with them before. We’ve had John on the show before and they have a lot of great programs and tools and classes that are focused, not on a certification necessarily, but on doing actual task, like log analysis, or our second one, which is packet analysis. Now, we talk about packet analysis, we’re moving up from the logs and we’re actually looking at the full capture packets that happen for a network. Now, Kip, what exactly is FPC or full packet capture?

Kip Boyle:
Okay. This is where you get to test my knowledge. My understanding of this is you could either capture headers or you can capture the full packet, which includes headers and payload. There’s a lot of, what do I want to say? Controversy right now about civil liberties and surveillance and that sort of thing, right? Depending on where you’re working, you may not actually be allowed to do full packet capture. You may only be allowed to look at headers or there may be some systems restrictions where you just don’t have enough storage space to capture an entire packet, headers and payloads. But I think this is what you’re getting at, right?

Jason Dion:
That’s exactly what I’m getting at. Right? When we talk about full packet capture, we’re capturing everything that was sent or received to that network or that network segment. If you wanted to do this on your home network, you could take an extra laptop, you could set it up and put it on SPAN port from your router and be able to capture all the traffic going in or out of your network. You can do a full packet capture using something like pcap or something like tcpdump or something like Wireshark.

Then you could open those things and start searching through the packets to identify what was actually sent. For example, if I set that up on my network and Kip was here and he decided he wanted to connect to FTP server, I can actually collect his username and password because FTP sends information in the clear, and in that [crosstalk] full packet capture, I’m going to have it, [crosstalk] and the entire file. [crosstalk] That’s right.

Kip Boyle:
If I upload or download, it doesn’t matter. It’s all in the clear and you can see all of it.

Jason Dion:
Exactly. That’s why packing capture is so useful. Now, most of us using FTPS, which does FTP over Secure Socket Layer or TLS, which is going to prevent that from happening. I’m still going to capture those packets, but they’re now encrypted and I can’t read them. But if you want to break that encryption, you could then read those packets later. Packet analysis is all about finding this information. Now, the reason I brought up full packet capture is, like Kip said, you have a lot of limitations with it.

But it is something that’s good to do, and you might want to do it on your own network, because there’s no privacy issues there because you own the network. Now, if you’re doing this for a company, you have to make sure that there is approval from those employees. Usually, it’s part of the AUP, the authorized user-

Kip Boyle:
Acceptable.

Jason Dion:
Yeah. Acceptable use program. I can’t even talk today. Acceptable use policy, the AUP. In that AUP, it’ll usually say employees understand they’re being monitored, and that means we can then do full packet capture. If you work for a government organization, you’re not going to be able to do full packet capture on the internet, for instance, because A, it’s too much data, and B, there’s a lot of civil liberties involved, especially here in the United States. We have laws against that.

But the idea here with packet analysis is you’re going to get some packets and you’re going to be able to look into them and see what is happening at layer one, layer two, layer three, layer four, layer five, layer six and layer seven of the TCP/IP protocol using the OSI model. Now, what we’re doing with [crosstalk] packet capture … Oh sorry. Go ahead.

Kip Boyle:
Well, I was going to say, listening to you talk right now, it reminds me when I first start to doing this kind of work, I didn’t understand what the hell I was looking at. Right? I understood, okay, pack it, you take some information, you encode it, you break it down, you break it apart, you shoot it all over the network. But my goodness, there’s so much variability, and I actually had to go purchase a two volume set of TCP/IP illustrated in order to be able to reference, right? Any particular packet that came through.

It could be a DNS packet, right? To try to do some kind of a name lookup or an IP address resolution, or could be FTP, could be SSH. There’s just so many different ways to build packets that when you start doing it for the first time, it could be very overwhelming and it could take you a while to come up to speed to the point where you can see things at a glance. It reminds me of The Matrix, right?

Where Cypher is sitting at this big massive display and Neo walks out and Cypher’s like, “I don’t even see the code anymore. I see blonde, redhead, brunette.” Because in the beginning, it just looks like a bunch of gobbledegook, but the more you get into it, the more this starts to become recognizable and then it starts to become second nature to you. I just wanted to point out that’s what it’s like to be on the learning curve of this.

Jason Dion:
Oh, most definitely. Right? Then when we start talking about packet analysis, we start with full packet capture where it means I’m capturing everything going in and out of the network. Well, if I did that to my home network, for instance, I send about 100 gigabytes a day of traffic, whether that’s because I’m uploading videos I’m recording like this podcast, or it’s because I’m downloading things like watching YouTube videos or Netflix. All of that is ones and zeros that are coming as packets.

If I’m doing full packet capture, I’m going to capture all of that, so I’m going to quickly run out of space. Especially in a large organization, you would need terabytes and petabytes of storage to be able to do full packet capture. Like Kip said earlier, sometimes we won’t do full packet capture, but we’ll do instead what’s called a sampling. By doing sampling, we might grab a packet every one second or every five seconds or whatever it is to get an idea of the kind of traffic that’s going through as opposed to the entire packet of everything that’s being said.

In addition to that, we can use something like NetFlow. NetFlow is going to give you just those headers. Now it’s going to collect all the headers and we’re going to be able to see where things are going and how much volume and what protocols we’re using, but we won’t know what’s inside those packets. Now, that’s good because it saves us all lot of space because headers are very small, the packet’s body is very large. But again, if I need to go back and say, “What was that malicious packet that happened at 2:35 AM yesterday?” I’m only going to have the header of where it came from and where it’s going, not what was inside of it. [crosstalk]

Kip Boyle:
Yeah. The headers, by the way, are sometimes referred as the metadata of the network traffic, right? Because it’s information about what’s being moved on the traffic, but it isn’t the actual traffic itself.

Jason Dion:
Exactly. Right? There are some systems that combine these two together, and this is what I recommend you use in your own networks, which is it’s going to basically work like an IDS, where it’s scanning everything coming in, it’s doing NetFlow and capturing the headers of everything. But when it sees something that matches its signature or it thinks is suspicious, it turns on that full packet capture for that period of time. That allows you to go back later and say, “Let me see those packets associated with that suspicious event,” and that is what you as a SOC analyst will usually be using.

You’re not going to be looking at 200 terabytes a day of data, but you’re only looking through the packets that were captured as suspicious or malicious. When you’re doing that analysis, you’re using a tool like Wireshark, it’s the most common one out there, to look through those packets and see what was being said. Now, that’s going to bring us to some other tools later because if I collected a binary, for instance, like an EXE file, how do I know if it’s suspicious, malicious or bad?

Well, I have to actually do malware analysis on it, but that’s a later thing. In packet analysis, we’re just putting those pieces back together to get that EXE file or that BAT file or that script, and then giving it to a malware analyst who can then analyze it further. In the SOC, a lot of times, everybody has their own piece and we hand things off from person to person. I might be able to collect that packet and then hand it to Kip who’s going to do the analysis on it depending on how we work inside of our organization.

Kip Boyle:
Yeah. I think of this whole workspace as like standing on the bank of a river, and there’s all these water molecules going by and it’s like, “Okay, there’s probably some cryptosporidium in there. Right? Some nasty little virus that’s going to actually hurt somebody if they take a drink of it,” but how in the world do you find it? Because it looks like every other drop of water at a glance, right? Then let’s say you take a little water sample out of that river.

Okay. Well, who do I give that to to actually test it to find out is it really raw sewage? What the heck’s all in this river? I don’t know. That’s a possibly useful way to think about this. If you’re in the audience right now and you’re struggling, you’re like, “I don’t even know what these guys are talking about,” because maybe you’ve never done this work before, I just wanted to give you a way possibly to start to thinking about it so you can get a little bit of traction on this.

Jason Dion:
Yeah. If you want to start building up your skills in packet analysis, the areas I would focus on is getting really comfortable using tools like tcpdump or Wireshark to do the packet captures. That’s the easy part. But then-

Kip Boyle:
And free.

Jason Dion:
… analyst part, and they’re both free tools, right? There’s lots of stuff online you can find on YouTube and other places that have great training on this stuff. Then you want to start doing packet analysis and actually digging through these things and finding those bad bits, that one bacteria out of the thousands of parts that you’ve been looking through, as Kip said. There are things that you can do for that as well.

There are Capture-the-Flag competitions focused specifically on packet analysis. I participated in a couple a few years ago, and literally, I would download a one gigabyte packet capture and then have to find what was that secret code that was in there based on the challenge they gave you. Learning how to use the filters inside of Wireshark and be able to cut through that data to find those things you’re looking for is really useful. If you want to get good training on this, I recommend taking a course.

Over in UNIMY, there’s a guy named Nathan House. He has a great course on Wireshark that will teach you every single thing of how to use that tool, including some practical examples. If you want to be a good SOC analyst, being able to do packet analysis is a critical function for you to learn. I definitely recommend going and learning that stuff.

Kip Boyle:
That’s going to help you really in the rest of your cybersecurity career, because understanding what the network looks like when you’re actually opening the hatch and looking inside of it, that is something that’s going to serve you for the rest of your career because people are going to bring problems to you all the time, and you’re going to have to think about, “Okay, well, what’s really going on there?”

You may not do the pocket analysis, but it’s going to help you think about what might be going on depending on what your role is. You might be some blue teamer and you’ve got to write a new firewall rule or something like that. This is foundational stuff. Now, let’s talk about the security information and event monitor. Are you ready to transition to that one now?

Jason Dion:
Definitely. Let’s talk about SIEMs. When we talk about SIEM, we’re talking about a security information and event monitor, or security information and event management system. Now, these systems basically work by taking in all that log data we talked about before. We have things from your routers, your switch, your firewalls, your IPS, your IDS, your DLP systems, your endpoints, all that stuff gets fed into this SIEM. It’s a central repository with all the information. Now, when I talked about log analysis earlier, I did mention the fact that we have all these systems coming in using something like CIS log into a generalized SIEM where we can conduct that analysis.

To do log analysis, you can just do it on one machine if you wanted to. For instance, if I was looking at Kip’s laptop because I thought it was infected with malware, that would be basic log analysis. But now if I want to look at it across all of the computers on my network and see who else might have been infected with that same piece of malware that Kip picked up, I can then do that using a SIEM. Learning how you use filtering and querying of SIEMs is essential.

One of the most popular ones that’s used out there in a proprietary environment is known as Splunk, S-P-L-U-N-K, and they have some great free training and free level one certifications that come on using their tools, and you can do this by going to splunk.com and you’ll find it over there. In addition to that, a lot of companies like to use open source technologies. I mentioned Kibana earlier and Security Onion. Those are two very popular systems that use as a SIEM to be able to go through and do all this log analysis inside of these data across all of your systems.

Kip Boyle:
Yeah. These tools are really helpful when you are trying to find bad things on the network and do something about it. If you’ve ever heard the term indicators of compromise, right? Oftentimes, you’ll get an alert from an ISAAC or something like that, or maybe from a vendor. If you’re a Window shop, Microsoft might issue some kind of an alert that there’s a problem, and they may give you indicators of compromise. You can use those indicators. Sometimes those are signatures, digital signatures.

Sometimes you’re looking for the presence of a certain type of packet on your network, whatever it is. But a security information and event monitor, you can load those indicators of compromise into there, and then it can create automatic filters to be looking for that stuff, and it will alert you when it finds something that matches that indicator of compromise. These are super powerful tools, but they also tend to be pretty expensive. You need to get trained on them, you need to use them a lot.

I think of this stuff as like learning a foreign language. You’ve got to get immersed in it, you’ve got to do it a lot, and that’s really going to prepare you to do well when you finally land a job in the security operation center. Okay. Just to recap, we’re going to talk about five things today. We’ve talked about three so far. Log analysis was the first, packet analysis was the second, and then using security information and event monitor. Was there anything else you wanted to mention about the SIEM, Jason, before we talk about the fourth area?

Jason Dion:
Yeah. The one thing I want to point out with the SIEM is we talked about log analysis, we talked about packet analysis, and in those, you might be looking at a single machine or a single log, you might looking at a single packet or single packet capture. With the SIEM, it puts all these things together. It consolidates all of this into one place. Then in the SIEM, as I’m searching for it, I might find that, “Hey, this looked like a malicious indicator of compromise based on this log,” and then I can right-click on that and say, “Show me the packets that were captured if you had full packet capture enabled,” and they actually tie together. Doing these things all in a coordinated way does work. I like learning log analysis first, then pack analysis, and then putting those together using a SIEM. [crosstalk]

Kip Boyle:
You’ll know why you need a SIEM if you do it in that order.

Jason Dion:
Exactly. For the fourth one … I’m going to kick it over to you, Kip. But the fourth one we have is vulnerability scanning and patch management. What does this involve?

Kip Boyle:
Right. Listen, when I talk to my customers, because I’m a practitioner, right? Just to remind everybody. I’m a virtual chief information security officer, and I’ve got customers that pay me to help them form security strategies and then execute on those, making sure that they have the right mitigation is in place so that their organization can become a smaller target, because, gosh, everybody’s getting attacked all the time and you can’t avoid every kind of attack.

But if you can make yourself a smaller target, then it decreases the risk that you’re going to get cyber exploited. Well, one of the things that you need to do in order to make yourself a smaller target is you to know when you have vulnerabilities. You can have vulnerabilities just standing there, just doing your normal business. Vulnerabilities can pop out of nowhere because people are doing research all the time trying to find new software bugs, trying to discover how certain configurations of computers can actually provide a toe hold or some traction for somebody to attack you.

Vulnerability scanning and patch management is a super important piece of what I call cyber hygiene in order to keep your organization as free as possible of opportunities for people to exploit you. All right. What does it take to do vulnerability scanning and patch management? By the way, yes, this is typically something that a security operations center will do. Sometimes, in some organizations, these functions are actually performed outside of the security operations center. Maybe the systems administrators do it, or this might be done by a dedicated team inside of the information security department.

You might not actually be involved in it directly, but you will be indirectly involved. Indirectly, you’ll see this stuff, but perhaps you’ll be asked to do it. What does it take? Well, first of all, you’ve got to scan for vulnerabilities. There’s a huge database of vulnerabilities that these scanning tools will be referencing. Now, in order to scan for a vulnerability, you need to have some kind of a signature loaded in your vulnerability scanning engine in order to be able to know what to look for, right?

Again, this is a different version of looking for needles in a needle stack, but you’re going to scan all of the network nodes, whether that’s going to be a workstation or that’s going to be a router or a switch or another piece of infrastructure or a server or something like that, and you’re going to be looking constantly, constantly looking for where are the vulnerabilities? Interestingly enough, if you do this long enough, you’ll actually see that vulnerability that you detected six months ago and patched can sometimes reappear, because why?

Well, maybe a server gets taken out and gets reloaded or something like that, and so then a patch that you installed actually falls off the server, so to speak. It’s really important that you do this vulnerability scanning. There’s going to be different tools that you can use. Some of them are free. You might use the community version of Nessus or OpenVAS or something like that, or you might be using Qualys. There’s all kinds of different tools that are available.

Your employer is probably going to pick the tool. You’re probably not going to pick the tool. You just need to show up and learn how to use it. Let’s say you find some vulnerabilities, and by the way, I’ve done an entire course on vulnerability management, implementing a vulnerability life cycle over on LinkedIn Learning. You might want to check that out if you’ve got access. But once you find vulnerabilities, now you’ve got to patch them.

There’s a completely different set of tools that you’ll use to actually deploy patches, and sometimes the problem isn’t that you’re missing a patch. Sometimes it’s a good configuration issue. Sometimes you’ve really got to get in there, roll up your sleeves and figure out, “Okay, I can’t just drop a patch on this thing. I’ve actually got to change the configuration.” In the SOC, you may not change the configuration, but what you might do is you might actually specify what the configuration change should be.

Then you’re going to reach out to the system administration team and you’re going to ask them, “Hey, we’ve got this vulnerability, we’ve done the research. We think this is the configuration change that needs to happen. Would you please take a look at this? If this lines up and isn’t going to cause any kind of outages, would you please deploy the configuration change and then we’ll scan it next week?” Typically, vulnerability scanning and patch management’s going to be some kind of a cadence, right?

You’re going to scan every week or you’re going to scan twice a week or something like that. There’s a whole structure that’s going to be built if you’re doing this for real, right? If you’re practicing good cyber hygiene, right? This is something that you’re going to do, and it’s going to be like a wash, rinse, repeat over and over and over again. Anyway, that’s my perspective as a virtual chief of information security officer on how vulnerability scanning and patch management is done. Jason, what’s it been like from your perspective?

Jason Dion:
Yeah. Most of the SOCs I’ve worked at have been very, very large. We’re dealing with millions of endpoint. We’re not dealing with the vulnerability scanning and patch management. We have that at lower level organizations that work for us. Instead, when I’m using it in my larger SOCs, I might be using it because I’m looking for a specific outage or a specific thing that just came out. For instance, a couple of years ago, I was working at the SOC and we had the Apache Struts vulnerability that came out. It was 2016, 2017.

Now, I wasn’t going to run a vulnerability scan across millions of endpoints to find out all the patches that were missing on those systems. But I was going through and running a vulnerability scan across those millions of endpoints to look for that one particular vulnerability that could be exploited by Apache Struts. Same thing when the EternalBlue came out.

I knew that I needed a search for MS17-010 and see if that was out there. I would do a specific vulnerability scan across my network for that particular vulnerability to see if they’ve been patched or not, and that way I know what is my exposure factor? What exactly is out there that could be hit by these different vulnerabilities that just came out at zero days? Knowing that, I could then put other things in place.

Kip Boyle:
Exactly. Just [crosstalk] Right? Log4J, right? We’re recording this in early January and a bunch of people I know just spent their entire holiday week, between Christmas and New Year’s, crawling through their networks looking for log4J instances in all kinds of different products. I’d hate to do that by hand, so having a vulnerability scanner could be super helpful.

Jason Dion:
Exactly, right? By able to narrow it down to a single particular plugin that you’re looking for or a particular patch, that allows us to identify what is our threat area? Then like you said, we might be, as a SOC, getting a weekly report from the vulnerability scanning team to say, “Hey, we’re at 88% patching. We need to get there at 12% done,” and be able to understand what things we’re missing and then we can follow up on it.

But again, this is something that really does depend on your organization and how you’re going to be built out. Smaller companies, a lot of times, they’ll have their SOC and their vulnerability scanning and patching all in one area. Larger organizations like I was in with millions of endpoints, we break it out because we had networks across six continents around the world and I couldn’t physically touch all my machines from my central point that I was working at.

Kip Boyle:
Right. Because if you’re going to scan over the internet, right? Geez, that … Scanning and patching over the internet is sometimes a crap shoot. Sometimes you can’t even find out what’s going on because internet unreliability. Also, you’ve got to transverse, usually, a bunch of security devices, firewalls, that sort of thing. Scanning and patching over the internet can be a real pain in the butt. It might not even be possible in some topologies, and so you are going to have to depend on the local teams in the different geographic areas. That’s probably what you were doing, right?

Jason Dion:
Yeah. We physically could scan from our central headquarters, but the reason was the bandwidth was just really limited, so it’d take a lot longer. My team was only so big, right? We had our own mission to do, and that wasn’t doing vulnerability scanning. If I wanted to centralize those functions back in the headquarters, I would’ve had to pull in an extra 100 or 200 or 300 people to do that, and I would end up having to find office space and all that for them, which was also at a premium. For us, we had local defenders that were responsible for their local networks, and then we just did the oversight over those local defenders and what they were doing.

Kip Boyle:
Yeah. In the past when I’ve deployed vulnerability scanning architectures, when we centralized it, one of the things that we would do is we would actually deploy a scanning engine into a data center and all of the scanning would happen on the local area network, and then what we would get back in the central location was a report, right? Over the WAN, we were only pushing reports back and forth. I would send a command and then a report would come back, and all the heavy scanning that actually happened would happen in the local engine in the data center. That’s another possibility.

Jason Dion:
We did a lot of that as well. Yep. All right.

Kip Boyle:
Okay. We’ve covered four areas. We got one more to go, a fifth, and that’s malware analysis, both dynamic and static. Jason, how does that fit into a security operation center?

Jason Dion:
Yeah. When we talk about the security operations center, right? We said that the majority people are looking at logs or doing packet analysis, and when they find something, you might find it executable and you need to figure out is this thing malware or not? You don’t really know. Well, to be able to figure that out, you’re going to do one or two things, either do dynamic analysis or static analysis. Now, dynamic analysis means you’re going to run that thing in a sandbox environment, like a VM, and basically detonate it and see what it does.

I’m going to go ahead and take a snapshot of the machine. I’m going to run that piece of malware, and then I’m going to take a snapshot again and compare the two. What registry entries were changed? What files were created? What processes respond? What services were installed? All those type of things, and that can start helping me figure out what is this thing doing. Now, the second thing you can do is you can actually take that binary and you can decompile it, and use something like IDA Pro and start doing a static analysis, line by line, through the code to figure out what it’s doing.

Now, are you, as a standard SOC level one analyst, going to do this? No, you may do dynamic analysis depending on your organization, but you’re never going to be doing stack analysis at that level. But I can tell you, for the larger SOCs that I worked in, we might have had 30 or 35 people on that SOC floor. We had one person who did analysis, right? If I found something, I had maybe four or five people that could do the dynamic analysis, and then we had that one person who could actually read assembly code and be able to put together what it was doing through the static environment.

Depending on the level that you need to do, it really does depend on your organization. For example, I was listening recently to another podcast that I love called American Innovations, and they did a series recently on the Stuxnet virus. This is a virus that went back in [inaudible] worm, a worm that came out back in 2011 that targeted SCADA power plants. The people who found that was actually Symantec. It was their SOC floor that found this new virus, this new malware, and they ran it through dynamic analysis and then they ran it through static analysis.

In those organizations, they need to understand exactly what it’s doing on a code level so they can write patches and reverse exploits against it. Those organizations, if you’re working for an antivirus company or anti-malware company, might do a lot more malware analysis, either dynamically or statically. If you’re running Kip’s cybersecurity SOC for the small businesses, he probably is not going to be doing dynamic and static analysis because he doesn’t need to. If he identifies it is bad, he’ll just block it and then move on, because now it’s not bad for us anymore. We don’t care.

But if you’re trying to write antivirus signatures for it, you have to know how that code works. Again, depending on what SOC you work on, you’re either going to do more or less of this malware dynamic and static analysis. Now, all five things we just talked about, right? We talked about five different things. We talked about log analysis, packet analysis, using both of those together while you’re using a configuring SIEMs, talked about vulnerability scanning and patch management, and then finally malware analysis, both dynamic and static.

As we talk about all these five things, these are all things that we cover in courses on certifications like CySA+. If you take my CySA+ course, I’m going to cover each of those five things, and you’re going to get about 10 to 20 minutes of me showing you what it looks like in the environment. I’m going to pull out Wireshark and show you how we do packet analysis. I’m going to pull out a SIEM and show you how it’s configured and how we go through and search for different logs and things like that.

I’m going to show you how vulnerability scanning is done with Nessus or OpenVAS. All these different things. Even malware analysis, I show you that in this course. Now, in you going in that course, you’re not going to become an expert, and if you get your CySA+ certification, no one is going to think you’re an expert in these five things, which is why I think it’s important for you to take those certifications to get the basic knowledge, but then you identify what areas you need to beef up on.

If I want to learn more about packet analysis, I now know that’s done in Wireshark. Let me go get a specific Wireshark course from some instructor who’s going to show me how to do real world packet analysis. If I want to do malware analysis, I need to learn, how is it done dynamically? How is it done statically? There are courses that dive into those areas. But if you take a generalized certification course, which is what most people do, you’re not going to have that level of knowledge, and it’s going to show when you go apply for a job to become a SOC analyst or a level one, level two or level three.

Depending on what level it is, they’re going to expect you to have more or less knowledge in these five areas. That’s why I think it’s important to understand what are these five areas and these hard skills you need to start beefing up? No one’s going to expect you to be an expert on all five of these, especially the last one. Malware dynamic and static analysis is a very specified field inside of SOC analyst work. But being able to understand this at a deeper level than what your certification is going to cover is something that’s going to help you out a lot in the real world.

Kip Boyle:
Yeah, definitely. I really appreciate the point that you’re making about. It’s not just about studying, right? It’s not just about desk knowledge. You actually have to get out into a network, some network, could be just your network, right? But you got to get into a network and you’ve got to unpack these tools. You got to try to use them and you need to get some hands-on perspective. But I find that a great way to start is to take the certification courses. Everything’s really well structured.

They don’t assume that you know anything, so all of the essential terms are defined upfront, sometimes with some of the … just a YouTube video or something like that. You might go and search on Wireshark on YouTube to try to watch a free YouTube video. But then the person making the video just is making all these assumptions about different terms that they assume that you already know, so they don’t take the time to explain it. Now, that video may be fantastically useful once you get the basic terminology and the basic concepts internalized.

But that’s something that I sometimes see people struggle with, is they say, “Well, I don’t need a certification. I’m just going to go out and figure it out, watch a bunch of YouTube videos.” For some people that works really well. But if you’re trying that approach and it’s seeming like you’re just standing in front of a smooth [inaudible] and somebody’s talking, but you can’t figure out how to do anything, then maybe you should go and do the certification work in order to build a base for yourself.

Okay. Well, I hope this was helpful today, these five skills, these five areas that you can expect, that you have to know something about in order to land a job at a security operation center. As a hiring manager, I can tell you that not only am I going to ask you questions, hard skill questions in these areas, but it’s very likely that I’m also going to give you a practical test, right? Some kind of a practical interview where I’m going to ask you to look at a slice of a log file.

Or I might give some packet headers and ask you to tell me, okay, what’s going on here? Right? Just from the information I’ve given you, what kind of a packet is this? Right? What kind of protocol is it related to? Is it ICMP? Is it TCP? Is it UDP? Whatever. This is what you need to be ready for. We want you to be ready for this, right? We want you to be irresistible to a hiring manager. That’s why we’re doing this. Anything else, Jason, before we wrap up the episode?

Jason Dion:
Yeah. I think the last thing, the last point I want to make here is, as Kip said, you can go on YouTube, you can go on all sorts of places and find this type of information and learn it. Like I said, you can go do a Capture-the-Fly competition and things like that. But if you don’t even have the basic skills of knowing how to use Wireshark and use filters, that’s not going to work for you. Right? We need to be able to figure those things out. Now, the other thing I want you to remember is that sometimes it helps to have somebody walk you through these step by step, as Kip said.

YouTube is a great resource when you have a specific question. How do I use Wireshark to do X? But if you don’t even know what X is, the question to ask, that’s where you get stuck. That’s why I think a place like Antisyphon training does a really good job of doing hands-on courses in packet analysis that’s a one or two or three day course that’s going to dig into all these things in a real world environment, taught by people who do this in the real world for SOCs. I think this is some of the great training you can find out there, and it’s not necessarily certification related, but it is hard skill related that’s going to make you a better analyst.

Kip Boyle:
Absolutely.

Jason Dion:
With that, I want to say thank you for joining us for this episode of Your Cyber Path Podcast. If you’d like to learn more about any of the things we talked about, you can always visit us at yourcyberpath.com for show notes, past episodes and other great content. Until next time, we’ll see you then.

Kip Boyle:
Bye.