Post Syndicated from Joshua Harr original https://blog.rapid7.com/2022/09/13/grey-time-the-hidden-cost-of-incident-response/

The time cost of incident response for security teams may be greater – and more complex – than we’ve been assuming. To see that in action, let’s look at a hypothetical scenario that should feel familiar to most cybersecurity analysts.

An everyday story

A security engineer, Casey, is tuning a SIEM to detect a specific threat that poses an increased risk to their organization. This project has been allotted some set amount of time to get completed. The research and testing that Casey must do in order to get the query and tuning correct, accurate, and effective are essential to the business. This is one of many projects this engineer has on their plate. They are getting into the research and starting to understand the attack at a level they will be able to begin writing some preliminary factors of the alert, and then…

An employee forwards an email that they believe to be phishy. Casey looks at the email and confirms it requires further investigation. However, the engineer must respond to the user by giving them the process to send the email as an attachment to look into headers and other details that could help identify the artifacts of a malicious email. After that, the engineer will do their assessment and respond appropriately to the event.

Now, 25 minutes have passed. Casey returns to focus on tuning the alert but needs to go back over the research a bit more to confirm where they left off. Another 10 minutes have passed, and they are back where they were then the phishing alert came in. Now they are gathering the right information for the project and trying to get the right people involved, then…

An EDR alert comes in. It is from a director’s laptop. This begins to take priority, as the director needs this laptop for their presentation to a customer, and they leave for the airport in 3 hours. Casey steps away to analyze the alert, eradicate the malware, and begin a scan across the organization to determine if the malware hash value is seen elsewhere. 30 minutes go by, because an incident report needs to be added to the ticket. Casey sits back down and, for another 20 minutes, must recalibrate their thoughts to focus on the task at hand.

Grey time

Scenarios like this are happening in almost every organization today. High-risk security projects are delayed because fires pop up and need to be responded to. In the scenario we’ve just laid out, this engineer has lost one hour and 25 minutes from their project work due to incidents. These incidents may have a risk to them if not dealt with promptly, but the project that this engineer is working on carries a high risk of impact if not completed.

Cal Newport, a computer science professor at Georgetown University, famously explained in his seminal book “Deep Work” that it takes each person a different amount of time to pivot from one task to another. It’s how our brains work. I’m calling that amount of time that it takes to pivot “grey time.” Grey time is not normally added into the time it takes to respond to incidents, but we should change that.

Whether it takes 30 seconds, 5 minutes, or 15 minutes to respond to an incident, you have to add 5 to 25 minutes of grey time to the process to pivot back to the work previously being performed. The longer the break from the task, the longer it may take to get back into the project fully. Grey time is just as detrimental to an organization as not responding to the incidents. There are quite a few statistics out there that help us quantify distractions and interruptions:

• People are twice as likely to make an error when they are distracted for as little as 3 seconds.
• Average interruptions per day: 56.
• The average employee gets 11 minutes between distractions.

Incidents can be distractions or interruptions. The fact is that some events that security professionals respond to are benign and do not lead to actioning an incident response plan or prevent prioritized work from being completed.

Here is where Security Orchestration, Automation, and Response (SOAR) comes into play. Those manual tasks security professionals are doing that take time away from risk-informed projects to secure the business can be automated. If tasks cannot be automated fully, we can at least automate the process of pivoting from tool to tool. SOAR can eliminate the manual notation in a ticketing system and the documentation of an incident report. It can also reduce time to respond and help eliminate grey time.

Grey time reduction through SOAR

In an industry where alert fatigue and employee attrition are pervasive issues, the need is high for SOAR’s extensive automation capabilities. Think about the tasks in your organization that you would automate if you could, because they are taking up more time than necessary. We can do some quick math to find your organization’s annual cost of manual response for each of those tasks, including grey time.

1. First, think of a repetitive action your team does repeatedly.
2. Assign a “task minutes” ™ value, which is approximately how long it takes to do that task.
3. Then, estimate the “task instances per week” (ti) value.
4. Multiply by 52 to find your “task minutes per year.”
5. Divide by 60 to find your “task hours per year.”
6. Multiply by your average hourly employee rate for the team that works on that task to find your annual cost of manual response.

I encourage you to do this for each playbook or process you have.

• Task minutes ™ x task instances per week (ti) = total task minutes per week (ttw)
• tw x 52 = total task minutes per year (tty)
• tty / 60 = total hours per year (ty)
• ty x hourly employee rate (hr) = cost of manual response

What we haven’t done here is add in the grey time. On average, it takes about 23 minutes and 15 seconds to regain focus on a task after a distraction. So, with that in mind, let’s round out this post by quantifying our story from earlier.

Let’s say that Casey, our engineer, takes 30 minutes for each phishing email, and malware compromises take 15 minutes to contain and eradicate. Both incident reports take about 20 minutes. Let’s also say that the organization sees about 16 phishing instances per week (ti) and phishing with the reporting takes 50 minutes. Let’s add in the grey time at 20 minutes to make it 70 minutes ™.

• 70 x 16 = 1,120 minutes (tw)
• 1,120 x 52 = 58,240 minutes (tty)
• 58,240 / 60 = 970.7 hours (ty)

Using the national average salary of an entry-level incident and intrusion analyst at $88,226, we can break that down to an hourly rate of $42.41. From there, 970.7 (ty) x 42.41 (hr) = $41,167.39.

That’s just over $41K spent on manual responses to phishing each year. What about the malware? I’ll shorthand it because I believe you get the picture. Let’s say malware incidents happen about 10 times a week.

• 25 min + 20 min = 45 min (Tm)
• 45 x 10 = 450 (TTw)
• 450 x 52 = 23,400 (TTy)
• 23,400 / 60 = 390 (THy)
• 390 x $42.41 = $16,539.90
• $16,539.90 + $41,167.39 = $57,707.29

That’s nearly a full-time employee salary for just two manual processes!

SOAR past grey time

SOAR is becoming increasingly needed within our information security programs. Not only are we wasting time on manual processes that could be automated, but we are adding grey time to our workday and decreasing the time we have to work on high-priority projects that are informed by business risk and necessary to protect revenue and business operations. With SOAR, you can refocus your efforts on risk-relevant tasks and limit manual task interruptions. You can also reduce grey time and increase the effectiveness of your security program. With SOAR, it’s all blue skies – and no grey time.

Additional reading:

• Cybersecurity Analysts: Job Stress Is Bad, but Boredom Is Kryptonite
• 360-Degree XDR and Attack Surface Coverage With Rapid7
• 5 SOAR Myths Debunked
• Deploying a SOAR Tool Doesn’t Have to Be Hard: I’ve Done It Twice