The software delivery process has been transformed in the last decade, with the adoption of well-understood paths that incorporate functions such as testing, release management and operational support. 

These changes have enabled organisations to speed up time to market and improve service reliability. So why does security still feel so hard for so many product teams? 

In our Secure Delivery playbook, we explain the principles and practices you can apply to security within continuous delivery. We’ll also explain the security pitfalls that we encounter with customers, and how you can avoid them. The top five software delivery security challenges we’ve identified are:  

1. Security gate bottlenecks
2. Disruptive chunks of security work
3. Lack of tangible value
4. Unsuitable security policies and checklists
5. Inefficient duplication of effort

1. Security Gate Bottlenecks

Product teams need to roll out new digital services to customers as quickly as possible, to gain rapid feedback and iterate on their minimum viable product. However, too often this process is held up by weeks or months, because of the need for lengthy risk assessments and penetration tests. These checks are carried out by an overloaded security team that doesn’t have the capacity to meet multiple iterative launch deadlines. 

You can reduce this bottleneck by embedding security assurance ownership into product teams. If product teams understand what assurance is required from a security assessment, then they can proactively provide it through code scanning, threat modelling and just-in-time security controls. This means security teams can change their role from fully assessing each release to simply checking the provided evidence. 

The success statement you’re looking for is:

“We no longer wait for security teams to sign off that our release is secure, we provide security scan results and documented security controls in an agreed format which they quickly verify and use for assurance”

2. Disruptive chunks of security work

It can be difficult to plan for the outcome of large, infrequent security reviews or audits. Teams are often required to take on large chunks of remedial security or compliance work, which can lead to combative negotiations, disruption and widespread risk acceptance. 

An effective way to reduce this disruption is by adopting security standards and controls earlier, in consultation with security teams. This means defining a clear set of standardised security requirements that are based on data sensitivity and system architecture. Additionally, shifting security requirements left towards the beginning of the delivery lifecycle means that product teams can design in controls from the start.

This looks like:

“We knew from product inception that our payment service would require service to service authentication and the encryption of card details. We planned this from the beginning and used agreed mechanisms that led to a very smooth delivery.”

3. Lack of tangible value

Too often, security activities such as code scanning, risk assessment and remedial work are carried out with little discernible value, apart from being able to say, “We did what we needed to to pass security sign off”.  This makes it difficult to justify security activities against product needs, and to measure their value.

This can be improved by measuring and owning security performance effectively. If you understand how security activities can be measured and used to provide assurance evidence, then you can prove vulnerabilities are fixed within timelines or that important security controls are widely adopted. This also means you can help teams take responsibility for owning security improvements and prioritising work.

“Using our security dashboards, as a delivery owner I can demonstrate that 100% of my critical systems have granular database permissions, are regularly scanned for vulnerabilities against the OWASP top 10, and 95% of any vulnerabilities found are fixed within 14 days.”

4. Unsuitable security checklists

It’s important to define a set of technical security requirements and standards if you want to scale out security assurance effectively. However, in many cases the result is hundreds of questions in a spreadsheet, based on old and high-level industry standards. Such checklists are created in isolation by security teams, and rarely updated. This leads to security activities that are unsuitable for fast-changing tech stacks and modern design paradigms, such as microservices, serverless and composable front-ends.

You can improve on this by co-developing useful, relevant and up-to-date security standards. Ensure standards are co-developed and co-owned by product and security teams, and regularly review and extend them to keep up with new technology or innovative system architectures.

This looks like:

“Our security standards are seen as a valuable design toolkit for engineers and architects that provide early indication of security requirements and act as a blueprint when implementing security controls. ”

5. Inefficient duplication of effort

Security scans and/or penetration tests for multiple digital services often find the same vulnerabilities over and over again, with multiple product teams carrying out the same remedial work or accepting the same business risks. This results in duplicated, expensive work and/or increased security risk from cross-cutting control deficiencies that are too big for any one team to fix.

You can remove the duplication with:

• Focused penetration testing. You should review penetration test findings globally, and move towards in-depth, regular testing and scanning on cross-cutting areas such as web front-ends or shared infrastructure, while reducing testing on individual microservices.
• In-depth remediation efforts. Tackle shared vulnerabilities by carefully standardising shared components such as base VM or container images, shared framework libraries or reverse-proxies, and fixing them regularly so teams get fixes “for free”.

This looks like:

“We no longer pen test each new customer-facing website component before release, instead we test the whole site in-depth every 3 months and since we standardised our web server configuration we no longer receive hundreds of duplicated findings.”

In upcoming blog posts, I’ll go into more detail on each of these pitfalls with some useful concrete techniques to avoid them. So, follow Equal Experts on Twitter and LinkedIn for updates!

Imagine that a few weeks ago you bought an outdoor theatre ticket for $100. You wake up on the day of the concert and it’s raining, with a storm due later, you’re feeling ill. On top of this, your car is in the repair shop, so you’ll need to take three different buses to get there. Yet, despite the fact that the current drawbacks outweigh the benefits, you still choose to go to the show.

This is known as the Sunk Cost Fallacy.

In simple terms, the Sunk Cost Fallacy is the phenomenon where “a person is reluctant to abandon a strategy or course of action because they have invested in it, even when it is clear that abandonment would be more beneficial.”

A point of research in behavioural economics and psychology for many years, it can also be applied to delivering software.

The crucial part of the definition above is this:

“even when it is clear that abandonment would be more beneficial.” 

The key question is, why do we keep falling for the Sunk Cost Fallacy? 

The Experts at Massachusett Institute of Technology (MIT) provide three hypotheses for this behaviour:

1. We don’t want to appear wasteful
2. We want to stick to our plans
3. We don’t want to accept any loss at all, so we follow through.

So whilst building software, how do you know if changing course would be more beneficial?

Stop, Start, Continue is often used when giving feedback. And it can also be used to apply to decision making in these situations. The principles are as follows. 

• Stop – stop doing what you are doing and abandon with no alternative 
• Start – start a new course of action to reach the goals but pivot away from the original solution 
• Continue – continue on the path you are already on. 

In these decision making situations, the first, and important point, is to take emotion away from the behaviour. This can be done simply by digging deeper into the cause and effects of stopping starting and continuing, and attributing a business value in terms of $ to each. 

By calculating the Net Value for each option, you will have a data-driven approach to making a decision (a different result for Stop, Start and Continue). 

However, when doing this it is important to remember that you don’t count the money you have already put in. This is now sunk cost. You can’t get it back no matter what you do, so the only important point, and all you can do at this stage, is look at which option will get you to your optimal goal, starting from now.

There are four clear steps to a data-driven answer.

Step 1.  Make sure your goal is still valid.  

Firstly it is crucial to revalidate the goal that has been set initially. 

• Is the original goal clear
• Does the initial goal still stand 
• And does your goal have clear measures of success so you know at what point you can say you have achieved it?

Step 2. Articulate the Business value for Stop, Start and Continue.

(STOP) Articulate the business value and costs if you were to stop. The likelihood is, in this situation, not only will you get no business value, but also no costs (if not including what is already sunk). However, you also need to be aware that not doing something could also result in loss of customers, revenue or increase in penalties, so ensure you think about the negative value that could also occur. 

 (START) Articulate the business value and costs for a different solution. Knowing what the costs would be in a new solution and when you would articulate that value (and even if the profile of the value is likely to change) will help you work out if you should start anew.

(CONTINUE) Articulating the business value and costs if you were to continue on the same path (ignore what you have paid in already, this is already sunk ), it is essential to understand what you still have left to pay and when you will get the business value at this rate. 

Step 3. Calculating the Net.

To calculate the net, use the following formula: Total Revenues – Total Expenses. 

Total Revenues (over a set period, usually 1 year)

• Sales revenue 
• Saving in operational costs

Total Expenses (over the same set period as revenue)

• Build costs
• Ongoing support costs 

Step 4. Compare the Results.

The net result that gives you a higher rate of return is the result to pursue from a business value perspective. That is because it will provide the most value over the duration. 

Some of the common pitfalls you will need to avoid.

So far it is pretty straightforward, right? OK, now we need to address some of the behaviours I have seen before. Some common, and other less common, traps that could derail a project.

Redesign at the expense of the immediate value.

One trap I have seen in clients is that solutions are abandoned weeks or even days before launch to re-architect, or change the technology stack. This is equally dangerous, as you could be willing to sink more costs for no added benefit. In this scenario, ensure you have looked at the business benefits based on where you are now.

Ruling out double development with no evidence of the impacts of that against value. 

Some projects are in a position where you are migrating to a new technology stack (or cloud provider). You may be unwilling to do work in the old world and the new world because it is deemed “wasted effort”. Although this may sometimes be the case, it is all relative to the value that work will bring. Apply these same principles when deciding if to do the job twice or wait. 

Assuming that this will demotivate engineering teams.

Here’s the facts as we see it, time after time. Most engineering teams that we have worked with ultimately want to be building something that will be used and add value. By changing course often, assumptions are made that this will demotivate the engineering teams. In my experience, this isn’t the problem.  The demotivation occurs because there hasn’t been the upfront and honesty around why there is a change of direction with the data that backs this up.  What you will find by doing this exercise and providing the data to your teams is it will result in motivated teams wanting to do what’s right because they see value in it.

***

The truth of the matter is, the Sunk Cost Fallacy is a natural tendency. But it is one that can be avoided by simply following a few principles, procedures and by recognising and avoiding the pitfalls. As soon as you recognise that the end result will benefit from your decisions at this stage, this approach becomes obvious to you, and everyone you need to convince to change. Because no-one wants to be the software delivery equivalent of someone who goes to the concert, and ends up standing alone, sick, cold and stranded in a muddy field.