How to overcome technology anarchy with aligned autonomy
Platform engineering means creating user-centric capabilities that enable teams to achieve their business outcomes faster than ever before. At Equal Experts, we’ve been doing platform engineering for over a decade, and we know it can be an effective solution to many scaling problems.
Unfortunately, it’s easy to get platform engineering wrong. In this series, I’m covering some of its pitfalls. Last time it was the power tools problem, and today it’s the technology anarchy problem.
Alignment and autonomy are needed
Teams need alignment and autonomy in product and technology to succeed at scale. Alignment connects vision, strategy, and execution, while autonomy empowers teams to act independently. Increasing alignment means better decisions, and increasing autonomy means faster decisions. Here’s a 2×2 from our alignment and autonomy 101.
Platform capabilities are a great way to share technical alignment with your teams. Baking alignment into capabilities such as deployment pipelines and observability dashboards makes engineering tasks much easier for teams. You might know this as an opinionated platform, paved roads, or golden paths. But when alignment is absent the magic doesn’t happen and your teams are in anarchy.
The long-term costs of technology anarchy
When your platform capabilities offer autonomy without alignment, your teams can quickly make technology decisions, but without guidance. In the short term, this allows teams to use familiar tech stacks to rapidly build services and deliver them to customers. However, in the long-term, it creates a fragmented ecosystem full of inefficiencies, staffing challenges, and a maintenance mountain.
I’ve previously described how to measure capabilities in internal customer value, internal customer costs, and platform costs. Here’s a v1 platform that lacks technical alignment with three teams building their own pipelines in different tech stacks. There’s some internal customer value, and low platform costs as the platform team doesn’t have to build much. But internal customer costs will skyrocket over time.
This pitfall occurs when these interconnected, incorrect beliefs exist in your organization:
- Command and control is the only method of alignment
- Alignment and autonomy are opposites
- Teams must be in autocracy or anarchy
- A platform team must impose strict rules, or have no rules at all
When your platform team chooses to build capabilities without technical alignment, it’s anarchy. Every team builds custom solutions, instead of leveraging shared capabilities. When a pipeline breaks, only one team can fix it. When a pipeline is enhanced, only one team benefits. When a team needs changing, people won’t want to move. When a team needs downsizing for maintenance mode, other teams won’t want to manage their services. There are no economies of scale.
For example, an American telco invited me to visit their 20 teams, who had adopted You Build It You Run It for all their digital services on Google Cloud Platform (GCP). Team boards didn’t show much progress, and there were blank looks when I asked about their platform team. It didn’t exist, so 20 teams were using GCP in 20 different ways. When we measured the teams on unplanned tech work, we learned 40-60% of team time was GCP work.
Similarly, a British retailer had 10 teams with 10 different RabbitMQ messaging solutions, until a single Pub/Sub solution was mandated for consistency. This meant 10 subtly different migrations, and there was a big dent in productivity and morale. More upfront technical alignment could have prevented all that unplanned tech work.
Achieving aligned autonomy
You escape the technology anarchy pitfall by replacing low alignment capabilities with aligned autonomy capabilities. This produces paved roads, which supply the friction-free guidance that teams need to make independent technology decisions in the same organizational direction.
In the above 2×2 grid, you’ll see aligned autonomy is in the top right. It’s possible to have high alignment and high autonomy when you implement technical alignment as contextual guidelines, not top-down rules. Here’s how your platform team can make it happen:
- Declare low alignment capabilities as v1, and restrict them to old services.
- Capture guidelines on tech stack, architecture, etc. from engineering leadership in Architectural Decision Records (ADRs).
- Ask teams to build services with the decision records built in.
- Rebuild v1 capabilities with decision records built in, and declare them as v2.
- Host new services on v2.
- Migrate old services to v2.
- Delete v1.
Your platform team needs to stay user-centered, and focus on a great platform experience. It might mean a customizable pipeline template for a single tech stack, a four golden signals observability dashboard, or an automated ServiceNow workflow. There’s a higher platform cost, but you minimize internal customer costs and boost internal customer value. That’s a good trade-off. Here’s v2 of that imaginary platform, showing the same three teams with one pipeline template for their common Python tech stack.
Conclusion
Technology anarchy is a dangerous pitfall with painful, long-term consequences for your organization. If you reject the false dichotomy of alignment and autonomy as opposites, your platform team can create aligned autonomy in platform capabilities and help your teams to achieve engineering excellence.
I’ll share more platform engineering insights in my talk “Three ways you’re screwing up platform engineering and how to fix it” at the Enterprise Technology Leadership Summit Las Vegas on 20 August 2024. If you’re attending, I’d love to connect and hear about your platform engineering challenges and solutions.
I speak with customers and consultants across the Equal Experts network, to help our customers understand how to speed up innovation and reduce total cost of ownership at scale. Sometimes, our customers refer to this broadly as ‘scaling agile delivery’.
We’ve helped a lot of organizations to scale agile delivery successfully. Forrester Research recently published a Total Economic Impact (TEI) study that shows partnering with Equal Experts achieved a 123% ROI and 60% reduction in time to market for:
- A food franchise scaleup: 10 teams, 40 workloads
- A consulting organization: 10 teams, 100 workloads
- A high end retailer: 40 teams, 75 workloads
- A government department: 80 teams, 900 workloads
It’s hard to implement cross-functional, outcome-oriented teams delivering complementary product capabilities. The good news is success leads to a faster time to market, better quality, reduced risks, lower costs, and better user outcomes which improve your bottom line. My colleague Jon Ayre recently talked about how to iteratively scale agile. I’ll take that a step further. Here are four ways to scale up your delivery capabilities:
- Set teams up to have alignment with autonomy
- Introduce a You Build It You Run It operating model
- Focus on frictionless onboarding
- Create paved roads to allow teams to focus on delivering business value.
Create alignment with autonomy
Teams make faster and better decisions when they understand the intent, purpose, and constraints behind their work, and when they can act for themselves. Teams need to be aligned to deliver on strategy and execution, and autonomous to deliver outcomes without handoffs.
The problem is organizations often implement alignment as command and control. Teams are forced to treat alignment and autonomy as opposites, and they become stuck in one of two scenarios:
- Autocracy. Teams have high alignment but low autonomy. They incur delays because they can’t make decisions. At a media conglomerate, I saw 8 teams who constantly had to wait for 2 shared product managers to make all their decisions for them.
- Anarchy. Teams have high autonomy but low alignment. They’re unaware if they’re making good decisions or not. At an insurer, I saw 10 teams spending ~80% of their time on unplanned BAU work, because they’d implemented 10 different services with 10 different AWS runtimes.
The solution is to replace command and control with contextual alignment, and give teams the freedom to make good decisions in aligned autonomy. See why you need alignment with autonomy at scale.
Adopt You Build It You Run It
Teams achieve higher throughput, greater reliability, and a continuous learning culture when they build and run their own workloads. Teams need to become accountable for reliability as well as functionality, and focussed on outcomes rather than outputs.
The problem is organizational inertia towards using a central operations team to run differentiated digital services. It’s not possible to achieve a high throughput, high reliability baseline when delivery teams and an operations team are set up to work at cross-purposes. I’ve seen many organizations struggle to innovate because handoffs, competing priorities, and misaligned incentives occur every day.
The solution is to adopt the You Build It You Run It operating model for differentiated services, and keep your operations team for foundational systems. Make your product managers accountable for reliability, and put product teams on-call. See You Build It You Run It sounds great but it won’t work here and the You Build It You Run It playbook.
Focus on frictionless team onboarding
Teams are more effective when new engineers can make meaningful contributions as soon as they join up. Teams need a friction-free onboarding process that allows people to join a team and be productive on day one.
The problem is organizations usually treat onboarding as an afterthought. The onboarding process is disjointed and time-consuming, because it lacks a clear owner and is split between multiple organizational functions. I’ve seen engineers on a new team wait for weeks to commit their first line of code, because they’re waiting for the necessary permissions.
The solution is to automate onboarding tasks and reserve team capacity for knowledge sharing, so new joiners are immediately productive. Replace gatekeeping with trust and verify, share business domain knowledge, and establish a Bring Your Own Devices (BYOD) policy. See seven onboarding tips to accelerate productivity.
Build paved roads
Teams can deliver many user outcomes at pace when a majority of technology challenges are solved for them ahead of time. Teams need tasks such as microservice creation and telemetry provisioning to be self-service, fully automated, and fault free.
The problem is centralized technology capabilities are rarely designed to be user-centric. Teams can’t accelerate because they depend on platform teams focussed on infrastructure standardization, rather than the needs of their consumers. I’ve seen teams wait for weeks for a microservice provisioning ticket to be actioned by a platform team.
The solution is for platform teams to partner with product teams to create bi-directional feedback loops, and deliver self-service paved roads formed around engineer user journeys. Create a positive engineering experience for teams, focus on solving business problems with minimal cognitive load, and prevent platform teams from becoming service desks. See the Digital Platform Playbook.
Find out more
If you’d like to know more about how to successfully scale for agile delivery, get in touch! We’d love to hear from you.
I speak with customers and consultants across the Equal Experts network, to help our customers understand what’s needed to speed up innovation and reduce total cost of ownership at scale. That could include design systems, data workloads, or digital services.
We’ve helped a lot of organizations to scale up, including a government tax department with 80+ teams, and a high end retailer with 40+ teams. When I talk with people about how to succeed at scale, alignment with autonomy is a popular topic. In this series, I’ll look at the topic in-depth, including how to implement aligned autonomy in your own organization. If you’re in a rush, I’d just say:
To succeed at scale, give your teams contextual alignment with autonomy
Alignment is the what and why
Alignment means a shared direction. It’s the what and why – the glue between vision, strategy, and execution. Teams make better decisions when they understand the intent, purpose, and constraints behind their work.
Examples from the Equal Experts network of teams lacking alignment include:
- 3 teams at a telco unnecessarily building 3 different versions of 1 product capability
- 2 government agencies designing 1 shared service as 2 conflicting sets of APIs
- 10 teams at an insurer running 10 similar workloads on 10 different AWS runtimes
Autonomy is the how
Autonomy means an ability to make decisions. It’s the how – the capability and confidence to do something. Teams make decisions faster when they can do it without handoffs, and make better decisions when they can learn from their own mistakes.
Examples of teams lacking autonomy include:
- 8 teams at a media conglomerate waiting on 2 product managers to prioritize everything from 2 shared backlogs
- 10 teams at a payments provider waiting on 1 head of engineering to approve each technical specification upfront
- Forty teams at a government agency waiting on 3 solution architects to review each technology solution
Alignment and autonomy as opposites leads to failure
Your organization needs alignment and autonomy in product, culture, technology, ways of working, and more. However, it’s likely they’re treated as opposites, and you regularly have to choose between alignment and autonomy. This happens when your leadership implements alignment as command and control. As a result, when decisions have to be made teams will oppose increasing alignment for fear of less autonomy, and your leadership will oppose increasing autonomy for fear of less alignment.
Alignment without autonomy is autocracy. In the above media conglomerate example, the product director resisted team autonomy and insisted on 2 backlogs to protect command and control product alignment. The 8 teams were unable to work independently on product capabilities, and often had to wait for decisions to be made. Feature launches were only bi-monthly, and innovation was thought to be impossible.
Autonomy without alignment is anarchy. In the above AWS insurer example, the 10 tech leads resisted alignment on AWS runtimes, and insisted on the status quo to protect their autonomy. Those teams frequently had to upgrade, secure, deploy, monitor, and repair their 10 workloads in 10 different ways. Teams spent ~80% of their time on unplanned BAU work, AWS billing costs were higher than necessary, and there was little innovation.
Alignment and autonomy as opposites leads to failure. It doesn’t have to be this way.
Why you need alignment and autonomy to succeed at scale
Customers and consultants across the Equal Experts network agree that alignment and autonomy are key to success at scale. You can’t unlock speed, creativity, and innovation across 10, 30, or 50+ teams if they are unmotivated, and surrounded by dependencies. You have to increase alignment so teams understand the big picture, and increase autonomy so they can solve problems themselves with zero handoffs.
To succeed at scale, your organization has to implement alignment and autonomy together. This means replacing command and control with contextual alignment, and giving teams the freedom to to make decisions in that context. This is known as aligned autonomy, from Henrik Kniberg’s work on Spotify culture and Stephen Bungay’s The Art of Action.
Implementing aligned autonomy starts with your leadership creating the right conditions. This includes outlining an organizational vision, communicating a product-technology context of desired customer impacts and technology direction, and creating a high trust culture in which outcome-oriented teams can make decisions for themselves. Those conditions then need to be protected by your teams using process and technology changes, such as automated metrics for customer impacts and paved roads for engineer enablement.
Independent, empowered product teams can achieve spectacular results. Later in this series, I’ll share how we help our customers to implement alignment with autonomy, look at the trade-offs in the Spotify Model and SAFe, and ask some Equal Experts consultants to contribute their own experiences. For now, I’ll just say:
To succeed at scale, give your teams contextual alignment with autonomy