HMRC’s original physical infrastructure consisted of 32 servers in total.

Eight servers were live. Six were staging servers. Four were for a testing environment. The live and staging sites were duplicated for disaster recovery purposes.

Outside of the main peak period, the applications only required about 25% of their total live staging and test capacity servers to support business-as-usual traffic (in other words, 0% of the DR capacity was used). But despite HMRC’s annual rush period only lasting between 24 and 48 hours, the servers were set for the highest levels of utilisation, 365 days per year.

If the highest levels of computing resources weren’t available during a peak period, every digital application housed on a specific server would fall over. This would prevent HMRC from processing critical revenue streams, such as VAT or PAYG. Further, loss of services would create additional pressure by triggering enormous amounts of customer enquiries and traffic to help desks.

Unfortunately, it wasn’t as simple as ‘turning down the server capacity’ in quiet periods.

Any change in resources throughout the year would require manual effort from the data centre supplier. They’d have to plan, scale, test, and provision DR testing. They’d have to manually add or reconfigure computers, storage, and networking, then manually ramp things up for performance and stability when the inevitable tax-time tsunami rolled in.

All told, any change in server capacity triggered a months-long process. In order to avoid the costs associated with manual reconfiguring, the infrastructure was essentially set to deal with HMRC’s busiest hour, of their busiest day, for the entire year.

After re-homing, and with the services now operating via Amazon Web Services (AWS), things can automatically scale as needed.

For example, from April 2017 to April 2018, 10.1 million Self-Assessment tax returns were filed. Of those total returns, 4.1 million were submitted in January.

Thanks to the new flexible and automated infrastructure, HMRC were able to scale up from 4 to 32 releases of the Self-Assessment filing application—coping with the rapid demand for additional performance and stability—before scaling back down once the filing deadline had passed.

The goal was clear: rapidly shift HMRC’s legacy applications from ‘A’ to ‘B’ while maintaining service continuity at all times.

To deliver a speedy migration while still serving huge numbers of people using the applications, the team took a unique approach to re-homing.

The approach began with creating generic services for authentication, auditing, user routing, and filing submissions, which could be used across multiple Portal applications.

Then, the team looked at Portal application architectures.

Each app was a monolithic Java application: a front-end and back-end with business logic built into the application. We decided to decouple the applications, splitting them into a frontend for the presentation layer, and a backend for business logic and data access.

De-coupling the monoliths provided HMRC with greater flexibility and the ability to deploy specific front- or back-end changes without changing the regression of the system.

Next, the applications were strategically sliced up so that any common servers or integration points—like authentication or simple content serving—could be shared by the remaining portions of unique code. These could then be easily ‘plugged in’ to use the shared parts.

Any datastore remained untouched. All the data also resided on separate infrastructures. Data was stored locally until a submission had been lodged, then stored in a data layer that was the existing un-migrated layer. This caused a negligible amount of latency issue, which was mitigated through improving latency of authentication, routing, and more.

The benefit? By migrating incrementally, we could decommission servers gradually and transition spend away from the expensive physical infrastructure, faster.

Re-routing traffic for service continuity, controlled monitoring and risk mitigation

Thanks to an innovative approach to traffic management throughout the migration process, HMRC could dynamically re-route traffic between the legacy infrastructure and the Multi-Channel Digital Tax Platform at a service, user, or page level.

This level of granular control empowered HMRC to send small amounts of traffic to the newly re-homed services as they shifted to the Digital Tax Platform, proactively monitoring the performance of that service at all times.

While it’s a common team structure, operating with siloed, distinct teams for each stage of development, regression testing, deployment, and support is largely inefficient.
Let’s consider the following team structure, which typically functions in a waterfall structure, and consists of:

• Development team
• Regression testing team
• Deployment team
• Post-production support team

Hypothetically, let’s imagine there’s an issue with any one of HMRC’s applications.

The post-production support team will likely look for issues in deployment as the first port of call. If there’s no identifiable problem with deployment—and the team knows the regression test passed, because otherwise it would not have gone through the change process—they’d raise the issue with the development team, assuming an issue within the code itself.

More than likely, the development team has moved on to another project. Halting progress on the current project, they switch context to review the code, trawling back through previous work to identify the potential issue.

Once the problem is identified and a fix is implemented, the fix needs to be reviewed by the regression testing team. They’ll perform testing and regression, before the fix is passed to the deployment team to push the change to production.

From there, the support team will identify that the fix has worked. Or, worst-case scenario, the fix hasn’t been successful, and the entire process is repeated.

While migrating applications to the cloud, we noticed the opportunity to improve team structure. By ensuring that the support teams also have the capability to deploy and test new fixes and features, we consolidate separate teams into one, and eliminate the inefficiencies created by multiple handovers.

Additionally, this creates:

• Vastly improved collaboration between technical teams and business stakeholders.
• Reduced lead times due to a Continuous Delivery pipeline.
• Removed any external dependencies on third-party suppliers, as internal teams have full ownership of support, maintenance and enhancement of services.
• Less commercial documentation and approval flows to request and deliver work.
• Proactive identification of issues, with no dependence on external suppliers to detect issues.
• More possibility to improve and refactor services through increased speed of delivery. Examples include improving Content Management and improving automated test coverage.