Using event-driven architecture for real-time stock visibility

Some of the crucial, and most constant, questions for every store are: do we have this stock available to promise to our customers? Where is it available? How will it be shipped to them?

In addition to being difficult questions to answer, each proved a massive strain on the existing technical architecture.

The sophisticated rules involved in answering these questions (referred to as Availability, Sourcing and Fulfilment capabilities) were extremely resource-intensive for the legacy monolith. For example, around 40% of the total calls processed in the monolith are related to product availability queries.

The answer to the crucial availability question—‘is this product available?’—is constantly changing. This can be impacted by a range of factors, including:

• Successful purchases made both online and instore, and the subsequent impact on stock levels

• Digital shoppers placing items in their cart, then exiting the checkout process before completing the purchase

• People grabbing items off the shelf in physical stores and either purchasing the items or placing them elsewhere in the store without completing checkout

• Items being lost, damaged, or potentially stolen

The vendor or location of a particular stock item. For example, our partner —like many large retailers— use drop-ship vendors for certain items. In some cases, products may be held in alternative third-party locations (a vendor might hold and deliver all trampolines to help warehouses save on space, for example), but still need to be catered for within the available inventory. Some stores will use a warehouse attached to a retail centre, while others have stand-alone distribution warehouses.

In short, the retail metric of ‘availability to promise’ (ATP) is a constantly moving target. Regardless, it is absolutely crucial that it’s accurate. Event-driven architecture offers real-time stock visibility with immutable precision, while securely surfacing data for multiple business intelligence units.

Accuracy and rapidity of information is critical. If the ATP number is too low, products will be incorrectly listed as ‘unavailable for purchase’, needlessly foregoing potential sales.

On the other hand, if the ATP is too high, they risk:

• overselling,

• an inability to fulfil customer orders with correct items,

• cancelled orders, and

• general customer dissatisfaction.

To provide an accurate representation of inventory levels, the partner was using a feed of real-time events pulling various datapoints from point-of-sale, warehouse stock levels, and other data sources throughout the business.

These feeds—called event streams—help to calculate a real-time, constantly updating the representation of stock, defined as “total inventory, minus confirmed orders minus, buffer” (‘buffer’ counts for people with items on the way to checkout that have not confirmed a complete purchase, for example).

Thanks to the event-based nature of the architecture, the data is immediate, can be consumed by multiple microservices and capabilities, initiates a wide range of automated business processes, and is immediately served to multiple business intelligence units—in the format they require— for necessary action or processing.

The architecture runs within the Microsoft Azure cloud platform, with events managed via Microsoft Event Hubs. Consumed events are written to a NoSQL database, Azure Cosmos DB, with a cache layer over the top to ensure the solution remains highly performant.

In terms of the architecture itself, the main storefront for ecommerce is the customer facing website. The customer website calls into the monolith—or at least the remaining capabilities contained within the monolith, while we’re in the process of strangling individual capabilities out as microservices—with all of the logic.

As part of the strangulation process, we’ve created a method for partitioning and processing subsets of customer traffic based on specific criteria. For example, we can implement rules to stipulate that all orders being fulfilled within South Australia are processed via the new microservices capabilities, while all other states are managed via the legacy capabilities housed within the monolith.

This approach—while providing granular control—allows the team to effectively ‘learn in production’ with comparatively lower levels of live orders, and thus minimal risk to both the end-customer and the retailer. This is crucial; even with enterprise level-quality assurance processes, you can never discount the value of seeing systems and architectures in practical application. You need data; you need live orders to process as a means to validate and identify anything that requires re-thinking or iterative enhancement.

Inevitably, the best way to endorse a system is to use it within a controlled environment.

Build, buy, borrow: a flexible combination to maximise value

As with any major system transformation, there were decisions to be made around buying off-the-shelf, building green-field solutions from scratch, or borrowing from existing componentry within the wider group.

Buying an entirely new overarching system to replace the existing one was always an option our partner could consider. However, it was discounted relatively early in the process as it would limit the team’s ability to innovate quickly and test new services. Ultimately, this was too great a tradeoff for a brand striving to consolidate its reputation at the forefront of in-store and digital customer experience.

At this scale, there’s a fundamental requirement to continuously test, trial, learn and improve.
This dynamic is simply not feasible within the constraints of large, standardised product packages.

By engaging with our partner early, it became clear that a Strangler Fig design pattern was the ideal approach. This process involved gradually migrating capabilities and business logic from the legacy monolith to a more flexible, microservices based architecture.

The first capabilities to transition were those that provided visibility of Availability, Sourcing and Fulfilment, while other capabilities with less demanding compute and resourcing requirements— like Personalisation, Payment, Promotions, and Order Management—remain within the monolith.

The Benefits of moving to Microservices

By moving to cloud services with a microservice-based architecture, we facilitate the necessary scalability and flexibility that our partner requires, both now and in future.

Microservices and event-driven architectures, by their nature, provide a wide range of advantages in comparison to a monolith. These include:

SCALABILITY

A nimble microservices architecture allows the retailer to rapidly scale in response to the traffic and infrastructure demands associated with large seasonal retail events in a way that simply isn’t possible in a monolithic architecture.

FASTER DELIVERY

Unlike monoliths, microservices readily lend themselves to less onerous and more independent deployment processes. As a result, the team can innovate at pace with access to more modular, independent components and a vastly reduced deployment effort.

GREATER EXTENSIBILITY

From a system perspective, event-driven architectures make it incredibly easy and efficient to slot in various third-party systems—as well as building out or ‘annexing’ additional functionalities and capabilities—as business requirements evolve over time. Learn more about flexibility and extensibility in event-driven architectures.

IMPROVED TALENT ACQUISITION

A revised approach to systems and development practice means we’re able to provide better tools and processes that appeal to a premium tier of technical team members. As a result, their people and culture departments are now in a position to attract stronger talent to the digital team.

TECHNOLOGIES AND TOOLS

Throughout the project we have implemented a variety of best-in-breed third-party technical solutions: