Enterprise integration and the API-first cargo cult

A lot has been written on the potential of a sound API strategy in the enterprise. APIs are everywhere, and even in the enterprise, the so-called API-first approach is getting traction. IT departments are waving the benefits of reuse and investing in enterprise architectures to support API cataloging, gateways and release management. Many companies are doing the right things, but we also see superficial implementations. Vendors of API toolboxes are keen to praise API-first to sell more licenses but do not always tell the full story. The enterprise system integration practitioners are boarding the API train. Every low-code integration is now called an API.

Still, migrating from EDI to JSON is not enough to become a digital company. Significant business value can only be unlocked by rethinking API design practices. By focusing on easy and superficial problems, we only entertain the API-first cargo cult in the enterprise.

State of API strategy in the enterprise, or lack thereof

It might seem trivial but with the advent of data engineering and techniques like event sourcing, we tend to forget that an API can be more than an end-point to extract data from a system. In generic terms, an Application Programming Interface, or API, is the programmatic interface developers rely on to interact with a computer system. A piece of code “talks” to another piece of code via its API. Whatever a system does, its API makes it available to developers programmatically.

The ultimate consumer of an API is the developer of the system. An API contract is a user manual for the developers. An inconsistent API has the same effect on a software engineer as an inconsistent graphical interface on an end-user. Similarly, an API projects a mental model of a system. How concepts interact with each other. What can be achieved or not. Expecting adoption across an ocean of inconsistent models leads to disappointing results. As a comparison, most companies have standard web UI design systems to enforce quality and consistency across their customer experience. Too many enterprises don’t have anything close to an API design system.

An API is also not just a data feed, at least if your system is a little bit more than a data store. There are low-level, technical APIs of course. Within the boundaries of a business domain, we integrate with databases, logging and monitoring tools, and plenty of other technical components. There is nothing wrong with that. When considering business applications though, offering a programmatic interface means giving access to business capabilities. It is intriguing to see enterprises deploying elaborate business capability models entirely decoupled from their API strategy.

The business capabilities offered by a composition of systems is poorly represented by the sum of its internal points of integration. At least it is not a good recipe for a consistent, simple and productive developer experience. We can describe a mechanical watch from all the interactions between gears and strings, but it doesn’t tell much about the capability of reading time. We need enterprise API catalogs to articulate the business capabilities around the available programmatic interfaces. We do not help API reuse by exposing every single point-to-point and technical integration in a catalog. Doing so breaks a valuable encapsulation of the domain models and confuses API consumers.

Domain driven API design

There are well documented techniques to shape a sound API strategy for the enterprise. Let’s explore some them.

API as a product

We acknowledge that APIs are consumed by developers. Putting a product lens means caring for the goal, journey and experience of the user. This is also known as the API-first approach.

As mentioned before, too many APIs are crafted as naive data interfaces, e.g. CRUD (Create Read Update Delete). Without being explicit about the underlying business process we take the risk of focusing only on data and ignoring behavior. There are pure data interfaces in specific domains, e.g. in business intelligence (BI) and management information (MI), but that is not the entirety of what most businesses do.

For example, whereas the purpose of a BI/MI interface is to deliver digital insight, the purpose of a shopping basket API is to power a marketplace. The goal of a user interacting with this API is to programmatically execute a business process, i.e. purchasing products from a shop. The role of the API product manager is to deliver an interface that captures as much knowledge as possible from the actual business process. This is hard, requires working with subject matter experts and is often not reducible to a simple CRUD interface.

Managing an API as a product also implies a deliberate SLA (Service Level Agreement), release and lifecycle management. Not all system integrations need to be managed as a product, but business APIs should. Instead of a catalog of system integrations, the enterprise needs a catalog of business API products.

Domain driven design

When taking an API-first approach, we give the API the privilege to model the business domain. We leave the system integration details behind and only retain what is true to both the business process and its digital implementation. This distillation process has been studied and documented for many years now, under the name Domain Driven Design, or DDD.

DDD is a vast topic and here is not the place to go into too much details. To summarize, this approach:

• promotes domain knowledge and the establishment of an ubiquitous language;
• is a model-driven design technique for business domains. It sits between the application and infrastructure layers;
• defines categories of objects within a domain that capture specific characteristics and behaviors: entities, repositories, value objects, aggregates, services, modules and factories;
• favors continuous improvement and refactoring of the domain models, a.k.a. distillation;
• scales across teams and business boundaries by being explicit about domain bounded contexts.

DDD is not a silver bullet but still an efficient technique when facing complex business domains distributed over multiple teams. It is a powerful tool to capture business knowledge from subject matter experts and to keep a consistent view of the world, shared by both business stakeholders and digital product teams. There is no reason for DDD to be any less efficient when the core product is an API. As a matter of fact, the synergies between DDD and REST API design have been explored before.

Another interesting aspect of DDD in the context of enterprise architecture is the ability to integrate with business capability modeling. Such a model captures the hierarchy of activities that a business needs to master, e.g. from sales, finance, logistics, down to pricing, shipping …etc. At a certain level of details, each capability maps naturally to a bounded context, in DDD parlance. This is not to say that DDD and business capabilities live at the same level of abstraction, but there is valuable continuity. We should exploit that continuity to build more actionable capability maps and enterprise API catalogs.

API design first

Let’s take one more step to go from API-first, i.e. promoting the API as the canonical product, to API-design-first, i.e. iterating on API design before writing any implementation code.

Let’s start with the disclaimer that not all APIs need to be designed that way. There are many valid reasons to generate API contracts from code when the scope of work and team setup is appropriate. Code-first works well on internal APIs, collocated teams, tightly coupled release cycles, homogeneous technology stacks or small and trivial domains. Unfortunately this is usually not the case at enterprise level. By definition, enterprise architecture has to cover heterogeneous business domains and technology stacks, operated by multiple teams and independent release cycles.

It seems completely natural for a digital product team to adopt a UI-design-first approach. It is the only way to deliver a consistent and pleasant user experience across the product lifecycle. It enables different disciplines to work in parallel: the frontend engineer implements the design from the last iteration while the product designer works on the next. A design prototype can be validated with users without having to write or deploy any code. Fixing a defect early in the product design is much cheaper than reworking a product in production.

Successful tools (e.g. Figma) and processes are tailored to the collaboration between UI designers and frontend developers. Those processes integrate nicely with agile software development practices and allow for incremental product design. When the core product is an API and not a graphical user interface, the same principles and benefits apply:

• the developer experience is consistent across API domains and use cases;
• API producer and consumer teams can work in parallel;
• API prototypes can be generated from a specification and tested before writing production grade software;
• quality increases as fixing API design defects early becomes trivial.

At the core of API-design-first is the design language. Even if not perfect, the de-facto standard when working with REST APIs is OpenAPI. Choosing an API definition language is only the easy part though. Many companies adopt OpenAPI definitions, but only as a by-product of an API implementation in a general purpose language (e.g. Go or Python).

Adopting API-design-first means treating the OpenAPI definition as a first-class code base. It needs its own:

• source version control, for distributed collaboration;
• pull request and code review process, to govern the iterative design;
• coding style and linting rules, to enforce consistency;
• integration tests, to check whether the API implementation complies with its definition;
• CI/CD pipelines, to run tests and deploy human-readable documentations.

Ways forward

As we have seen, API-first is not about systems integration. A digital company achieves efficiency gains by offering programmable and consistent interfaces for its business capabilities, as API products.

There is continuity between business capability models and API design. Domain Driven Design is a powerful methodology to deliver the right product incrementally, including APIs.

Thinking of APIs as products means we can apply what works for other kinds of digital products. Agile software development and product design processes fit perfectly with the iterative API-design-first approach. With the right tools and processes, it is not a novelty.