API design first: a case for a better language

Building web APIs as products and thinking of digital products as APIs first is a well established practice. API-first makes digital products more versatile by giving control to the end-user. Instead of a rigid graphical user interface, a business offers a programmable interface over its domain. Adopting a product mindset means caring for the goal, journey and experience of the user of the API, the developers. Unleashing automation enables emergent use-cases from an ecosystem of external partners, open-source contributors and other internal product teams.

API-design-first is a natural next step for many organizations. This approach acknowledges that, if the API is the product and the domain is relatively complex, then we better invest in API design before we build anything. It brings similar benefits as digital product design applied to web user interfaces. A product designer delivers incremental UI designs and validates clickable prototypes before the frontend developer writes the first line of Javascript. With an API-design-first approach, distributed teams collaborate on web services incrementally, validating API designs independently of any implementation or technical stack.

OpenAPI Specification (a.k.a OAS) is the most popular web API specification and description format. The format is specified in JSON but naturally extends to YAML for better readability. It is the de facto standard for generating code from specification and generating specification from code. It is available in most general purpose programming languages via third-party libraries and supported by many web development frameworks. A large ecosystem of compatible tools is maintained by a dynamic community.

Manually maintaining the OpenAPI definition of a couple of HTTP end-points in a handful of YAML files is straightforward. When the business domain gets bigger or more complex, e.g. a few dozen resources and JSON schemas, the lack of modularity intrinsic to the OpenAPI format becomes visible. YAML files quickly grow beyond thousands of lines. They are harder to refactor. Code reviews are impaired as the semantic of a change and its lexicographic context are less and less consistent. OpenAPI editors can help to some extent. Unfortunately, as they only offer a UI layer on top of the same OpenAPI format, it doesn’t solve the collaboration and modularity problems at scale.

OpenAPI started as an API description and documentation format long before the industry coined the term API-design-first. Years have passed and the community ended up promoting the OpenAPI Specification to an API design language. At its core, OpenAPI still remains optimized for machine generation and consumption rather than readability. Writing Abstract Syntax Trees in YAML by hand only gets you so far. When dealing with large and complex business domains, we need a better API specification language to make the most of API-design-first.

What is a web API definition?

RAML, API Blueprint and OpenAPI are all specifications to define HTTP interfaces. In that sense, they all belong to the class of interface description languages (IDLs). They enable interoperability over web services between different general purpose languages and technologies. For example, an HTTP server written in Go can provide an OpenAPI definition that is consumed by a team writing the corresponding HTTP client in TypeScript. The same interface definition can later be used to write a client in Java or Rust.

The scope of those interface languages being HTTP services, their domain is mainly about:

• what HTTP end-points are available, as URIs;
• what HTTP methods are supported by each URI;
• what parameters can be passed in the query string part of the URI;
• what HTTP status codes can be returned and why;
• what media type is potentially expected and/or returned;
• for specific media types, e.g. JSON or XML, what schema the request or response must comply with.

Example of a minimalistic service definition in RAML:

# GET /message
+ Response 200 (text/plain)

        Hello World!

OpenAPI

The OpenAPI Specification defines an OpenAPI document as a JSON or YAML object. The choice of a data interchange format suggests a declarative configuration rather than a programming language. That is in line with the origins of OpenAPI, back in 2011, then named Swagger, not as an API design language but as a formal mechanism to communicate an API contract for client generation and documentation purposes.

Swagger was invented to fill a gap. At a time when SOAP and WSDL dominated the enterprise web services landscape, the emergence of REST pushed younger digital companies to prefer plain-old XML or JSON over HTTP. It was cheaper, more flexible and worked just fine, excepted for the lack of standards and tools. Swagger was not perfect but it proposed a JSON schema to describe HTTP web services, and that is what the community needed. Some alternatives emerged a couple of years later, e.g. RAML, API Blueprint, but the community behind Swagger, and later OpenAPI, made the difference. Nowadays, Swagger 2 and OpenAPI 3.x are by far the most popular choices.

Over the years, the industry evolved from everything-is-XML to everything-is-JSON. OpenAPI does provide some XML schema capabilities, but the vast majority of the web APIs released today are JSON-based. Even if only partially compatible at first, the last revision of OpenAPI offers full feature parity with JSON Schema, the industry standard for describing JSON data.

Example of OpenAPI 3.x document, in YAML:

openapi: 3.0.3
info:
  title: OpenAPI definition
  version: 0.1.0
servers:
- url: https://example.org
paths:
  /hello:
    get:
      summary: get-hello
      operationId: get-hello
      responses:
        default:
          description: Greetings
          content:
            application/json:
              schema:
                type: object
                properties:
                  greetings:
                    type: array
                    items:
                      type: string

JSON Schema

The first draft for JSON Schema was released in 2009. It introduced a JSON format for defining the structure of JSON data. In a world where most of the web APIs are defined in terms of JSON requests and responses, we needed a formal mechanism to communicate the structure of JSON documents. As with OpenAPI, a JSON Schema document is either a JSON or a YAML object. It is today the most popular specification format for data interchange over the web.

An additional benefit of JSON Schema is to equally apply to popular NoSQL databases and streaming technologies. MongoDB supports automatic schema validation on collections. The Schema Registry of Confluent Kafka allows both producers and consumers to serialize and deserialize messages according to a JSON Schema. This ability to share JSON object models end-to-end, from the web API down to the data store, is powerful. It does not mean that the schema of the web API is the same as the database and the event stream. The envelop of the documents will change to adapt to the medium in each layer of the stack. Nevertheless, what is embedded in those documents, the core domain models, are likely consistent across the stack.

Example of JSON Schema document, in JSON:

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "https://example.com/product.schema.json",
  "title": "Product",
  "description": "A product in the catalog",
  "type": "object"
}

Limitations

When used as part of the so-called code-first practice, i.e. writing the implementation of an API and then generating its contract, OpenAPI and JSON Schema are very powerful tools. All mainstream programming languages support JSON and YAML. It is straightforward to map a native data structure in any language to its JSON or YAML equivalent. The generated OpenAPI definition can then be used to produce interactive HTML documentation with tools like Swagger UI or Redoc, generate client code or enforce API and JSON document structure. It is a slightly different picture when adopting API-design-first practices.

Before proceeding further, it is important to note that API-design-first does not imply that no design takes place when doing code-first. Writing correct code requires a design phase, API or not. Identifying the problem, studying potential solutions, and agreeing on a direction is all design work. The main difference is whether the API definition is considered as a first-class design document, or only a by-product of an API implementation.

Promoting OpenAPI and JSON Schema to design documents brings additional requirements. Not only those specification formats need to cater for the communication of API contracts, they also need to support the lifecycle of incremental API definition, in terms of API developer experience and tools. Incremental development is a given in any general purpose programming language. We assume the capacity to start with something small, to add features sprint after sprint, to refactor to keep the code true to the domain model. Unfortunately, this is not an intrinsic capacity of neither OpenAPI nor JSON Schema.

Expressiveness

JSON means JavaScript Object Notation. It is a human and machine-readable representation, as a string of characters, of the canonical JavaScript data structure, the JavaScript object. A JavaScript object is a collection of key-value pairs, also called properties. For what OpenAPI and JSON Schema are concerned, keys are fixed strings (the property name), and values are of any JavaScript type (the property value). YAML employs a slightly different syntax but the structure is identical.

The expressiveness of both OpenAPI Specification and JSON Schema is therefore limited to a tree of key-value pairs, potentially spread across multiple files. At small scale, the tree might span a hundred of properties. It still fits on a couple of pages on a screen. It is not too difficult to navigate. When touching upon enterprise domain models, even a single bounded context requires thousands of properties. This is when problems start.

Abstraction

The essence of software engineering is to keep the important concepts in our head at any given time. When reasoning about a problem, there are cognitive boundaries to how much information we can keep. It is utterly expensive to reason about an API definition at the scale of thousands of properties laid down sequentially on a screen.

Software engineers invented abstractions to deal with this very problem, be it for programming languages:

• structured programming: programs are control flows;
• objected oriented programming: programs are objects;
• logic programming: programs are logic relations;
• functional programming: programs are function compositions.

Or computer systems:

• POSIX: operating system interfaces;
• World Wide Web: distributed communications;
• Kubernetes: container orchestration.

Or application domains:

• HTML: linked documents;
• MATLAB: mathematics;
• SQL: structured data management.

The value and challenge of choosing the right abstraction is at the core of creating a programming language. To quote Yukihiro Matsumoto, the creator of the Ruby programming language:

I believe that the purpose of life is, at least in part, to be happy. Based on this belief, Ruby is designed to make programming not only easy, but also fun. It allows you to concentrate on the creative side of programming, with less stress.

Not all HTTP interfaces live at the same level of abstraction. Leonard Richardson proposed a model to reason about the so-called REST maturity levels. Following this proposal, the mental model to reason about a web API is structured around:

1. HTTP as a transport;
2. composition of resources;
3. semantic of HTTP verbs;
4. relations between resources, a.k.a. hypermedia.

Whichever maturity level we chose, none of the crucial concepts of web API design are properly captured by a JSON or YAML representation, as defined by OpenAPI and JSON Schema. Everything here has the same type, and lives at the same level of abstraction: a key-value in a gigantic tree. Details cannot be encapsulated into higher level concepts. Concepts do not compose to enable incremental development. Making small semantic changes often involves convoluted pruning and cutting, just to keep the JSON object tree alive.

Doing better

A language where refactoring is painful or error-prone entertains fear for change. This is the opposite of what we need an API design language to be. Designing means exploring different options, putting aside implementation details, failing fast and correcting the path to land on the desired solution. A design language must communicate design decisions clearly, abstracting away unnecessary details, without ignoring them. Neither JSON nor YAML are suitable language substrates to achieve that goal.

When dealing with complex business domains in distributed organizations, API-design-first is the way to go. OpenAPI and JSON Schema have conquered the web API world and are here to stay. They are the assembly languages of API development, the common interface at the core of a vibrant ecosystem. We only need better abstractions on top of the assembly. To paraphrase Yukihiro Matsumoto, we must make API design not only easy, but also fun. We must concentrate on the creative side of API design, with less stress.

If you are interested in a concrete example of what such a language could look like, feel free to check out Oxlip.