Introspection is a heavy word, but in systems architecture it essentially means: “I will be able to tell you what I can do programmatically.”

So if I am a database, I can tell you the tables I have, the columns and views, etc. If I am a REST backend, I can tell you the curl you need to make, the optional parameters and what I return as a response, including error codes. And if I am a GraphQL endpoint, I can tell you the queries, mutations and types that I support.

As a developer, you can always read docs; make SQL, curl, or GraphQL calls; and learn the backend capabilities by trying. Then why bother about introspection? Because introspection is the only way for a program to build something extraordinary on top of it. It could be a tool like Swagger Docs that creates docs. It could be a tool like GraphiQL that lets developers submit queries and see responses. It could be a database optimization program that tries to create indexes automatically. It could be an IDE that needs a pulldown of the queries supported by an application. It could be a change management tool that runs periodically and determines the impacts of the backend changes on the application. It could be an autonomous agent that bootstraps itself by starting an understanding of what the landscape of systems is. It could be many things, but the important thing is that it is not a human being.

The State of Introspection for REST APIs

The current gold standard for describing REST APIs is OpenAPI. OpenAPI 3.0 was the first official release of the specification after being donated to the OpenAPI Initiative and renamed from the Swagger Specification to OpenAPI specification in 2015. OpenAPI is a description, in YAML or JSON, of what the backend capabilities are. So, for example, here is a snippet from devto.io:

---
paths:
  /articles:
    get:
      description: |
        ...
      parameters:
        - $ref: "#/components/parameters/pageParam"
        - $ref: "#/components/parameters/perPageParam30to1000"

As you can see, it specifies that the API supports GET /articles?pageParam.

I won’t get into the exact syntax here — there are plenty of references. The great thing about the OpenAPI spec is that it solves the documentation problem for a machine; with tooling, it also solves the documentation problem for human beings. Two birds with one stone!

However, creating an OpenAPI spec for an endpoint is entirely voluntary. Twitter does not publish its OpenAPI spec, and the OpenAPI spec for HolidayAPI is out of date! I estimate that less than 5 percent of the more than 100,000 APIs that are publicly available have good, consistent and up-to-date OpenAPI specs. So that leaves 95,000 APIs with no OpenAPI specs and no way for any programmatic access to them.

The State of Introspection for GraphQL APIs

GraphQL has emerged as the preferred paradigm for frontend developers to fetch and modify backend data in the past few years. GraphQL is intuitive and performant, and the amount of tooling available around it is impressive. And GraphQL, by default, is introspective. What that means is that GraphQL endpoints support a GraphQL query (wow!) like this:

{
  __schema {
    queryType {
      name
    }
  }
}

Fundamentally, we ask a GraphQL endpoint to tell us what queries it supports using a GraphQL query (this reminds me of SQL introspection in most databases!). This is convenient, actually awesome, and allows a whole set of third-party tools to work against any GraphQL endpoint.

Furthermore, if the GraphQL endpoint changes, the introspection query returns a different — current or updated — result. Consequently, as long as the program issues the query periodically, the production of an API endpoint and its use stays up to date. How awesome is that? In contrast, in OpenAPI, the spec is a separate doc. So if that is not updated, the endpoint will do something, and the spec will say something else. Bad!

The ‘We Don’t Have OpenAPI Specs for 95% of REST APIs’ Problem

Postman has some tooling to take a Postman collection and generate an OpenAPI spec out of it. However, it is experimental, and there is no sure shot that it works. Some people have created some OpenAPI specs against the APIs they like, but again, they are typically incomplete or wrong as a third-party effort. For example, one can get the OpenAPI spec for a small subset of AccuWeather API, but it is woefully outdated. So, in essence, if the API provider does not want to provide, or does not provide for whatever reason, an OpenAPI spec, the developer is basically out of luck.

However, if you consume a GraphQL API, you get introspection for free. And you get all the benefits of GraphQL. So no surprise that developers are increasingly demanding that the APIs that they deal with are GraphQL-ized.

If an API provider decides to GraphQL-ize their REST endpoints, some tricks help make the investment worthwhile:

  1. Write the OpenAPI spec. Then it is simple, with tools like IBM’s openapi-to-graphql and StepZen’s OpenAPI introspection, to create a GraphQL endpoint. Thus, you kill two birds with one stone: You get an OpenAPI spec for your REST API consumers, and you also get a fantastic GraphQL API for your GraphQL consumers. We recommend this path.
  2. Take your existing curl and Postman collections and generate a pretty good GraphQL out of it (using tools like StepZen). The advantage of this is that you get GraphQL through simple clicks and responses.

Summary

There are two decades of investment in REST, yet the future of APIs that deal with modern frontend experiences is primarily being written around GraphQL. Unfortunately, the investment in introspection in REST has not kept up with its popularity. Meanwhile, GraphQL comes with auto-introspection.

We believe that some targeted investments by the API providers, and using an increasing number of REST-to-GraphQL open source and vendor tools, will make the transition much more straightforward. Come on, let’s get a move on!

Originally published: The New Stack