A Docker Compose project based on the code in this tutorial is available in our public GitHub repository.

A game-changing aspect of GraphQL is that it is built from the ground up to support API-of-APIs. So when the data required by your application is sourced from multiple providers, GraphQL can help you design a tailored API layer on top of these. StepZen is a cloud solution for automatically aggregating data sources, such as REST APIs, relational databases, and other GraphQL APIs, into a single GraphQL endpoint.

The StepZen CLI lets you build the API from its constituent data sources and deploy it to the StepZen cloud. This tutorial describes how to replace the deployment target with a custom setup, such as your own computer or a sealed contained environment (CI) environment. Here are some reasons that you may want to do so:

  • You are in the first stages of developing your app and want to defer hosting and deployment decisions to later.
  • You want to run end-to-end tests against your app in a contained environment (CI) as a quality gate for production deployment.
  • You are testing your app against mocks of data sources that are not implemented / made public yet.
  • You are evaluating StepZen and want to keep data contained while you do your proof-of-concept.

The Docker Compose tool - part of the Docker platform - allows you to define and spin up multi-container Docker applications. You create a YAML file to configure a set of services, based either on existing images (from public sources such as Docker Hub or from your own private repository) or built on-demand from local  Dockerfiles. Containers talk to each other over a virtual network, and you have fine-grained control over which ports are exposed to other containers and/or to the host.

In this article we use Docker Compose to run a mix of services: two data source APIs, the StepZen service providing a common API for the data sources, and an API client. For simplicity, the data sources are toy database and REST services, but are easily substituted with your own services or public Docker images. The example does not rely on cloud services (except for pulling the base images from Docker Hub), and so can even be run offline. You will need Docker Desktop and Node.JS installed on your system.

A SQL data source

The first data source in our example is a PostgreSQL database, containing a single table representing a stock portfolio. In an empty directory, create a file docker-compose.yaml with the following content:

version: "3.9"
services:
  portfolio:
    image: postgres:14
    environment:
      - POSTGRES_USER=c
      - POSTGRES_PASSWORD=c
      - POSTGRES_DB=portfolio
    volumes:
      - ./portfolio:/docker-entrypoint-initdb.d/

This configures the service portfolio from the official PostgreSQL image, sets up the database credentials, and maps the initialization script directory to the host directory portfolio. Next, create the new subdirectory portfolio and add the following SQL to a file inside of it (the name of the file is not important):

CREATE TABLE portfolio (
    symbol VARCHAR(16) NOT NULL,
    nshares INT NOT NULL
);
INSERT INTO portfolio (symbol, nshares)
VALUES ('AAPL', 100),
       ('MSFT', 200),
       ('TSLA', 300),
       ('GOOG', 400);

We can now check that the database is provisioned correctly by running docker-compose up. Note that docker-compose does not map exposed ports to the host by default; in order to connect to the database at the default PostgreSQL port, we could add the following lines to the portfolio service definition:

services:
  portfolio:
    # ...
    ports:
      - "5432:5432"

We are then able to connect using a PostgreSQL client on the host machine, e.g:

 psql postgres://c:c@localhost/portfolio

For the remainder of this example, host port mapping is only required by the StepZen service in order to talk to the StepZen CLI, while the ports of the data source containers only need to be accessible within the Docker virtual network. However, host-mapping may be useful for verifying that containers work as expected during development. To stop all services and delete the associated containers, run:

docker-compose down

A REST data source

As a second data source, we add a small REST endpoint based on Node.JS Express serving up random stock quotes. First, add the following section under the services section of docker-compose.yaml:

services:
  # ...
  quote:
    image: node:latest
    working_dir: "/quote"
    volumes:
      - "./quote:/quote"
    command: [ "sh", "-c", "npm install; npm start"]

Then create a directory called quote and create two files inside of it, package.json and server.js:

{
   "name": "quote-service",
   "dependencies": {
      "express": "^4.13"
   }
}

const express = require('express')
express()
  .use(express.json())
  .post("/", (request, response) => {
    const json = request.body
    if (json.symbol && json.symbol.length > 1) {
      response.send({
        'symbol': json.symbol,
        'price': (Math.random() * 1000).toFixed(2)
      })
    } else {
      response.status(500).send({
          'error': 'Unable to obtain quote'
      })
    }})
  .listen(80)

When the quote service starts, it installs the npm dependencies and then executes server.js. As the quote directory is mapped to the container, it will contain the cached node_modules after the first launch of the container, speeding up subsequent launches.

As we now have two data source services defined in our Docker Compose project, we are ready to add StepZen to the mix.

Adding the StepZen service

The StepZen service offers both introspection, the development-time extraction of GraphQL schemas out of existing data sources, and the run-time API server that materializes GraphQL queries from these data sources. Both of these features are bundled into a single Docker image:

us-docker.pkg.dev/stepzen-public/images/stepzen:production

Normally you will want to use the production-tagged image, as this matches the StepZen cloud service (you can alternatively pull latest, but note that this requires installing the latest beta version of the CLI as well).

The container is stateless; it relies on a PostgreSQL metadata database to store the deployed GraphQL endpoints. Next, let’s create another PostgreSQL container for this purpose (re-using our portfolio database is not a good idea as we should avoid coupling between data source APIs and the GraphQL layer). In a production deployment scenario, the metadata database is hosted by StepZen and does not need to be provisioned separately.

To add the metadata database and StepZen service container, extend the services section in docker-compose.yaml with the following service definitions.

services:
  # ...
  stepzen:
    image: us-docker.pkg.dev/stepzen-public/images/stepzen:production
    environment:
      - STEPZEN_CONTROL_DB_DSN=postgresql://stepzen:pw@stepzen_metadata/stepzen
    ports:
      - "9000:9000"
    depends_on:
      - portfolio
      - quote
      - stepzen_metadata
  stepzen_metadata:
    image: postgres:14
    environment:
      - POSTGRES_USER=stepzen
      - POSTGRES_PASSWORD=pw
      - POSTGRES_DB=stepzen

The environment variable STEPZEN_CONTROL_DB_DSN links the stepzen container to the stepzen_metadata container. On startup of stepzen, if the connection is established successfully, the metadata database is automatically initialized if it does not already exist. We have also mapped container port 9000 used for communicating with the StepZen CLI to the host.

Next, we are ready to spin up the containers:

docker-compose up

Check the container log for errors from any of the services (if there are errors, they are most likely due to a port already being in use by some process. If so, terminate the process or re-map the ports to free ones on the host). If everything started fine let’s next have a look at how StepZen can create an API for us from the data sources we defined in the previous two sections.

Connecting to the service

For the following steps, we use the StepZen CLI (version 0.21.0 or newer). Install it as follows:

npm i -g stepzen

After successful installation, the stepzen command is available globally. The first step will be to log in to the default graphql account, which is used to communicate with the local service. There are two keys associated with StepZen accounts: the admin key and the API key. The former is used during development for API administration tasks, such as adding endpoints or modifying existing endpoints by uploading a new schema, while the latter is used by the deployed app to invoke the API. We can retrieve the admin key from the StepZen service by running the key tool in the stepzen container:

docker-compose exec stepzen key admin

The output of the command looks like this:

graphql::local.io+1000::901304d9317ea6bd3735268bcdc1961e3f0f2f42a2eae3f58df055d891e97577

Note that the key is the entire line (i.e, including the graphql::local.io+1000:: part). To obtain the API key instead, run docker-compose exec stepzen key api. The API key will be needed later to access your endpoint after deployment.

Note that StepZen creates a random key when it initializes the database; as long as you keep your database container, the keys will remain the same even when recycling the stepzen container. On the other hand, if you shut down your services with docker-compose down, the next docker-compose up will result in generation of new keys as Docker Compose recycles the stepzen_metadata container.

Before proceeding with the login, we should note that the CLI by default connects to the StepZen cloud service, which is not what we want in this case. We can point it to our local service by adding an environment variable file .env in the current directory:

STEPZEN_DEPLOYMENT_TYPE=local
STEPZEN_ZENCTL_API_URL=http://localhost:9000
STEPZEN_DBINTROSPECTION_SERVER_URL=http://localhost:9000/dbintrospection
STEPZEN_JSON2SDL_SERVER_URL=http://localhost:9000/introspection

These environment variables ensure that the CLI connects to the local service instead (note that the override only applies when executing stepzen from the directory containing the .env file). Note that if you have mapped the port to a value different from 9000 in docker-compose.yaml, this needs to be reflected in the .env file as well.

We are now ready to log in. Run the following command:

stepzen login --account graphql --adminkey $(docker-compose exec stepzen key admin)

Alternatively, omit the flags and enter the account name (graphql) and admin key when prompted. Either way, the CLI will confirm that you are successfully logged in.

Building the API

A GraphQL schema defines the available queries, and materializers populate the queries with data. We will here leverage StepZen’s introspection, which generates both for us; we need only to point it to a data source, and it creates the schema and links it up to the source. Starting with the portfolio database created in the first step, run the following command to instruct StepZen to introspect the database with the passed credentials:

stepzen import postgresql --name portfolio_schema --db-host=portfolio --db-database=portfolio --db-user=c --db-password=c

The StepZen CLI will prompt for an endpoint URL fragment. Because this is the first introspection we run, select the default or choose something descriptive, such as api/financial.  Next, the introspection service pulls the schema from the database and turns it into GraphQL. Once complete, you will see a couple of new files have been created:

|- config.yaml
|- index.grapqhl
|- portfolio_schema   
|-  |- index.graphql
|- stepzen.config.json
  • config.yaml contains the database credentials (DSN) passed via the command-line flags.
  • The top-level index.graphql aggregates all imported APIs; at the moment it includes only portfolio_schema/index.graphql, which is the schema generated from the database. In the latter file, you can see that StepZen has generated a number of both queries and mutations for the portfolio table.
  • The file stepzen.config.json stores the chosen endpoint name. Our directory is now a StepZen workspace, meaning that it can be deployed to the StepZen service any time using the stepzen deploy command.

However, before deploying we should also add the quote service to the API. In this case, we will use another import command - stepzen import curl - to introspect REST endpoints:

stepzen import curl --name quote_schema --query-name getQuote --query-type Quote http://quote -H 'Content-Type: application/json' --data '{ "symbol": "MSFT" }'

We need to provide the content type and a piece of JSON data that the introspection can include in the request, as the quote service expects a JSON POST request. Note that we also provide a desired name for our query and the type returned by the query, as there is no context in the response from a REST response from which the introspection service could derive suitable names. (Note: We could also have omitted --query-name and --query-type, in which case default names would have been generated (we could then modify the schema by hand afterwards.)

If we now look in the generated quote_schema/index.graphql file, we can see that there is one query which is linked via the @rest connector to http://quote. This means that StepZen materializes getQuote queries by sending a POST request to this URL.

Deploying the API is straightforward, simply run: stepzen deploy

StepZen uploads your API configuration in the current directory (index.graphql, **/index.graphql, config.yaml and stepzen.config.json) to the metadata database. When complete, it shows an example curl command you can use to test your GraphQL endpoint (which should now be served at http://localhost:9000/api/financial/__graphql).

We are now done with the API design and deployment, and we can log out: stepzen logout

A GraphQL client

Next, to exercise the API, we add a service that prints the value of our stock portfolio to the console. Add the following definition of report to the services section of docker-compose.yaml:

services:
  # ...
  report:
    image: node:18
    working_dir: "/report"
    volumes:
      - "./report:/report"
    command: [ "sh", "-c", "npm install; node report.js"]
    depends_on: 
      - stepzen

Then, create a directory called report and add the files package.json and report.js to it:

{
    "name": "report",
    "scripts": {
      "start": "node report.js"
    },
    "dependencies": {
        "node-fetch": "2.6.7"
    }
}

const fetch = require('node-fetch')
const API_KEY = process.env.API_KEY
async function graphql(query) {
    const response = await fetch(
        'http://stepzen:9000/api/financial/__graphql',
        {
          method: 'POST',
          headers: {
              'Host': 'graphql.local.net',
              'Authorization': `Apikey ${API_KEY}`,
              'Content-Type': 'application/json'
          },
          body: JSON.stringify({
              'query': query
          }),
        })
    const json = await response.json()
    if (json.data) {
        return json.data
    } else {
        throw new Error(`GraphQL error: ${JSON.stringify(json.errors)}`)
    }
}
async function report() {
    const portfolio = (await graphql('query { getPortfolioList { symbol nshares} }')).getPortfolioList
    portfolio.forEach(async ({symbol, nshares}) => {
        const price = (await graphql(`query { getQuote(symbol: "${symbol}") { price } }`)).getQuote.price
        console.log(`${symbol} ${nshares} ${(nshares * price).toFixed(2)}`)
    })
}
report()

The above code invokes the getPortfolioList query to obtain a list of stock symbols and the number of shares held in the portfolio, and then for each symbol fetches the quote and prints the total value.

  • For simplicity, we use the node-fetch library (in a real-world application, you would probably want to use a fully-featured GraphQL client library such as @apollo/client). Note that we use the URL http://stepzen:9000/api/financial/__graphql to access the API - this is the address of the StepZen server within the container network.
  • In the Host header, the client must pass the value graphql.local.net or the request will be denied by the StepZen server.
  • Finally, we read the API key for the Authorization header from the environment variable API_KEY; this should be passed from the host when running the report container:
docker-compose run -e API_KEY=$(docker-compose exec stepzen key api) report

The above command results in output similar to the following:

AAPL 100 97554.00
MSFT 200 137626.00
GOOG 400 345364.00
TSLA 300 109701.00

An alternative way of obtaining the API key is by running the command stepzen whoami --apikey (note that this requires the CLI being logged in to the graphql account).

Summary

In this article, we described how to orchestrate a StepZen-based application together with its dependencies (the data source) and the StepZen service providing the GraphQL API endpoint using Docker Compose. In summary, the setup steps were:

  1. Defining containers for your data source APIs; these could be based on Docker images that you produce in other build pipelines, or local Dockerfiles.
  2. Adding the StepZen service and metadata database.
  3. Generating the StepZen configuration by introspecting the data source APIs and/or creating your own schemas.

The files produced by these steps depend only on the underlying data sources; they could be committed to version control and only updated when data source APIs change. Given this, the run-time steps, which you could easily automate in your CI environment, consist of:

  1. Starting the source APIs and StepZen: docker-compose up stepzen
  2. Logging in: stepzen login --account graphql --adminkey $(docker-compose exec stepzen key admin)
  3. Deploying: stepzen deploy
  4. Logging out: stepzen logout
  5. Obtaining the API key: docker-compose exec stepzen key api
  6. Running your app / API test suite against http://stepzen:9000/api/myapi/__graphql, authorizing the request with the API key obtained in step 5.

StepZen also supports other local deployment scenarios. In our example, the API client was another Docker container. A variation of this approach is to run your app/API test suite on the host and use Docker Compose only for StepZen and the source APIs. This scenario is also supported by the configuration presented in this article as StepZen serves both the admin interface and the APIs from the same host: the only difference will be in the client, where the API URL’s host should be localhost:9000 instead of stepzen:9000.

Another option is to use the stepzen service command to manage the StepZen service and metadata database instead of Docker Compose - see Run StepZen in Docker for Local Development for more information.

A Docker Compose project based on the code in this tutorial is available in our public GitHub repository.