Adding a GraphQL Response Cache

A Brief Introduction to Caching

Huge GraphQL query operations can slow down your server as deeply nested selection sets can cause a lot of subsequent database reads or calls to other remote services. Tools like DataLoader can reduce the amount of concurrent and subsequent requests via batching and caching during the execution of a single GraphQL operation. Features like @defer and @stream can help with streaming slow-to-retrieve result partials to the clients progressively. However, for subsequent requests we hit the same bottleneck over and over again.

What if we don’t need to go through the execution phase at all for subsequent requests that execute the same query operation with the same variables?

A common practice for reducing slow requests is to leverage caching. There are many types of caching available. E.g. We could cache the whole HTTP responses based on the POST body of the request or an in memory cache within our GraphQL field resolver business logic in order to hit slow services less frequently.

Having a cache comes with the drawback of requiring some kind of cache invalidation mechanism. Expiring the cache via a TTL (time to live) is a widespread practice, but can result in hitting the cache too often or too scarcely. Another popular strategy is to incorporate cache invalidation logic into the business logic. Writing such logic can potentially become too verbose and hard to maintain. Other systems might use database write log observers for invalidating entities based on updated database rows.

In a strict REST API environment, caching entities is significantly easier, as each endpoint represents one resource, and thus a GET method can be cached and a PATCH method can be used to automatically invalidate the cache for the corresponding GET request, which is described via the HTTP path (/api/user/12).

With GraphQL such things become much harder and more complicated. First of all, we usually only have a single HTTP endpoint /graphql that only accepts POST requests. A query operation execution result could contain many different types of entities, thus, we need different strategies for caching GraphQL APIs.

SaaS services like GraphCDN started popping up providing proxies for your existing GraphQL API, that magically add response based caching. But how does this even work?

How Does GraphQL Response Caching Work?

Caching Query Operations

In order to cache a GraphQL execution result (response) we need to build an identifier based on the input that can be used to identify whether a response can be served from the cache or must be executed and then stored within the cache.

query UserProfileQuery($id: ID!) {
  user(id: $id) {
    __typename
    id
    login
    repositories
    friends(first: 2) {
      __typename
      id
      login
    }
  }
}

{
  "id": "1"
}

Usually, those inputs are the Query operation document and the variables for such an operation document.

Thus a response cache can store the execution result under a cache key that is built from those inputs:

OperationCacheKey (e.g. SHA1) = hash(GraphQLOperationString, Stringify(GraphQLVariables))

Under some circumstances, it is also required to cache based on the request initiator. For example, a user requesting his profile should not receive the cached profile of another user. In such a scenario, building the operation cache key should also include a part that uniquely identifies the requestor. This could be a user ID extracted from an authorization token.

OperationCacheKey (e.g. SHA1) = hash(GraphQLOperationString, Stringify(GraphQLVariables), RequestorId)

This allows us to identify recurring operations with the same variables and serve them from the cache for subsequent requests. If we can serve a response from the cache we don’t need to parse the GraphQL operation document and furthermore can skip the expensive execution phase. That will result in significant speed improvements.

But to make our cache smart we still need a suitable cache invalidation mechanism.

Invalidating Cached GraphQL Query Operations

Let’s take a look at a possible execution result for the GraphQL operation.

{
  "data": {
    "user": {
      "__typename": "User",
      "id": "1",
      "login": "dotan",
      "repositories": ["codegen"],
      "friends": [
        {
          "__typename": "User",
          "id": "2",
          "login": "urigo"
        },
        {
          "__typename": "User",
          "id": "3",
          "login": "n1ru4l"
        }
      ]
    }
  }
}

Many frontend frameworks cache GraphQL operation results in a normalized cache. The identifier for storing the single entities of a GraphQL operation result within the cache is usually the id field of object types for schemas that use global unique IDs or a compound of the __typename and id field for schemas that use non-global ID fields.

{
  "User:1": {
    "__typename": "User",
    "id": "1",
    "login": "dotan",
    "repositories": ["codegen"],
    "friends": ["$$ref:User:2", "$$ref:User:3"]
  },
  "User:2": {
    "__typename": "User",
    "id": "2",
    "login": "urigo"
  },
  "User:3": {
    "__typename": "User",
    "id": "3",
    "login": "n1ru4l"
  }
}

Interestingly, the same strategy for constructing cache keys on the client can also be used on the backend for tracking which GraphQL operations contain which entities. That allows invalidating GraphQL query operation results based on entity IDs.

For the execution result entity IDS that could be used for invalidating the operation are the following: User:1, User:2 and User:3.

And also keep a register that maps entities to operation cache keys.

Entity   List of Operation cache keys that reference an entity

User:1   OperationCacheKey1, OperationCacheKey2, ...
User:2   OperationCacheKey2, OperationCacheKey3, ...
User:3   OperationCacheKey3, OperationCacheKey1, ...

This allows us to keep track of which GraphQL operations must be invalidated once a certain entity becomes stale.

The remaining question is, how can we track an entity becoming stale?

As mentioned before, listening to a database write log is a possible option - but the implementation is very specific and differs based on the chosen database type. Time to live is also possible, but a very inaccurate solution.

Another solution is to add invalidation logic within our GraphQL mutation resolver. By the GraphQL Specification mutations are meant to modify our GraphQL graph.

A common pattern when sending mutations from clients is to select and return affected/mutated entities with the selection set.

For our example from above, the following could be a possible mutation for adding a new repository to the repositories field on the user entity.

mutation RepositoryAddMutation($userId: ID, $repositoryName: String!) {
  repositoryAdd(userId: $userId, repositoryName: $repositoryName) {
    user {
      id
      repositories
    }
  }
}

{
  "data": {
    "repositoryAdd": {
      "user": {
        "id": "1",
        "repositories": ["codegen", "envelop"]
      }
    }
  }
}

Similar to how we build entity identifiers from the execution result of query operations for identifying what entities are referenced in which operations, we can extract the entity identifiers from the mutation operation result for invalidating affected operations.

In this specific case, all operations that select User:1 should be invalidated.

Such an implementation assumes that all mutations by default select affected entities and, furthermore, all mutations of underlying entities are done through the GraphQL gateway via mutations. In a scenario where we have actors that are not GraphQL services or services that operate directly on the database, we can use this approach in a hybrid model with other methods such as listening to database write logs.

Envelop Response Cache

The envelop response cache plugin now provides primitives and a reference in memory store implementation for adopting such a cache with all the features mentioned above with any GraphQL server.

The goal of the response cache plugin is to educate how such mechanisms are implemented and furthermore give developers the building blocks for constructing their own global cache with their cloud provider of choice.

Adding a response cache to an existing envelop GraphQL server setup is as easy as adding the plugin:

import * as GraphQLJS from 'graphql'
import { envelop, useEngine } from '@envelop/core'
import { useResponseCache } from '@envelop/response-cache'
 
const getEnveloped = envelop({
  plugins: [
    useEngine(GraphQLJS),
    // ... other plugins ...
    useResponseCache()
  ]
})

If you need to imperatively invalidate you can do that by providing the cache to the plugin:

import * as GraphQLJS from 'graphql'
import { envelop, useEngine } from '@envelop/core'
import { createInMemoryCache, useResponseCache } from '@envelop/response-cache'
import { emitter } from './event-emitter'
 
const cache = createInMemoryCache()
 
emitter.on('invalidate', entity => {
  cache.invalidate([
    {
      typename: entity.type,
      id: entity.id
    }
  ])
})
 
const getEnveloped = envelop({
  plugins: [
    useEngine(GraphQLJS),
    // ... other plugins ...
    useResponseCache({ cache })
  ]
})

The caching behavior can be fully customized. A TTL can be provided as global or more granular per type or schema coordinate.

import * as GraphQLJS from 'graphql'
import { envelop, useEngine } from '@envelop/core'
import { useResponseCache } from '@envelop/response-cache'
 
const getEnveloped = envelop({
  plugins: [
    useEngine(GraphQLJS),
    // ... other plugins ...
    useResponseCache({
      // cache operations for 1 hour by default
      ttl: 60 * 1000 * 60,
      ttlPerType: {
        // cache operation containing Stock object type for 500ms
        Stock: 500
      },
      ttlPerSchemaCoordinate: {
        // cache operation containing Query.rocketCoordinates selection for 100ms
        'Query.rocketCoordinates': 100
      },
      // never cache responses that include a RefreshToken object type.
      ignoredTypes: ['RefreshToken']
    })
  ]
})

Need to cache based on the user? No problem.

import * as GraphQLJS from 'graphql'
import { envelop, useEngine } from '@envelop/core'
import { useResponseCache } from '@envelop/response-cache'
 
const getEnveloped = envelop({
  plugins: [
    useEngine(GraphQLJS),
    // ... other plugins ...
    useResponseCache({
      // context is the GraphQL context that would be used for execution
      session: context => (context.user ? String(context.user.id) : null),
      // never serve cache for admin users
      enabled: context => (context.user ? isAdmin(context.user) === false : true)
    })
  ]
})

Don’t want to automatically invalidate based on mutations? Also configurable!

import * as GraphQLJS from 'graphql'
import { envelop, useEngine } from '@envelop/core'
import { useResponseCache } from '@envelop/response-cache'
 
const getEnveloped = envelop({
  plugins: [
    useEngine(GraphQLJS),
    // ... other plugins ...
    useResponseCache({
      // some might prefer invalidating only based on a database write log
      invalidateViaMutation: false
    })
  ]
})

Want a Global Cache on Redis?

Maybe you are in a server-less environment and the In-Memory Cache isn’t an option. Also, when having multiple server replicas, you might want to have a shared cache between all the replicas.

npm i @envelop/response-cache-redis

First, create a Redis database with your favorite hosting provider.

Once you have that, gather up the necessary connection settings (e.g., host, port, username, password, tls). Or even easier is to just find the connection string, so that you can create and configure a Redis client and set any additional options.

Then, with that instance of the Redis Cache setup, provide it to the useResponseCache plugin options, and you’re done. Here’s an example:

import * as GraphQLJS from 'graphql'
import { envelop, useEngine } from '@envelop/core'
import { useResponseCache } from '@envelop/response-cache'
import { createRedisCache } from '@envelop/response-cache-redis'
import Redis from 'ioredis'
 
const redis = new Redis({
  host: 'my-redis-db.example.com',
  port: '30652',
  password: '1234567890'
})
 
const redis = new Redis('rediss://:1234567890@my-redis-db.example.com:30652')
 
const cache = createRedisCache({ redis })
 
const getEnveloped = envelop({
  plugins: [
    useEngine(GraphQLJS),
    // ... other plugins ...
    useResponseCache({ cache })
  ]
})

More information about all possible configuration options can be found on the response cache docs on the Plugin Hub.