Data Feed API is out now: build custom indexers on Ceramic

From the very beginning, the Ceramic network was meant to be an open and customizable platform for developers to build on. The core Ceramic team is excited to keep delivering on that promise and launch Ceramic’s new Data Feed API.

The Data Feed provides a new API for Ceramic that allows developers to receive notifications of updates to JSON documents that are managed via Ceramic. This feed of document updates allows developers to customize the way their data is indexed and queried, and enables the development of new custom database products on top of Ceramic. Databases built on top of the Data Feed API get the advantage of powerful querying capabilities custom-tailored to the needs of the application, while retaining interoperability and composability of the underlying data with all other databases built on the Data Feed API throughout the Ceramic ecosystem.

What is the Data Feed API

The Data Feed API enables developers to keep track of all the new state changes happening to documents in the Ceramic network, as they are observed on a Ceramic node. These state updates can either be from writes sent to the Ceramic node through the HTTP Client, or writes discovered from the network for Streams belonging to Models that are indexed on the Ceramic node. Ultimately, the Data Feed API can be used to build custom indexers and other database products, e.g., OrbisDB.

Supported API Methods

The Data Feed is a read-only API, allowing you to retrieve the feed data through HTTP GET methods. The Data Feed operates on the aggregated state of documents (such as Models and ModelInstanceDocuments), abstracting away the lower-level concerns of the underlying event streaming system. This allows developers to build with familiar JSON documents while taking advantage of automatic application of updates to document state with built-in schema validation.

Feed data can be accessed via a new HTTP endpoint on the js-ceramic node with the URL path api/v0/feed/aggregation/documents. The feed utilizes Server-Sent Events (SSE) to push messages with the following schema:

FeedDocument = {
  resumeToken: string
  commitId: CommitID
  content: any
  metadata: StreamMetadata
  eventType: EventType
}

So for example, the following request:

curl http://localhost:7007/api/v0/feed/aggregation/documents

will return the following response:

data: {
  "resumeToken": "1714742204565000000"
  "commitId": "k6zn3t2py84tn1dpy24625xjv65g4r23wuqpch6mmrywshreivaqiyaqctrz2ba5kk0qjvec61pbmyl15b49zxfd8qd3aiiupltnpveh45oiranqr4njj40",
  "content": "{...}",
  "metadata": {
    "controllers": [
      "did:key:z6MknE3RuK7XU2W1KGCQrsSVhzRwCUJ9uMb6ugwbELm9JdP6"
    ],
    "model": "kh4q0ozorrgaq2mezktnrmdwleo1d"
  },
  "eventType": 2
}

The Data Feed API can also be consumed through event listeners to continuously pull and process events from the feed. For example:

import { EventSource  } from "cross-eventsource";
import { JsonAsString, AggregationDocument } from '@ceramicnetwork/codecs';
import { decode } from "codeco";

const source = new EventSource('<http://localhost:7007/api/v0/feed/aggregation/documents>')
const Codec = JsonAsString.pipe(AggregationDocument)

source.addEventListener('message', (event) => {
	console.log('message', event)
	//use JsonAsString, and AggregationDocument to decode and use event.data
	const parsedData = decode(Codec, event.data);
	console.log('parsed', parsedData)
})

source.addEventListener('error', error => {
	console.log('error', error)
})

console.log('listening...')

Resumability

The Data Feed API also comes with a resumability feature that provides “at least once” delivery semantics. Resumability is the ability for a consumer of the feed to pick up “where it left off” after downtime. Without resumability, an indexer that goes down has no way of receiving and indexing data that was propagated while the indexer was offline. The goal of resumability is to ensure that a consumer of the Data Feed API can be guaranteed to receive the most recent data for each Stream at least once, even in the event of failure.

Resumability in Data Feed API is enabled by the resumeToken property, which is added to every object emitted by the Data Feed API. This property enables you to initiate a connection and ask for the entries starting immediately after resumeToken.

For example, let’s say your application got an entry containing resumeToken: "1714742204565000000" . When connecting, you can pass the token value as a query parameter, as follows:

// ... same as a code snipped above
const url = new URL("<http://localhost:7007/api/v0/feed/aggregation/documents>")
url.searchParams.set('after', '1714742204565000000') // Value of the last resumeToken
// Connection to <http://localhost:7007/api/v0/feed/aggregation/documents?after=1714742204565000000>
const source = new EventSource(url)

Note that the internal structure of the resumeToken is meant to be considered opaque and may change in the future. You should not try to craft resumeTokens manually, instead, you should always use the resumeToken of the last event you received and processed successfully from the Data Feed resuming the feed with a new connection.

What’s coming next?

This release marks the initial public release of the Data Feed API with quite a few significant updates already on the horizon.

One of the upcoming significant updates will be introduced with the new Event Streaming API, powered by ceramic-one (a new implementation of the Ceramic protocol, written in Rust), which will allow developers direct access to the underlying event streams that have historically been hidden behind document-based APIs.

In addition to that, the core Ceramic team is committed to listening to your feedback and feature requests and taking them into account for future releases.

At this stage, the core Ceramic team would like to see early adopters trying out the new API and sharing the early feedback with the team.

Share your feedback with the Ceramic team!

Do you have ideas for how Data Feed API can enable the development of your projects? Or maybe you have questions or feedback on what could be improved? Share your thoughts with the core Ceramic team on the Ceramic Forum. We would love to hear from you!