What is Ceramic?

Ceramic is a public, permissionless, open source protocol that provides computation, state transformations, and consensus for all types of data structures stored on the decentralized web. Ceramic's stream processing enables developers to build with dynamic information without trusted database servers to create powerful, secure, trustless, censorship-resistant applications.

This overview introduces how:

• Decentralized content computation gives rise to a new era of open source information
• Stream processing provides an appropriate framework for dynamic, decentralized content
• You can use Ceramic to replace your database with a truly decentralized alternative

To skip ahead and get started building, try Playground to demo Ceramic in a browser application, the Quick Start guide to learn the basics using the Ceramic CLI, or follow the Installation page to integrate Ceramic into your project.

This post was originally published on the Ceramic documentation site. Contents here may fall out of date, but you can always find the most up to date version here.

The internet of open source information

At its core, the internet is a collection of applications running on stateful data sources – from identity systems and user tables to databases and feeds for storing all kinds of content generated by users, services, or machines.

Most of the information on today's internet is locked away on application-specific database servers designed to protect data as a proprietary resource. Acting as a trusted middleman, applications make it difficult and opaque for others to access this information by requiring explicit permissions, one-off API integrations, and trust that returned state is correct. This siloed and competitive environment results in more friction for developers and worse experiences for users.

Along other dimensions, the web has rapidly evolved into a more open source, composable, and collaborative ecosystem. We can observe this trend in open source software enabled by Git's distributed version control and in open source finance enabled by blockchain's double-spend protection. The same principles of open source have not yet been applied to content.

The next wave of transformative innovation will be in applying the same open source principles to the world's information, unlocking a universe of content that can be frictionlessly shared across application or organizational boundaries. Achieving this requires a decentralized computation network designed specifically for content with flexibility, scalability, and composability as first class requirements.

Decentralized content computation

Open sourcing the content layer for applications requires deploying information to a public, permissionless environment where files can be stored, computation can be performed, state can be tracked, and others can easily access content.

Advancements in other Web3 protocols have already achieved success in decentralized file storage. As a universal file system for the decentralized web, IPFS (including IPLD and Libp2p) provides an extremely flexible content naming and routing system. As a storage disk, durable persistence networks (such as Filecoin, Arweave, and Sia) ensure that the content represented in IPFS files are persisted and kept available. This stack of Web3 protocols performs well for storing static files, but on its own lacks the computation and state management capacity for more advanced database-like features such as mutability, version control, access control, and programmable logic. These are required to enable developers to build fully-featured decentralized applications.

Ceramic enables static files to be composed into higher-order mutable data structures, programmed to behave in any desired manner, and whose resulting state is stored and replicated across a decentralized network of nodes. Ceramic builds upon and extends the IPFS file system and underlying persistence networks, as well as other open standards in the decentralized ecosystem, with a general-purpose decentralized content computation substrate. Due to Ceramic's permissionless design and unified global network, anyone in the world can openly create, discover, query, and build upon existing data without needing to trust a centralized server, integrate one-off APIs, or worry if the state of information being returned is correct.

Streams

Ceramic's decentralized content computation network is modeled after various stream processing frameworks found in Web2. In these types of systems, events are ingested, processed as they arrive, and the resulting output is applied to a log. When queried and reduced, this log represents the current state of a piece of information. This is an appropriate framework for conceptualizing how dynamic information should be modeled on the decentralized web. Furthermore because the function that processes incoming events on any particular stream can be custom written with logic for any use case, it provides the general-purpose flexibility and extensibility needed to represent the diversity of information that may exist on the web.

On Ceramic, each piece of information is represented as an append-only log of commits, called a Stream. Each stream is a DAG stored in IPLD, with an immutable name called a StreamID, and a verifiable state called a StreamState. Streams are similar in concept to Git trees, and each stream can be thought of as its own blockchain, ledger, or event log.

StreamTypes

Each stream must specify a StreamType, which is the processing logic used by the particular stream. A StreamType is essentially a function that is executed by a Ceramic node upon receipt of a new commit to the stream that governs the stream's state transitions and resulting output. StreamTypes are responsible for enforcing all rules and logic for the stream, such as data structure, content format, authentication or access control, and consensus algorithm. If an update does not conform to the logic specified by the StreamType, the update is disregarded. After applying a valid commit to the stream, the resulting StreamState is broadcast out to the rest of the nodes on the Ceramic Network. Each of the other nodes that are also maintaining this stream will update their StreamState to reflect this new transaction.

Ceramic's flexible StreamTypes framework enables developers to deploy any kind of information that conforms to any set of arbitrary rules as a stateful stream of events. Ceramic clients come pre-packaged with a standard set of StreamTypes that cover a wide range of common use cases, making it easy to get started building applications:

• Tile Document: a StreamType that stores a JSON document, providing similar functionality as a NoSQL document store. Tile Documents are frequently used as a database replacement for identity metadata (profiles, social graphs, reputation scores, linked social accounts), user-generated content (blog posts, social media, etc), indexes of other StreamIDs to form collections and user tables (IDX), DID documents, verifiable claims, and more. Tile Documents rely on DIDs for authentication and all valid updates to a stream must be signed by the DID that controls the stream.
• CAIP-10 Link: a StreamType that stores a cryptographically verifiable proof that links a blockchain address to a DID. A DID can have an unlimited number of CAIP-10 Links that bind it to many different addresses on many different blockchain networks. CAIP-10 Links also rely on DIDs for authentication, same as the Tile Document.
• Custom: You can implement your own StreamType and deploy it to your Ceramic node if the pre-packaged StreamTypes are not suitable for your use case.

Authentication

StreamTypes are able to specify their authentication requirements for how new data is authorized to be added to a particular stream. Different StreamTypes may choose to implement different authentication requirements. One of the most powerful and important authentication mechanisms that Ceramic StreamTypes support is DIDs, the W3C standard for decentralized identifiers. DIDs are used by the default StreamTypes (Tile Documents and CAIP-10 Links).

DIDs provide a way to go from a globally-unique, platform-agnostic string identifier to a DID document containing public keys for signature verification and encryption. Ceramic is capable of supporting any DID method implementation. Below, find the DID methods that are currently supported by Ceramic:

• 3ID DID Method: A DID method that uses Ceramic's Tile Document StreamType to represent a mutable DID document. 3IDs are typically used for end-user accounts. When 3IDs are used in conjunction with IDX and the 3ID Keychain (as is implemented in 3ID Connect), a 3ID can easily be controlled with any number of blockchain accounts from any L1 or L2 network. This provides a way to unify a user's identity across all other platforms.
• Key DID Method: A DID method statically generated from any Ed25519 key pair. Key DIDs are typically used for developer accounts. Key DID is lightweight, but the drawback is that its DID document is immutable and has no ability to rotate keys if it is compromised.
• NFT DID Method (coming soon): A DID method for any NFT on any blockchain. The DID document is statically generated from on-chain data. The DID associated to the blockchain account of the asset's current owner (using CAIP-10 Links) is the only entity authorized to act on behalf of the NFT DID, authenticate in DID-based systems, and make updates to streams or other data owned by the NFT DID. When ownership of the NFT changes, so does the controller permissions.
• Safe DID Method (coming soon): A DID method for a Gnosis Safe smart contract on any blockchain. Typically used for organizations, DAOs, and other multi-sig entities.

Ceramic Network

The Ceramic Network is a decentralized, worldwide network of nodes running the Ceramic protocol that communicate over a dedicated topic on the Libp2p peer-to-peer networking protocol. Ceramic is able to achieve maximum horizontal scalability, throughput, and performance due to its unique design.

Sharded execution environment

Unlike traditional blockchain systems where scalability is limited to a single global virtual execution environment (VM) and the state of a single ledger is shared between all nodes, each Ceramic node acts as an individual execution environment for performing computations and validating transactions on streams – there is no global ledger. This "built-in" execution sharding enables the Ceramic Network to scale horizontally to parallelize the processing of an increasing number of simultaneous stream transactions as the number of nodes on the network increases. Such a design is needed to handle the scale of the world's data, which is orders of magnitude greater than the throughput needed on a financial blockchain. Another benefit of this design is that a Ceramic node can perform stream transactions in an offline-first environment and then later sync updates with the rest of the network when it comes back online.

Global namespace

Since all nodes are part of the same Ceramic Network, every stream on Ceramic exists within a single global namespace where it can be accessed by any other node or referenced by any other stream. This creates a public data web of open source information.

Additional node responsibilities

In addition to executing stream transactions according to StreamType logic, Ceramic nodes also maintain a few other key responsibilities:

• StreamState storage: A Ceramic node only persists StreamStates for the streams it cares to keep around, a process called "pinning." Different nodes will maintain StreamStates for different streams, but multiple nodes can maintain the state of a single stream.
• Commit log storage: A Ceramic node maintains a local copy of all commits to the streams it is pinning.
• Persistence connectors: Ceramic nodes can optionally utilize an additional durable storage backend for backing up commits for streams it is pinning. This can be any of the persistence networks mentioned above, including Filecoin, Arweave, Sia, etc. (coming soon).
• Query responses: Ceramic nodes respond to stream queries from clients. If the node has the stream pinned it will return the response; if not, it will ask the rest of the network for the stream over libp2p and then return the response.
• Broadcasting transactions: When a Ceramic node successfully performs a transaction on a stream, it broadcasts this transaction out the rest of the network over libp2p so other nodes also pinning this stream can update their StreamState to reflect this new transaction.

Clients

Clients provide standard interfaces for performing transactions and queries on streams, and are installed into applications. Clients are also responsible for authenticating users and signing transactions.

Currently there are three clients for Ceramic. Additional client implementations can easily be developed in other programming languages:

• JS HTTP client: A lightweight JavaScript client which connects to a remote Ceramic node over HTTP. The JS HTTP client is recommended for application developers.
• JS Core client: A JavaScript client which also includes a full Ceramic node. The JS Core client is for those who want the maximum decentralization of running the full Ceramic protocol directly in a browser application.
• CLI: A command line interface for interacting with a Ceramic node.

Getting started

Try Ceramic

To experience how Ceramic works in a browser application, try the Playground app.

Installation

Getting started with Ceramic is simple. Visit the Quick Start guide to learn the basics using the Ceramic CLI or follow the Installation page to integrate Ceramic into your project.

Tools and services

In addition to various standards referenced throughout this document, the Ceramic community has already begun developing many different open source protocols, tools, and services that simplify the experience of developing on Ceramic. Here are a few notable examples:

• 3ID Connect: A authentication SDK for browser-based applications that allows your users to transact with Ceramic using their blockchain wallet.
• IDX: A protocol for decentralized identity that allows a DID to aggregate an index of all their data from across all apps in one place. IDX enables user-centric data storage, discovery, and interoperability. It is effectively a decentralized, cross-platform user table. IDX can reference all data source types, including Ceramic streams and other peer-to-peer databases and files.
• IdentityLink: A service that issues verifiable claims which prove a DID owns various other Web2 social accounts such as Twitter, Github, Discord, Discourse, Telegram, Instagram, etc. Once issued, claims are stored in the DID's IDX.
• Tiles: An explorer for the Ceramic Network.
• Documint: A browser-based IDE for creating and editing streams.