The next architecture for building Web3 data apps

The Ceramic community is committed to providing the best possible developer experience for building Web3 data applications. Over the past few months, the community has made significant upgrades to Ceramic's emerging open source developer tools and data standards ecosystem, including replacing the popular IDX.js client with a more powerful and flexible set of tools. This post describes the best ways to build applications on Ceramic today as of October, 2021.

The Web3 data architecture

The primary difference between Web3 and Web2 applications is their data architecture. Web3 applications rely on a user-centric ("self-sovereign" or "sovereign") data storage architecture. In this model, application data is hosted on a decentralized network, stored with each individual user, and is written to and queried by applications client-side. This contrasts to Web2 applications, which rely on an application-centric design where all data for all users of an application is stored in a single database server siloed to that application.

What's needed to build sovereign apps?

Therefore, building Web3 data apps requires a few foundational primitives:

1. Network for decentralized data availability with fast mutability
2. Protocol for identity-centric, model-based data storage and retrieval
3. Client for performing CRUD operations on users' stores at runtime

User-centric data models

As you can see, Web3 data applications are built on the concept of data models for storing and retrieving structured data from user-controlled storage. Web3 takes a model-based approach vs. an application-based approach in order to allow for simple data reuse and composability across applications. It's much easier for applications to share individual data models with other applications (e.g. user profile, social graph, skills, linked blockchain accounts) than it is to force these applications to share entire databases containing all different kinds of data, many of which are not shared between applications.

Creating a great developer experience around building applications with user-centric data models is important to accelerating adoption of this new Web3 data architecture. Tools that improve the developer experience and increase cross-application data interoperability include:

• Tooling to create and deploy data models
• Tooling to share, discover, and reuse data models

Example: Decentralized social network

As a simple example of how this all works, let's consider a decentralized social network. Behind the scenes this application can be reduced to just four data models, where each user maintains their own data in their own store for each model:

• PostList: stores an index of a user's posts
• Post: stores a single post
• Profile: stores a user's profile
• FollowList: stores a list of users they follow

Application deployment and usage

To deploy this data application, a developer would deploy each of these data models to Ceramic. After they're available on the network, they can be used by any application to CRUD a user's data store for that model. For example, if another developer is building a new social network and wants to bootstrap it with existing users and content, they can simply reuse the same data models as the initial social network and their application will be able to automatically load and support storage interactions on this existing data. This is how multiple applications can collaborate on shared data sets, and how open data standards and protocols can emerge from any developer in the community.

For an example of two applications building on the same data models to achieve data interoperability, try the Self.ID and DNS applications. Any change to data in one application is automatically reflected in the other.

IDX: ID-centric, model-based storage protocol

Ceramic provides a decentralized network for data availability with fast mutability, the foundation upon which Web3 data applications can be deployed and run. About a year ago the Ceramic community first introduced CIP-11 "Identity Index", commonly referred to as the IDX protocol. It specifies a protocol for identity-centric, model-based data storage and retrieval implemented on top of Ceramic – the second requirement for building sovereign applications.

IDX.js: The most popular IDX client

IDX protocol (CIP-11) is just a specification and not working code. To make the protocol usable by application developers, the 3Box Labs team created IDX.js, a runtime library for performing CRUD operations on users' data stores – the third and final requirement for building Web3 data applications. They also created supporting tools such as the IDX CLI, and a reference application, Self.id, to showcase IDX.js and related Ceramic technologies.

By serving as a simple entry-point to building applications on Ceramic, adoption of IDX.js has accelerated in lockstep with growth of the Ceramic ecosystem and has become the most popular way to build applications on Ceramic. Almost all of the applications currently deployed on Ceramic mainnet are using IDX.js.

Why change a good thing?

In an attempt to offer more features, service more use cases and provide the best developer experience, the scope of IDX.js grew beyond simply providing a bare-bones implementation of the IDX protocol to a point where it became unnecessarily monolithic and opinionated.

As a result, IDX.js and associated libraries idx-constants, idx-tools, idx-cli are being deprecated. They are replaced by a new collection of tools outlined in the rest of this post.

If you are currently using IDX.js and/or 3ID Connect, there is no rush to upgrade your code. There are no breaking changes to the IDX protocol itself, and the data interactions of the new libraries are fully compatible with IDX.js.

DataModels: Open source for data models

DataModels allows developers to create, share, reuse, and discuss open source data models for Ceramic applications that can be used with the IDX protocol, enabling cross-application data interoperability.

DataModels Registry

The DataModels Registry is an open-source, community-created repository of reusable application data models for Ceramic. It provides a single place where developers can openly register, discover, and reuse existing data models – the foundation for interoperable applications built on shared data models. Currently the registry is implemented as an open source Github repository dual-licensed under MIT and Apache 2. In the future, it will be decentralized on Ceramic.

Installation & Usage

All data models added to the registry are automatically published to npm under the @datamodels organization. Any developer can install one or more data models by using the @datamodels/model-name convention, making those models available for storing or retrieving data during runtime with any IDX client, including DID DataStore or Self.ID, both of which are described below.

DataModels Forum

The DataModels Forum is located within the DataModels Registry repository. Every model in the DataModel Registry has its own discussion thread which the community can use to leave reviews, have discussions, discuss upgrades, etc. Developers can also post ideas for DataModels to solicit input from the community prior to adding it to the Registry.

Glaze: Ceramic Developer Suite

Glaze is a suite of low-level developer tools for building applications on Ceramic. Glaze aims to provide developers with more flexibility and control by unbundling IDX.js into smaller, more specific packages.

DID DataStore.js: Core IDX client

DID DataStore is a bare-bones IDX runtime client – a stripped-down replacement for IDX.js – with less scope and fewer dependencies. If you've used IDX.js, you'll notice that DID DataStore does not come bundled with default data models, such as basicProfile, nor does it support CAIP-10 blockchain accounts links out of the box. If you want to use DID DataStore with data models or blockchain accounts you will need to separately install these as packages. If you want a more bundled, drop-in solution similar to IDX.js, see Self.ID below.

DataModels.js: A runtime for DataModels

DataModels.js is a runtime library that simplifies the process of using data models in your application. It is frequently used with DID DataStore, and supports a few useful runtime features for DataModels:

1. Availability – ensures the data models used your application are available on your Ceramic node, so they can be used with a Ceramic or IDX client, such as DID Datastore
2. Aliasing – allows developers to assign human-readable names to their data models, so they can more easily be used within application code instead of needing to reference data models via their streamID

If you've used IDX.js or the IDX CLI, DataModels serves as a replacement for running the idx bootstrap command or using similar bootstrap files, and eliminates the need to create aliases at runtime.

More Glaze Tools

• DevTools is a library for managing the full lifecycle of DataModels, such as creating custom models, adding to existing or creating new schemas and definitions, and publishing DataModels to any Ceramic node.
• Glaze CLI is an upgraded command line interface that is designed to support Glaze development flows. It provides more functionality than the core Ceramic CLI, and adds native functionality for DataModels and DID Datastore.

Self.ID: Bundles for building Ceramic apps

Self.ID is a suite of high-level bundles that simplify the development process for building Ceramic applications.

Self.ID Core: Read-only bundle

Self.ID Core is a package that comes pre-installed with everything you need to build applications that read data from accounts on Ceramic. Self.ID Core offers cross-platform support, but read-only APIs. If you are currently using IDX.js, Self.ID Core is very similar. It includes:

1. Ceramic client: JS Ceramic HTTP Client with all required setup. The only thing you need to do is pass in your node's CeramicURL
2. IDX client: DID DataStore.js
3. DataModels runtime: DataModels.js
4. Popular DataModels: Basic Profile, 3ID Keychain, Crypto Accounts, Web Accounts
5. Popular utilities: Support for CAIP-10 blockchain accounts

Self.ID Web: Browser bundle

Self.ID Web extends the Self.ID Core bundle with write APIs and provides built-in user authentication. Self.ID Web only supports browser applications, and is now the most popular bundle for building Ceramic applications.

1. 3ID Connect: Browser-based, cross-chain user authentication system that is compatible with blockchain wallets. 3ID Connect allows users to connect one or more blockchain accounts from any chain to their Ceramic decentralized identity. 3ID Connect supports all EVM chains and other blockchain platforms such as NEAR, Cosmos, Filecoin, and Tezos.

More Self.ID Tools

• Image Utilities: for uploading images to IPFS, storing them in multiple sizes and resolutions, and retrieving one most desirable to your application UI. Great for optimizing application performance when handling images such as user profiles photos. Sometimes you want to show a full-resolution image (e.g. profile page), and other times you only low-res thumbnails (e.g. dashboard).
• 3Box Legacy Profiles: A utility function for loading a legacy 3Box profile as a BasicProfile. Previously, this was provided by IDX.js.

FAQs

Which library should I use?

If you are already familiar with IDX or are more advanced in your development needs, DID DataStore is likely a better option; it offers the most flexibility and new features will be implemented there first. If you use custom data models (schemas and definitions), you should get familiar with the new set of tools for data models including the DataModels Registry and DataModels.js.

If you are just getting started with Ceramic and IDX, Self.ID should provide a smoother experience. It allows you to build Web3 data applications without having to understand all the concepts in order to have a working setup. You should still learn about Ceramic and the rest of the ecosystem if you want a better understanding of the full set of possibilities.

What should I do now?

IDX.js and related packages are being deprecated. Moving forward new features and improvements will only be implemented in the Glaze and Self.ID packages. We recommend that you update your app to use the new packages whenever possible.

Join the developer community

The Ceramic community is passionate about building Web3 data applications. If you want to learn more about anything mentioned in this post, encounter any issues trying to use the new tools, either for a new project or migrating from IDX.js, please join the Ceramic Discord!

To get started building Ceramic applications, head over to the developer documentation for Glaze and Self.ID now!