How to store signed and encrypted data on IPFS

Storing authenticated and encrypted data on IPFS is a core building block for many Web3 applications, but to date there has not been a standardized way to encode this type of data.

Without a standard, many developers have been forced to create custom formats for their signed and encrypted data. This has been prohibitive to the openness and interoperability of information stored in IPFS by siloing data to their particular implementation. Another approach to authenticated data has been to put data in IPFS and put the CID of the data in a smart contract on a blockchain, such as Ethereum. This is essentially an expensive way of adding a signature on top of the data and persisting the signature record on the blockchain.

With the introduction of EIP-2844, a standard that allows wallets to support a few new methods for signing and decrypting data based on DIDs and the dag-jose IPLD codec, we can now simply put authenticated and encrypted data directly into IPFS. In this tutorial, you will learn how you can utilize these primitives with two libraries, js-did and 3ID Connect!

What are DIDs and JOSE?

DIDs is the W3C standard for Decentralized Identifiers. It specifies a general way of going from a string identifier, e.g. did:3:bafy..., to a DID document which contains public keys for signature verification and key exchange. In most DID methods the document can be updated when keys are rotated for security reasons.

JOSE is a standard from IETF which stands for JSON Object Signing and Encryption, and that pretty much explains what it is. There are two main primitives in this standard: JWS (JSON Web Signatures) and JWE (JSON Web Encryption). Both of these formats allow for multiple participants: in JWS there can be one or multiple signatures over the payload, and in JWE there might be one or multiple recipients for the encrypted cleartext.

Building with dag-jose and EIP2844

As we have been building out Ceramic with dag-jose and EIP-2844 as basic building blocks, we've created a few lower-level tools which allow us to more easily use these technologies. This tutorial will show you how to use these powerful tools directly.

js-3id-did-provider is an implementation of EIP-2844 using 3ID as the DID method. It can be used standalone as a DID Provider, or more conveniently within the 3ID Connect library. 3ID Connect allows users to use their Ethereum wallet (support for more blockchains coming soon) to get access to a DID Provider.

key-did-provider-ed25519 is an implementation of EIP-2844 using the Key DID method. It's the most simple DID Provider that supports both signing and encryption.

js-did is a library that allows developers to represent a user in the form of a DID. This is the main interface we're going to be looking at in this tutorial. It allows us to sign data with the currently authenticated user, encrypt data to any user (DID), and decrypt data with the currently authenticated user.

Signed data in IPFS

By using the dag-jose IPLD codec we can create data structures that are linked and signed. This is done by creating JSON Web Signatures (JWS) that contain a link to additional data. One of the main problems that the dag-jose codec solves is that the payload of a JWS is traditionally encoded as base64url which means that if it contains any IPLD links you can't traverse those links. Instead what we do with DagJWS is enforce the payload to be the bytes of a CID. The codec then transforms the payload into a CID instance and sets it to the link property of the DagJWS. This allows us to easily traverse the resulting DAG.

Setup your environment

This section will cover how to set up some specific dependencies needed for this tutorial. If you just want to skip the setup part we have prepared a simple playground which bundles ipfs, 3id-connect, and dids. You can use it by opening the web page, clicking on connect, then opening the developer console where you can run the commands. If you decide to do so, skip the following two sections.

Setup IPFS with dag-jose support

The new dag-jose IPLD codec is now included in js-ipfs by default. Simply create an instance of IPFS as described below.

import {create as createIPFS} from 'ipfs-core'

const ipfs = await createIPFS()

Setup a DID instance

In the example setup below we use the key-did-provider-ed25519. If you choose to use the web playground from above you will be using 3ID Connect and js-3id-did-provider in the background.

import { DID } from 'dids'
import { Ed25519Provider } from 'key-did-provider-ed25519'
import KeyResolver from '@ceramicnetwork/key-did-resolver'
import { randomBytes } from '@stablelib/random'

// generate a seed, used as a secret for the DID
const seed = randomBytes(32)

// create did instance
const provider = new Ed25519Provider(seed)
const did = new DID({ provider, resolver: KeyResolver.getResolver() })
await did.authenticate()
window.did = did
console.log('Connected with DID:', did.id)

Create a signed data structure

We can now start signing and adding data to IPFS! First lets create a simple function that takes a payload, signs it using the did.createDagJWS method, and adds the resulting data to IPFS. As we can see in the code below we get two objects back from this method: jws which is the DagJWS itself and linkedBlock which is the raw bytes of the encoded payload. What happens in the background is that the payload gets encoded using dag-cbor, after this the CID of the encoded payload is used as the payload of the created jws. We can access this payload CID on the DagJWS instance as jws.link.

async function addSignedObject(payload) {
  // sign the payload as dag-jose
  const { jws, linkedBlock } = await did.createDagJWS(payload)
  
  // put the JWS into the ipfs dag
  const jwsCid = await ipfs.dag.put(jws, { storeCodec: dagJose.name, hashAlg: 'sha2-256' })
  
  // put the payload into the ipfs dag
  await ipfs.block.put(linkedBlock, jws.link)
    
  return jwsCid
}

Using this function, let's create our first signed data objects:

// Create our first signed object
const cid1 = await addSignedObject({ hello: 'world' })

// Log the DagJWS:
console.log((await ipfs.dag.get(cid1)).value)
// > {
// >   payload: "AXESIHhRlyKdyLsRUpRdpY4jSPfiee7e0GzCynNtDoeYWLUB",
// >   signatures: [{
// >     signature: "h7bHmTaBGza_QlFRI9LBfgB3Nw0m7hLzwMm4nLvcR3n9sHKRoCrY0soWnDbmuG7jfVgx4rYkjJohDuMNgbTpEQ",
// >     protected: "eyJraWQiOiJkaWQ6MzpiYWdjcWNlcmFza3hxeng0N2l2b2tqcW9md295dXliMjN0aWFlcGRyYXpxNXJsem4yaHg3a215YWN6d29hP3ZlcnNpb24taWQ9MCNrV01YTU1xazVXc290UW0iLCJhbGciOiJFUzI1NksifQ"
// >   }],
// >   link: CID(bafyreidykglsfhoixmivffc5uwhcgshx4j465xwqntbmu43nb2dzqwfvae)
// > }

// Log the payload:
await ipfs.dag.get(cid1, { path: '/link' }).then(b => console.log(b.value))
// > { hello: 'world' }

// Create another signed object that links to the previous one
const cid2 = await addSignedObject({ hello: 'getting the hang of this', prev: cid1 })

// Log the new payload:
await ipfs.dag.get(cid2, { path: '/link' }).then(b => console.log(b.value))
// > {
// >   hello: 'getting the hang of this'
// >   prev: CID(bagcqcerappi42sb4uyrjkhhakqvkiaibkl4pfnwpyt53xkmsbkns4y33ljzq)
// > }

// Log the old payload:
await ipfs.dag.get(cid2, { path: '/link/prev/link' }).then(b => console.log(b.value))
// > { hello: 'world' }

Note that the values of the CIDs and JWS will be different for you since the payload will be signed by your DID.

Verify a signed data structure

Verifying a JWS is very straight forward. Simply retrieve the JWS object and pass it to the verifyJWS method. If the signature is invalid, this function will throw an error. If the signature is valid, it will will return the DID (with key fragment) that was used to sign the JWS.

const jws1 = await ipfs.dag.get(cid1)
const jws2 = await ipfs.dag.get(cid2)

const signingDID1 = await did.verifyJWS(jws1.value)
await did.verifyJWS(jws2.value)

Encrypted data in IPFS

Signed data in IPFS is one piece of the puzzle, but perhaps more interesting is encrypted data. With the use of dag-jose and EIP-2844 we can encrypt data to one or multiple DIDs and store it directly in IPFS. Below we demonstrate how to use the convenient tools provided by the js-did library to do this.

Encrypt IPLD data

There is a simple method to create a DagJWE object which is encrypted to one or multiple DIDs, createDagJWE. This method accepts an IPLD object (a JSON object that may also include CID links) and an array of DIDs. It will resolve the DIDs to retrieve the public encryption keys found in their DID document and create a JWE that is encrypted to these keys. To get going, let's create a helper function that creates a JWE and puts it into IPFS.

async function addEncryptedObject(cleartext, dids) {
    const jwe = await did.createDagJWE(cleartext, dids)
    return ipfs.dag.put(jwe, { storeCodec: 'dag-jose', hashAlg: 'sha2-256' })
}

Once we have this function we can create a few encrypted objects. In the example below we first create a simple encrypted object, then we create an additional encrypted object that links to the previous one.

const cid3 = await addEncryptedObject({ hello: 'secret' }, [did.id])

const cid4 = await addEncryptedObject({ hello: 'cool!', prev: cid3 }, [did.id])

Note that in the example above we use [did.id](<http://did.id>) to encrypt the data to the currently authenticated DID. We can of course also encrypt the data to the DID of a user that is not locally authenticated, such as another user!

Decrypt IPLD data

When the data is retrieved from IPFS we will get the encrypted JWE. This means that we need to decrypt the data after we fetch it. Since we have created objects that link to each other, let's create a function that retrieves these objects and decrypts them recursively.

async function followSecretPath(cid) {
    const jwe = (await ipfs.dag.get(cid)).value
    const cleartext = await did.decryptDagJWE(jwe)
    console.log(cleartext)
    if (cleartext.prev) {
        await followSecretPath(cleartext.prev)
    }
}

The function above is a toy example that just logs the decrypted objects. We can use it to look at the content of these objects.

// Retrieve a single object
await followSecretPath(cid3)
// > { hello: 'secret' }

// Retrive multiple linked objects
await followSecretPath(cid4)
// > { hello: 'cool!', prev: CID(bagcqceraqittnizulygv6qldqgezp3siy2o5vpg66n7wms3vhffvyc7pu7ba) }
// > { hello: 'secret' }

That’s it for this tutorial! Hope you enjoyed diving deep into IPFS-enabled authenticated and encrypted data. I'm really looking forward to what you build on top of these powerful standards.

If you want to go deeper into how we are using these technologies in Ceramic, have a look at the js-ceramic Github repo and jump into our Discord.