How to store encrypted secrets using IDX

Paul Le Cam -

IDX library has been deprecated, for future interactions with the Identity Index protocol we've built DID-DataStore. For more details on how to store encrypted secrets on the Identity Index protocol, please take a look at our updated documentation here.

This tutorial will demonstrate how to allow users of your application or platform to encrypt secrets with a DID and store them in their IDX. Apps and libraries can use this pattern to easily store and recover seeds, secret keys, personal information, application data, and credentials that they may need for their own systems, simply based on the ability to authenticate a DID.

Because IDX allows any data to be associated to a DID in a decentralized manner, the secrets and information stored in IDX are secure, 100% owned by the user, and can easily be discovered and accessed by any application or platform allowed by the user. (Interop!) Under the hood, IDX uses Ceramic as a decentralized data storage protocol.

In this tutorial we will use the following technologies:

  • DIDs: a W3C standard for decentralized identifiers
  • JWE: an IETF standard for encrypted content
  • Ceramic HTTP client: a JS client for interacting with a remote Ceramic node over HTTP
  • IDX: a protocol for user-centric data management based on DIDs
  • Ed25519 Key DID provider: a library for using the Key DID method
  • uint8arrays: a utility library for converting between binary and text data

Environment setup

To get started using IDX, we'll first need to install the Ceramic and IDX CLIs using npm. You'll also need to have node installed.

npm install -g @ceramicnetwork/cli @ceramicstudio/idx-cli

Then we'll start the Ceramic daemon to run a local Ceramic node:

ceramic daemon

We will also need to make sure the Ceramic node supports IDX documents, using the following command:

idx bootstrap

Now let's create a DID and give it the label local so we can more easily reference it throughout the rest of this tutorial. We will use this DID for authentication when interacting with the CLI as a developer (not an end-user):

idx did:create --label=local

Data model

To store our encrypted secret using Ceramic and IDX, we will need two things:

  • A JSON schema for the encrypted payload (JWE)
  • An IDX definition to uniquely identify our encrypted secret in the user's IDX index

JWE schema

We will use the following JSON schema to validate that the encrypted payloads match the JWE specification when storing them on Ceramic:

{
  $schema: 'http://json-schema.org/draft-07/schema#',
  title: 'JWE',
  type: 'object',
  properties: {
    protected: { type: 'string' },
    iv: { type: 'string' },
    ciphertext: { type: 'string' },
    tag: { type: 'string' },
    aad: { type: 'string' },
    recipients: {
      type: 'array',
      items: {
        type: 'object',
        properties: {
          header: {
            type: 'object',
            properties: {
              alg: { type: 'string' },
              iv: { type: 'string' },
              tag: { type: 'string' },
              epk: { type: 'object' },
              kid: { type: 'string' },
            },
            required: ['alg', 'iv', 'tag'],
          },
          encrypted_key: { type: 'string' },
        },
        required: ['header', 'encrypted_key'],
      },
    },
  },
  required: ['protected', 'iv', 'ciphertext', 'tag'],
}

Using the DID previously created, we can publish the schema to the local Ceramic node. Use the idx schema:publish command as shown below:

idx schema:publish local '{"$schema":"http://json-schema.org/draft-07/schema#","title":"JWE","type":"object","properties":{"protected":{"type":"string"},"iv":{"type":"string"},"ciphertext":{"type":"string"},"tag":{"type":"string"},"aad":{"type":"string"},"recipients":{"type":"array","items":{"type":"object","properties":{"header":{"type":"object","properties":{"alg":{"type":"string"},"iv":{"type":"string"},"tag":{"type":"string"},"epk":{"type":"object"},"kid":{"type":"string"}},"required":["alg","iv","tag"]},"encrypted_key":{"type":"string"}},"required":["header","encrypted_key"]}}},"required":["protected","iv","ciphertext","tag"]}'

Successfully running this command will display the schema URL, which will be used in the next step. In my case the URL is ceramic://k3y52l7qbv1fryd65pd3dyu9dim46vx39z4bgzrrligbi5sfd0e3mwump2p6pdn9c but when you run the command with your own DID, it will be different.

Encrypted secret definition

Now that we have a generic JWE schema, we need to create a unique definition for the secret data we want to store using the idx definition:create command. Multiple definitions can use the same schema, but each definition functions as a unique key in the user's IDX index.

idx definition:create local --schema=<schema URL from previous command> --name="my secret data" --description="encrypted payload"

Running this command with display the unique StreamID for the definition, which we will need in the following step. In my case this ID is kjzl6cwe1jw14adxazb6clhiiniffhkedz69mrkvi4i1miad62iwazktfiy8rgm but it will be different for you.

Using our definition

Let's create a simple script that uses our definition to check if the encrypted secret is present in a user's IDX index, or create and store a new one if needed.

Dependencies

We'll need to install the following dependencies from npm: Ceramic HTTP client, IDX, Ed25519 Key DID provider, Key DID resolver, DIDs, and uint8arrays.

npm install @ceramicnetwork/http-client @ceramicstudio/idx key-did-provider-ed25519 key-did-resolver dids uint8arrays

Now let's create a simple script in a secret.js file, importing these dependencies along with randomBytes from the crypto module:

const { randomBytes } = require('crypto')
const Ceramic = require('@ceramicnetwork/http-client').default
const { IDX } = require('@ceramicstudio/idx')
const { Ed25519Provider } = require('key-did-provider-ed25519')
const KeyDidResolver = require('key-did-resolver').default
const { DID } = require('dids')
const { fromString, toString } = require('uint8arrays')
secret.js file

Ceramic authentication

In order to allow a user to encrypt their secret and store it in their IDX index, we will need our Ceramic client to be authenticated. Here we will use the Ed25519Provider requiring a 32 bytes seed that we will either get from the environment, or by generating a random one:

function getSeed() {
  let seed
  if (process.env.SEED) {
    seed = fromString(process.env.SEED, 'base16')
    console.log('Using provided seed')
  } else {
    seed = new Uint8Array(randomBytes(32))
    console.log(`Created seed: ${toString(seed, 'base16')}`)
  }
  return seed
}
secret.js file

In order to access secret data already associated to a DID, we will need to use the same seed by setting the SEED=<base16-encoded string> environment variable, otherwise a new DID will be created.

Now let's use this getSeed() function to authenticate our Ceramic client:

async function getCeramic() {
  const ceramic = new Ceramic('http://localhost:7007')
  const resolver = KeyDidResolver.getResolver()
  const provider = new Ed25519Provider(getSeed())
  await ceramic.setDID(new DID({ resolver, provider }))
  await ceramic.did.authenticate()
  return ceramic
}
secret.js file

Encrypted secret interactions

We will now create two functions to interact with our encrypted secret: one to encrypt and store, and another one to load and decrypt, using the definition we created from the CLI. Note, make sure to replace kjzl6cwe1jw14adxazb6clhiiniffhkedz69mrkvi4i1miad62iwazktfiy8rgm in the code below with the definition ID you created:

const KEY = 'kjzl6cwe1jw14adxazb6clhiiniffhkedz69mrkvi4i1miad62iwazktfiy8rgm'

async function uploadSecret(idx, payload) {
  const jwe = await idx.did.createJWE(payload, [idx.did.id])
  await idx.set(KEY, jwe)
}

async function downloadSecret(idx) {
  const jwe = await idx.get(KEY)
  return jwe ? await idx.did.decryptJWE(jwe) : null
}
secret.js file

We provide our own DID idx.did.id when calling idx.did.createJWE() so that only the authenticated DID can decrypt the created JWE payload. It's not the case in this example, but if you would like to allow other DIDs to decrypt the payload you would include them here.

As an alternative to createJWE and decryptJWE, we could use the createDagJWE and decryptDagJWE methods to encrypt JSON data instead of binary as we are doing in this example.

Interaction logic

Finally, let's create a function to implement a simple flow of checking if the encrypted secret exists and display it, or create and store a new secret:

async function run() {
  const ceramic = await getCeramic()
  const idx = new IDX({ ceramic })
  console.log(`Connected with DID: ${idx.id}`)

  const existing = await downloadSecret(idx)
  if (existing == null) {
    const secret = new Uint8Array(randomBytes(32))
    await uploadSecret(idx, secret)
    console.log('Created secret:')
    console.log(secret)
  } else {
    console.log('Found existing secret:')
    console.log(existing)
  }
}

run().catch(console.error)
secret.js

Running our script

Let's run our script a first time, this will generate both a random seed for the DID and a random secret:

node ./secret.js

This will log the created seed, DID and created secret.

Now let's run the script again but providing the created seed as logged above:

SEED=<base16-encoded string> node ./secret.js

The script should log the same DID and decrypted secret.

Full script

Here is the full secret.js script for convenience, don't forget to change the KEY value with your own definition ID before running it:

const { randomBytes } = require('crypto')
const Ceramic = require('@ceramicnetwork/http-client').default
const { IDX } = require('@ceramicstudio/idx')
const { Ed25519Provider } = require('key-did-provider-ed25519')
const KeyDidResolver = require('key-did-resolver').default
const { DID } = require('dids')
const { fromString, toString } = require('uint8arrays')

function getSeed() {
  let seed
  if (process.env.SEED) {
    seed = fromString(process.env.SEED, 'base16')
    console.log('Using provided seed')
  } else {
    seed = new Uint8Array(randomBytes(32))
    console.log(`Created seed: ${toString(seed, 'base16')}`)
  }
  return seed
}

async function getCeramic() {
  const ceramic = new Ceramic('http://localhost:7007')
  const resolver = KeyDidResolver.getResolver()
  const provider = new Ed25519Provider(getSeed())
  await ceramic.setDID(new DID({ resolver, provider }))
  await ceramic.did.authenticate()
  return ceramic
}

const KEY = 'kjzl6cwe1jw14adxazb6clhiiniffhkedz69mrkvi4i1miad62iwazktfiy8rgm'

async function uploadSecret(idx, payload) {
  const jwe = await idx.did.createJWE(payload, [idx.did.id])
  await idx.set(KEY, jwe)
}

async function downloadSecret(idx) {
  const jwe = await idx.get(KEY)
  return jwe ? await idx.did.decryptJWE(jwe) : null
}

async function run() {
  const ceramic = await getCeramic()
  const idx = new IDX({ ceramic })
  console.log(`Connected with DID: ${idx.id}`)

  const existing = await downloadSecret(idx)
  if (existing == null) {
    const secret = new Uint8Array(randomBytes(32))
    await uploadSecret(idx, secret)
    console.log('Created secret:')
    console.log(secret)
  } else {
    console.log('Found existing secret:')
    console.log(existing)
  }
}

run().catch(console.error)
secret.js file

Use case: Encrypted seeds

There are many exciting use cases for storing and retrieving encrypted secrets using IDX. Let's walk through a very important one – storing encrypted seeds.

Many decentralized systems rely on public key cryptography for performing authenticated transactions, such as signing and encrypting information. In these systems, a private key (seed) is used to sign a message which can then be validated against the corresponding public key (identifier) belonging to the user. Public key cryptography allows the decentralized system to verify that an interaction has originated from a given user without requiring the use of servers. Most of these types of systems require users to manage a dedicated seed for their system, which means that as users use more systems, they have and ever-growing list of keys to manage.

This model poses many challenges to users and application developers. Here are some common questions that arise when using decentralized technologies dependent on seeds for authenticated transactions:

  • How should I securely store this seed to make sure only the intended user can access it?
  • How can an app or library check if an account (seed) has already been created for a given user?
  • How can my app make use of multiple decentralized technologies without requiring users to manage a different seed for each one?

As we have demonstrated throughout this tutorial, storing encrypted seeds directly with users on IDX using their DID can solve all of these challenges. And as the number of decentralized systems that users will need to interact with is only increasing, the need to simplify secret management in a platform-agnostic way is becoming ever more valuable.

What's next?

The examples in this tutorial were intentionally kept simple but more complex schemas could be created, for example, to store public data along with encrypted secrets to support more specific use cases.

Learn more about IDX by visiting the IDX website and documentation.

If you need any help or just want to say hi, join the IDX Discord channel!