Skip to main content

Running in the Cloud

Run a ComposeDB server in the cloud

Things to Know​

  • This guide is focused on running in the cloud using Docker and Kubernetes. For local deployment instructions check out Running Locally.
  • Interacting with ComposeDB requires running a Ceramic node as an interface for Ceramic applications, ceramic-one binary for data network access, and a Postgres DB. Each of these components should be running within a separate Docker container.
  • Docker images to run a Ceramic server are built from the js-ceramic repository. Images built from the main branch are tagged with latest, the git commit hash of the code from which the image was built, and the npm package version of the corresponding @ceramicnetwork/cli release.

Cloud Requirements​

Supported Operating Systems

  • Linux

Compute requirements

You’ll need sufficient compute resources to power Ceramic, IPFS, and Postgres. Below are the recommended requirements:

  • 4 vCPUs
  • 8GB RAM
note

If you are just getting started with a brand new project, you can start with a smaller instance and scale afterwards.

Running Ceramic server on Kubernetes​

You can run Ceramic Server on Kubernetes on the cloud, such as Google Kubernetes Engine or Amazon Elastic Kubernetes Service. You can also run Ceramic Server on DigitalOcean Kubernetes.

Running Kubernetes on the Cloud means a provider will manage the underlying infrastructure for you. You can also run Kubernetes on your own infrastructure, but that is outside the scope of this guide.

Running Ceramic server on DigitalOcean Kubernetes​

DigitalOcean Kubernetes (DOKS) allows developers to deploy Kubernetes clusters using simple managed service. Ceramic deployment on DigitalOcean Kubernetes will require 2 tools:

  • kubectl - the Kubernetes command line tool
  • doctl - the DigitalOcean command line tool

Make sure you have these tools installed on your machine before proceeding to the next step of this guide.

Creating a Kubernetes Cluster​

First, you will have to create your DigitalOcean Kubernetes cluster. To do that, follow an official DigitalOcean tutorial. The process of setting up your Kubernetes cluster will take about 10 minutes. Once it’s up and running, you are good to continue with the next step.

note

When it comes to choosing your cluster capacity, we recommend starting with the most cost-effective option - starting with the smallest cluster size and upgrading later. Also, keep in mind that Digital Ocean offers free credits for the new users to start building their projects.

Connecting to Kubernetes cluster​

First, you will have to configure the authentication of your cluster and retrieve the credentials. This can be achieved using the doctl command below and substituting the authentication number provided to you by Digital Ocean right after your cluster is launched:

doctl kubernetes cluster kubeconfig save 362dda8b-b555-4c47-9bf0-1a81cf58e0a8

After authenticating your cluster, it’s a good idea to verify the connectivity. This can be achieved using the following command which should list your cluster name, user and namespace:

kubectl config get-contexts

Deploy a Ceramic node​

In this section we will focus on deploying the Ceramic node on the DigitalOcean Kubernetes cluster.

  1. Clone the simpledeploy repository and enter the created directory:
git clone https://github.com/ceramicstudio/simpledeploy.git
cd simpledeploy/k8s/base/ceramic-one
  1. Run the following commands to deploy the stack:
# Create a namespace for the deployment
export CERAMIC_NAMESPACE=ceramic-one-0-17-0
kubectl create namespace ${CERAMIC_NAMESPACE}

# Create the necessary secrets
./scripts/create-secrets.sh

# Apply the deployment
kubectl apply -k .
  1. It will take a few minutes for the deployment to pull the docker images and start the containers. You can watch the process with the following command:
kubectl get pods --watch --namespace ceramic-one-0-17-0

You will know that your deployment is up and running when all of the processes have a status Running as follows:

NAME          READY   STATUS    RESTARTS   AGE
ceramic-one-0  1/1     Running   0           77s
ceramic-one-1  1/1     Running   0           77s
js-ceramic-0   1/1     Running   0           77s
js-ceramic-1   1/1     Running   0           77s
postgres-0     1/1     Running   0           77s

Hit ^C on your keyboard to exit this view.

note

You can easily access the logs of each of the containers by using the command below and configuring the container name. For example, to access the Ceramic node logs, you can run:

kubectl logs --follow --namespace ceramic-one-0-17-0 js-ceramic-0

Access the Ceramic node using the API​

You can use local port forwarding to access the Ceramic node from your local machine. Open a new terminal and run the command below. The port forward will stop when the command is exited so make sure to keep this command running for the rest of this guide.

kubectl port-forward --namespace ceramic-one-0-17-0 js-ceramic-0 7007:7007

Once you run the command you should see the following output in your terminal:

Forwarding from 127.0.0.1:7007 -> 7007
Forwarding from [::1]:7007 -> 7007
note

The Ceramic node must be ready to accept connections before you can access it. The pod's state must be Running and the READY column must be 1/1. You can check the status of the node by running the command below:

$ kubectl get pods --namespace ceramic-one-0-17-0 js-ceramic-0 NAME READY STATUS RESTARTS AGE js-ceramic-0 1/1 Running 1 (28h ago) 28h

To check the connection, open a new terminal and run the command below. A successful connection should utter a response Alive! as follows:

curl http://127.0.0.1:7007/api/v0/node/healthcheck

Expected output:

Alive!

Expose the node endpoint to the internet​

The last step is to expose your Ceramic node to the internet so that it’s accessible for your application. This can be done using a DigitalOcean Load Balancer which comes pre-configured for using using the SimpleDeploy scripts. You can get the EXTERNAM IP of your js-ceramic node (as well as ceramic-one) using the following command:

kubectl get svc --namespace ceramic-one-0-17-0 js-ceramic-lb-0

The result of this command will be an output similar to the one below. Keep in mind that might take a few minutes for the EXTERMAL-IP to be configured and change the status from <pending>:

NAME              TYPE           CLUSTER-IP      EXTERNAL-IP       PORT(S)          AGE
js-ceramic-lb-1   LoadBalancer   10.245.10.130   174.138.109.159   7007:31284/TCP   4m4s

This external IP address can now be used for accessing your node. To test it out, copy the external IP address provided above and substitute it in the following health check command:

curl http://174.138.109.159:7007/api/v0/node/healthcheck

Once again, a successful connection will provide an output Alive!:

Alive!

Optional - Add an SSL Cert and Domain Name​

If you wish to direct a domain to your ceramic node and acquire an SSL Certificate, you may follow the steps under cert-ingress to modify the kubernetes setup. Of course you may use other methods to add a domain name and certificate depending on what provider you wish to use.

Utilize the Deployed Assets with ComposeDB CLI and Graphiql Server​

Now that you have a Ceramic server deployed, you can utilize the ComposeDB Cli to create models and composites, as well as standing up a Graphiql server backed by the Ceramic with ComposeDB server.

First you will need to install ComposeDB Cli. Next you will need to setup, your environment to properly talk to your server

export CERAMIC_URL="http://"$(kubectl get service js-ceramic-lb-0 --namespace ceramic-one-0-17-0 -o json | jq -r '.status.loadBalancer.ingress[0].ip')":7007"
export DID_PRIVATE_KEY=$(kubectl get secrets --namespace ceramic-one-0-17-0 ceramic-admin -o json | jq -r '.data."private-key"' | base64 -d)

You can now follow the existing guides, adding --ceramic-url or --did-private-key to your composdb calls. For example:

composedb composite:from-model kjzl6hvfrbw6c5ajfmes842lu09vjxu5956e3xq0xk12gp2jcf9s90cagt2god9 --output=my-first-composite-single.json --ceramic-url=$CERAMIC_URL --did-private-key=$DID_PRIVATE_KEY

will create a new composite, utilizing your remote Ceramic server. You can also run Graphiql locally

composedb graphql:server --graphiql runtime-composite.json --port=5005 --did-private-key=$DID_PRIVATE_KEY

You can access the graphiql server at http://localhost:5005/graphql

Commonly asked questions​

Where is my data stored?​

Each part of the stack (js-ceramic, postgres) has its own Persistent Volume. You can view the volumes with the following command:

kubectl get PersistentVolumeClaim --namespace ceramic-one-0-17-0

This output includes identifiers for the volume on the cloud provider as well as the size and storage class, which defines the properties of the volume.

What is my admin DID and how do I use it to connect?​

The ceramic node is configured with an admin DID. This DID is used to authenticate with the ceramic node. The DID is derived from a seed, which is stored in a kubernetes secret named ceramic-admin and the private-key key's value is the base64 encoded seed.

While the example deployment creates a random seed for the admin DID, you can use your own seed by creating a secret with the same name and key instead of using the create-secrets.sh script.

Example:

$ kubectl create secret generic ceramic-admin --from-literal=private-key=<YOUR SEED>

To view the currently configured admin DID seed, you can use the following command (requires jq):

kubectl get secrets --namespace ceramic-one-0-17-0 ceramic-admin -o json | jq -r '.data."private-key"' | base64 -d

How do I connect to the Postgres database?​

You can create a session to the postgres database with the following command:

kubectl exec --namespace ceramic-one-0-17-0 -ti postgres-0 -- psql -U ceramic

A postgres service is also created and can be exposed locally with port-forwarding:

kubectl port-forward --namespace ceramic-one-0-17-0 svc/postgres 5432

The ceramic user password randomly generated during deployment. It is also available in the postgres-auth secret:

kubectl --namespace ceramic-one-0-17-0 get secrets postgres-auth -o yaml

Here you should get the following output:

apiVersion: v1
data:
  password: NzNjNzQ4ZDkxM2Y5NGQ2MmQwOTRiYzQ2YzIzMmM4YzdlYzFhODA2MA==
  username: Y2VyYW1pYw==
kind: Secret
...

How do I shut it all down?​

To remove the workload from the cluster, you can delete the namespace. For example:

kubectl delete namespace ceramic-one-0-17-0

Docker Hub​

You can find the ComposeDB server and IPFS Docker images on Docker Hub. Below, you can find examples of how you can run IPFS, Postgres and Ceramic processes using Docker.

Running Postgres​

An example below demonstrates how you can run a Postgres process. Make sure to update the variables to fit your use case:

docker pull postgres

docker run -d \
  -e POSTGRES_PASSWORD=mysecretpassword \
  -e PGDATA=/var/lib/postgresql/data/pgdata \
  -v /path_on_volume_for_postgres_data:/var/lib/postgresql/data \
  -p 5432:5432 \
  --name postgres \
  postgres

You can also follow the examples from the official Postgres Docker image documentation.

Running Ceramic​

An example below demonstrates how you can run a Ceramic server using Docker. Make sure to update the variables to fit your use case:

docker pull ceramicnetwork/js-ceramic:latest

docker run -d \
  -p 7007:7007 \
  -v /path_on_volume_for_daemon_config:/root/.ceramic/daemon.config.json \
  -v /path_on_volume_for_ceramic_logs:/root/.ceramic/logs \
  -v /path_on_volume_for_ceramic_statestore:/root/.ceramic/statestore \
  -e NODE_ENV=production \
  -e CERAMIC_INDEXING_DB_URI=postgres://username:password@host:5432/dbname \
  --name ceramic \
  js-ceramic --ipfs-api http://ipfs_ip_address:5101

Editing the daemon.config.json file​

To have the settings persist in your Ceramic node, edit the daemon.config.json file to include the configurations. The default location is ~/.ceramic/daemon.config.json. For a full file example, see the Ceramic docs.

...
    "ipfs": {
        "mode": "remote",
        "host": "http://ipfs_ip_address:5101"
    },
...
...
"indexing": {
    "db": "postgres://username:password@host:5432/dbname",
    "allow-queries-before-historical-sync": true,
    "enable-historical-sync": false
  }
...

Next Steps​