fix: fixes bug where nodes were connecting to clay

The changes to rust-ceramic that use the network to connect to a predefined set of nodes based on the network meant that the CAS ipfs node was connecting to clay and therefore the rest of the nodes would learn about and connect to clay. This changes is so that CAS uses a shared IPFS spec with Ceramic, thus fixing the bug that the CAS ipfs network was not being configured. This is a breaking change in that the pubsub_topic is no longer needed and the ipfs_resource_limits is removed from CAS as its not part of `ipfs` spec directly.
3box · Dec 7, 2023 · 75e3e37 · 75e3e37
1 parent 55060c5
commit 75e3e37
Show file tree

Hide file tree

Showing 12 changed files with 604 additions and 481 deletions.
diff --git a/Cargo.lock b/Cargo.lock
diff --git a/k8s/operator/manifests/operator.yaml b/k8s/operator/manifests/operator.yaml
@@ -102,7 +102,7 @@ spec:
       containers:
       - name: keramik-operator
         image: "keramik/operator"
-        imagePullPolicy: IfNotPresent # Should be IfNotPresent when using imageTag: dev, but Always if using imageTag: latest
+        imagePullPolicy: Always # Should be IfNotPresent when using imageTag: dev, but Always if using imageTag: latest
         command:
           - "/usr/bin/keramik-operator"
           - "daemon"

diff --git a/keramik/src/advanced_configuration.md b/keramik/src/advanced_configuration.md
@@ -7,18 +7,16 @@ You can configure the Ceramic nodes to use an external instance of the CAS inste
 CAS running in 3Box Labs infrastructure, you will also need to specify the Ceramic network type associated with the
 node, e.g. `dev-unstable`.
 
-In this case, the Ceramic network PubSub topic must be specified as an empty string in order to clear it from the
-Ceramic configuration. Ceramic nodes do not permit the PubSub topic to be specified for a network type that is not one
-of `local` or `inmemory`.
-
 You may also specify an Ethereum RPC endpoint for the Ceramic nodes to be able to verify anchors, or set it to an empty
 string to clear it from the Ceramic configuration. In the latter case, the Ceramic nodes will come up but will not be
 able to verify anchors.
 
-If left unspecified, `networkType` will default to `local`, `pubsubTopic` to `/ceramic/local-keramik`, `ethRpcUrl` to
-`http://ganache:8545`, and `casApiUrl` to `http://cas:8081`. These defaults point to an internal CAS using a local
+If left unspecified, `networkType` will default to `local`, `ethRpcUrl` to `http://ganache:8545`,
+and `casApiUrl` to `http://cas:8081`. These defaults point to an internal CAS using a local
 pubsub topic in a fully isolated network.
 
+Additionally IPFS can be [configured](./ipfs.md) with custom images and resources for both CAS and Ceramic.
+
 ```yaml
 # network configuration
 ---
@@ -30,12 +28,12 @@ spec:
   replicas: 2
   privateKeySecret: "small"
   networkType: "dev-unstable"
-  pubsubTopic: ""
   ethRpcUrl: ""
   casApiUrl: "https://some-anchor-service.com"
 ```
 
 # Adjusting Ceramic Environment
+
 Ceramic environment can be adjusted by specifying environment variables in the network configuration
 
 ```yaml
@@ -53,6 +51,7 @@ spec:
 ```
 
 # Disabling AWS Functionality
+
 Certain functionality in CAS depends on AWS services. If you are running Keramik in a non-AWS environment, you can
 disable this by editing the statefulset for CAS
 
@@ -70,6 +69,7 @@ and adding the following environment variables to the `spec/template/spec/contai
 *Note* statefulsets must be edited every time the network is recreated.
 
 # Image Resources
+
 You can also use the [network](./setup_network.md) specification to specify resources for the pods that are running
 
 ```yaml

diff --git a/keramik/src/deploy_images.md b/keramik/src/deploy_images.md
diff --git a/keramik/src/ipfs.md b/keramik/src/ipfs.md
@@ -1,10 +1,12 @@
 # IPFS
 
-The IPFS behavior used by Ceramic can be customized.
+The IPFS behavior used by CAS and Ceramic can be customized using the same IPFS spec.
 
 ## Rust IPFS
 
-Example [network config](./setup_network.md) that uses Rust based IPFS (i.e. ceramic-one) with its defaults.
+### Ceramic
+
+Example [network config](./setup_network.md) that uses Rust based IPFS (i.e. ceramic-one) with its defaults for Ceramic.
 
 ```yaml
 apiVersion: "keramik.3box.io/v1alpha1"
@@ -18,7 +20,7 @@ spec:
         rust: {}
 ```
 
-Example [network config](./setup_network.md) that uses Rust based IPFS (i.e. ceramic-one) with a specific image.
+Example [network config](./setup_network.md) that uses Rust based IPFS (i.e. ceramic-one) with a specific image for Ceramic.
 
 ```yaml
 apiVersion: "keramik.3box.io/v1alpha1"
@@ -34,9 +36,43 @@ spec:
          imagePullPolicy: IfNotPresent
 ```
 
+### CAS
+
+Example [network config](./setup_network.md) that uses Rust based IPFS (i.e. ceramic-one) with a specific image for CAS.
+
+```yaml
+apiVersion: "keramik.3box.io/v1alpha1"
+kind: Network
+metadata:
+  name: example-vanilla-ceramic-one
+spec:
+  replicas: 5
+  cas:
+    ipfs:
+      rust: {}
+```
+
+Example [network config](./setup_network.md) that uses Rust based IPFS (i.e. ceramic-one) with a specific image for CAS.
+
+```yaml
+apiVersion: "keramik.3box.io/v1alpha1"
+kind: Network
+metadata:
+  name: example-custom-ceramic-one
+spec:
+  replicas: 5
+  cas:
+    ipfs:
+     rust:
+       image: rust-ceramic/ceramic-one:dev
+       imagePullPolicy: IfNotPresent
+```
+
 ## Kubo IPFS
 
-Example [network config](./setup_network.md) that uses Go based IPFS (i.e. Kubo) with its defaults.
+### Ceramic
+
+Example [network config](./setup_network.md) that uses Go based IPFS (i.e. Kubo) with its defaults for Ceramic.
 
 ```yaml
 apiVersion: "keramik.3box.io/v1alpha1"
@@ -50,7 +86,7 @@ spec:
         go: {}
 ```
 
-Example [network config](./setup_network.md) that uses Go based IPFS (i.e. Kubo) with a specific image.
+Example [network config](./setup_network.md) that uses Go based IPFS (i.e. Kubo) with a specific image for Ceramic.
 
 ```yaml
 apiVersion: "keramik.3box.io/v1alpha1"
@@ -66,7 +102,7 @@ spec:
          imagePullPolicy: IfNotPresent
 ```
 
-Example [network config](./setup_network.md) that uses Go based IPFS (i.e. Kubo) with extra configuration commands.
+Example [network config](./setup_network.md) that uses Go based IPFS (i.e. Kubo) with extra configuration commands for Ceramic.
 
 ```yaml
 apiVersion: "keramik.3box.io/v1alpha1"
@@ -83,3 +119,53 @@ spec:
          commands:
            - ipfs config --json Swarm.RelayClient.Enabled false
 ```
+
+### CAS
+
+Example [network config](./setup_network.md) that uses Go based IPFS (i.e. Kubo) with its defaults for CAS.
+
+```yaml
+apiVersion: "keramik.3box.io/v1alpha1"
+kind: Network
+metadata:
+  name: example-vanilla-kubo
+spec:
+  replicas: 5
+  cas:
+    ipfs:
+      go: {}
+```
+
+Example [network config](./setup_network.md) that uses Go based IPFS (i.e. Kubo) with a specific image for CAS.
+
+```yaml
+apiVersion: "keramik.3box.io/v1alpha1"
+kind: Network
+metadata:
+  name: example-custom-kubo
+spec:
+  replicas: 5
+  cas:
+    ipfs:
+     go:
+       image: ceramicnetwork/go-ipfs-daemon:develop
+       imagePullPolicy: IfNotPresent
+```
+
+Example [network config](./setup_network.md) that uses Go based IPFS (i.e. Kubo) with extra configuration commands for CAS.
+
+```yaml
+apiVersion: "keramik.3box.io/v1alpha1"
+kind: Network
+metadata:
+  name: example-custom-kubo
+spec:
+  replicas: 5
+  cas:
+    ipfs:
+     go:
+       image: ceramicnetwork/go-ipfs-daemon:develop
+       imagePullPolicy: IfNotPresent
+       commands:
+         - ipfs config --json Swarm.RelayClient.Enabled false
+```
diff --git a/operator/src/network/cas.rs b/operator/src/network/cas.rs
@@ -16,23 +16,30 @@ use k8s_openapi::{
 };
 use kube::core::ObjectMeta;
 
-use crate::labels::{managed_labels, selector_labels};
-use crate::network::{resource_limits::ResourceLimitsConfig, CasSpec};
-
 use crate::network::{
+    ceramic::NetworkConfig,
     controller::{
         CAS_APP, CAS_IPFS_APP, CAS_IPFS_SERVICE_NAME, CAS_POSTGRES_APP, CAS_POSTGRES_SERVICE_NAME,
         CAS_SERVICE_NAME, GANACHE_APP, GANACHE_SERVICE_NAME, LOCALSTACK_APP,
         LOCALSTACK_SERVICE_NAME,
     },
     datadog::DataDogConfig,
+    ipfs::{IpfsConfig, IPFS_DATA_PV_CLAIM},
+    resource_limits::ResourceLimitsConfig,
+    CasSpec,
+};
+use crate::{
+    labels::{managed_labels, selector_labels},
+    network::ipfs::IpfsInfo,
 };
 
+const CAS_IPFS_INFO_SUFFIX: &str = "cas";
+
 pub struct CasConfig {
     pub image: String,
     pub image_pull_policy: String,
+    pub ipfs: IpfsConfig,
     pub cas_resource_limits: ResourceLimitsConfig,
-    pub ipfs_resource_limits: ResourceLimitsConfig,
     pub ganache_resource_limits: ResourceLimitsConfig,
     pub postgres_resource_limits: ResourceLimitsConfig,
     pub localstack_resource_limits: ResourceLimitsConfig,
@@ -49,11 +56,7 @@ impl Default for CasConfig {
                 memory: Quantity("1Gi".to_owned()),
                 storage: Quantity("1Gi".to_owned()),
             },
-            ipfs_resource_limits: ResourceLimitsConfig {
-                cpu: Quantity("250m".to_owned()),
-                memory: Quantity("512Mi".to_owned()),
-                storage: Quantity("1Gi".to_owned()),
-            },
+            ipfs: Default::default(),
             ganache_resource_limits: ResourceLimitsConfig {
                 cpu: Quantity("250m".to_owned()),
                 memory: Quantity("1Gi".to_owned()),
@@ -92,10 +95,7 @@ impl From<CasSpec> for CasConfig {
                 value.cas_resource_limits,
                 default.cas_resource_limits,
             ),
-            ipfs_resource_limits: ResourceLimitsConfig::from_spec(
-                value.ipfs_resource_limits,
-                default.ipfs_resource_limits,
-            ),
+            ipfs: value.ipfs.map(Into::into).unwrap_or(default.ipfs),
             ganache_resource_limits: ResourceLimitsConfig::from_spec(
                 value.ganache_resource_limits,
                 default.ganache_resource_limits,
@@ -112,6 +112,12 @@ impl From<CasSpec> for CasConfig {
     }
 }
 
+pub fn config_maps(config: impl Into<CasConfig>) -> BTreeMap<String, BTreeMap<String, String>> {
+    let config = config.into();
+    let ipfs_info = IpfsInfo::new(CAS_IPFS_INFO_SUFFIX.to_string());
+    config.ipfs.config_maps(ipfs_info)
+}
+
 // TODO make this a deployment
 pub fn cas_stateful_set_spec(
     ns: &str,
@@ -513,8 +519,24 @@ pub fn cas_service_spec() -> ServiceSpec {
     }
 }
 
-pub fn cas_ipfs_stateful_set_spec(config: impl Into<CasConfig>) -> StatefulSetSpec {
+pub fn cas_ipfs_stateful_set_spec(
+    config: impl Into<CasConfig>,
+    net_config: &NetworkConfig,
+) -> StatefulSetSpec {
     let config = config.into();
+
+    let mut volumes = vec![Volume {
+        name: IPFS_DATA_PV_CLAIM.to_owned(),
+        persistent_volume_claim: Some(PersistentVolumeClaimVolumeSource {
+            claim_name: IPFS_DATA_PV_CLAIM.to_owned(),
+            ..Default::default()
+        }),
+        ..Default::default()
+    }];
+
+    let ipfs_info = IpfsInfo::new(CAS_IPFS_INFO_SUFFIX.to_string());
+    volumes.append(&mut config.ipfs.volumes(ipfs_info.clone()));
+
     StatefulSetSpec {
         replicas: Some(1),
         selector: LabelSelector {
@@ -528,83 +550,14 @@ pub fn cas_ipfs_stateful_set_spec(config: impl Into<CasConfig>) -> StatefulSetSp
                 ..Default::default()
             }),
             spec: Some(PodSpec {
-                containers: vec![Container {
-                    command: Some(vec![
-                        "/usr/bin/ceramic-one".to_owned(),
-                        "daemon".to_owned(),
-                        "--store-dir".to_owned(),
-                        "/data/ipfs".to_owned(),
-                        "-b".to_owned(),
-                        "0.0.0.0:5001".to_owned(),
-                    ]),
-                    env: Some(vec![
-                        EnvVar {
-                            name: "CERAMIC_ONE_KADEMLIA_REPLICATION".to_owned(),
-                            value: Some("6".to_owned()),
-                            ..Default::default()
-                        },
-                        EnvVar {
-                            name: "CERAMIC_ONE_KADEMLIA_PARALLELISM".to_owned(),
-                            value: Some("1".to_owned()),
-                            ..Default::default()
-                        },
-                        EnvVar {
-                            name: "RUST_LOG".to_owned(),
-                            value: Some("info,ceramic_one=debug,quinn_proto=error".to_owned()),
-                            ..Default::default()
-                        },
-                    ]),
-                    image: Some("public.ecr.aws/r5b3e0r5/3box/ceramic-one".to_owned()),
-                    image_pull_policy: Some("Always".to_owned()),
-                    name: "ipfs".to_owned(),
-                    ports: Some(vec![
-                        ContainerPort {
-                            container_port: 4001,
-                            name: Some("swarm-tcp".to_owned()),
-                            ..Default::default()
-                        },
-                        ContainerPort {
-                            container_port: 5001,
-                            name: Some("api".to_owned()),
-                            ..Default::default()
-                        },
-                        ContainerPort {
-                            container_port: 8080,
-                            name: Some("gateway".to_owned()),
-                            ..Default::default()
-                        },
-                        ContainerPort {
-                            container_port: 9090,
-                            name: Some("metrics".to_owned()),
-                            ..Default::default()
-                        },
-                    ]),
-                    resources: Some(ResourceRequirements {
-                        limits: Some(config.ipfs_resource_limits.clone().into()),
-                        requests: Some(config.ipfs_resource_limits.into()),
-                        ..Default::default()
-                    }),
-                    volume_mounts: Some(vec![VolumeMount {
-                        mount_path: "/data/ipfs".to_owned(),
-                        name: "cas-ipfs-data".to_owned(),
-                        ..Default::default()
-                    }]),
-                    ..Default::default()
-                }],
-                volumes: Some(vec![Volume {
-                    name: "cas-ipfs-data".to_owned(),
-                    persistent_volume_claim: Some(PersistentVolumeClaimVolumeSource {
-                        claim_name: "cas-ipfs-data".to_owned(),
-                        ..Default::default()
-                    }),
-                    ..Default::default()
-                }]),
+                containers: vec![config.ipfs.container(ipfs_info, net_config)],
+                volumes: Some(volumes),
                 ..Default::default()
             }),
         },
         volume_claim_templates: Some(vec![PersistentVolumeClaim {
             metadata: ObjectMeta {
-                name: Some("cas-ipfs-data".to_owned()),
+                name: Some(IPFS_DATA_PV_CLAIM.to_owned()),
                 ..Default::default()
             },
             spec: Some(PersistentVolumeClaimSpec {