--- title: "Configure multiple Calico Cloud networks on a pod" description: "Add extra Calico Cloud networks to each pod in a connected cluster with the Multus-CNI plugin, then control access with tiered network policy on every interface." product: "Calico Cloud" version: "v23.0.1" section: "Networking" canonical_url: "https://docs.tigera.io/calico-cloud/networking/configuring/multiple-networks" --- # Configure multiple Calico Cloud networks on a pod ## Big picture Configure a Kubernetes cluster with multiple Calico Cloud networks on each pod, and enforce security using Calico Cloud tiered network policy. ## Value By default, you can configure only one CNI (network and pod interface) in a cluster. But many deployments require multiple networks (for example, one that is faster or more secure) for sending different types of data. Calico Cloud supports configuring additional Calico Cloud networks and interfaces in your pods using the Multus-CNI plugin. You can then use Calico Cloud tiered policy and other features to enforce security on all of your workload traffic. ## Concepts ### About the Multus-CNI plugin Calico Cloud uses the [Multus-CNI plugin](https://github.com/intel/multus-cni/) to create multiple Calico Cloud networks and multiple pod interfaces to access these networks. This extends the default network and pod interface that comes with the Calico CNI. You install Multus on a cluster, then simply enable Multus in the Calico Cloud Installation resource. Using the Multus **NetworkAttachmentDefinition**, you define the new networks and reference them as an annotation in the pod resource. ### Labels, workload endpoints, and policy When you set the `MultiInterfaceMode` field to `Multus` in the Installation resource, the following network and network interface labels are automatically added to new workload endpoints. - `projectcalico.org/network` - `projectcalico.org/network-namespace` - `projectcalico.org/network-interface` You can then create Calico Cloud policies using these label selectors to target specific networks or network interfaces. ### Limitations **Maximum additional networks per pod** You can define a maximum of nine additional Calico Cloud networks on a pod. If you add a network that exceeds the limit for the pod, networking is not configured and the pod fails to start with an associated error. **Calico Cloud features** Although the following Calico Cloud features are supported for your default Calico Cloud network, they are not supported at this time for additional networks/network interfaces using Multus: - Floating IPs - Specific IPs - Specifying IP pools on a per-namespace or per-pod basis - Egress gateways ## Before you begin... **Required** - Calico CNI > **SECONDARY:** Verify that you are using the Calico Cloud CNI. The CNI plugin used by Kubernetes for AKS, EKS, and GKE may be different, which means this feature will not work. - [Install Multus 3.0+ on your Kubernetes cluster](https://github.com/intel/multus-cni/) > **SECONDARY:** Multus is installed on OpenShift 4.0+ clusters. ## How to 1. [Configure cluster for multiple networks](#configure-cluster-for-multiple-networks) 2. [Create a new network](#create-a-new-network) 3. [Create a pod interface for the new network](#create-a-pod-interface-for-the-new-network) 4. [Configure the IP pool for the network](#configure-the-ip-pool-for-the-network) 5. [Enforce policy on the new network and pod interface](#enforce-policy-on-the-new-network-and-pod-interface) 6. [View workload endpoints](#view-workload-endpoints) ### Configure cluster for multiple networks In the [Installation custom resource](https://docs.tigera.io/calico-cloud/reference/installation/api.md#caliconetworkspec), set the `MultiInterfaceMode` to **Multus**. ### Create a new network Create a new network using the Multus **NetworkAttachmentDefinition**, and set the following required field to `"type":"calico"`. ```yaml apiVersion: 'k8s.cni.cncf.io/v1' kind: NetworkAttachmentDefinition metadata: name: additional-calico-network spec: config: '{ "cniVersion": "0.3.1", "type": "calico", "log_level": "info", "datastore_type": "kubernetes", "mtu": 1410, "nodename_file_optional": false, "ipam": { "type": "calico-ipam", "assign_ipv4" : "true", "assign_ipv6" : "false" }, "policy": { "type": "k8s" }, "kubernetes": { "kubeconfig": "/etc/cni/net.d/calico-kubeconfig" } }' ``` ### Create a pod interface for the new network Create a pod interface that specifies the new network using an annotation. In the following example, we create a pod with an additional pod interface named, `cali1`. The pod interface is attached to the network named, `additional-calico-network`, using the `k8s.v1.cni.cncf.io/networks` annotation. Note that all networks in `k8s.v1.cni.cncf.io/networks` are assumed to be Calico Cloud networks. ```yaml apiVersion: v1 kind: Pod metadata: name: multus-test-pod-1 namespace: default annotations: k8s.v1.cni.cncf.io/networks: additional-calico-network@cali1 spec: nodeSelector: kubernetes.io/os: linux containers: - name: multus-test command: ['/bin/sh', '-c', 'trap : TERM INT; sleep infinity & wait'] image: alpine ``` ### Configure the IP pool for the network Although not required, you may want to assign IPs from specific pools to specific network interfaces. If you are using the [Calico Cloud IPAM plugin](https://docs.tigera.io/calico-cloud/reference/component-resources/configuration.md#specifying-ip-pools-on-a-per-namespace-or-per-pod-basis), specify the IP pools in the **NetworkAttachmentDefinition** custom resource. For example: ```text "ipam": { "type": "calico-ipam", "assign_ipv4" : "true", "assign_ipv6" : "false" "ipv4_pools": ["10.0.0.0/24", "20.0.0.0/16", "default-ipv4-ippool"], }, ``` ### Enforce policy on the new network and pod interface When MultiInterfaceMode is set to Multus, WorkloadEndpoints are created with these labels: - `projectcalico.org/network` - `projectcalico.org/network-namespace` - `projectcalico.org/network-interface` You can use these labels to enforce policies on specific interfaces and networks using policy label selectors. > **SECONDARY:** Prior to Calico Cloud 3.0, if you were using Kubernetes datastore (kdd mode), the workload endpoint field and name suffix were always **eth0**. In 3.0, the value for workload labels may not be what you expect. Before creating policies targeting WorkloadEndpoints using the new labels, you should verify label values using the commands in [View workload endpoints](#view-workload-endpoints). In this policy example, we use the selector field to target all WorkloadEndpoints with the network interface of, `cali1`. ```yaml apiVersion: projectcalico.org/v3 kind: NetworkPolicy metadata: name: internal-access.allow-tcp-6379 namespace: production spec: tier: internal-access selector: projectcalico.org/network-interface == cali1 types: - Ingress - Egress ingress: - action: Allow metadata: annotations: from: frontend to: database protocol: TCP source: selector: role == 'frontend' destination: ports: - 6379 egress: - action: Allow ``` ### View workload endpoints **In the Calico Cloud web console**, go to the **WorkloadEndpoint** page to see all of the WorkloadEndpoints, including the network labels are for targeting WorkloadEndpoints with policy. **Using the CLI...** To view all WorkloadEndpoints for pods (default and new), use the following command. ```text MULTI_INTERFACE_MODE=multus calicoctl get workloadendpoints -o wide ``` ```text NAME WORKLOAD NODE NETWORKS INTERFACE PROFILES NATS test--bo--72vg--kadm--infra--0-k8s-multus--test--pod--1-eth0 multus-test-pod-1 bryan-bo-72vg-kadm-infra-0 192.168.53.129/32 calif887e436e8b kns.default,ksa.default.default test--bo--72vg--kadm--infra--0-k8s-multus--test--pod--1-net1 multus-test-pod-1 bryan-bo-72vg-kadm-infra-0 192.168.53.140/32 calim17CD6INXIX kns.default,ksa.default.default test--bo--72vg--kadm--infra--0-k8s-multus--test--pod--1-testiface multus-test-pod-1 bryan-bo-72vg-kadm-infra-0 192.168.53.142/32 calim27CD6INXIX kns.default,ksa.default.default test--bo--72vg--kadm--infra--0-k8s-multus--test--pod--1-net3 multus-test-pod-1 bryan-bo-72vg-kadm-infra-0 192.168.52.143/32 calim37CD6INXIX kns.default,ksa.default.default ``` To view specific WorkloadEndpoints, use the following command. ```text MULTI_INTERFACE_MODE=multus calicoctl get workloadendpoint test--bz--72vg--kadm--infra--0-k8s-multus--test--pod--1-net1 -o yaml ``` ```yaml apiVersion: projectcalico.org/v3 kind: WorkloadEndpoint metadata: creationTimestamp: '2020-05-04T22:23:05T' labels: projectcalico.org/namespace: default projectcalico.org/network: calico projectcalico.org/network-interface: net1 projectcalico.org/network-namespace: default projectcalico.org/orchestrator: k8s projectcalico.org/serviceaccount: default name: test--bz--72vg--kadm--infra--0-k8s-multus--test--pod--1-net1 namespace: default resourceVersion: '73572' uid: b9bb7482-cdb8-48d4-9ae5-58322d48391a spec: endpoint: net1 interfaceName: calim16CD6INXIX ipNetworks: - 192.168.52.141/32 node: bryan-bo-72vg-kadm-infra-0 orchestrator: k8s pod: multus-test-pod-1 profiles: - kns.default - ksa.default.default ```