--- title: "Configuring the Calico Cloud CNI plugins" description: "Reference for configuring the CNI plugin in Calico Cloud connected clusters covering operator-managed and manifest-mode CNI options." product: "Calico Cloud" version: "v23.0.1" section: "Reference" canonical_url: "https://docs.tigera.io/calico-cloud/reference/component-resources/configuration" --- # Configuring the Calico Cloud CNI plugins **Tab: Operator** The Calico Cloud CNI plugins do not need to be configured directly when installed by the operator. For a complete operator configuration reference, see [the installation API reference documentation](https://docs.tigera.io/calico-cloud/reference/installation/api.md). **Tab: Manifest** The Calico Cloud CNI plugin is configured through the standard CNI [configuration mechanism](https://github.com/containernetworking/cni/blob/master/SPEC.md#network-configuration) A minimal configuration file that uses Calico Cloud for networking and IPAM looks like this ```json { "name": "any_name", "cniVersion": "0.1.0", "type": "calico", "ipam": { "type": "calico-ipam" } } ``` If the `cnx-node` container on a node registered with a `NODENAME` other than the node hostname, the CNI plugin on this node must be configured with the same `nodename`: ```json { "name": "any_name", "nodename": "", "type": "calico", "ipam": { "type": "calico-ipam" } } ``` Additional configuration can be added as detailed below. ## Generic ### Datastore type The Calico Cloud CNI plugin supports the following datastore: - `datastore_type` (kubernetes) ### Logging Logging is always to `stderr`. Logs are also written to `/var/log/calico/cni/cni.log` on each host by default. Logging can be configured using the following options in the netconf. | Option name | Default | Description | | -------------------- | ----------------------------- | --------------------------------------------------------------------------------------------------------- | | `log_level` | INFO | Logging level. Allowed levels are `ERROR`, `WARNING`, `INFO`, and `DEBUG`. | | `log_file_path` | `/var/log/calico/cni/cni.log` | Location on each host to write CNI log files to. Logging to file can be disabled by removing this option. | | `log_file_max_size` | 100 | Max file size in MB log files can reach before they are rotated. | | `log_file_max_age` | 30 | Max age in days that old log files will be kept on the host before they are removed. | | `log_file_max_count` | 10 | Max number of rotated log files allowed on the host before they are cleaned up. | ```json { "name": "any_name", "cniVersion": "0.1.0", "type": "calico", "log_level": "DEBUG", "log_file_path": "/var/log/calico/cni/cni.log", "ipam": { "type": "calico-ipam" } } ``` ### IPAM When using Calico Cloud IPAM, the following flags determine what IP addresses should be assigned. NOTE: These flags are strings and not boolean values. - `assign_ipv4` (default: `"true"`) - `assign_ipv6` (default: `"false"`) A specific IP address can be chosen by using [`CNI_ARGS`](https://github.com/appc/cni/blob/master/SPEC.md#parameters) and setting `IP` to the desired value. By default, Calico Cloud IPAM will assign IP addresses from all the available IP pools. Optionally, the list of possible IPv4 and IPv6 pools can also be specified via the following properties: - `ipv4_pools`: An array of CIDR strings or pool names. (e.g., `"ipv4_pools": ["10.0.0.0/24", "20.0.0.0/16", "default-ipv4-ippool"]`) - `ipv6_pools`: An array of CIDR strings or pool names. (e.g., `"ipv6_pools": ["2001:db8::1/120", "namedpool"]`) Example CNI config: ```json { "name": "any_name", "cniVersion": "0.1.0", "type": "calico", "ipam": { "type": "calico-ipam", "assign_ipv4": "true", "assign_ipv6": "true", "ipv4_pools": ["10.0.0.0/24", "20.0.0.0/16", "default-ipv4-ippool"], "ipv6_pools": ["2001:db8::1/120", "default-ipv6-ippool"] } } ``` > **SECONDARY:** `ipv6_pools` will be respected only when `assign_ipv6` is set to `"true"`. Any IP pools specified in the CNI config must have already been created. It is an error to specify IP pools in the config that do not exist. ### Container settings The following options allow configuration of settings within the container namespace. - allow\_ip\_forwarding (default is `false`) ```json { "name": "any_name", "cniVersion": "0.1.0", "type": "calico", "ipam": { "type": "calico-ipam" }, "container_settings": { "allow_ip_forwarding": true } } ``` ### Readiness Gates The following option makes CNI plugin wait for specified endpoint(s) to be ready before configuring pod networking. - `readiness_gates` This is an optional property that takes an array of URLs. Each URL specified will be polled for readiness and pod networking will continue startup once all readiness\_gates are ready. Example CNI config: ```json { "name": "any_name", "cniVersion": "0.1.0", "type": "calico", "ipam": { "type": "calico-ipam" }, "readiness_gates": ["http://localhost:9099/readiness", "http://localhost:8888/status"] } ``` ## Kubernetes specific When using the Calico Cloud CNI plugin with Kubernetes, the plugin must be able to access the Kubernetes API server to find the labels assigned to the Kubernetes pods. The recommended way to configure access is through a `kubeconfig` file specified in the `kubernetes` section of the network config. e.g. ```json { "name": "any_name", "cniVersion": "0.1.0", "type": "calico", "kubernetes": { "kubeconfig": "/path/to/kubeconfig" }, "ipam": { "type": "calico-ipam" } } ``` As a convenience, the API location can also be configured directly, e.g. ```json { "name": "any_name", "cniVersion": "0.1.0", "type": "calico", "kubernetes": { "k8s_api_root": "http://127.0.0.1:8080" }, "ipam": { "type": "calico-ipam" } } ``` ### Enabling Kubernetes policy If you wish to use the Kubernetes `NetworkPolicy` resource then you must set a policy type in the network config. There is a single supported policy type, `k8s`. When set, you must also run `tigera/kube-controllers` with the policy, profile, and workloadendpoint controllers enabled. ```json { "name": "any_name", "cniVersion": "0.1.0", "type": "calico", "policy": { "type": "k8s" }, "kubernetes": { "kubeconfig": "/path/to/kubeconfig" }, "ipam": { "type": "calico-ipam" } } ``` When using `type: k8s`, the Calico Cloud CNI plugin requires read-only Kubernetes API access to the `Pods` resource in all namespaces. ## IPAM ### Using host-local IPAM Calico can be configured to use [host-local IPAM](https://www.cni.dev/plugins/current/ipam/host-local/) instead of the default `calico-ipam`. Host local IPAM uses a pre-determined CIDR per-host, and stores allocations locally on each node. This is in contrast to Calico IPAM, which dynamically allocates blocks of addresses and single addresses alike in response to cluster needs. Host local IPAM is generally only used on clusters where integration with the Kubernetes [route controller](https://kubernetes.io/docs/concepts/architecture/cloud-controller/#route-controller) is necessary. Note that some Calico features - such as the ability to request a specific address or pool for a pod - require Calico IPAM to function, and will not work with host-local IPAM enabled. - Operator - Manifest The `host-local` IPAM plugin can be configured by setting the `Spec.CNI.IPAM.Plugin` field to `HostLocal` on the [operator.tigera.io/Installation](https://docs.tigera.io/calico-cloud/reference/installation/api.md#installation) API. Calico will use the `host-local` IPAM plugin to allocate IPv4 addresses from the node's IPv4 pod CIDR if there is an IPv4 pool configured in `Spec.IPPools`, and an IPv6 address from the node's IPv6 pod CIDR if there is an IPv6 pool configured in `Spec.IPPools`. The following example configures Calico to assign dual-stack IPs to pods using the host-local IPAM plugin. ```yaml kind: Installation apiVersion: operator.tigera.io/v1 metadata: name: default spec: calicoNetwork: ipPools: - cidr: 192.168.0.0/16 - cidr: 2001:db8::/64 cni: type: Calico ipam: type: HostLocal ``` When using the CNI `host-local` IPAM plugin, two special values - `usePodCidr` and `usePodCidrIPv6` - are allowed for the subnet field (either at the top-level, or in a "range"). This tells the plugin to determine the subnet to use from the Kubernetes API based on the Node.podCIDR field. Calico Cloud does not use the `gateway` field of a range so that field is not required and it will be ignored if present. > **SECONDARY:** `usePodCidr` and `usePodCidrIPv6` can only be used as the value of the `subnet` field, it cannot be used in `rangeStart` or `rangeEnd` so those values are not useful if `subnet` is set to `usePodCidr`. Calico Cloud supports the host-local IPAM plugin's `routes` field as follows: - If there is no `routes` field, Calico Cloud will install a default `0.0.0.0/0`, and/or `::/0` route into the pod (depending on whether the pod has an IPv4 and/or IPv6 address). - If there is a `routes` field then Calico Cloud will program *only* the routes in the routes field into the pod. Since Calico Cloud implements a point-to-point link into the pod, the `gw` field is not required and it will be ignored if present. All routes that Calico Cloud installs will have Calico Cloud's link-local IP as the next hop. Calico Cloud CNI plugin configuration: - `node_name` - The node name to use when looking up the CIDR value (defaults to current hostname) ```json { "name": "any_name", "cniVersion": "0.1.0", "type": "calico", "kubernetes": { "kubeconfig": "/path/to/kubeconfig", "node_name": "node-name-in-k8s" }, "ipam": { "type": "host-local", "ranges": [[{ "subnet": "usePodCidr" }], [{ "subnet": "usePodCidrIPv6" }]], "routes": [{ "dst": "0.0.0.0/0" }, { "dst": "2001:db8::/96" }] } } ``` When making use of the `usePodCidr` or `usePodCidrIPv6` options, the Calico Cloud CNI plugin requires read-only Kubernetes API access to the `Nodes` resource. #### Configuring node and typha When using `host-local` IPAM with the Kubernetes API datastore, you must configure both cnx-node and the Typha deployment to use the `Node.podCIDR` field by setting the environment variable `USE_POD_CIDR=true` in each. ### Using Kubernetes annotations #### Specifying IP pools on a per-namespace or per-pod basis In addition to specifying IP pools in the CNI config as discussed above, Calico Cloud IPAM supports specifying IP pools per-namespace or per-pod using the following [Kubernetes annotations](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/). - `cni.projectcalico.org/ipv4pools`: A list of configured IPv4 Pools from which to choose an address for the pod. Example: ```yaml annotations: 'cni.projectcalico.org/ipv4pools': '["default-ipv4-ippool"]' ``` - `cni.projectcalico.org/ipv6pools`: A list of configured IPv6 Pools from which to choose an address for the pod. Example: ```yaml annotations: 'cni.projectcalico.org/ipv6pools': '["2001:db8::1/120"]' ``` If provided, these IP pools will override any IP pools specified in the CNI config. > **SECONDARY:** This requires the IP pools to exist before `ipv4pools` or `ipv6pools` annotations are used. Requesting a subset of an IP pool is not supported. IP pools requested in the annotations must exactly match a configured [IPPool](https://docs.tigera.io/calico-cloud/reference/resources/ippool.md) resource. > **SECONDARY:** The Calico Cloud CNI plugin supports specifying an annotation per namespace. If both the namespace and the pod have this annotation, the pod information will be used. Otherwise, if only the namespace has the annotation the annotation of the namespace will be used for each pod in it. #### Requesting a specific IP address You can also request a specific IP address through [Kubernetes annotations](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/) with Calico Cloud IPAM. There are two annotations to request a specific IP address: - `cni.projectcalico.org/ipAddrs`: A list of IPv4 and/or IPv6 addresses to assign to the Pod. The requested IP addresses will be assigned from Calico Cloud IPAM and must exist within a configured IP pool. Example: ```yaml annotations: 'cni.projectcalico.org/ipAddrs': '["192.168.0.1"]' ``` - `cni.projectcalico.org/ipAddrsNoIpam`: A list of IPv4 and/or IPv6 addresses to assign to the Pod, bypassing IPAM. Any IP conflicts and routing have to be taken care of manually or by some other system. Calico Cloud will only distribute routes to a Pod if its IP address falls within a Calico Cloud IP pool using BGP mode. Calico will not distribute ipAddrsNoIpam routes when operating in VXLAN mode. If you assign an IP address that is not in a Calico Cloud IP pool or if its IP address falls within a Calico Cloud IP pool that uses VXLAN encapsulation, you must ensure that routing to that IP address is taken care of through another mechanism. Example: ```yaml annotations: 'cni.projectcalico.org/ipAddrsNoIpam': '["10.0.0.1"]' ``` The ipAddrsNoIpam feature is disabled by default. It can be enabled in the feature\_control section of the CNI network config: ```json { "name": "any_name", "cniVersion": "0.1.0", "type": "calico", "ipam": { "type": "calico-ipam" }, "feature_control": { "ip_addrs_no_ipam": true } } ``` > **WARNING:** This feature allows for the bypassing of network policy via IP spoofing. Users should make sure the proper admission control is in place to prevent users from selecting arbitrary IP addresses. > **SECONDARY:** > > - The `ipAddrs` and `ipAddrsNoIpam` annotations can't be used together. > - You can only specify one IPv4/IPv6 or one IPv4 and one IPv6 address with these annotations. > - When `ipAddrs` or `ipAddrsNoIpam` is used with `ipv4pools` or `ipv6pools`, `ipAddrs` / `ipAddrsNoIpam` take priority. #### Requesting a floating IP You can request a floating IP address for a pod through [Kubernetes annotations](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/) with Calico Cloud. > **SECONDARY:** The specified address must belong to an IP Pool for advertisement to work properly. - `cni.projectcalico.org/floatingIPs`: A list of floating IPs which will be assigned to the pod's workload endpoint. Example: ```yaml annotations: 'cni.projectcalico.org/floatingIPs': '["10.0.0.1"]' ``` The floatingIPs feature is disabled by default. It can be enabled in the feature\_control section of the CNI network config: ```json { "name": "any_name", "cniVersion": "0.1.0", "type": "calico", "ipam": { "type": "calico-ipam" }, "feature_control": { "floating_ips": true } } ``` > **WARNING:** This feature can allow pods to receive traffic which may not have been intended for that pod. Users should make sure the proper admission control is in place to prevent users from selecting arbitrary floating IP addresses. ### Using IP pools node selectors Nodes will only assign workload addresses from IP pools which select them. By default, IP pools select all nodes, but this can be configured using the `nodeSelector` field. Check out the [IP pool resource document](https://docs.tigera.io/calico-cloud/reference/resources/ippool.md) for more details. Example: 1. Create (or update) an IP pool that only allocates IPs for nodes where it contains a label `rack=0`. ```bash kubectl create -f -< **SECONDARY:** Be aware of the following [portmap plugin CNI issue](https://github.com/containernetworking/cni/issues/605) where draining nodes may take a long time with a cluster of 100+ nodes and 4000+ services. To disable it, remove the portmap section from the CNI network configuration in the Calico Cloud manifests. ```json { "type": "portmap", "snat": true, "capabilities": { "portMappings": true } } ``` **Traffic shaping plugin** The [traffic shaping Kubernetes CNI plugin](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/) supports pod ingress and egress traffic shaping. This bandwidth management technique delays the flow of certain types of network packets to ensure network performance for higher priority applications. It is enabled by default. You can add the `kubernetes.io/ingress-bandwidth` and `kubernetes.io/egress-bandwidth` annotations to your pod. For example, the following sets a 1 megabit-per-second connection for ingress and egress traffic. ```bash apiVersion: v1 kind: Pod metadata: annotations: kubernetes.io/ingress-bandwidth: 1M kubernetes.io/egress-bandwidth: 1M ... ``` To disable it, remove the bandwidth section from the CNI network configuration in the Calico Cloud manifests. ```json { "type": "bandwidth", "capabilities": { "bandwidth": true } } ``` ### Order of precedence If more than one of these methods are used for IP address assignment, they will take on the following precedence, 1 being the highest: 1. Kubernetes annotations 2. CNI configuration 3. IP pool node selectors > **SECONDARY:** Calico Cloud IPAM will not reassign IP addresses to workloads that are already running. To update running workloads with IP addresses from a newly configured IP pool, they must be recreated. We recommend doing this before going into production or during a maintenance window. ### Specify num\_queues for veth interfaces `num_rx_queues` and `num_tx_queues` can be set using the `num_queues` option in the CNI configuration. Default: 1 For example: ```json { "num_queues": 3 } ``` **Tab: Tab 3** The `host-local` IPAM plugin can be configured by setting the `Spec.CNI.IPAM.Plugin` field to `HostLocal` on the [operator.tigera.io/Installation](https://docs.tigera.io/calico-cloud/reference/installation/api.md#installation) API. Calico will use the `host-local` IPAM plugin to allocate IPv4 addresses from the node's IPv4 pod CIDR if there is an IPv4 pool configured in `Spec.IPPools`, and an IPv6 address from the node's IPv6 pod CIDR if there is an IPv6 pool configured in `Spec.IPPools`. The following example configures Calico to assign dual-stack IPs to pods using the host-local IPAM plugin. ```yaml kind: Installation apiVersion: operator.tigera.io/v1 metadata: name: default spec: calicoNetwork: ipPools: - cidr: 192.168.0.0/16 - cidr: 2001:db8::/64 cni: type: Calico ipam: type: HostLocal ``` **Tab: Tab 4** When using the CNI `host-local` IPAM plugin, two special values - `usePodCidr` and `usePodCidrIPv6` - are allowed for the subnet field (either at the top-level, or in a "range"). This tells the plugin to determine the subnet to use from the Kubernetes API based on the Node.podCIDR field. Calico Cloud does not use the `gateway` field of a range so that field is not required and it will be ignored if present. > **SECONDARY:** `usePodCidr` and `usePodCidrIPv6` can only be used as the value of the `subnet` field, it cannot be used in `rangeStart` or `rangeEnd` so those values are not useful if `subnet` is set to `usePodCidr`. Calico Cloud supports the host-local IPAM plugin's `routes` field as follows: - If there is no `routes` field, Calico Cloud will install a default `0.0.0.0/0`, and/or `::/0` route into the pod (depending on whether the pod has an IPv4 and/or IPv6 address). - If there is a `routes` field then Calico Cloud will program *only* the routes in the routes field into the pod. Since Calico Cloud implements a point-to-point link into the pod, the `gw` field is not required and it will be ignored if present. All routes that Calico Cloud installs will have Calico Cloud's link-local IP as the next hop. Calico Cloud CNI plugin configuration: - `node_name` - The node name to use when looking up the CIDR value (defaults to current hostname) ```json { "name": "any_name", "cniVersion": "0.1.0", "type": "calico", "kubernetes": { "kubeconfig": "/path/to/kubeconfig", "node_name": "node-name-in-k8s" }, "ipam": { "type": "host-local", "ranges": [[{ "subnet": "usePodCidr" }], [{ "subnet": "usePodCidrIPv6" }]], "routes": [{ "dst": "0.0.0.0/0" }, { "dst": "2001:db8::/96" }] } } ``` When making use of the `usePodCidr` or `usePodCidrIPv6` options, the Calico Cloud CNI plugin requires read-only Kubernetes API access to the `Nodes` resource. #### Configuring node and typha When using `host-local` IPAM with the Kubernetes API datastore, you must configure both cnx-node and the Typha deployment to use the `Node.podCIDR` field by setting the environment variable `USE_POD_CIDR=true` in each.