--- title: "Enabling the eBPF data plane" description: "Switch a running Calico Open Source cluster to the eBPF data plane through automatic Tigera Operator detection on kubeadm clusters or a manual configuration path." product: "Calico Open Source" version: "3.32 (latest)" section: "Installing and upgrading" canonical_url: "https://docs.tigera.io/calico/latest/operations/ebpf/enabling-ebpf" --- # Enabling the eBPF data plane This page shows you how to enable the eBPF data plane on an existing cluster. For self-managed clusters that use `kubeadm` or similar tools to create the cluster, you can configure the Tigera Operator to enable eBPF automatically. For all other clusters, there is a manual configuration process to enable eBPF. ## Enable the eBPF data plane automatically for self-managed clusters You can quickly enable the eBPF data plane by configuring Tigera Operator. This is the recommended approach for most self-managed clusters that use `kubeadm` or similar tools to create the cluster. ***Prerequisites*** - Your cluster was created using `kubeadm` or `kubeadm`-based tools. - Calico Open Source was installed on your cluster using the Tigera Operator. - Your cluster has `kube-proxy` running in the `kube-system` namespace. - `kube-proxy` is not managed by an automated tool, such as Helm or ArgoCD. - The Tigera Operator can access `kubernetes` service and endpoints. ***Procedure*** - To enable eBPF mode with the automatic configuration, set the `spec.calicoNetwork.bpfNetworkBootstrap` and `spec.calicoNetwork.kubeProxyManagement` parameters in the operator's `Installation` resource to "Enabled", and the `spec.calicoNetwork.linuxDataplane` parameter to "BPF". ```bash kubectl patch installation.operator.tigera.io default --type merge -p '{"spec":{"calicoNetwork":{"linuxDataplane":"BPF", "bpfNetworkBootstrap":"Enabled", "kubeProxyManagement":"Enabled"}}}' ``` > **SECONDARY:** The operator rolls out the change with a rolling update (non-disruptive) and then swiftly transitions all nodes t o eBPF mode. However, it's inevitable that some nodes will enter eBPF mode before others. This can disrupt the flow of traffic through node ports. After this change, the operator will configure the API Server addresses and disable `kube-proxy`. ### Next steps - For best performance, configure your cluster to use [direct server return mode](#enable-direct-server-return-mode). ## Enable the eBPF data plane manually on any cluster This section explains how to enable the eBPF data plane on all compatible clusters. ### Before you begin #### Supported - x86-64 - arm64 (little endian) - Kubernetes datastore driver. - Distributions: - Generic or kubeadm - kOps - OpenShift - EKS - AKS with limitations: - [AKS with Azure CNI and Calico network policy](https://docs.tigera.io/calico/latest/getting-started/kubernetes/managed-public-cloud/aks.md) works, but it is not possible to disable kube-proxy resulting in wasted resources and suboptimal performance. - [AKS with Calico networking](https://docs.tigera.io/calico/latest/getting-started/kubernetes/managed-public-cloud/aks.md#install-aks-with-self-managed-calico-for-networking-and-network-policy) is in testing with the eBPF data plane. This should be a better solution overall but, at time of writing, the testing was not complete. - RKE (RKE2 recommended because it supports disabling `kube-proxy`) - MKE - Linux distribution/kernel: - Ubuntu 22.04. - Red Hat v8.4 with Linux kernel v4.18.0-305 or above (Red Hat have backported the required features to that build). - Another [supported distribution](https://docs.tigera.io/calico/latest/getting-started/kubernetes/requirements.md) with Linux kernel v5.10 or above. - An underlying network fabric that allows VXLAN traffic between hosts. In eBPF mode, VXLAN is used to forward Kubernetes NodePort traffic. - IPv6 Limitations: - IPIP is not supported (Calico iptables does not support it either). VXLAN is the recommended overlay for eBPF mode. To enable IPv6 in eBPF mode, see [Configure dual stack or IPv6 only](https://docs.tigera.io/calico/latest/networking/ipam/ipv6.md). You may be able to run with non-Calico IPAM. eks-cni is known to work. #### Not supported - Other processor architectures. - etcd datastore driver. The etcd datastore driver doesn't support watching Kubernetes services, which is required for some features in eBPF mode. - Distributions: - GKE. This is because of an incompatibility with the GKE CNI plugin. - Clusters with some eBPF nodes and some standard data plane and/or Windows nodes. - Floating IPs. - SCTP (either for policy or services). This is due to lack of kernel support for the SCTP checksum in BPF. - VLAN-based traffic without hardware offloading. > **SUCCESS:** You can use a VLAN device to connect a node to the cluster by using the `bpfDataIfacePattern` Felix configuration variable if the underlying/physical device supports VLAN offloading. For more information, see [Debugging Connectivity in Calico eBPF](https://www.tigera.io/blog/deep-dive/debugging-connectivity-in-calico-ebpf-the-mysterious-bpfdataifaceregexp-co/). #### Verify that your cluster is ready for eBPF mode This section explains how to make sure your cluster is suitable for eBPF mode. To check that the kernel on a node is suitable, you can run ```bash uname -rv ``` The output should look like this: ```text 5.10.0-26-generic #28~20.04.1-Ubuntu SMP Fri Jan 27 14:30:10 UTC 2023 ``` In this case the kernel version is v5.10, which is suitable. On Red Hat-derived distributions, you may see something like this: ```text 4.18.0-305.el8.x86_64 (mockbuild@x86-vm-08.build.eng.bos.redhat.com) ``` Since the Red Hat kernel is v4.18 with at least build number 305 (RHEL 8.4), this kernel is suitable. #### Performance For best pod-to-pod performance, we recommend using an underlying network that doesn't require Calico to use an overlay. For example: - A cluster within a single AWS subnet. - A cluster using a compatible cloud provider's CNI (such as the AWS VPC CNI plugin). - An on-prem cluster with BGP peering configured. If you must use an overlay, we recommend that you use VXLAN, not IPIP. VXLAN has much better performance than IPIP in eBPF mode due to various kernel optimisations. ### Configure Calico to talk directly to the API server In eBPF mode, Calico implements Kubernetes service networking directly (rather than relying on `kube-proxy`). Of course, this makes it highly desirable to disable `kube-proxy` when running in eBPF mode to save resources and avoid confusion over which component is handling services. To be able to disable `kube-proxy`, Calico needs to communicate to the API server *directly* rather than going through `kube-proxy`. To make *that* possible, we need to find a persistent, static way to reach the API server. The best way to do that varies by Kubernetes distribution: - If you created a cluster manually (for example by using `kubeadm`) then the right address to use depends on whether you opted for a high-availability cluster with multiple API servers or a simple one-node API server. - If you opted to set up a high availability cluster then you should use the address of the load balancer that you used in front of your API servers. As noted in the Kubernetes documentation, a load balancer is required for a HA set-up but the precise type of load balancer is not specified. - If you opted for a single control plane node then you can use the address of the control plane node itself. However, it's important that you use a *stable* address for that node such as a dedicated DNS record, or a static IP address. If you use a dynamic IP address (such as an EC2 private IP) then the address may change when the node is restarted causing Calico to lose connectivity to the API server. - `kops` typically sets up a load balancer of some sort in front of the API server. You should use the FQDN and port of the API load balancer, for example `api.internal.` as the `KUBERNETES_SERVICE_HOST` below and 443 as the `KUBERNETES_SERVICE_PORT`. - OpenShift requires various DNS records to be created for the cluster; one of these is exactly what we need: `api-int..` should point to the API server or to the load balancer in front of the API server. Use that (filling in the `` and `` as appropriate for your cluster) for the `KUBERNETES_SERVICE_HOST` below. OpenShift uses 6443 for the `KUBERNETES_SERVICE_PORT`. - MKE runs a reverse proxy in each node that can be used to reach the API server. You should use `proxy.local` as the `KUBERNETES_SERVICE_HOST` and `6444` as the `KUBERNETES_SERVICE_PORT`. - For AKS and EKS clusters you should use the FQDN of the API server's load balancer. This can be found with ```text kubectl cluster-info ``` which gives output like the following: ```text Kubernetes master is running at https://60F939227672BC3D5A1B3EC9744B2B21.gr7.us-west-2.eks.amazonaws.com ... ``` In this example, you would use `60F939227672BC3D5A1B3EC9744B2B21.gr7.us-west-2.eks.amazonaws.com` for `KUBERNETES_SERVICE_HOST` and `443` for `KUBERNETES_SERVICE_PORT` when creating the config map. **The next step depends on whether you installed Calico using the operator, or a manifest:** **Tab: Operator** If you installed Calico using the operator, create the following config map in the `tigera-operator` namespace using the host and port determined above: ```yaml kind: ConfigMap apiVersion: v1 metadata: name: kubernetes-services-endpoint namespace: tigera-operator data: KUBERNETES_SERVICE_HOST: '' KUBERNETES_SERVICE_PORT: '' ``` The operator will pick up the change to the config map automatically and do a rolling update of Calico to pass on the change. Confirm that pods restart and then reach the `Running` state with the following command: ```text watch kubectl get pods -n calico-system ``` If you do not see the pods restart then it's possible that the `ConfigMap` wasn't picked up (sometimes Kubernetes is slow to propagate `ConfigMap`s (see Kubernetes [issue #30189](https://github.com/kubernetes/kubernetes/issues/30189))). You can try restarting the operator. **Tab: Manifest** If you installed Calico using a manifest, create the following config map in the `kube-system` namespace using the host and port determined above: ```yaml kind: ConfigMap apiVersion: v1 metadata: name: kubernetes-services-endpoint namespace: kube-system data: KUBERNETES_SERVICE_HOST: '' KUBERNETES_SERVICE_PORT: '' ``` Wait 60s for kubelet to pick up the `ConfigMap` (see Kubernetes [issue #30189](https://github.com/kubernetes/kubernetes/issues/30189)); then, restart the Calico pods to pick up the change: ```text kubectl delete pod -n kube-system -l k8s-app=calico-node kubectl delete pod -n kube-system -l k8s-app=calico-kube-controllers ``` And, if using Typha: ```text kubectl delete pod -n kube-system -l k8s-app=calico-typha ``` Confirm that pods restart and then reach the `Running` state with the following command: ```text watch "kubectl get pods -n kube-system | grep calico" ``` You can verify that the change was picked up by checking the logs of one of the calico/node pods. ```text kubectl get po -n kube-system -l k8s-app=calico-node ``` Should show one or more pods: ```text NAME READY STATUS RESTARTS AGE calico-node-d6znw 1/1 Running 0 48m ... ``` Then, to search the logs, choose a pod and run: ```text kubectl logs -n kube-system | grep KUBERNETES_SERVICE_HOST ``` You should see the following log, with the correct `KUBERNETES_SERVICE_...` values. ```text 2020-08-26 12:26:29.025 [INFO][7] daemon.go 182: Kubernetes server override env vars. KUBERNETES_SERVICE_HOST="172.16.101.157" KUBERNETES_SERVICE_PORT="6443" ``` ### Configure kube-proxy In eBPF mode Calico replaces `kube-proxy` so it wastes resources (and reduces performance) to run both. This section explains how to disable `kube-proxy` in some common environments. > **WARNING:** If you are running `kube-proxy` in IPVS mode, you must switch it to iptables mode before disabling `kube-proxy` or enabling eBPF mode. This applies to all platforms and is required for a successful migration. After switching to iptables mode, restart your nodes before proceeding. ##### Clusters that run `kube-proxy` with a `DaemonSet` (such as `kubeadm`) For a cluster that runs `kube-proxy` in a `DaemonSet` (such as a `kubeadm`-created cluster), you can disable `kube-proxy` reversibly by adding a node selector to `kube-proxy`'s `DaemonSet` that matches no nodes, for example: ```text kubectl patch ds -n kube-system kube-proxy -p '{"spec":{"template":{"spec":{"nodeSelector":{"non-calico": "true"}}}}}' ``` Then, should you want to start `kube-proxy` again, you can simply remove the node selector. > **SECONDARY:** This approach is not suitable for AKS with Azure CNI since that platform makes use of the Kubernetes add-on manager. the change will be reverted by the system. For AKS, you should follow [Avoiding conflicts with kube-proxy](#avoiding-conflicts-with-kube-proxy) below. #### OpenShift If you are running OpenShift, you can disable `kube-proxy` as follows: ```text kubectl patch networks.operator.openshift.io cluster --type merge -p '{"spec":{"deployKubeProxy": false}}' ``` To re-enable it: ```text kubectl patch networks.operator.openshift.io cluster --type merge -p '{"spec":{"deployKubeProxy": true}}' ``` #### MKE If you are running MKE, you can disable `kube-proxy` as follows: Follow the step procedure in [Modify an existing MKE configuration](https://docs.mirantis.com/mke/current/ops/administer-cluster/configure-an-mke-cluster/use-an-mke-configuration-file.html#modify-an-existing-mke-configuration) to download, edit, and upload your MKE configuration. During the editing step, add the following configuration: `kube_proxy_mode=disabled` and `kube_default_drop_masq_bits=true`. #### Avoiding conflicts with kube-proxy If you cannot disable `kube-proxy` (for example, because it is managed by your Kubernetes distribution), then you *must* change Felix configuration parameter `BPFKubeProxyIptablesCleanupEnabled` to `false`. This can be done with `kubectl` as follows: ```text kubectl patch felixconfiguration default --patch='{"spec": {"bpfKubeProxyIptablesCleanupEnabled": false}}' ``` If both `kube-proxy` and `BPFKubeProxyIptablesCleanupEnabled` is enabled then `kube-proxy` will write its iptables rules and Felix will try to clean them up resulting in iptables flapping between the two. You should also set `bpfKubeProxyHealthzPort` to `0` to disable the health check server in Calico's BPF kube-proxy replacement, which by default binds to port 10256 and would conflict with the Kubernetes `kube-proxy` already running on the node. The Kubernetes `kube-proxy` can serve the health check equally well, so there is no degradation. ```text kubectl patch felixconfiguration default --patch='{"spec": {"bpfKubeProxyHealthzPort": 0}}' ``` ### MKE: Change the VXLAN port before enabling eBPF > **WARNING:** MKE uses Docker Swarm overlay networking, which creates VXLAN devices on UDP port 4789 inside Docker network namespaces. When Calico switches to eBPF mode on kernels where BTF is available (typically v5.8+), Felix creates the `vxlan.calico` device in flow mode, which acts as a catch-all on its UDP port. The kernel rejects this when another VXLAN device (Docker Swarm's) already holds the same port, causing `vxlan.calico` to stay DOWN with `address already in use` errors. > > **You must change the VXLAN port before enabling eBPF on MKE clusters:** > > ```bash > kubectl patch felixconfiguration default --type merge -p '{"spec":{"vxlanPort":4790}}' > ``` > > Wait for all calico-node pods to recreate the VXLAN device on the new port, then verify on each node: > > ```bash > kubectl exec -n calico-system -- ip -d link show vxlan.calico > ``` > > Confirm the device shows `dstport 4790` (or your chosen port) and is UP before proceeding. Ensure that the chosen UDP port is allowed by your underlying network between all nodes. ### Enable eBPF mode **The next step depends on whether you installed Calico using the operator, or a manifest:** **Tab: Operator** To enable eBPF mode, change the `spec.calicoNetwork.linuxDataplane` parameter in the operator's `Installation` resource to "BPF". ```bash kubectl patch installation.operator.tigera.io default --type merge -p '{"spec":{"calicoNetwork":{"linuxDataplane":"BPF"}}}' ``` > **SECONDARY:** The operator rolls out the change with a rolling update (non-disruptive) and then swiftly transitions all nodes to eBPF mode. However, it's inevitable that some nodes will enter eBPF mode before others. This can disrupt the flow of traffic through node ports. **Tab: Manifest** If you installed Calico using a manifest, change Felix configuration parameter `BPFEnabled` to `true`. This can be done with `calicoctl`, as follows: ```text calicoctl patch felixconfiguration default --patch='{"spec": {"bpfEnabled": true}}' ``` When enabling eBPF mode, preexisting connections continue to use the non-BPF datapath; such connections should not be disrupted, but they do not benefit from eBPF mode’s advantages. ### Next steps - For best performance, configure your cluster to use [direct server return mode](#enable-direct-server-return-mode). ## Enable direct server return mode Direct server return (DSR) mode skips a hop through the network for traffic to services (such as node ports) from outside the cluster. This reduces latency and CPU overhead but it requires the underlying network to allow nodes to send traffic with each other's IPs. In AWS, this requires all your nodes to be in the same subnet and for the source/dest check to be disabled. In GCP, the source/dest check should also be disabled, which can be done by enabling IP forwarding. DSR mode is disabled by default; to enable it, set the `BPFExternalServiceMode` Felix configuration parameter to `"DSR"`. This can be done with `calicoctl`: ```text calicoctl patch felixconfiguration default --patch='{"spec": {"bpfExternalServiceMode": "DSR"}}' ``` To switch back to tunneled mode, set the configuration parameter to `"Tunnel"`: ```text calicoctl patch felixconfiguration default --patch='{"spec": {"bpfExternalServiceMode": "Tunnel"}}' ``` Switching external traffic mode can disrupt in-progress connections. ## Reversing the process To revert to standard Linux networking: 1. (Depending on whether you installed Calico with the operator or with a manifest) reverse the changes to the operator's `Installation` or the `FelixConfiguration` resource: **Tab: Operator** ```bash kubectl patch installation.operator.tigera.io default --type merge -p '{"spec":{"calicoNetwork":{"linuxDataplane":"Iptables"}}}' ``` **Tab: Manifest** ```text calicoctl patch felixconfiguration default --patch='{"spec": {"bpfEnabled": false}}' ``` 2. If you disabled `kube-proxy` **manually**, re-enable it (for example, by removing the node selector added above). If you chose the **Auto config**, Operator will revert the changes and re-deploy `kube-proxy`. ```text kubectl patch ds -n kube-system kube-proxy --type merge -p '{"spec":{"template":{"spec":{"nodeSelector":{"non-calico": null}}}}}' ``` 3. If you are running MKE, follow the step procedure in [Modify an existing MKE configuration](https://docs.mirantis.com/mke/current/ops/administer-cluster/configure-an-mke-cluster/use-an-mke-configuration-file.html#modify-an-existing-mke-configuration) to download, edit, and upload your MKE configuration. During the editing step, add the following configuration: `kube_proxy_mode` to `iptables`. 4. Since disabling eBPF mode is disruptive to existing connections, monitor existing workloads to make sure they re-establish any connections that were disrupted by the switch.