--- title: "Install in eBPF mode" description: "Install Calico Enterprise with the eBPF data plane during initial cluster setup as an alternative to the iptables data plane." product: "Calico Enterprise" version: "3.23 (latest)" section: "Operations" canonical_url: "https://docs.tigera.io/calico-enterprise/latest/operations/ebpf/install" --- # Install in eBPF mode ## Big picture Install the eBPF data plane during the initial installation of Calico Enterprise. ## Value The eBPF data plane mode has several advantages over standard Linux networking pipeline mode: - It scales to higher throughput. - It uses less CPU per GBit. - It has native support for Kubernetes services (without needing kube-proxy) that: - Reduces first packet latency for packets to services. - Preserves external client source IP addresses all the way to the pod. - Supports DSR (Direct Server Return) for more efficient service routing. - Uses less CPU than kube-proxy to keep the data plane in sync. To learn more and see performance metrics from our test environment, see the blog, [Introducing the Calico eBPF data plane](https://www.projectcalico.org/introducing-the-calico-ebpf-dataplane/). ## Concepts ### eBPF eBPF (or "extended Berkeley Packet Filter"), is a technology that allows safe mini programs to be attached to various low-level hooks in the Linux kernel. eBPF has a wide variety of uses, including networking, security, and tracing. You’ll see a lot of non-networking projects leveraging eBPF, but for Calico Enterprise our focus is on networking, and in particular, pushing the networking capabilities of the latest Linux kernels to the limit. ## Before you begin **Supported architecture and versions** - x86-64 - arm64 (little-endian) - Linux distribution/kernel: - Ubuntu 22.04 or above. - Red Hat v8.4 with Linux kernel v4.18.0-305 or above (Red Hat have backported the required features to that build). - Another [supported distribution](https://docs.tigera.io/calico-enterprise/latest/getting-started/install-on-clusters/requirements.md) with Linux kernel v5.10 or above. - Immutable operating systems (e.g., Talos Linux): Ensure the `CgroupV2Path` in the `FelixConfiguration` CRD is set to a writable path. #### Kernel version requirements for eBPF features Some eBPF features require a higher kernel version than the base eBPF data plane. | Feature | Minimum kernel version | Details | | ------------------------------------------------------------------------------------------------------------- | ------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Base eBPF data plane | v5.10 (RHEL: v4.18.0-305) | CO-RE supported at this version for better performance | | Log rules in eBPF mode | v5.16 | Logs sent to trace pipe via `bpftool prog tracelog` | | [QoS bandwidth controls](https://docs.tigera.io/calico-enterprise/latest/networking/configuring/qos-controls.md) | v6.6 | Requires `tcx` support. Established connection limits not supported with eBPF | | [DNS policy inline mode](https://docs.tigera.io/calico-enterprise/latest/network-policy/domain-based-policy.md) | v5.17 (RHEL: v5.14) | `BPFDNSPolicyMode: Inline` parses DNS responses in eBPF before they reach the application. Only wildcard prefixes (`*.x.y.z`) supported. Falls back to `NoDelay` on older kernels | > **WARNING:** The minimum kernel version required for the eBPF data plane is v5.10, which includes support for [CO-RE (Compile Once - Run Everywhere)](https://docs.ebpf.io/concepts/core/). CO-RE significantly improves compatibility and performance across kernel versions. For access to all eBPF features, we recommend kernel v6.6 or above. - An underlying network fabric that allows VXLAN traffic between hosts. In eBPF mode, VXLAN is used to forward Kubernetes NodePort traffic. **Unsupported platforms** - GKE - TKG - RKE > **SECONDARY:** eBPF supports AKS with Calico CNI and Calico Enterprise network policy. However, with [AKS with Azure CNI and Calico Enterprise network policy](https://docs.tigera.io/calico-enterprise/latest/getting-started/install-on-clusters/aks.md#install-aks-with-azure-cni-networking), kube-proxy cannot be disabled so the performance benefits of eBPF are lost. However, there are other reasons to use eBPF other than performance gains, as described in [eBPF use cases](https://docs.tigera.io/calico-enterprise/latest/operations/ebpf/use-cases-ebpf.md). **Unsupported features** - Clusters with some eBPF nodes and some standard data plane and/or Windows nodes - Floating IPs - SCTP (either for policy or services) - Tagged VLAN devices - L7 logs - Application layer policies - Web application firewall (WAF) **Recommendations for performance** For best pod-to-pod performance, we recommend using an underlying network that doesn't require Calico Enterprise to use an overlay. For example: - A cluster within a single AWS subnet - A cluster using a compatible cloud provider's CNI (such as the AWS VPC CNI plugin) - An on-prem cluster with BGP peering configured If you must use an overlay, we recommend that you use VXLAN, not IPIP. VXLAN has better performance than IPIP in eBPF mode due to various kernel optimizations. ## How to Installing Calico Enterprise normally consists of the following steps, which are covered by the main installation guide: - Create a cluster suitable to run Calico Enterprise - Install the Tigera Operator, and the associated Custom Resource Definitions - Apply a set of Custom Resources to tell the operator what to install - Wait for the operator to provision all the associated resources and report back via its status resource To install directly in eBPF is very similar; this guide explains the differences: - [Create a cluster](#create-a-suitable-cluster) suitable to run Calico Enterprise **with the added requirement that the nodes must use a recent enough kernel**. - [**Create a config map with the "real" address of the API server.**](#create-kubernetes-service-endpoint-config-map) This allows the operator to install Calico Enterprise with a direct connection to the API server so that it can take over from `kube-proxy`. - [Install the Tigera Operator](#install-the-tigera-operator) (possibly via a Helm chart), and the associated Custom Resource Definitions. - **[Download and tweak the installation Custom Resource](#tweak-and-apply-installation-custom-resources) to tell the operator to use eBPF mode.** - [Apply a set of Custom Resources](#tweak-and-apply-installation-custom-resources) to tell the operator what to install. - [Wait for the operator to provision all the associated resources and report back via its status resource](#monitor-the-progress-of-the-installation). - [Disable kube-proxy or avoid conflicts.](#disable-kube-proxy-or-avoid-conflicts) These steps are explained in more detail below. ### Create a suitable cluster The basic requirement for eBPF mode is to have a recent-enough kernel (see [supported architectures and versions](#before-you-begin)). Select the appropriate tab below for distribution-specific instructions: **Tab: Generic or kubeadm** `kubeadm` supports a number of base OSes; as long as the base OS chosen (such as Ubuntu 22.04) meets the kernel requirements, `kubeadm`-provisioned clusters are supported. Since `kube-proxy` is not required in eBPF mode, you must disable `kube-proxy` at install time. With `kubeadm` you can do that by passing the `--skip-phases=addon/kube-proxy` flag to `kubeadm init`: ```text kubeadm init --skip-phases=addon/kube-proxy ``` **Tab: kOps** `kops` supports a number of base OSes; as long as the base OS chosen (such as Ubuntu 22.04 or RHEL 8.4) meets the kernel requirements, `kops`-provisioned clusters are supported. Since `kube-proxy` is not required in eBPF mode, you must disable `kube-proxy` at install time. With `kops` you can do that by setting the following in your `kops` configuration: ```yaml kubeProxy: enabled: false ``` **Tab: OpenShift** OpenShift supports a number of base OSes; as long as the base OS chosen has a recent enough kernel, OpenShift clusters are fully supported. Since Red Hat have backported the eBPF features required by Calico Enterprise the Red Hat kernel version required is lower than the mainline: v4.18.0-305 or above. **Tab: AKS** Azure Kubernetes Service (AKS) supports a number of base OSes. The most recent Ubuntu 18.04 image has a recent enough kernel to use with eBPF mode. AKS does not support disabling `kube-proxy` so it's necessary to tell Calico Enterprise not to try to clean up `kube-proxy`'s iptables rules at a later stage. **Tab: EKS** Amazon's Elastic Kubernetes Service (EKS) supports a number of base OSes for nodes. At the time of writing, the default kernel used by Amazon Linux is recent enough to run eBPF mode, as is the Bottlerocket kernel. The Ubuntu 18.04 image did not have a recent-enough kernel (but that may have changed by the time you read this). **Tab: MKE** The eBPF data plane is supported on MKE with any Linux operating system that meets the minimum kernel requirements. Since `kube-proxy` is not required in eBPF mode, you must disable `kube-proxy` at install time. With `MKE` you can do that by setting `--kube-proxy-mode=disabled` and `--kube-default-drop-masq-bits` when installing the cluster. More details can be found in [the MKE documentation](https://docs.mirantis.com/mke/current/install/predeployment/configure-networking/cluster-service-networking-options.html) > **WARNING:** MKE's Docker Swarm overlay networking uses UDP port 4789 for its own VXLAN devices. In eBPF mode, when BTF is available on the node (typically kernel v5.8+), Calico Enterprise creates the `vxlan.calico` device in flow mode, which conflicts with Docker Swarm's use of the same port. You must change the VXLAN port before enabling eBPF: > > ```bash > kubectl patch felixconfiguration default --type merge -p '{"spec":{"vxlanPort":4790}}' > ``` > > Ensure the chosen UDP port is allowed by your underlying network between all nodes. See [Troubleshoot eBPF mode](https://docs.tigera.io/calico-enterprise/latest/operations/ebpf/troubleshoot-ebpf.md#mke-vxlan-device-down) for more details. ### Create kubernetes-service-endpoint config map In eBPF mode, Calico Enterprise takes over from `kube-proxy`. This means that, like `kube-proxy`, it needs to be able to reach the API server *directly* rather than by using the API server's `ClusterIP`. To tell Calico Enterprise how to reach the API server we create a `ConfigMap` with the API server's "real" address. In this guide we do that before installing the Tigera Operator. That means that the operator itself can also use the direct connection and hence it doesn't require `kube-proxy` to be running. The tabs below explain how to find the "real" address of the API server for a range of distributions. **Note:** In all cases it's important that the address used is stable even if your API server is restarted or scaled up/down. If you have multiple API servers, with DNS or other load balancing in front it's important to use the address of the load balancer. This prevents Calico Enterprise from being disconnected if the API servers IP changes. **Tab: Generic or kubeadm** If you created a cluster manually (for example by using `kubeadm`) then the right address to use depends on whether you opted for a high-availability cluster with multiple API servers or a simple one-node API server. - If you opted to set up a high availability cluster then you should use the address of the load balancer that you used in front of your API servers. As noted in the Kubernetes documentation, a load balancer is required for a HA set-up but the precise type of load balancer is not specified. - If you opted for a single control plane node then you can use the address of the control plane node itself. However, it's important that you use a *stable* address for that node such as a dedicated DNS record, or a static IP address. If you use a dynamic IP address (such as an EC2 private IP) then the address may change when the node is restarted causing Calico Enterprise to lose connectivity to the API server. **Tab: kOps** When using `kops`, `kops` typically sets up a load balancer of some sort in front of the API server. You should use the FQDN and port of the API load balancer: `api.internal.`. **Tab: OpenShift** OpenShift requires various DNS records to be created for the cluster; one of these is exactly what we need: `api..` should point to the API server or to the load balancer in front of the API server. Use that (filling in the `` and `` as appropriate for your cluster) for the `KUBERNETES_SERVICE_HOST` below. OpenShift uses 6443 for the `KUBERNETES_SERVICE_PORT`. **Tab: AKS** For AKS clusters, you should use the FQDN of your API server. This can be found by running the following command: ```bash kubectl cluster-info ``` which should give output similar to the following: ```text Kubernetes master is running at https://mycalicocl-calicodemorg-03a087-36558dbb.hcp.canadaeast.azmk8s.io:443 ``` In this example, you would use `mycalicocl-calicodemorg-03a087-36558dbb.hcp.canadaeast.azmk8s.io` for `KUBERNETES_SERVICE_HOST` and `443` for `KUBERNETES_SERVICE_PORT` when creating the config map. **Tab: EKS** For an EKS cluster, it's important to use the domain name of the EKS-provided load balancer that is in front of the API server. This can be found by running the following command: ```bash kubectl cluster-info ``` which should give output similar to the following: ```text Kubernetes master is running at https://60F939227672BC3D5A1B3EC9744B2B21.gr7.us-west-2.eks.amazonaws.com ... ``` In this example, you would use `60F939227672BC3D5A1B3EC9744B2B21.gr7.us-west-2.eks.amazonaws.com` for `KUBERNETES_SERVICE_HOST` and `443` for `KUBERNETES_SERVICE_PORT` when creating the config map. **Tab: MKE** MKE runs a reverse proxy in each node which can be used to reach the api-server. `KUBERNETES_SERVICE_HOST` must be set to `proxy.local` and `KUBERNETES_SERVICE_PORT` must be set to `6444`. ### Install the Tigera Operator Follow the steps in the main install for your platform that installs the Tigera Operator, without applying the custom-resources.yaml (you will update this file in a later step in this doc). For clusters in AWS, such as kOps and EKS, you must also patch the tigera-operator deployment with DNS config so the operator can resolve the apiserver DNS. AWS DNS server's address is 169.254.169.253. ```bash kubectl patch deployment -n tigera-operator tigera-operator -p '{"spec":{"template":{"spec":{"dnsConfig":{"nameservers":["169.254.169.253"]}}}}}' ``` ### Create the Config Map Create the following config map in the `tigera-operator` namespace using the host and port determined above: ```bash kubectl apply -f - <" KUBERNETES_SERVICE_PORT: "" EOF ``` ### Tweak and apply installation Custom Resources When the main install guide tells you to apply the `custom-resources.yaml`, typically by running `kubectl create` with the URL of the file directly, you should instead download the file, so that you can edit it: ```bash curl -o custom-resources.yaml https://downloads.tigera.io/ee/v3.23.2/manifests/custom-resources.yaml ``` Edit the file in your editor of choice and find the `Installation` resource, which should be at the top of the file. To enable eBPF mode, we need to add a new `calicoNetwork` section inside the `spec` of the Installation resource, including the `linuxDataplane` field. For EKS Bottlerocket OS only, you should also add the `flexVolumePath` setting as shown below. For example: ```yaml # This section includes base Calico Enterprise installation configuration. kind: Installation metadata: name: default spec: # Added calicoNetwork section with linuxDataplane field calicoNetwork: linuxDataplane: BPF # EKS with Bottlerocket as node image only: # flexVolumePath: /var/lib/kubelet/plugins # ... remainder of the Installation resource varies by platform ... # Install Calico Enterprise variant: CalicoEnterprise ``` Then apply the edited file: ```bash kubectl create -f custom-resources.yaml ``` > **SUCCESS:** If you already created the custom resources, you can switch your cluster over to eBPF mode by updating the installation resource. The operator will automatically roll out the change. > > ```bash > kubectl patch installation.operator.tigera.io default --type merge -p '{"spec":{"calicoNetwork":{"linuxDataplane":"BPF", "hostPorts":null}}}' > ``` ### Monitor the progress of the installation You can monitor progress of the installation with the following command: ```bash watch kubectl get tigerastatus ``` ### Disable `kube-proxy` (or avoid conflicts) In eBPF mode, to avoid conflicts with `kube-proxy` it's necessary to either disable `kube-proxy` or to configure Calico Enterprise not to clean up `kube-proxy`'s iptables rules. If you didn't disable `kube-proxy` when starting your cluster then follow the steps below to avoid conflicts: > **WARNING:** If you are running `kube-proxy` in IPVS mode, you must switch it to iptables mode before disabling `kube-proxy` or enabling eBPF mode. This applies to all platforms and is required for a successful migration. After switching to iptables mode, restart your nodes before proceeding. **Tab: Generic or kubeadm** For a cluster that runs `kube-proxy` in a `DaemonSet` (such as a `kubeadm`-created cluster), you can disable `kube-proxy`, reversibly, by adding a node selector to `kube-proxy`'s `DaemonSet` that matches no nodes, for example: ```bash kubectl patch ds -n kube-system kube-proxy -p '{"spec":{"template":{"spec":{"nodeSelector":{"non-calico": "true"}}}}}' ``` Then, should you want to start `kube-proxy` again, you can simply remove the node selector. **Tab: kOps** `kops` allows `kube-proxy` to be disabled by setting ```yaml kubeProxy: enabled: false ``` in its configuration. You will need to do `kops update cluster` to roll out the change. **Tab: OpenShift** In OpenShift, you can disable `kube-proxy` as follows: ```bash kubectl patch networks.operator.openshift.io cluster --type merge -p '{"spec":{"deployKubeProxy": false}}' ``` If you need to re-enable it later: ```bash kubectl patch networks.operator.openshift.io cluster --type merge -p '{"spec":{"deployKubeProxy": true}}' ``` **Tab: AKS** AKS with Azure CNI does not allow `kube-proxy` to be disabled, `kube-proxy` is deployed by the add-on manager, which will reconcile away any manual changes made to its configuration. To ensure `kube-proxy` and Calico Enterprise don't fight, set the Felix configuration parameter `bpfKubeProxyIptablesCleanupEnabled` to false. This can be done with `kubectl` as follows: ```bash kubectl patch felixconfiguration default --type merge --patch='{"spec": {"bpfKubeProxyIptablesCleanupEnabled": false}}' ``` **Tab: EKS** In EKS, you can disable `kube-proxy`, reversibly, by adding a node selector that doesn't match and nodes to `kube-proxy`'s `DaemonSet`, for example: ```bash kubectl patch ds -n kube-system kube-proxy -p '{"spec":{"template":{"spec":{"nodeSelector":{"non-calico": "true"}}}}}' ``` Then, should you want to start `kube-proxy` again, you can simply remove the node selector. ## Next steps **Recommended** - [Learn more about eBPF](https://docs.tigera.io/calico-enterprise/latest/operations/ebpf/use-cases-ebpf.md) **Recommended - Security** - [Get started with Calico Enterprise tiered network policy](https://docs.tigera.io/calico-enterprise/latest/network-policy/policy-tiers/tiered-policy.md)