--- title: "K3s multi-node install" description: "Install Calico Open Source on a multi-node K3s cluster for testing or development workloads." product: "Calico Open Source" version: "3.32 (latest)" section: "Installing and upgrading" canonical_url: "https://docs.tigera.io/calico/latest/getting-started/kubernetes/k3s/multi-node-install" --- # K3s multi-node install ## Big picture This tutorial gets you a multi node K3s cluster with Calico in approximately 10 minutes. ## Value K3s is a lightweight implementation of Kubernetes packaged as a single binary. The geeky details of what you get: | Policy | IPAM | CNI | Overlay | Routing | Datastore | | ------ | ------ | ------ | ------- | ------- | ---------- | | Calico | Calico | Calico | VXLAN | Calico | Kubernetes | ? ## Before you begin - Make sure you have a linux host that meets the following requirements - x86-64 processor - 1CPU - 1GB Ram - 10GB free disk space - Ubuntu 18.04 (amd64), Ubuntu 20.04 (amd64) > **SECONDARY:** K3s supports ARM processors too, this tutorial was tested against x86-64 processor environment. For more detail please visit [this link](https://rancher.com/docs/k3s/latest/en/installation/installation-requirements/#operating-systems). ## How to ### Initializing control plane instance K3s installation script can be modified by [environment variables](https://rancher.com/docs/k3s/latest/en/installation/install-options/#options-for-installation-with-script). Here you are providing some extra arguments to disable `flannel`, disable k3s default network policy and change the pod ip CIDR. > **SECONDARY:** Full list of arguments can be viewed [at this link](https://rancher.com/docs/k3s/latest/en/installation/install-options/server-config/). curl -sfL [https://get.k3s.io](https://get.k3s.io) | INSTALL\_K3S\_EXEC="--flannel-backend=none --disable-network-policy --cluster-cidr=192.168.0.0/16" sh - > **WARNING:** If 192.168.0.0/16 is already in use within your network you must select a different pod network CIDR by replacing 192.168.0.0/16 in the above command. ### Enable remote access to your K3s instance To set up remote access to your cluster first ensure you have installed `kubectl` on your system. > **SECONDARY:** If you are not sure how to install kubectl in your OS [visit this link](https://kubernetes.io/docs/tasks/tools/install-kubectl/). K3s stores a kubeconfig file in your server at `/etc/rancher/k3s/k3s.yaml`, copy all the content of `k3s.yaml` from your server into `~/.kube/config` on the system that you like to have remote access to the cluster. ### Add extra nodes to K3s cluster To add additional nodes to your cluster you need two piece of information. - `K3S_URL` which is going to be your main node ip address. - `K3S_TOKEN` which is stored in `/var/lib/rancher/k3s/server/node-token` file in main Node [(Step 1)](#initializing-control-plane-instance). Execute following command in your node instance and join it to the cluster. > **SECONDARY:** Remember to change `serverip` and `mytoken`. ```bash curl -sfL https://get.k3s.io | K3S_URL=https://serverip:6443 K3S_TOKEN=mytoken sh - ``` ### Install Calico **Tab: Operator** Install the Calico operator and custom resource definitions. ```bash kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.32.1/manifests/v1_crd_projectcalico_org.yaml kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.32.1/manifests/tigera-operator.yaml ``` > **SECONDARY:** Due to the large size of the CRD bundle, `kubectl apply` might exceed request limits. Instead, use `kubectl create` or `kubectl replace`. Install Calico by creating the necessary custom resource. For more information on configuration options available in this manifest, see [the installation reference](https://docs.tigera.io/calico/latest/reference/installation/api.md). ```bash kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.32.1/manifests/custom-resources.yaml ``` > **SECONDARY:** Before creating this manifest, read its contents and make sure its settings are correct for your environment. For example, you may need to change the default IP pool CIDR to match your pod network CIDR. **Tab: Manifest** Install Calico by using the following command. ```bash kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.32.1/manifests/calico.yaml ``` > **SECONDARY:** You can also [view the YAML in a new tab](https://raw.githubusercontent.com/projectcalico/calico/v3.32.1/manifests/calico.yaml). You should see the following output. ```text configmap/calico-config created customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created clusterrole.rbac.authorization.k8s.io/calico-node created clusterrolebinding.rbac.authorization.k8s.io/calico-node created daemonset.apps/calico-node created serviceaccount/calico-node created deployment.apps/calico-kube-controllers created serviceaccount/calico-kube-controllers created ``` ### Check the installation 1. Confirm that all of the pods are running using the following command. **Tab: Operator** ```text NAMESPACE NAME READY STATUS RESTARTS AGE tigera-operator tigera-operator-c9cf5b94d-gj9qp 1/1 Running 0 107s calico-system calico-typha-7dcd87597-npqsf 1/1 Running 0 88s calico-system calico-node-rdwwz 1/1 Running 0 88s kube-system local-path-provisioner-6d59f47c7-4q8l2 1/1 Running 0 2m14s kube-system metrics-server-7566d596c8-xf66d 1/1 Running 0 2m14s kube-system coredns-8655855d6-wfdbm 1/1 Running 0 2m14s calico-system calico-kube-controllers-89df8c6f8-7hxc5 1/1 Running 0 87s ``` **Tab: Manifest** ```text NAMESPACE NAME READY STATUS RESTARTS AGE kube-system calico-node-9hn9z 1/1 Running 0 23m kube-system local-path-provisioner-6d59f47c7-drznc 1/1 Running 0 38m kube-system calico-kube-controllers-789f6df884-928lt 1/1 Running 0 23m kube-system metrics-server-7566d596c8-qxlfz 1/1 Running 0 38m kube-system coredns-8655855d6-blzl5 1/1 Running 0 38m ``` 1. Confirm that you now have two nodes in your cluster with the following command. ```bash kubectl get nodes -o wide ``` It should return something like the following. ```text NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME k3s-master Ready master 40m v1.18.2+k3s1 172.16.2.128 Ubuntu 18.04.3 LTS 4.15.0-101-generic containerd://1.3.3-k3s2 k3s-node1 Ready 30m v1.18.2+k3s1 172.16.2.129 Ubuntu 18.04.3 LTS 4.15.0-101-generic containerd://1.3.3-k3s2 ``` Congratulations! You now have a multi node K3s cluster equipped with Calico and Traefik. ## Next steps - Try running the [Kubernetes Network policy demo](https://docs.tigera.io/calico/latest/network-policy/get-started/kubernetes-policy/kubernetes-demo.md) to see live graphical view of network policy in action