---
title: "Bring your own Prometheus"
description: "Scrape Calico Enterprise component metrics from an existing bring-your-own Prometheus deployment instead of the bundled operator-managed Prometheus."
product: "Calico Enterprise"
version: "3.23 (latest)"
section: "Operations"
canonical_url: "https://docs.tigera.io/calico-enterprise/latest/operations/monitor/prometheus/byo-prometheus"
---

# Bring your own Prometheus

## Big picture

Scrape Calico Enterprise metrics for Bring Your Own (BYO) Prometheus.

## Value

Calico Enterprise uses the Prometheus monitoring tool to scrape metrics from instrumented jobs, and displays time-series data in a visualizer such as Grafana. You can scrape the following time-series metrics for Calico Enterprise components to your own Prometheus:

- elasticsearch
- fluentd
- calico-node
- kube-controllers
- felix
- typha (not enabled by default)

## Before you begin

**Supported**

For the supported version of Prometheus in this release, see the [Release Notes](https://docs.tigera.io/calico-enterprise/latest/release-notes.md) (`coreos-prometheus`).

## How to

- [Scrape all enabled metrics](#scrape-all-enabled-metrics)
- [Scrape metrics from specific components directly](#scrape-metrics-from-specific-components-directly)
- [Verify BYO Prometheus](#verify-byo-prometheus)
- [Create policy to secure traffic between pods](#create-policy-to-secure-traffic-between-pods)
- [Troubleshooting](#troubleshooting)

### Scrape all enabled metrics

In this section we create a service monitor that scrapes all enabled metrics. To enable metrics that are not enabled by default, please consult the [next section](#scrape-metrics-from-specific-components-directly).

The following example shows a Prometheus server installed in namespace "external-prometheus" with a `serviceMonitorSelector` that selects all service monitors with the label `k8s-app=tigera-external-prometheus`.

1. Save the following configuration in a file called `monitor.yaml`.

   ```yaml
   apiVersion: operator.tigera.io/v1
   kind: Monitor
   metadata:
     name: tigera-secure
   spec:
     externalPrometheus:
       namespace: external-prometheus
       serviceMonitor:
         labels:
           k8s-app: tigera-external-prometheus
   ```

   For a list of all configuration options, see the [Installation API reference](https://docs.tigera.io/calico-enterprise/latest/reference/installation/api.md).

2. Apply the manifest to your cluster.

   ```bash
   kubectl apply -f monitor.yaml
   ```

3. Verify that the new configuration has been added to your cluster

   ```bash
   export NS=external-prometheus
   kubectl get servicemonitor -n $NS tigera-external-prometheus
   kubectl get serviceaccount -n $NS tigera-external-prometheus
   kubectl get secret -n $NS tigera-external-prometheus
   kubectl get clusterrole tigera-external-prometheus
   kubectl get clusterrolebinding tigera-external-prometheus
   ```

   That's it. You should be seeing the new metrics show up in your Prometheus instance within a minute. For more information on verifying metrics, see the section, [Verify BYO Prometheus](#verify-byo-prometheus).

### Scrape metrics from specific components directly

We recommend the previous section for scraping all enabled metrics. Read on if you wish to scrape metrics from specific components directly using mTLS, or if you wish to enable metrics that are disabled by default.

<!-- tabs -->

**Tab: elasticsearch**

**Configure TLS certificates**

1. Copy the required secret and configmap to your namespace.

2. Save the manifest of the required TLS secret and CA configmap.

   ```bash
   kubectl get secret calico-node-prometheus-client-tls -n tigera-prometheus -o yaml >  calico-node-prometheus-client-tls.yaml
   ```

   ```bash
   kubectl get configmap -n tigera-prometheus tigera-ca-bundle -o yaml > tigera-ca-bundle.yaml
   ```

3. Edit `calico-node-prometheus-client-tls.yaml` and `tigera-ca-bundle.yaml` by changing the namespace to the namespace where your prometheus instance is running.

4. Apply the manifests to your cluster.

   ```bash
   kubectl apply -f calico-node-prometheus-client-tls.yaml
   ```

   ```bash
   kubectl apply -f tigera-ca-bundle.yaml
   ```

**Create the service monitor**

Apply the ServiceMonitor to the namespace where Prometheus is running.

```bash
export NAMESPACE=<my-prometheus-namespace>
```

```bash
kubectl apply -f https://downloads.tigera.io/ee/v3.23.2/manifests/prometheus/elasticsearch-metrics-service-monitor.yaml -n $NAMESPACE
```

The .yamls have no namespace defined so when you apply `kubectl`, it is applied in the $NAMESPACE.

**Tab: fluentd**

**Configure TLS certificates**

1. Copy the required secret and configmap to your namespace.

2. Save the manifest of the required TLS secret and CA configmap.

   ```bash
   kubectl get secret calico-node-prometheus-client-tls -n tigera-prometheus -o yaml >  calico-node-prometheus-client-tls.yaml
   ```

   ```bash
   kubectl get configmap -n tigera-prometheus tigera-ca-bundle -o yaml > tigera-ca-bundle.yaml
   ```

3. Edit `calico-node-prometheus-client-tls.yaml` and `tigera-ca-bundle.yaml` and change the namespace to the namespace where your prometheus instance is running.

4. Apply the manifests to your cluster.

   ```bash
   kubectl apply -f calico-node-prometheus-client-tls.yaml
   ```

   ```bash
   kubectl apply -f tigera-ca-bundle.yaml
   ```

**Create the service monitor**

Apply the ServiceMonitor to the namespace where Prometheus is running.

```bash
export NAMESPACE=<my-prometheus-namespace>
```

```bash
kubectl apply -f  https://downloads.tigera.io/ee/v3.23.2/manifests/prometheus/fluentd-metrics-service-monitor.yaml -n $NAMESPACE
```

The .yamls have no namespace defined so when you apply `kubectl`, it is applied in the $NAMESPACE.

**Tab: calico node**

**Configure TLS certificates**

1. Copy the required secret and configmap to your namespace.

2. Save the manifest of the required TLS secret and CA configmap.

   ```bash
   kubectl get secret calico-node-prometheus-client-tls -n tigera-prometheus -o yaml >  calico-node-prometheus-client-tls.yaml
   ```

   ```bash
   kubectl get configmap -n tigera-prometheus tigera-ca-bundle -o yaml > tigera-ca-bundle.yaml
   ```

3. Edit `calico-node-prometheus-client-tls.yaml` and `tigera-ca-bundle.yaml` by changing the namespace to the namespace where your prometheus instance is running.

4. Apply the manifests to your cluster.

   ```bash
   kubectl apply -f calico-node-prometheus-client-tls.yaml
   ```

   ```bash
   kubectl apply -f tigera-ca-bundle.yaml
   ```

**Create the service monitor**

Apply the ServiceMonitor to the namespace where Prometheus is running.

```bash
export NAMESPACE=<my-prometheus-namespace>
```

```bash
kubectl apply -f  https://downloads.tigera.io/ee/v3.23.2/manifests/prometheus/calico-node-monitor-service-monitor.yaml -n $NAMESPACE
```

The .yamls have no namespace defined so when you apply `kubectl`, it is applied in $NAMESPACE.

**Tab: kube-controllers**

**Configure TLS certificates**

1. Copy the required secret and configmap to your namespace.

2. Save the manifest of the required TLS secret and CA configmap.

   ```bash
   kubectl get secret calico-node-prometheus-client-tls -n tigera-prometheus -o yaml >  calico-node-prometheus-client-tls.yaml
   ```

   ```bash
   kubectl get configmap -n tigera-prometheus tigera-ca-bundle -o yaml > tigera-ca-bundle.yaml
   ```

3. Edit `calico-node-prometheus-client-tls.yaml` and `tigera-ca-bundle.yaml` by changing the namespace to the namespace where your prometheus instance is running.

4. Apply the manifests to your cluster.

   ```bash
   kubectl apply -f calico-node-prometheus-client-tls.yaml
   ```

   ```bash
   kubectl apply -f tigera-ca-bundle.yaml
   ```

**Create the service monitor**

Apply the ServiceMonitor to the namespace where Prometheus is running.

```bash
export NAMESPACE=<my-prometheus-namespace>
```

```bash
kubectl apply -f  https://downloads.tigera.io/ee/v3.23.2/manifests/prometheus/kube-controller-metrics-service-monitor.yaml -n $NAMESPACE
```

The .yamls have no namespace defined so when you apply `kubectl`, it is applied in the $NAMESPACE.

**Tab: Felix**

**Enable metrics**

Felix metrics are not enabled by default.

By default, Felix uses **port 9091 TCP** to publish metrics.

Use the following command to enable Felix metrics.

```bash
kubectl patch felixconfiguration default --type merge --patch '{"spec":{"prometheusMetricsEnabled": true}}'
```

You should see a result similar to:

```text
felixconfiguration.projectcalico.org/default patched
```

For all Felix configuration values, see [Felix configuration](https://docs.tigera.io/calico-enterprise/latest/reference/component-resources/node/felix/configuration.md).

For all Prometheus Felix configuration values, see [Felix Prometheus](https://docs.tigera.io/calico-enterprise/latest/reference/component-resources/node/felix/prometheus.md).

**For Windows nodes, create a service to expose Felix metrics**

If you're running Calico Enterprise for Windows, you must create a service to expose Felix metrics for Windows nodes:

```bash
kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  name: felix-windows-metrics-svc
  namespace: calico-system
  labels:
    k8s-app: felix-metrics
spec:
  clusterIP: None
  selector:
    k8s-app: calico-node-windows
  ports:
  - port: 9091
    targetPort: 9091
EOF
```

By default, the Windows firewall blocks listening on ports. For Calico Enterprise to manage the Prometheus metrics ports Windows firewall rules, enable the `windowsManageFirewallRules` setting in FelixConfiguration:

```bash
kubectl patch felixConfiguration default --type merge --patch '{"spec":{"windowsManageFirewallRules": "Enabled"}}'
```

[See the FelixConfiguration reference for more details](https://docs.tigera.io/calico-enterprise/latest/reference/resources/felixconfig.md). You can also add a Windows firewall rule that allows listening on the Prometheus port(s) instead of having Calico Enterprise manage it.

**Create the service monitor**

Apply the ServiceMonitor to the namespace where Prometheus is running.

```bash
export NAMESPACE=<my-prometheus-namespace>
```

```bash
kubectl apply -f  https://downloads.tigera.io/ee/v3.23.2/manifests/prometheus/felix-metrics-service-monitor.yaml -n $NAMESPACE
```

The .yamls have no namespace defined so when you apply `kubectl`, it is applied in the $NAMESPACE.

**Tab: Typha**

**Enable metrics**

Typha metrics are not enabled by default.

By default, Typha uses **port 9091** TCP to publish metrics. However, if Calico Enterprise is installed using the Amazon yaml file, this port will be 9093 because it is set manually using the **TYPHA\_PROMETHEUSMETRICSPORT** environment variable.

Use the following command to enable Typha metrics.

```bash
kubectl patch installation default --type=merge -p '{"spec": {"typhaMetricsPort":9093}}'
```

You should see a result similar to:

```bash
installation.operator.tigera.io/default patched
```

**Create the service monitor**

Apply the ServiceMonitor to the namespace where Prometheus is running.

```bash
export NAMESPACE=<my-prometheus-namespace>
```

```bash
kubectl apply -f  https://downloads.tigera.io/ee/v3.23.2/manifests/prometheus/typha-metrics-service-monitor.yaml -n $NAMESPACE
```

The .yamls have no namespace defined so when you apply `kubectl`, it is applied in the $NAMESPACE.

<!-- /tabs -->

### Verify BYO Prometheus

1. Access the Prometheus dashboard using the port-forwarding feature.

   ```bash
   kubectl port-forward pod/byo-prometheus-pod 9090:9090 -n $NAMESPACE
   ```

2. Browse to the Prometheus dashboard: [http://localhost:9090](http://localhost:9090).

3. In the Expression text box, enter your metric name and click the **Execute** button.

   The Console table is populated with all of your nodes with the number of endpoints.

### Troubleshooting

This section is applicable only if you experience issues with mTLS after following the [Scrape metrics from specific components directly](#scrape-metrics-from-specific-components-directly) section.

1. Extract the TLS credentials and CA bundle from the cluster.

   ```bash
   kubectl get secret -n $NAMESPACE calico-node-prometheus-client-tls -o jsonpath='{.data.tls\.key}' | base64 -d > key.pem
   kubectl get secret -n $NAMESPACE calico-node-prometheus-client-tls -o jsonpath='{.data.tls\.crt}' | base64 -d > cert.pem
   kubectl get cm -n $NAMESPACE tigera-ca-bundle -o jsonpath='{.data.tigera-ca-bundle\.crt}' > bundle.pem
   ```

2. Port-forward the Prometheus pods and run this command with the forwarded port.

   ```bash
   curl --cacert bundle.pem --key key.pem  --cert cert.pem https://localhost:8080/metrics
   ```

You should be able to see the metrics.

### Create policy to secure traffic between pods

To support zero trust, we recommend that you create Calico Enterprise network policy to allow the traffic between BYO Prometheus pods, and the respective metrics pods. For samples of ingress and egress policies, see [Get started with Calico network policy](https://docs.tigera.io/calico-enterprise/latest/network-policy/beginners/calico-network-policy.md).