--- title: "Troubleshoot egress gateways" description: "Troubleshooting guide for Calico Cloud egress gateways covering connection failures, source IP mismatches, BGP route propagation, and required pod metadata." product: "Calico Cloud" version: "v23.0.1" section: "Networking" canonical_url: "https://docs.tigera.io/calico-cloud/networking/egress/troubleshoot" --- # Troubleshoot egress gateways - [Checklist of common problems](#checklist-of-common-problems) - [Connection to an egress gateway cannot be established](#connection-to-an-egress-gateway-cannot-be-established) - [Connection to an egress gateway is established, but destination is not getting correct IP](#connection-to-an-egress-gateway-is-established-but-destination-is-not-getting-correct-ip) ## Checklist of common problems Use the following checklist to troubleshoot, or to collect details before opening a Support ticket. ### Is the egress gateway feature enabled? Egress gateway is disabled by default.  Have you enabled it in [Felix configuration](https://docs.tigera.io/calico-cloud/reference/resources/felixconfig.md) by setting `egressIPSupport` to `EnabledPerNamespace` or `EnabledPerNamespaceOrPerPod`? ### Does your egress gateway routing go through a router? (On-prem only.) As shown in the following diagram, from the gateway to the destination, the source IP is the egress IP.  On the return path, from the destination back to the gateway, the destination IP is the egress IP. If there are any routers between the gateway and the destination, they must all know how to route the egress IP back to the gateway. If they don’t, the attempted connection cannot be established. ![egress-basic-routing](data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHhtbG5zOnhsaW5rPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5L3hsaW5rIiB3aWR0aD0iNjExIiBoZWlnaHQ9IjEyOCIgc3R5bGU9ImJhY2tncm91bmQtY29sb3I6I2ZmZiIgdmlld0JveD0iLTAuNSAtMC41IDYxMSAxMjgiPjxyZWN0IHdpZHRoPSIyNDAiIGhlaWdodD0iMTE4IiB5PSI5IiBmaWxsPSIjOGViYmZjIiBmaWxsLW9wYWNpdHk9Ii4zIiBzdHJva2U9IiNkZGQiIHN0cm9rZS1vcGFjaXR5PSIuMyIgcG9pbnRlci1ldmVudHM9Im5vbmUiIHJ4PSIxIiByeT0iMSIvPjxwYXRoIGZpbGw9IiMwNmMiIGQ9Ik04NC4wNCA0NkM3NS45IDQ2IDY5IDM5LjY2IDY5IDMwLjkgNjkgMjIuNDIgNzYuMDcgMTYgODQuMDEgMTYgOTIuMjUgMTYgOTkgMjMuMTkgOTkgMzAuOTMgOTkgMzkuMDkgOTIuMzYgNDYgODQuMDQgNDYiIHBvaW50ZXItZXZlbnRzPSJub25lIi8+PHBhdGggZmlsbD0iI2ZmZiIgZD0iTTc4LjYgMjYuOTFjLS44IDAtMS4zMi0uNjItMS4zMi0xLjM4IDAtLjY0LjUyLTEuMyAxLjMyLTEuMy43NiAwIDEuMzYuNjEgMS4zNiAxLjMgMCAuNzItLjUyIDEuMzgtMS4zNiAxLjM4bTEwLjc0IDBjLS44IDAtMS4zMi0uNjItMS4zMi0xLjM4IDAtLjY0LjUyLTEuMyAxLjMyLTEuMy43NiAwIDEuMzYuNjEgMS4zNiAxLjMgMCAuNzItLjUyIDEuMzgtMS4zNiAxLjM4bTAgMTAuNzhjLS44IDAtMS4zMi0uNjItMS4zMi0xLjM4IDAtLjY0LjUyLTEuMyAxLjMyLTEuMy43NiAwIDEuMzYuNjEgMS4zNiAxLjMgMCAuNzItLjUyIDEuMzgtMS4zNiAxLjM4bS0xMi4wNi0zLjkyYy0xLjU0IDAtMi44LTEuMjEtMi44LTIuNzkgMC0xLjIxIDEtMi43OCAyLjgtMi43OCAxLjU4IDAgMi43OCAxLjI1IDIuNzggMi43OGEyLjc3IDIuNzcgMCAwIDEtMi43OCAyLjc5bTYuNzQtNi43NWMtMS41NSAwLTIuODEtMS4yMS0yLjgxLTIuNzkgMC0xLjIyIDEuMDEtMi43OCAyLjgxLTIuNzggMS41NyAwIDIuNzggMS4yNSAyLjc4IDIuNzhhMi43NyAyLjc3IDAgMCAxLTIuNzggMi43OW02LjcxIDYuNzVjLTEuNTUgMC0yLjgxLTEuMjEtMi44MS0yLjc5IDAtMS4yMSAxLjAxLTIuNzggMi44MS0yLjc4IDEuNTcgMCAyLjc4IDEuMjUgMi43OCAyLjc4YTIuNzcgMi43NyAwIDAgMS0yLjc4IDIuNzltLTYuNzQtLjY5Yy0xLjEzIDAtMi4wOS0uODctMi4wOS0yLjEgMC0xLjA2LjgzLTIuMDkgMi4xLTIuMDkgMS4wOSAwIDIuMTEuODQgMi4xMSAyLjEgMCAxLjIxLS45NSAyLjA5LTIuMTIgMi4wOW0tNS4zOSA0LjYxYy0uOCAwLTEuMzItLjYyLTEuMzItMS4zOCAwLS42NC41Mi0xLjMgMS4zMi0xLjMuNzYgMCAxLjM2LjYxIDEuMzYgMS4zIDAgLjcyLS41MiAxLjM4LTEuMzYgMS4zOG01LjQyIDIuODZjLTEuNTUgMC0yLjgxLTEuMjEtMi44MS0yLjc5IDAtMS4yMSAxLjAxLTIuNzggMi44MS0yLjc4IDEuNTcgMCAyLjc4IDEuMjUgMi43OCAyLjc4YTIuNzcgMi43NyAwIDAgMS0yLjc4IDIuNzkiIHBvaW50ZXItZXZlbnRzPSJub25lIi8+PHJlY3Qgd2lkdGg9IjkwIiBoZWlnaHQ9IjQ1IiB4PSIxMCIgeT0iNjIiIGZpbGw9IiNmZmYiIHBvaW50ZXItZXZlbnRzPSJub25lIiByeD0iNi43NSIgcnk9IjYuNzUiLz48c3dpdGNoIHRyYW5zZm9ybT0idHJhbnNsYXRlKC0uNSAtLjUpIj48Zm9yZWlnbk9iamVjdCB3aWR0aD0iMTAwJSIgaGVpZ2h0PSIxMDAlIiBwb2ludGVyLWV2ZW50cz0ibm9uZSIgcmVxdWlyZWRGZWF0dXJlcz0iaHR0cDovL3d3dy53My5vcmcvVFIvU1ZHMTEvZmVhdHVyZSNFeHRlbnNpYmlsaXR5IiBzdHlsZT0ib3ZlcmZsb3c6dmlzaWJsZTt0ZXh0LWFsaWduOmxlZnQiPjxkaXYgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkveGh0bWwiIHN0eWxlPSJkaXNwbGF5OmZsZXg7YWxpZ24taXRlbXM6dW5zYWZlIGNlbnRlcjtqdXN0aWZ5LWNvbnRlbnQ6dW5zYWZlIGNlbnRlcjt3aWR0aDo4OHB4O2hlaWdodDoxcHg7cGFkZGluZy10b3A6ODVweDttYXJnaW4tbGVmdDoxMXB4Ij48ZGl2IHN0eWxlPSJib3gtc2l6aW5nOmJvcmRlci1ib3g7Zm9udC1zaXplOjA7dGV4dC1hbGlnbjpjZW50ZXIiPjxkaXYgc3R5bGU9ImRpc3BsYXk6aW5saW5lLWJsb2NrO2ZvbnQtc2l6ZToxMnB4O2ZvbnQtZmFtaWx5OlZlcmRhbmE7Y29sb3I6IzAwMDtsaW5lLWhlaWdodDoxLjI7cG9pbnRlci1ldmVudHM6bm9uZTtmb250LXdlaWdodDo3MDA7d2hpdGUtc3BhY2U6bm9ybWFsO292ZXJmbG93LXdyYXA6bm9ybWFsIj48Zm9udCBzdHlsZT0iZm9udC1zaXplOjExcHg7Zm9udC13ZWlnaHQ6NDAwIj5DbGllbnQ8L2ZvbnQ+PC9kaXY+PC9kaXY+PC9kaXY+PC9mb3JlaWduT2JqZWN0Pjx0ZXh0IHg9IjU1IiB5PSI4OCIgZm9udC1mYW1pbHk9IlZlcmRhbmEiIGZvbnQtc2l6ZT0iMTIiIGZvbnQtd2VpZ2h0PSJib2xkIiB0ZXh0LWFuY2hvcj0ibWlkZGxlIj5DbGllbnQ8L3RleHQ+PC9zd2l0Y2g+PHJlY3Qgd2lkdGg9IjkwIiBoZWlnaHQ9IjQ1IiB4PSIxNDAiIHk9IjYyIiBmaWxsPSIjZmZmIiBwb2ludGVyLWV2ZW50cz0ibm9uZSIgcng9IjYuNzUiIHJ5PSI2Ljc1Ii8+PHN3aXRjaCB0cmFuc2Zvcm09InRyYW5zbGF0ZSgtLjUgLS41KSI+PGZvcmVpZ25PYmplY3Qgd2lkdGg9IjEwMCUiIGhlaWdodD0iMTAwJSIgcG9pbnRlci1ldmVudHM9Im5vbmUiIHJlcXVpcmVkRmVhdHVyZXM9Imh0dHA6Ly93d3cudzMub3JnL1RSL1NWRzExL2ZlYXR1cmUjRXh0ZW5zaWJpbGl0eSIgc3R5bGU9Im92ZXJmbG93OnZpc2libGU7dGV4dC1hbGlnbjpsZWZ0Ij48ZGl2IHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5L3hodG1sIiBzdHlsZT0iZGlzcGxheTpmbGV4O2FsaWduLWl0ZW1zOnVuc2FmZSBjZW50ZXI7anVzdGlmeS1jb250ZW50OnVuc2FmZSBjZW50ZXI7d2lkdGg6ODhweDtoZWlnaHQ6MXB4O3BhZGRpbmctdG9wOjg1cHg7bWFyZ2luLWxlZnQ6MTQxcHgiPjxkaXYgc3R5bGU9ImJveC1zaXppbmc6Ym9yZGVyLWJveDtmb250LXNpemU6MDt0ZXh0LWFsaWduOmNlbnRlciI+PGRpdiBzdHlsZT0iZGlzcGxheTppbmxpbmUtYmxvY2s7Zm9udC1zaXplOjEycHg7Zm9udC1mYW1pbHk6VmVyZGFuYTtjb2xvcjojMDAwO2xpbmUtaGVpZ2h0OjEuMjtwb2ludGVyLWV2ZW50czpub25lO2ZvbnQtd2VpZ2h0OjcwMDt3aGl0ZS1zcGFjZTpub3JtYWw7b3ZlcmZsb3ctd3JhcDpub3JtYWwiPjxmb250IHN0eWxlPSJmb250LXNpemU6MTFweDtmb250LXdlaWdodDo0MDAiPkdhdGV3YXk8L2ZvbnQ+PC9kaXY+PC9kaXY+PC9kaXY+PC9mb3JlaWduT2JqZWN0Pjx0ZXh0IHg9IjE4NSIgeT0iODgiIGZvbnQtZmFtaWx5PSJWZXJkYW5hIiBmb250LXNpemU9IjEyIiBmb250LXdlaWdodD0iYm9sZCIgdGV4dC1hbmNob3I9Im1pZGRsZSI+R2F0ZXdheTwvdGV4dD48L3N3aXRjaD48cmVjdCB3aWR0aD0iMTIwIiBoZWlnaHQ9IjgwIiB4PSI0OTAiIHk9IjM3IiBmaWxsPSIjZmZmIiBzdHJva2U9IiM3ZWE2ZTAiIHBvaW50ZXItZXZlbnRzPSJub25lIiByeD0iMTIiIHJ5PSIxMiIvPjxzd2l0Y2ggdHJhbnNmb3JtPSJ0cmFuc2xhdGUoLS41IC0uNSkiPjxmb3JlaWduT2JqZWN0IHdpZHRoPSIxMDAlIiBoZWlnaHQ9IjEwMCUiIHBvaW50ZXItZXZlbnRzPSJub25lIiByZXF1aXJlZEZlYXR1cmVzPSJodHRwOi8vd3d3LnczLm9yZy9UUi9TVkcxMS9mZWF0dXJlI0V4dGVuc2liaWxpdHkiIHN0eWxlPSJvdmVyZmxvdzp2aXNpYmxlO3RleHQtYWxpZ246bGVmdCI+PGRpdiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMTk5OS94aHRtbCIgc3R5bGU9ImRpc3BsYXk6ZmxleDthbGlnbi1pdGVtczp1bnNhZmUgY2VudGVyO2p1c3RpZnktY29udGVudDp1bnNhZmUgY2VudGVyO3dpZHRoOjExOHB4O2hlaWdodDoxcHg7cGFkZGluZy10b3A6NzdweDttYXJnaW4tbGVmdDo0OTFweCI+PGRpdiBzdHlsZT0iYm94LXNpemluZzpib3JkZXItYm94O2ZvbnQtc2l6ZTowO3RleHQtYWxpZ246Y2VudGVyIj48ZGl2IHN0eWxlPSJkaXNwbGF5OmlubGluZS1ibG9jaztmb250LXNpemU6MTJweDtmb250LWZhbWlseTpWZXJkYW5hO2NvbG9yOiMwMDA7bGluZS1oZWlnaHQ6MS4yO3BvaW50ZXItZXZlbnRzOm5vbmU7Zm9udC13ZWlnaHQ6NzAwO3doaXRlLXNwYWNlOm5vcm1hbDtvdmVyZmxvdy13cmFwOm5vcm1hbCI+PHNwYW4gc3R5bGU9ImZvbnQtd2VpZ2h0OjQwMCI+RGVzdGluYXRpb248L3NwYW4+PC9kaXY+PC9kaXY+PC9kaXY+PC9mb3JlaWduT2JqZWN0Pjx0ZXh0IHg9IjU1MCIgeT0iODEiIGZvbnQtZmFtaWx5PSJWZXJkYW5hIiBmb250LXNpemU9IjEyIiBmb250LXdlaWdodD0iYm9sZCIgdGV4dC1hbmNob3I9Im1pZGRsZSI+RGVzdGluYXRpb248L3RleHQ+PC9zd2l0Y2g+PHN3aXRjaCB0cmFuc2Zvcm09InRyYW5zbGF0ZSgtLjUgLS41KSI+PGZvcmVpZ25PYmplY3Qgd2lkdGg9IjEwMCUiIGhlaWdodD0iMTAwJSIgcG9pbnRlci1ldmVudHM9Im5vbmUiIHJlcXVpcmVkRmVhdHVyZXM9Imh0dHA6Ly93d3cudzMub3JnL1RSL1NWRzExL2ZlYXR1cmUjRXh0ZW5zaWJpbGl0eSIgc3R5bGU9Im92ZXJmbG93OnZpc2libGU7dGV4dC1hbGlnbjpsZWZ0Ij48ZGl2IHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5L3hodG1sIiBzdHlsZT0iZGlzcGxheTpmbGV4O2FsaWduLWl0ZW1zOnVuc2FmZSBjZW50ZXI7anVzdGlmeS1jb250ZW50OnVuc2FmZSBjZW50ZXI7d2lkdGg6NThweDtoZWlnaHQ6MXB4O3BhZGRpbmctdG9wOjMycHg7bWFyZ2luLWxlZnQ6MTAwcHgiPjxkaXYgc3R5bGU9ImJveC1zaXppbmc6Ym9yZGVyLWJveDtmb250LXNpemU6MDt0ZXh0LWFsaWduOmNlbnRlciI+PGRpdiBzdHlsZT0iZGlzcGxheTppbmxpbmUtYmxvY2s7Zm9udC1zaXplOjEycHg7Zm9udC1mYW1pbHk6VmVyZGFuYTtjb2xvcjojMDAwO2xpbmUtaGVpZ2h0OjEuMjtwb2ludGVyLWV2ZW50czpub25lO2ZvbnQtd2VpZ2h0OjcwMDt3aGl0ZS1zcGFjZTpub3JtYWw7b3ZlcmZsb3ctd3JhcDpub3JtYWwiPkNsdXN0ZXI8L2Rpdj48L2Rpdj48L2Rpdj48L2ZvcmVpZ25PYmplY3Q+PHRleHQgeD0iMTI5IiB5PSIzNiIgZm9udC1mYW1pbHk9IlZlcmRhbmEiIGZvbnQtc2l6ZT0iMTIiIGZvbnQtd2VpZ2h0PSJib2xkIiB0ZXh0LWFuY2hvcj0ibWlkZGxlIj5DbHVzdGVyPC90ZXh0Pjwvc3dpdGNoPjxwYXRoIGZpbGw9Im5vbmUiIHN0cm9rZT0iIzAwMCIgc3Ryb2tlLW1pdGVybGltaXQ9IjEwIiBkPSJNMTA2LjM3IDg0aDI3LjI2IiBwb2ludGVyLWV2ZW50cz0ibm9uZSIvPjxwYXRoIHN0cm9rZT0iIzAwMCIgc3Ryb2tlLW1pdGVybGltaXQ9IjEwIiBkPSJtMTAxLjEyIDg0IDctMy41LTEuNzUgMy41IDEuNzUgMy41Wk0xMzguODggODRsLTcgMy41IDEuNzUtMy41LTEuNzUtMy41WiIgcG9pbnRlci1ldmVudHM9Im5vbmUiLz48cGF0aCBmaWxsPSIjN2VhNmUwIiBkPSJNMzY0Ljc2IDExN0MzNDMuNTUgMTE3IDMyOSAxMDEgMzI5IDg0LjQydi0zLjcxYzAtMTYuMTcgMTQuMDctMzIuNjggMzUuNTEtMzIuNjggMTguOTYgMCAzNS40NyAxNC4wMyAzNS40NyAzMy4wNGwuMDIgMi44OWMwIDE4Ljc4LTE2LjQ0IDMzLjA0LTM1LjI0IDMzLjA0IiBwb2ludGVyLWV2ZW50cz0ibm9uZSIvPjxwYXRoIGZpbGwtb3BhY2l0eT0iLjMiIGQ9Ik0zNjQuNjYgMTE0LjE3YzE5LjAxIDAgMzUuMzItMTQuMzcgMzUuMzItMzMuMWwuMDIgMi44OWMwIDE4LjgtMTYuNDUgMzMuMDQtMzUuMjQgMzMuMDQtMjEuMjggMC0zNS43Ni0xNS45OC0zNS43Ni0zMi41OHYtMy43MWMwIDE5Ljk3IDE3LjMxIDMzLjQ2IDM1LjY2IDMzLjQ2IiBwb2ludGVyLWV2ZW50cz0ibm9uZSIvPjxwYXRoIGZpbGw9IiNmZmYiIGQ9Im0zNjQuNDkgMTEwLjkxLTcuMjQtMTEuNzVoNC4zMlY4NC4wM2gtMTYuMjJ2NC4wMmwtMTIuNjEtNi43MyAxMi42MS02Ljg2djQuMTNoMTYuMjJWNjMuMzZoLTQuMzJsNy4yNC0xMS43NCA3LjI0IDExLjc0aC00LjMxdjE1LjIzaDE2LjIydi00LjEzbDEyLjYyIDYuODYtMTIuNjIgNi43NHYtNC4wM2gtMTYuMjJ2MTUuMTNoNC4zMVoiIHBvaW50ZXItZXZlbnRzPSJub25lIi8+PHN3aXRjaCB0cmFuc2Zvcm09InRyYW5zbGF0ZSgtLjUgLS41KSI+PGZvcmVpZ25PYmplY3Qgd2lkdGg9IjEwMCUiIGhlaWdodD0iMTAwJSIgcG9pbnRlci1ldmVudHM9Im5vbmUiIHJlcXVpcmVkRmVhdHVyZXM9Imh0dHA6Ly93d3cudzMub3JnL1RSL1NWRzExL2ZlYXR1cmUjRXh0ZW5zaWJpbGl0eSIgc3R5bGU9Im92ZXJmbG93OnZpc2libGU7dGV4dC1hbGlnbjpsZWZ0Ij48ZGl2IHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5L3hodG1sIiBzdHlsZT0iZGlzcGxheTpmbGV4O2FsaWduLWl0ZW1zOnVuc2FmZSBjZW50ZXI7anVzdGlmeS1jb250ZW50OnVuc2FmZSBjZW50ZXI7d2lkdGg6NjlweDtoZWlnaHQ6MXB4O3BhZGRpbmctdG9wOjM0cHg7bWFyZ2luLWxlZnQ6MzMwcHgiPjxkaXYgc3R5bGU9ImJveC1zaXppbmc6Ym9yZGVyLWJveDtmb250LXNpemU6MDt0ZXh0LWFsaWduOmNlbnRlciI+PGRpdiBzdHlsZT0iZGlzcGxheTppbmxpbmUtYmxvY2s7Zm9udC1zaXplOjEycHg7Zm9udC1mYW1pbHk6VmVyZGFuYTtjb2xvcjojMDAwO2xpbmUtaGVpZ2h0OjEuMjtwb2ludGVyLWV2ZW50czpub25lO2ZvbnQtd2VpZ2h0OjcwMDt3aGl0ZS1zcGFjZTpub3JtYWw7b3ZlcmZsb3ctd3JhcDpub3JtYWwiPlJvdXRlcnM8L2Rpdj48L2Rpdj48L2Rpdj48L2ZvcmVpZ25PYmplY3Q+PHRleHQgeD0iMzY1IiB5PSIzOCIgZm9udC1mYW1pbHk9IlZlcmRhbmEiIGZvbnQtc2l6ZT0iMTIiIGZvbnQtd2VpZ2h0PSJib2xkIiB0ZXh0LWFuY2hvcj0ibWlkZGxlIj5Sb3V0ZXJzPC90ZXh0Pjwvc3dpdGNoPjxwYXRoIGZpbGw9Im5vbmUiIHN0cm9rZT0iIzAwMCIgc3Ryb2tlLW1pdGVybGltaXQ9IjEwIiBkPSJNMjMwIDc2LjVoMTAwIiBwb2ludGVyLWV2ZW50cz0ibm9uZSIvPjxzd2l0Y2ggdHJhbnNmb3JtPSJ0cmFuc2xhdGUoLS41IC0uNSkiPjxmb3JlaWduT2JqZWN0IHdpZHRoPSIxMDAlIiBoZWlnaHQ9IjEwMCUiIHBvaW50ZXItZXZlbnRzPSJub25lIiByZXF1aXJlZEZlYXR1cmVzPSJodHRwOi8vd3d3LnczLm9yZy9UUi9TVkcxMS9mZWF0dXJlI0V4dGVuc2liaWxpdHkiIHN0eWxlPSJvdmVyZmxvdzp2aXNpYmxlO3RleHQtYWxpZ246bGVmdCI+PGRpdiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMTk5OS94aHRtbCIgc3R5bGU9ImRpc3BsYXk6ZmxleDthbGlnbi1pdGVtczp1bnNhZmUgY2VudGVyO2p1c3RpZnktY29udGVudDp1bnNhZmUgY2VudGVyO3dpZHRoOjU4cHg7aGVpZ2h0OjFweDtwYWRkaW5nLXRvcDo2MnB4O21hcmdpbi1sZWZ0OjI2MXB4Ij48ZGl2IHN0eWxlPSJib3gtc2l6aW5nOmJvcmRlci1ib3g7Zm9udC1zaXplOjA7dGV4dC1hbGlnbjpjZW50ZXIiPjxkaXYgc3R5bGU9ImRpc3BsYXk6aW5saW5lLWJsb2NrO2ZvbnQtc2l6ZToxMXB4O2ZvbnQtZmFtaWx5OlZlcmRhbmE7Y29sb3I6IzAwMDtsaW5lLWhlaWdodDoxLjI7cG9pbnRlci1ldmVudHM6bm9uZTt3aGl0ZS1zcGFjZTpub3JtYWw7b3ZlcmZsb3ctd3JhcDpub3JtYWwiPlNvdXJjZTogRWdyZXNzIElQPC9kaXY+PC9kaXY+PC9kaXY+PC9mb3JlaWduT2JqZWN0Pjx0ZXh0IHg9IjI5MCIgeT0iNjUiIGZvbnQtZmFtaWx5PSJWZXJkYW5hIiBmb250LXNpemU9IjExIiB0ZXh0LWFuY2hvcj0ibWlkZGxlIj5Tb3VyY2U6IEVnci4uLjwvdGV4dD48L3N3aXRjaD48cGF0aCBmaWxsPSJub25lIiBzdHJva2U9IiMwMDAiIHN0cm9rZS1taXRlcmxpbWl0PSIxMCIgZD0iTTQwMCA3Ni41aDgzLjYzIiBwb2ludGVyLWV2ZW50cz0ibm9uZSIvPjxwYXRoIHN0cm9rZT0iIzAwMCIgc3Ryb2tlLW1pdGVybGltaXQ9IjEwIiBkPSJtNDg4Ljg4IDc2LjUtNyAzLjUgMS43NS0zLjUtMS43NS0zLjVaIiBwb2ludGVyLWV2ZW50cz0ibm9uZSIvPjxzd2l0Y2ggdHJhbnNmb3JtPSJ0cmFuc2xhdGUoLS41IC0uNSkiPjxmb3JlaWduT2JqZWN0IHdpZHRoPSIxMDAlIiBoZWlnaHQ9IjEwMCUiIHBvaW50ZXItZXZlbnRzPSJub25lIiByZXF1aXJlZEZlYXR1cmVzPSJodHRwOi8vd3d3LnczLm9yZy9UUi9TVkcxMS9mZWF0dXJlI0V4dGVuc2liaWxpdHkiIHN0eWxlPSJvdmVyZmxvdzp2aXNpYmxlO3RleHQtYWxpZ246bGVmdCI+PGRpdiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMTk5OS94aHRtbCIgc3R5bGU9ImRpc3BsYXk6ZmxleDthbGlnbi1pdGVtczp1bnNhZmUgY2VudGVyO2p1c3RpZnktY29udGVudDp1bnNhZmUgY2VudGVyO3dpZHRoOjU4cHg7aGVpZ2h0OjFweDtwYWRkaW5nLXRvcDoxMTJweDttYXJnaW4tbGVmdDo0MTFweCI+PGRpdiBzdHlsZT0iYm94LXNpemluZzpib3JkZXItYm94O2ZvbnQtc2l6ZTowO3RleHQtYWxpZ246Y2VudGVyIj48ZGl2IHN0eWxlPSJkaXNwbGF5OmlubGluZS1ibG9jaztmb250LXNpemU6MTFweDtmb250LWZhbWlseTpWZXJkYW5hO2NvbG9yOiMwMDA7bGluZS1oZWlnaHQ6MS4yO3BvaW50ZXItZXZlbnRzOm5vbmU7d2hpdGUtc3BhY2U6bm9ybWFsO292ZXJmbG93LXdyYXA6bm9ybWFsIj5EZXN0aW5hdGlvbjogRWdyZXNzIElQPC9kaXY+PC9kaXY+PC9kaXY+PC9mb3JlaWduT2JqZWN0Pjx0ZXh0IHg9IjQ0MCIgeT0iMTE1IiBmb250LWZhbWlseT0iVmVyZGFuYSIgZm9udC1zaXplPSIxMSIgdGV4dC1hbmNob3I9Im1pZGRsZSI+RGVzdGluYXRpb24uLi48L3RleHQ+PC9zd2l0Y2g+PHBhdGggZmlsbD0ibm9uZSIgc3Ryb2tlPSIjMDAwIiBzdHJva2UtbWl0ZXJsaW1pdD0iMTAiIGQ9Ik00MDAgOTRoOTBNMjM2LjM3IDk1LjggMjU5IDk2aDcwIiBwb2ludGVyLWV2ZW50cz0ibm9uZSIvPjxwYXRoIHN0cm9rZT0iIzAwMCIgc3Ryb2tlLW1pdGVybGltaXQ9IjEwIiBkPSJtMjMxLjEyIDk1Ljc2IDcuMDMtMy40NC0xLjc4IDMuNDggMS43MiAzLjUyWiIgcG9pbnRlci1ldmVudHM9Im5vbmUiLz48c3dpdGNoPjxnIHJlcXVpcmVkRmVhdHVyZXM9Imh0dHA6Ly93d3cudzMub3JnL1RSL1NWRzExL2ZlYXR1cmUjRXh0ZW5zaWJpbGl0eSIvPjxhIHhsaW5rOmhyZWY9Imh0dHBzOi8vd3d3LmRpYWdyYW1zLm5ldC9kb2MvZmFxL3N2Zy1leHBvcnQtdGV4dC1wcm9ibGVtcyIgdGFyZ2V0PSJfYmxhbmsiIHRyYW5zZm9ybT0idHJhbnNsYXRlKDAgLTUpIj48dGV4dCB4PSI1MCUiIHk9IjEwMCUiIGZvbnQtc2l6ZT0iMTAiIHRleHQtYW5jaG9yPSJtaWRkbGUiPlZpZXdlciBkb2VzIG5vdCBzdXBwb3J0IGZ1bGwgU1ZHIDEuMTwvdGV4dD48L2E+PC9zd2l0Y2g+PC9zdmc+) Options to make routers aware of egress IP: - Program routes statically on routers - Peer routers with the cluster, directly or indirectly using BGP or some other protocol, or other method so routers learn about the egress IP ### Does your egress gateway have required metadata? Review important egress gateway metadata (for example, namespace and labels); they are required for a client to identify the gateway(s) that it should use. ### Is natOutgoing on your IPPool set up correctly? For most egress gateway scenarios you should have: `natOutgoing: false` on the egress IPPool. If you have `natOutgoing: true`, the egress gateway will SNAT to its own IP, which is the intended egress gateway IP. But then the egress gateway’s node will also SNAT to its own IP (i.e. the node IP), which immediately overrides the egress gateway IP. ### Do clients and nodes have required selectors? Review the following annotations that are required for the client to identify its egress gateways: - egress.projectcalico.org/egressGatewayPolicy - egress.projectcalico.org/selector - egress.projectcalico.org/namespaceSelector on - Client pod - Client pod’s namespace ### Does Calico have enough route tables? On Linux, Calico claims a number of routing tables for various purposes. The exact number of tables allocated is determined by the [FelixConfiguration](https://docs.tigera.io/calico-cloud/reference/resources/felixconfig.md) option `routeTableRanges`. Failing to allocate enough route tables will result in Calico failing to program the networking needed for Egress Gateways to function. Under these conditions, `calico-node` pods will log error messages and report [unhealthy](#check-calico-node-health). [See more about routeTableRanges](https://docs.tigera.io/calico-cloud/reference/resources/felixconfig.md#routetableranges) ### Check egress gateway health As of v3.15.0, the egress gateway `Deployment` includes a Kubernetes `readinessProbe`. The egress gateway will only report ready if it is able to connect to the local `calico-node` pod and if any configured HTTP/ICMP probes are succeeding. If the egress gateway reports not-Ready then more information can be found in its logs. The egress gateway logs to `stdout` so its logs are available via `kubectl logs -n `. ### Check health of calico-node to egress gateway probes As of v3.15.0, each `calico-node` pod probes the health of the remote egress gateways that its local pods are using. If probes fail, the failures are logged in `calico-node`'s log (search for `egress_gw_tracker`) and reported via Prometheus metrics: ```text felix_egress_gateway_remote_polls{status="probe-failed"} 0 felix_egress_gateway_remote_polls{status="total"} 2 felix_egress_gateway_remote_polls{status="up"} 2 ``` Where, the `total` metric reports the total number of remote egress gateways that are being polled and the `up` and `probe-failed` metrics report the number of egress gateways in each of those states. ### Check calico-node health Check that your calico-node pods are consistently running and ready, especially on the nodes hosting the client and gateway pods. Confirming healthy pods will rule out possible issues. If you find that `calico-node` is not ready, describing the pod should show which health check is failing: ```bash kubectl describe pod ... ``` In AWS, issues such as permissions problems will report a problem with the `"aws-eni-addresses-in-sync"` health check. For more information on the problem, review the `calico-node` log. For example, a permissions issue will result in a log such as the following: ```text 2021-11-16 13:11:59.292 [ERROR][26606] felix/aws_iface_provisioner.go 343: Failed to resync with AWS. Will retry after backoff. error=failed to create ENI: operation error EC2: CreateNetworkInterface, https response error StatusCode: 403, RequestID: 13dead98-7da0-4695-9be8-80cab4d5528e, api error UnauthorizedOperation: You are not authorized to perform this operation. Encoded authorization failure message: j4x3cFwZdJ......ShGkw ``` If you see such a log, check the AWS IAM permissions assigned to the nodes in your cluster to ensure that the nodes have the [required permissions](https://docs.tigera.io/calico-cloud/networking/egress/egress-gateway-aws.md#configure-aws-iam-roles). It is also possible to decode the "encoded authorization failure message" in the log by following [this guide](https://aws.amazon.com/premiumsupport/knowledge-center/aws-backup-encoded-authorization-failure/); this gives more detail on the error. ### Check IP rule and routing setup on the client node **Run `ip rule`** On the client node, run: ```text ip rule ``` **Sample output** You will see a line for each pod on the node that is configured to use an egress gateway. ```text from 192.168.24.35 fwmark 0x80000/0x80000 lookup 250 ``` Where: - `192.168.24.35` is the relevant client's pod IP - `250` is the routing table number - `fwmark 0x80000/0x80000` is the bit/mask If you don’t see this, it means one of the following: - egressIPSupport is not enabled - egressIPSupport is enabled, but you have not configured egress annotations on the client pod or on its namespace - egressIPSupport is EnabledPerNamespace and you have configured egress annotations on the client pod, but not on its namespace **Run `ip route show table`** On the client node, run the following command using the routing table number from the `ip rule` command. For example: `250`. ```text ip route show table ``` **Sample output: clients using a single egress gateway** ```text default via 11.11.11.1 dev egress.calico onlink ``` **Sample: clients using multiple gateways** ```text default onlink nexthop via 11.11.11.1 dev egress.calico weight 1 onlink nexthop via 11.11.11.2 dev egress.calico weight 1 onlink ``` **Sample: clients using the following egress gateway policy** ```yaml apiVersion: projectcalico.org/v3 kind: EgressGatewayPolicy metadata: name: "egw-policy1" spec: rules: - destination: cidr: 10.0.0.0/8 description: "Local: no gateway" - destination: cidr: 11.0.0.0/8 description: "Gateway to on prem" gateway: namespaceSelector: "projectcalico.org/name == 'default'" selector: "egress-code == 'blue'" maxNextHops: 2 - description: "Gateway to internet" gateway: namespaceSelector: "projectcalico.org/name == 'default'" selector: "egress-code == 'red'" ``` ```text default onlink nexthop via 11.11.11.1 dev egress.calico weight 1 onlink nexthop via 11.11.11.2 dev egress.calico weight 1 onlink nexthop via 11.11.11.3 dev egress.calico weight 1 onlink throw 10.0.0.0/8 11.0.0.0/8 onlink nexthop via 11.11.12.1 dev egress.calico weight 1 onlink nexthop via 11.11.12.2 dev egress.calico weight 1 onlink ``` If you see nothing at all, or the following: ```text unreachable default scope link ``` - Verify that you have provisioned the gateways - Review the selectors, and gateway namespace and labels to determine why they aren’t matching each other ### Do you have egress IPs in BGPConfiguration svcExternalIPs? You should not have any egress IPs or pod IP ranges in BGPConfiguration `serviceExternalIPs` or `serviceClusterIPs` fields; it causes problems if you do. By default, Calico Cloud BGP exports all pod IPs, which includes egress gateway IPs because they are pod IPs. But you can also use [BGPConfiguration resource parameters](https://docs.tigera.io/calico-cloud/reference/resources/bgpconfig.md) like `BGPConfiguration`, `serviceClusterIPs`, `serviceExternalIPs` and `serviceLoadBalancerIPs` to export additional IP ranges, in particular Kubernetes Service IPs. Because Calico Cloud exports additional IP ranges in a different way from pod IPs, things can go wrong if you include pod IPs in the additional ranges. ## Connection to an egress gateway cannot be established If the outbound connection cannot be established, the policy may be denying the flow. As shown in the following diagram, policy is enforced at more points in an egress gateway flow. ![egress-basic-routing](data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHhtbG5zOnhsaW5rPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5L3hsaW5rIiB3aWR0aD0iNjExIiBoZWlnaHQ9IjEyOCIgc3R5bGU9ImJhY2tncm91bmQtY29sb3I6I2ZmZiIgdmlld0JveD0iLTAuNSAtMC41IDYxMSAxMjgiPjxyZWN0IHdpZHRoPSIyNDAiIGhlaWdodD0iMTE4IiB5PSI5IiBmaWxsPSIjOGViYmZjIiBmaWxsLW9wYWNpdHk9Ii4zIiBzdHJva2U9IiNkZGQiIHN0cm9rZS1vcGFjaXR5PSIuMyIgcG9pbnRlci1ldmVudHM9Im5vbmUiIHJ4PSIxIiByeT0iMSIvPjxwYXRoIGZpbGw9IiMwNmMiIGQ9Ik04NC4wNCA0NkM3NS45IDQ2IDY5IDM5LjY2IDY5IDMwLjkgNjkgMjIuNDIgNzYuMDcgMTYgODQuMDEgMTYgOTIuMjUgMTYgOTkgMjMuMTkgOTkgMzAuOTMgOTkgMzkuMDkgOTIuMzYgNDYgODQuMDQgNDYiIHBvaW50ZXItZXZlbnRzPSJub25lIi8+PHBhdGggZmlsbD0iI2ZmZiIgZD0iTTc4LjYgMjYuOTFjLS44IDAtMS4zMi0uNjItMS4zMi0xLjM4IDAtLjY0LjUyLTEuMyAxLjMyLTEuMy43NiAwIDEuMzYuNjEgMS4zNiAxLjMgMCAuNzItLjUyIDEuMzgtMS4zNiAxLjM4bTEwLjc0IDBjLS44IDAtMS4zMi0uNjItMS4zMi0xLjM4IDAtLjY0LjUyLTEuMyAxLjMyLTEuMy43NiAwIDEuMzYuNjEgMS4zNiAxLjMgMCAuNzItLjUyIDEuMzgtMS4zNiAxLjM4bTAgMTAuNzhjLS44IDAtMS4zMi0uNjItMS4zMi0xLjM4IDAtLjY0LjUyLTEuMyAxLjMyLTEuMy43NiAwIDEuMzYuNjEgMS4zNiAxLjMgMCAuNzItLjUyIDEuMzgtMS4zNiAxLjM4bS0xMi4wNi0zLjkyYy0xLjU0IDAtMi44LTEuMjEtMi44LTIuNzkgMC0xLjIxIDEtMi43OCAyLjgtMi43OCAxLjU4IDAgMi43OCAxLjI1IDIuNzggMi43OGEyLjc3IDIuNzcgMCAwIDEtMi43OCAyLjc5bTYuNzQtNi43NWMtMS41NSAwLTIuODEtMS4yMS0yLjgxLTIuNzkgMC0xLjIyIDEuMDEtMi43OCAyLjgxLTIuNzggMS41NyAwIDIuNzggMS4yNSAyLjc4IDIuNzhhMi43NyAyLjc3IDAgMCAxLTIuNzggMi43OW02LjcxIDYuNzVjLTEuNTUgMC0yLjgxLTEuMjEtMi44MS0yLjc5IDAtMS4yMSAxLjAxLTIuNzggMi44MS0yLjc4IDEuNTcgMCAyLjc4IDEuMjUgMi43OCAyLjc4YTIuNzcgMi43NyAwIDAgMS0yLjc4IDIuNzltLTYuNzQtLjY5Yy0xLjEzIDAtMi4wOS0uODctMi4wOS0yLjEgMC0xLjA2LjgzLTIuMDkgMi4xLTIuMDkgMS4wOSAwIDIuMTEuODQgMi4xMSAyLjEgMCAxLjIxLS45NSAyLjA5LTIuMTIgMi4wOW0tNS4zOSA0LjYxYy0uOCAwLTEuMzItLjYyLTEuMzItMS4zOCAwLS42NC41Mi0xLjMgMS4zMi0xLjMuNzYgMCAxLjM2LjYxIDEuMzYgMS4zIDAgLjcyLS41MiAxLjM4LTEuMzYgMS4zOG01LjQyIDIuODZjLTEuNTUgMC0yLjgxLTEuMjEtMi44MS0yLjc5IDAtMS4yMSAxLjAxLTIuNzggMi44MS0yLjc4IDEuNTcgMCAyLjc4IDEuMjUgMi43OCAyLjc4YTIuNzcgMi43NyAwIDAgMS0yLjc4IDIuNzkiIHBvaW50ZXItZXZlbnRzPSJub25lIi8+PHJlY3Qgd2lkdGg9IjkwIiBoZWlnaHQ9IjQ1IiB4PSIxMCIgeT0iNjIiIGZpbGw9IiNmZmYiIHBvaW50ZXItZXZlbnRzPSJub25lIiByeD0iNi43NSIgcnk9IjYuNzUiLz48c3dpdGNoIHRyYW5zZm9ybT0idHJhbnNsYXRlKC0uNSAtLjUpIj48Zm9yZWlnbk9iamVjdCB3aWR0aD0iMTAwJSIgaGVpZ2h0PSIxMDAlIiBwb2ludGVyLWV2ZW50cz0ibm9uZSIgcmVxdWlyZWRGZWF0dXJlcz0iaHR0cDovL3d3dy53My5vcmcvVFIvU1ZHMTEvZmVhdHVyZSNFeHRlbnNpYmlsaXR5IiBzdHlsZT0ib3ZlcmZsb3c6dmlzaWJsZTt0ZXh0LWFsaWduOmxlZnQiPjxkaXYgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkveGh0bWwiIHN0eWxlPSJkaXNwbGF5OmZsZXg7YWxpZ24taXRlbXM6dW5zYWZlIGNlbnRlcjtqdXN0aWZ5LWNvbnRlbnQ6dW5zYWZlIGNlbnRlcjt3aWR0aDo4OHB4O2hlaWdodDoxcHg7cGFkZGluZy10b3A6ODVweDttYXJnaW4tbGVmdDoxMXB4Ij48ZGl2IHN0eWxlPSJib3gtc2l6aW5nOmJvcmRlci1ib3g7Zm9udC1zaXplOjA7dGV4dC1hbGlnbjpjZW50ZXIiPjxkaXYgc3R5bGU9ImRpc3BsYXk6aW5saW5lLWJsb2NrO2ZvbnQtc2l6ZToxMnB4O2ZvbnQtZmFtaWx5OlZlcmRhbmE7Y29sb3I6IzAwMDtsaW5lLWhlaWdodDoxLjI7cG9pbnRlci1ldmVudHM6bm9uZTtmb250LXdlaWdodDo3MDA7d2hpdGUtc3BhY2U6bm9ybWFsO292ZXJmbG93LXdyYXA6bm9ybWFsIj48Zm9udCBzdHlsZT0iZm9udC1zaXplOjExcHg7Zm9udC13ZWlnaHQ6NDAwIj5DbGllbnQ8L2ZvbnQ+PC9kaXY+PC9kaXY+PC9kaXY+PC9mb3JlaWduT2JqZWN0Pjx0ZXh0IHg9IjU1IiB5PSI4OCIgZm9udC1mYW1pbHk9IlZlcmRhbmEiIGZvbnQtc2l6ZT0iMTIiIGZvbnQtd2VpZ2h0PSJib2xkIiB0ZXh0LWFuY2hvcj0ibWlkZGxlIj5DbGllbnQ8L3RleHQ+PC9zd2l0Y2g+PHJlY3Qgd2lkdGg9IjkwIiBoZWlnaHQ9IjQ1IiB4PSIxNDAiIHk9IjYyIiBmaWxsPSIjZmZmIiBwb2ludGVyLWV2ZW50cz0ibm9uZSIgcng9IjYuNzUiIHJ5PSI2Ljc1Ii8+PHN3aXRjaCB0cmFuc2Zvcm09InRyYW5zbGF0ZSgtLjUgLS41KSI+PGZvcmVpZ25PYmplY3Qgd2lkdGg9IjEwMCUiIGhlaWdodD0iMTAwJSIgcG9pbnRlci1ldmVudHM9Im5vbmUiIHJlcXVpcmVkRmVhdHVyZXM9Imh0dHA6Ly93d3cudzMub3JnL1RSL1NWRzExL2ZlYXR1cmUjRXh0ZW5zaWJpbGl0eSIgc3R5bGU9Im92ZXJmbG93OnZpc2libGU7dGV4dC1hbGlnbjpsZWZ0Ij48ZGl2IHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5L3hodG1sIiBzdHlsZT0iZGlzcGxheTpmbGV4O2FsaWduLWl0ZW1zOnVuc2FmZSBjZW50ZXI7anVzdGlmeS1jb250ZW50OnVuc2FmZSBjZW50ZXI7d2lkdGg6ODhweDtoZWlnaHQ6MXB4O3BhZGRpbmctdG9wOjg1cHg7bWFyZ2luLWxlZnQ6MTQxcHgiPjxkaXYgc3R5bGU9ImJveC1zaXppbmc6Ym9yZGVyLWJveDtmb250LXNpemU6MDt0ZXh0LWFsaWduOmNlbnRlciI+PGRpdiBzdHlsZT0iZGlzcGxheTppbmxpbmUtYmxvY2s7Zm9udC1zaXplOjEycHg7Zm9udC1mYW1pbHk6VmVyZGFuYTtjb2xvcjojMDAwO2xpbmUtaGVpZ2h0OjEuMjtwb2ludGVyLWV2ZW50czpub25lO2ZvbnQtd2VpZ2h0OjcwMDt3aGl0ZS1zcGFjZTpub3JtYWw7b3ZlcmZsb3ctd3JhcDpub3JtYWwiPjxmb250IHN0eWxlPSJmb250LXNpemU6MTFweDtmb250LXdlaWdodDo0MDAiPkdhdGV3YXk8L2ZvbnQ+PC9kaXY+PC9kaXY+PC9kaXY+PC9mb3JlaWduT2JqZWN0Pjx0ZXh0IHg9IjE4NSIgeT0iODgiIGZvbnQtZmFtaWx5PSJWZXJkYW5hIiBmb250LXNpemU9IjEyIiBmb250LXdlaWdodD0iYm9sZCIgdGV4dC1hbmNob3I9Im1pZGRsZSI+R2F0ZXdheTwvdGV4dD48L3N3aXRjaD48cmVjdCB3aWR0aD0iMTIwIiBoZWlnaHQ9IjgwIiB4PSI0OTAiIHk9IjM3IiBmaWxsPSIjZmZmIiBzdHJva2U9IiM3ZWE2ZTAiIHBvaW50ZXItZXZlbnRzPSJub25lIiByeD0iMTIiIHJ5PSIxMiIvPjxzd2l0Y2ggdHJhbnNmb3JtPSJ0cmFuc2xhdGUoLS41IC0uNSkiPjxmb3JlaWduT2JqZWN0IHdpZHRoPSIxMDAlIiBoZWlnaHQ9IjEwMCUiIHBvaW50ZXItZXZlbnRzPSJub25lIiByZXF1aXJlZEZlYXR1cmVzPSJodHRwOi8vd3d3LnczLm9yZy9UUi9TVkcxMS9mZWF0dXJlI0V4dGVuc2liaWxpdHkiIHN0eWxlPSJvdmVyZmxvdzp2aXNpYmxlO3RleHQtYWxpZ246bGVmdCI+PGRpdiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMTk5OS94aHRtbCIgc3R5bGU9ImRpc3BsYXk6ZmxleDthbGlnbi1pdGVtczp1bnNhZmUgY2VudGVyO2p1c3RpZnktY29udGVudDp1bnNhZmUgY2VudGVyO3dpZHRoOjExOHB4O2hlaWdodDoxcHg7cGFkZGluZy10b3A6NzdweDttYXJnaW4tbGVmdDo0OTFweCI+PGRpdiBzdHlsZT0iYm94LXNpemluZzpib3JkZXItYm94O2ZvbnQtc2l6ZTowO3RleHQtYWxpZ246Y2VudGVyIj48ZGl2IHN0eWxlPSJkaXNwbGF5OmlubGluZS1ibG9jaztmb250LXNpemU6MTJweDtmb250LWZhbWlseTpWZXJkYW5hO2NvbG9yOiMwMDA7bGluZS1oZWlnaHQ6MS4yO3BvaW50ZXItZXZlbnRzOm5vbmU7Zm9udC13ZWlnaHQ6NzAwO3doaXRlLXNwYWNlOm5vcm1hbDtvdmVyZmxvdy13cmFwOm5vcm1hbCI+PHNwYW4gc3R5bGU9ImZvbnQtd2VpZ2h0OjQwMCI+RGVzdGluYXRpb248L3NwYW4+PC9kaXY+PC9kaXY+PC9kaXY+PC9mb3JlaWduT2JqZWN0Pjx0ZXh0IHg9IjU1MCIgeT0iODEiIGZvbnQtZmFtaWx5PSJWZXJkYW5hIiBmb250LXNpemU9IjEyIiBmb250LXdlaWdodD0iYm9sZCIgdGV4dC1hbmNob3I9Im1pZGRsZSI+RGVzdGluYXRpb248L3RleHQ+PC9zd2l0Y2g+PHN3aXRjaCB0cmFuc2Zvcm09InRyYW5zbGF0ZSgtLjUgLS41KSI+PGZvcmVpZ25PYmplY3Qgd2lkdGg9IjEwMCUiIGhlaWdodD0iMTAwJSIgcG9pbnRlci1ldmVudHM9Im5vbmUiIHJlcXVpcmVkRmVhdHVyZXM9Imh0dHA6Ly93d3cudzMub3JnL1RSL1NWRzExL2ZlYXR1cmUjRXh0ZW5zaWJpbGl0eSIgc3R5bGU9Im92ZXJmbG93OnZpc2libGU7dGV4dC1hbGlnbjpsZWZ0Ij48ZGl2IHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5L3hodG1sIiBzdHlsZT0iZGlzcGxheTpmbGV4O2FsaWduLWl0ZW1zOnVuc2FmZSBjZW50ZXI7anVzdGlmeS1jb250ZW50OnVuc2FmZSBjZW50ZXI7d2lkdGg6NThweDtoZWlnaHQ6MXB4O3BhZGRpbmctdG9wOjMycHg7bWFyZ2luLWxlZnQ6MTAwcHgiPjxkaXYgc3R5bGU9ImJveC1zaXppbmc6Ym9yZGVyLWJveDtmb250LXNpemU6MDt0ZXh0LWFsaWduOmNlbnRlciI+PGRpdiBzdHlsZT0iZGlzcGxheTppbmxpbmUtYmxvY2s7Zm9udC1zaXplOjEycHg7Zm9udC1mYW1pbHk6VmVyZGFuYTtjb2xvcjojMDAwO2xpbmUtaGVpZ2h0OjEuMjtwb2ludGVyLWV2ZW50czpub25lO2ZvbnQtd2VpZ2h0OjcwMDt3aGl0ZS1zcGFjZTpub3JtYWw7b3ZlcmZsb3ctd3JhcDpub3JtYWwiPkNsdXN0ZXI8L2Rpdj48L2Rpdj48L2Rpdj48L2ZvcmVpZ25PYmplY3Q+PHRleHQgeD0iMTI5IiB5PSIzNiIgZm9udC1mYW1pbHk9IlZlcmRhbmEiIGZvbnQtc2l6ZT0iMTIiIGZvbnQtd2VpZ2h0PSJib2xkIiB0ZXh0LWFuY2hvcj0ibWlkZGxlIj5DbHVzdGVyPC90ZXh0Pjwvc3dpdGNoPjxwYXRoIGZpbGw9Im5vbmUiIHN0cm9rZT0iIzAwMCIgc3Ryb2tlLW1pdGVybGltaXQ9IjEwIiBkPSJNMTA2LjM3IDg0aDI3LjI2IiBwb2ludGVyLWV2ZW50cz0ibm9uZSIvPjxwYXRoIHN0cm9rZT0iIzAwMCIgc3Ryb2tlLW1pdGVybGltaXQ9IjEwIiBkPSJtMTAxLjEyIDg0IDctMy41LTEuNzUgMy41IDEuNzUgMy41Wk0xMzguODggODRsLTcgMy41IDEuNzUtMy41LTEuNzUtMy41WiIgcG9pbnRlci1ldmVudHM9Im5vbmUiLz48cGF0aCBmaWxsPSIjN2VhNmUwIiBkPSJNMzY0Ljc2IDExN0MzNDMuNTUgMTE3IDMyOSAxMDEgMzI5IDg0LjQydi0zLjcxYzAtMTYuMTcgMTQuMDctMzIuNjggMzUuNTEtMzIuNjggMTguOTYgMCAzNS40NyAxNC4wMyAzNS40NyAzMy4wNGwuMDIgMi44OWMwIDE4Ljc4LTE2LjQ0IDMzLjA0LTM1LjI0IDMzLjA0IiBwb2ludGVyLWV2ZW50cz0ibm9uZSIvPjxwYXRoIGZpbGwtb3BhY2l0eT0iLjMiIGQ9Ik0zNjQuNjYgMTE0LjE3YzE5LjAxIDAgMzUuMzItMTQuMzcgMzUuMzItMzMuMWwuMDIgMi44OWMwIDE4LjgtMTYuNDUgMzMuMDQtMzUuMjQgMzMuMDQtMjEuMjggMC0zNS43Ni0xNS45OC0zNS43Ni0zMi41OHYtMy43MWMwIDE5Ljk3IDE3LjMxIDMzLjQ2IDM1LjY2IDMzLjQ2IiBwb2ludGVyLWV2ZW50cz0ibm9uZSIvPjxwYXRoIGZpbGw9IiNmZmYiIGQ9Im0zNjQuNDkgMTEwLjkxLTcuMjQtMTEuNzVoNC4zMlY4NC4wM2gtMTYuMjJ2NC4wMmwtMTIuNjEtNi43MyAxMi42MS02Ljg2djQuMTNoMTYuMjJWNjMuMzZoLTQuMzJsNy4yNC0xMS43NCA3LjI0IDExLjc0aC00LjMxdjE1LjIzaDE2LjIydi00LjEzbDEyLjYyIDYuODYtMTIuNjIgNi43NHYtNC4wM2gtMTYuMjJ2MTUuMTNoNC4zMVoiIHBvaW50ZXItZXZlbnRzPSJub25lIi8+PHN3aXRjaCB0cmFuc2Zvcm09InRyYW5zbGF0ZSgtLjUgLS41KSI+PGZvcmVpZ25PYmplY3Qgd2lkdGg9IjEwMCUiIGhlaWdodD0iMTAwJSIgcG9pbnRlci1ldmVudHM9Im5vbmUiIHJlcXVpcmVkRmVhdHVyZXM9Imh0dHA6Ly93d3cudzMub3JnL1RSL1NWRzExL2ZlYXR1cmUjRXh0ZW5zaWJpbGl0eSIgc3R5bGU9Im92ZXJmbG93OnZpc2libGU7dGV4dC1hbGlnbjpsZWZ0Ij48ZGl2IHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5L3hodG1sIiBzdHlsZT0iZGlzcGxheTpmbGV4O2FsaWduLWl0ZW1zOnVuc2FmZSBjZW50ZXI7anVzdGlmeS1jb250ZW50OnVuc2FmZSBjZW50ZXI7d2lkdGg6NjlweDtoZWlnaHQ6MXB4O3BhZGRpbmctdG9wOjM0cHg7bWFyZ2luLWxlZnQ6MzMwcHgiPjxkaXYgc3R5bGU9ImJveC1zaXppbmc6Ym9yZGVyLWJveDtmb250LXNpemU6MDt0ZXh0LWFsaWduOmNlbnRlciI+PGRpdiBzdHlsZT0iZGlzcGxheTppbmxpbmUtYmxvY2s7Zm9udC1zaXplOjEycHg7Zm9udC1mYW1pbHk6VmVyZGFuYTtjb2xvcjojMDAwO2xpbmUtaGVpZ2h0OjEuMjtwb2ludGVyLWV2ZW50czpub25lO2ZvbnQtd2VpZ2h0OjcwMDt3aGl0ZS1zcGFjZTpub3JtYWw7b3ZlcmZsb3ctd3JhcDpub3JtYWwiPlJvdXRlcnM8L2Rpdj48L2Rpdj48L2Rpdj48L2ZvcmVpZ25PYmplY3Q+PHRleHQgeD0iMzY1IiB5PSIzOCIgZm9udC1mYW1pbHk9IlZlcmRhbmEiIGZvbnQtc2l6ZT0iMTIiIGZvbnQtd2VpZ2h0PSJib2xkIiB0ZXh0LWFuY2hvcj0ibWlkZGxlIj5Sb3V0ZXJzPC90ZXh0Pjwvc3dpdGNoPjxwYXRoIGZpbGw9Im5vbmUiIHN0cm9rZT0iIzAwMCIgc3Ryb2tlLW1pdGVybGltaXQ9IjEwIiBkPSJNMjMwIDc2LjVoMTAwIiBwb2ludGVyLWV2ZW50cz0ibm9uZSIvPjxzd2l0Y2ggdHJhbnNmb3JtPSJ0cmFuc2xhdGUoLS41IC0uNSkiPjxmb3JlaWduT2JqZWN0IHdpZHRoPSIxMDAlIiBoZWlnaHQ9IjEwMCUiIHBvaW50ZXItZXZlbnRzPSJub25lIiByZXF1aXJlZEZlYXR1cmVzPSJodHRwOi8vd3d3LnczLm9yZy9UUi9TVkcxMS9mZWF0dXJlI0V4dGVuc2liaWxpdHkiIHN0eWxlPSJvdmVyZmxvdzp2aXNpYmxlO3RleHQtYWxpZ246bGVmdCI+PGRpdiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMTk5OS94aHRtbCIgc3R5bGU9ImRpc3BsYXk6ZmxleDthbGlnbi1pdGVtczp1bnNhZmUgY2VudGVyO2p1c3RpZnktY29udGVudDp1bnNhZmUgY2VudGVyO3dpZHRoOjU4cHg7aGVpZ2h0OjFweDtwYWRkaW5nLXRvcDo2MnB4O21hcmdpbi1sZWZ0OjI2MXB4Ij48ZGl2IHN0eWxlPSJib3gtc2l6aW5nOmJvcmRlci1ib3g7Zm9udC1zaXplOjA7dGV4dC1hbGlnbjpjZW50ZXIiPjxkaXYgc3R5bGU9ImRpc3BsYXk6aW5saW5lLWJsb2NrO2ZvbnQtc2l6ZToxMXB4O2ZvbnQtZmFtaWx5OlZlcmRhbmE7Y29sb3I6IzAwMDtsaW5lLWhlaWdodDoxLjI7cG9pbnRlci1ldmVudHM6bm9uZTt3aGl0ZS1zcGFjZTpub3JtYWw7b3ZlcmZsb3ctd3JhcDpub3JtYWwiPlNvdXJjZTogRWdyZXNzIElQPC9kaXY+PC9kaXY+PC9kaXY+PC9mb3JlaWduT2JqZWN0Pjx0ZXh0IHg9IjI5MCIgeT0iNjUiIGZvbnQtZmFtaWx5PSJWZXJkYW5hIiBmb250LXNpemU9IjExIiB0ZXh0LWFuY2hvcj0ibWlkZGxlIj5Tb3VyY2U6IEVnci4uLjwvdGV4dD48L3N3aXRjaD48cGF0aCBmaWxsPSJub25lIiBzdHJva2U9IiMwMDAiIHN0cm9rZS1taXRlcmxpbWl0PSIxMCIgZD0iTTQwMCA3Ni41aDgzLjYzIiBwb2ludGVyLWV2ZW50cz0ibm9uZSIvPjxwYXRoIHN0cm9rZT0iIzAwMCIgc3Ryb2tlLW1pdGVybGltaXQ9IjEwIiBkPSJtNDg4Ljg4IDc2LjUtNyAzLjUgMS43NS0zLjUtMS43NS0zLjVaIiBwb2ludGVyLWV2ZW50cz0ibm9uZSIvPjxzd2l0Y2ggdHJhbnNmb3JtPSJ0cmFuc2xhdGUoLS41IC0uNSkiPjxmb3JlaWduT2JqZWN0IHdpZHRoPSIxMDAlIiBoZWlnaHQ9IjEwMCUiIHBvaW50ZXItZXZlbnRzPSJub25lIiByZXF1aXJlZEZlYXR1cmVzPSJodHRwOi8vd3d3LnczLm9yZy9UUi9TVkcxMS9mZWF0dXJlI0V4dGVuc2liaWxpdHkiIHN0eWxlPSJvdmVyZmxvdzp2aXNpYmxlO3RleHQtYWxpZ246bGVmdCI+PGRpdiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMTk5OS94aHRtbCIgc3R5bGU9ImRpc3BsYXk6ZmxleDthbGlnbi1pdGVtczp1bnNhZmUgY2VudGVyO2p1c3RpZnktY29udGVudDp1bnNhZmUgY2VudGVyO3dpZHRoOjU4cHg7aGVpZ2h0OjFweDtwYWRkaW5nLXRvcDoxMTJweDttYXJnaW4tbGVmdDo0MTFweCI+PGRpdiBzdHlsZT0iYm94LXNpemluZzpib3JkZXItYm94O2ZvbnQtc2l6ZTowO3RleHQtYWxpZ246Y2VudGVyIj48ZGl2IHN0eWxlPSJkaXNwbGF5OmlubGluZS1ibG9jaztmb250LXNpemU6MTFweDtmb250LWZhbWlseTpWZXJkYW5hO2NvbG9yOiMwMDA7bGluZS1oZWlnaHQ6MS4yO3BvaW50ZXItZXZlbnRzOm5vbmU7d2hpdGUtc3BhY2U6bm9ybWFsO292ZXJmbG93LXdyYXA6bm9ybWFsIj5EZXN0aW5hdGlvbjogRWdyZXNzIElQPC9kaXY+PC9kaXY+PC9kaXY+PC9mb3JlaWduT2JqZWN0Pjx0ZXh0IHg9IjQ0MCIgeT0iMTE1IiBmb250LWZhbWlseT0iVmVyZGFuYSIgZm9udC1zaXplPSIxMSIgdGV4dC1hbmNob3I9Im1pZGRsZSI+RGVzdGluYXRpb24uLi48L3RleHQ+PC9zd2l0Y2g+PHBhdGggZmlsbD0ibm9uZSIgc3Ryb2tlPSIjMDAwIiBzdHJva2UtbWl0ZXJsaW1pdD0iMTAiIGQ9Ik00MDAgOTRoOTBNMjM2LjM3IDk1LjggMjU5IDk2aDcwIiBwb2ludGVyLWV2ZW50cz0ibm9uZSIvPjxwYXRoIHN0cm9rZT0iIzAwMCIgc3Ryb2tlLW1pdGVybGltaXQ9IjEwIiBkPSJtMjMxLjEyIDk1Ljc2IDcuMDMtMy40NC0xLjc4IDMuNDggMS43MiAzLjUyWiIgcG9pbnRlci1ldmVudHM9Im5vbmUiLz48c3dpdGNoPjxnIHJlcXVpcmVkRmVhdHVyZXM9Imh0dHA6Ly93d3cudzMub3JnL1RSL1NWRzExL2ZlYXR1cmUjRXh0ZW5zaWJpbGl0eSIvPjxhIHhsaW5rOmhyZWY9Imh0dHBzOi8vd3d3LmRpYWdyYW1zLm5ldC9kb2MvZmFxL3N2Zy1leHBvcnQtdGV4dC1wcm9ibGVtcyIgdGFyZ2V0PSJfYmxhbmsiIHRyYW5zZm9ybT0idHJhbnNsYXRlKDAgLTUpIj48dGV4dCB4PSI1MCUiIHk9IjEwMCUiIGZvbnQtc2l6ZT0iMTAiIHRleHQtYW5jaG9yPSJtaWRkbGUiPlZpZXdlciBkb2VzIG5vdCBzdXBwb3J0IGZ1bGwgU1ZHIDEuMTwvdGV4dD48L2E+PC9zd2l0Y2g+PC9zdmc+) **Policy enforcement**: - From the client pod, egress - To the gateway pod, ingress - From the gateway pod, egress - Any relevant HostEndpoints that are configured in your cluster In [the web console](https://docs.tigera.io/calico-cloud/tutorials/calico-cloud-features/tour.md), check for dropped packets because of policy on the outbound connection path. If you are using the iptables data plane, you can also run the following command on the client and gateway nodes to look at a lower level. ```text watch iptables-save -c | grep DROP | grep -v 0:0 ``` ## Connection to an egress gateway is established, but destination is not getting correct IP If you see that the outbound connection established, but the source IP is incorrect at the destination, this can indicate that other SNAT or MASQUERADE is taking effect after the packet leaves the egress gateway pod and is overriding the egress gateway IP. If you intentionally have a MASQUERADE/SNAT for another general purpose, you must filter it so it does not apply to traffic whose source IP comes from the egress gateway pool. To check the egress gateway’s node, use iptables: ```text iptables-save -c | grep -i MASQUERADE iptables-save -c | grep -i SNAT ``` ## Finding leaked AWS ENIs In normal usage of the AWS-backed IP pools feature, the calico-node Pod on each node will manage the secondary ENIs used for networking AWS-backed IP pools. It also marks its secondary ENIs for deletion on instance termination to avoid leaking any ENIs when an instance is terminated. However, in certain highly unusual situations, such as the following sequence of events: - calico-node adds an ENI. - The AWS API call to mark the ENI for "delete on termination" fails. - The entire instance is deleted before the automatic retry of the above operation succeeds. Then, it would be possible for an ENI to be leaked. Calico Cloud marks all the ENIs that it creates with tags to identify them as Calico Cloud secondary ENIs and the ID of the instance they *should* belong to. To find potentially leaked ENIs, you can use the AWS command line tool as follows: ```bash aws ec2 describe-network-interfaces --filters Name=status,Values=available Name=tag-key,Values=calico:use ``` Then, examine the "Tag set" of the returned network interface values to see if any of them belong to nodes that have been deleted: ```text "TagSet": [ { "Key": "calico:use", "Value": "secondary" }, { "Key": "calico:instance", "Value": "i-00122bf604c6ab776" } ], ``` If the instance ID recorded in the "calico:instance" tag is for an instance that no longer exists then the ENI has been leaked; it is safe to delete the ENI. If the instance ID belongs to an active instance then there is no need to delete the ENI, it should be cleaned up (or put into use) by the calico-node Pod running on that instance.