tech:kubernetes-cluster:traefik-v2 [Laxwiki]

Laxwiki

Tech for the Homelab

User Tools

Site Tools


tech:kubernetes-cluster:traefik-v2

Setup Traefik V2 on Rancher

As the time of this writing, there is an experimental helm chart which actually works well. If you use the helm chart you just have to edit the traefik load balancer service to add an External IP (first run the helm chart then edit the service from rancher GUI). Also add the admin port (9000) in the load balancer. This should let you skip the creating CRDs, traefik workload and the load balancing section and jump down to the test workload section and the IngressRoute section.

Initial Setup

On rancher, go to the system project and create a new namespace called traefik. Everything related to this will be installed into this namespace. Your actual workloads can be in another namespace like default. You just have to make the IngressRoute in the same namespace as the workload. For example the whoami container can be deployed in the default namespace and the ingressroute needs to be in the same default namespace.

1. Create the Custom Resource Definitions

This is the crucial part of Traefik v2, it uses custom Kubernetes resources, in particular the IngressRoute and Middleware resources. This must be created via yaml files either using kubectl or the Import Yaml button on the rancher GUI.

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ingressroutes.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: IngressRoute
    plural: ingressroutes
    singular: ingressroute
  scope: Namespaced

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ingressroutetcps.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: IngressRouteTCP
    plural: ingressroutetcps
    singular: ingressroutetcp
  scope: Namespaced

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: middlewares.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: Middleware
    plural: middlewares
    singular: middleware
  scope: Namespaced

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: tlsoptions.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: TLSOption
    plural: tlsoptions
    singular: tlsoption
  scope: Namespaced

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: traefikservices.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: TraefikService
    plural: traefikservices
    singular: traefikservice
  scope: Namespaced

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller

rules:
  - apiGroups:
      - ""
    resources:
      - services
      - endpoints
      - secrets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses/status
    verbs:
      - update
  - apiGroups:
      - traefik.containo.us
    resources:
      - middlewares
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - traefik.containo.us
    resources:
      - ingressroutes
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - traefik.containo.us
    resources:
      - ingressroutetcps
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - traefik.containo.us
    resources:
      - tlsoptions
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - traefik.containo.us
    resources:
      - traefikservices
    verbs:
      - get
      - list
      - watch

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller

roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: traefik-ingress-controller
subjects:
  - kind: ServiceAccount
    name: traefik-ingress-controller
    namespace: default

You can see all the new CRDs using:

kubectl get crd

2.0 Note on Let's Encrypt

Let's encrypt provides free certificates for your website so that you can use HTTPS. This is done through a program called ACME which automatically downloads the certificates. ACME is built into traefik already so all you need to do is give it a few settings to tell it what you want to do.

Check the official documentation here to see all the details, but in short DNS based verification works very well because it allows wildcard certificates. Essentially that means that you can create a certificate for *.domain.com and the star can be anything. This is super useful in a kubernetes environment because you can now deploy an app called anything and then traefik will automatically get a certificate for it that will work at anything.domain.com. Basically using labels you can deploy an app, expose it to the internet and get a certificate for it immediately. I recommend DNS based verification for a kubernetes setup.

Cloudflare offers this for free as do other providers. See the link above for a list of providers.

  1. To setup the parameters for DNS verification, you need to get your Global API Key from the cloudflare website under the API Tokens section.
  2. Once you have the key, you need to make a kubernetes secret on rancher. Resources → Secrets → Add Secret. call it cloudflaredns. Make it available to a single namespace → traefik. Enter two keys: CF_API_EMAIL and CF_API_KEY with their associated values (the email is the email you used to sign up for your cloudflare account) .See above link for the names of keys for different providers.
  3. This secret will be passed to the traefik container (note that secrets are the correct way to pass protected information to a container).

2.1 Create the Traefik Wokload

This will be the container that actually runs traefik. It will run HTTP on port 8000, HTTPS on 8443 and admin on 9000. A load balancing service will then redirect traffic from the internet to this container as needed.

This section has to be created by yaml files and can't be dont via GUI because as far as I can tell, the Rancher GUI does not allow a selector → match labels field which is crucial. ie:

Spec:
  Selector:
    matchLabels:
      app: traefik
      release: traefik

The load balancer service is relying on these labels to know where to redirect traffic. If you add a label via the RancherGUI, it will not appear under selector → match label, but rather under metadata which will not work. You could change the load balancer to redirect to a specific workload, but using labels is more robust and the 'proper' way to do it.

Execute the following yaml either from your workstation by using the kubectl command or the rancher GUI Import YAML button on the workloads screen:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: traefik
    release: traefik
  name: traefik
  namespace: traefik
spec:
  replicas: 1
  selector:
    matchLabels:
      app: traefik
      release: traefik
  template:
    metadata:
      labels:
        app: traefik
        release: traefik
    spec:
      containers:
      - args:
        - --api.insecure
        - --accesslog
        - --global.checknewversion=true
        - --entryPoints.traefik.address=:9000
        - --entryPoints.web.address=:8000
        - --entryPoints.websecure.address=:8443
        - --certificatesresolvers.default.acme.dnschallenge=true
        - --certificatesresolvers.default.acme.dnschallenge.provider=cloudflare
        - --certificatesresolvers.default.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory
        - --certificatesresolvers.default.acme.email=EMAIL_ADDRESS
        - --certificatesresolvers.default.acme.storage=acme.json
        - --api.dashboard=true
        - --ping=true
        - --providers.kubernetescrd
        - --log.level=DEBUG
        envFrom:
         - secretRef:
             name: cloudflaredns
        image: traefik:2.1.1
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /ping
            port: 9000
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 2
        name: traefik
        ports:
        - containerPort: 9000
          name: traefik
          protocol: TCP
        - containerPort: 8000
          name: web
          protocol: TCP
        - containerPort: 8443
          name: websecure
          protocol: TCP
        readinessProbe:
          failureThreshold: 1
          httpGet:
            path: /ping
            port: 9000
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 2
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: traefik
      serviceAccountName: traefik
      terminationGracePeriodSeconds: 60
  • This will create the traefik workload, be sure to change the tag to the latest version
  • Don't use api.insecure in production, this allows you to access the API and dashboard over HTTP instead of HTTPS, good for testing for now, but delete later.
  • Change EMAIL_ADDRESS to your email address.
  • By default, traefik will use the real Let's Encrypt server but if you do it too often, you will get rate limited. Until you get everything working, use the stating server. Once everything works, just delete the line for the staging server and restart the pod, the real server will then get used.
    --certificatesresolvers.default.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory
  • The args section will tell traefik to use cloudflare DNS challenge. You can also use tls and http if you check the traefik docs.
  • log-level is set to debug, change it to warn when you are ready for production.
  • the envfrom section will allow the container to access the secret you created earlier
  • You have to wait at least a few minutes for Let's encrypt to get the certificates, you can check if you have them by opening a shell into the traefik container and running cat acme.json to see the certificates for each domain you have.
  • You can do some debugging with the following options if you have issues:
            - --log.level=DEBUG
            - --log.filePath=/traefik.log

    This will create a file with where you can see all the output. When you are done, change debug to warn

  • For DNS challenge, traefik will create a txt entry in your DNS on cloudflare, you can see it if you check the DNS section of your cloudflare account right after it tries to make a new certificate. Sometimes it doesn't propogate withing the alloted 2 mins so you can add a delay that might help with this option
    - --certificatesresolvers.default.acme.dnschallenge.delayBeforeCheck=10

    where the time is in seconds.

3a. Traefik Load Balancing Service

This service is a Layer 4 load balancer. It will have an external IP address so that when traffic comes to that IP on port 80 (HTTP), it will redirect it to the HTTP port on the traefik container. It will also redirect 443 (HTTPS) and 9000 (traefik's admin port which can be any number).

If you used a Helm chart to deploy traefik, this will likely have been created for you but you will need to edit it and specify the ExternalIP and then also add the admin port 9000 along with the other two ports already there.

This can be done from the rancher GUI, or you could just apply the yaml at the end of this section (you only need either 3a or 3b).

To create this yourself from scratch:

  1. Under Service Discover select Add Record
  2. Name it traefik and put it into the traefik namespace
  3. Select Resolves toThe set of pods which match a selector
  4. the selector will be the label on the right side, specify app:traefik and also release:traefik
  5. Press Show advanced options in the bottom right to get all the settings
  6. Select Layer-4 Load Balancer and add an ExternalIP which should be the node that the traefik container is running on (but it can be any IP from your Kubernetes cluster if you want).
  7. Traffic from your local network or the internet will hit this service on the Service Port, you need to map that to the Target Port on the traefik container. 80 → 8000, 443 → 8443, 9000 → 9000. Each port should have a randomly assigned NodePort. The name you give them is not important but kubernetes can reference the name instead of the number so if you make it something descriptive it can make like easier.
  8. You need to add labels for app:traefik and also release:traefik
  9. Hit save to finish. It will likely show up as Pending but actually works now. You can double check the status using the following.
    kubectl get svc traefik --namespace traefik

    You should see the External-IP listed as what you set.

3b. YAML for the above Traefik Load Balancing Service

Instead of using the GUI, you can apply the following yaml to do the same thing. Note you must change the ExternalIP to an IP of one of your cluster's nodes.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: traefik
  namespace: traefik
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: traefik
    release: traefik
  name: traefik
  namespace: traefik
spec:
  externalIPs:
  - 192.168.1.202
  externalTrafficPolicy: Cluster
  ports:
  - name: web
    nodePort: 31909
    port: 80
    protocol: TCP
    targetPort: 8000
  - name: websecure
    nodePort: 30584
    port: 443
    protocol: TCP
    targetPort: 8443
  - name: admin
    nodePort: 32316
    port: 9000
    protocol: TCP
    targetPort: 9000
  selector:
    app: traefik
    release: traefik
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer: {}

4. Deploy the test Workload

  1. Deploy a simple containous/whoami container in the default namespace with port 80 exposed as a ClusterIP

5. Deploy the IngressRoute for the Test Workload

This must be done as a yaml file

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: whoami-web
  namespace: default
spec:
  entryPoints:
    - web
  routes:
  - match: Host(`test.laxprad.ca`)
    kind: Rule
    services:
    - name: whoami
      port: 80

---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: whoami-websecure
  namespace: default
spec:
  entryPoints:
    - websecure
  routes:
  - match: Host(`test.laxprad.ca`) && PathPrefix(`/tls`)
    kind: Rule
    services:
    - name: whoami
      port: 80
  tls:
    certResolver: default
  • Note that the certResolver: default is the default resolver you specified in the traefik workload args section –certificatesresolvers.default…. You could have more that one certificate resolver, but most homelabs will only need one
  • For whoami-websecure the port should also be 80 even though it will use 443 from the internet and 8443 on the traefik container
  • Be sure the change the host to your domain and subdomain.
  • The Services → name section here corresponds to the label you set on the workload whoami. This give lots of flexibility because you can easily redirect to a new label or change an existing one.

6. Testing

At this point you should be able to open a private/incognito browser and enter your subdomain.domain.com and get to the HTTP version of whoami. https://subdomain.domain.com/tls should get to the HTTPS version. subdomain.domain.com:9000 should get to the traefik dashboard.

The procedure for deploying a new workload is:

  1. create a workload, the name is important, this is how the IngressRoute finds it
  2. be sure to expose post 80 as a ClusterIP on the workload
  3. set a label app:name, where name is the name of the workload
  4. Once the workload is done, create a IngressRoute based on the template from above, either HTTPS or HTTP, either way the IngressRoute port is 80, which is what you told the container to expose.
  5. Be sure the set the match to the subdomain.domain.com that you want. (On your DNS, you have to make sure there is a CNAME entry for this subdomain)
  6. Wait a few minutes for traefik to pick up the new route (you can monitor in the dashboard)
  7. Wait a few more minutes for Let's Encrypt to send you a new certificate (you can monitor on traefik via watch cat acme.json)
tech/kubernetes-cluster/traefik-v2.txt · Last modified: 2020/01/29 04:43 by superuser