HAProxy + KeyDB on AKS - take two

Last Article was all about putting quickly KeyDB in use on Kubernetes; hence, hardcoding of KeyDB's AUTH password.

Now, however, we'll go a little more professional.

Getting default values for KeyDB's Helm chart

There's a way to specify an already existing Secret as password source for KeyDB, and we're going to leverage that opportunity to provide needed AUTH informations to HAProxy.

ferdi@DESKTOP-NL6I2OD:~/kind-python-keydb$ helm show values enapter/keydb > keydb-default-values.yml

The relevant excerpt of the values file is there:

password: ""
existingSecret: ""
existingSecretPasswordKey: "password"

Create an appropriate Secret and reinstall the Helm chart using it

The Secret

ferdi@DESKTOP-NL6I2OD:~/kind-python-keydb$ kubectl create secret generic keydb-password -n default --dry-run=client -o yaml --from-literal=password="Savignone.2015" > keydb-password.yml

ferdi@DESKTOP-NL6I2OD:~/kind-python-keydb$ cat keydb-password.yml
apiVersion: v1
data:
  password: U2F2aWdub25lLjIwMTU=
kind: Secret
metadata:
  creationTimestamp: null
  name: keydb-password
  namespace: default

Referencing it

ferdi@DESKTOP-NL6I2OD:~/kind-python-keydb$ kubie ctx keydb

[keydb|default] ferdi@DESKTOP-NL6I2OD:~/kind-python-keydb$ kubectl apply -f keydb-password.yml
secret/keydb-password created

[keydb|default] ferdi@DESKTOP-NL6I2OD:~/kind-python-keydb$ helm list
NAME    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART           APP VERSION
keydb   default         1               2022-11-21 15:37:08.155778831 +0100 CET deployed        keydb-0.43.1    6.3.1

[keydb|default] ferdi@DESKTOP-NL6I2OD:~/kind-python-keydb$ helm upgrade --install keydb enapter/keydb --set existingSecret=keydb-password
Release "keydb" has been upgraded. Happy Helming!
NAME: keydb
LAST DEPLOYED: Wed Nov 23 09:44:34 2022
NAMESPACE: default
STATUS: deployed
REVISION: 2
TEST SUITE: None

Now our KeyDB direct public IP is gone, but we can still access it through HAProxy's one.

[keydb|default] ferdi@DESKTOP-NL6I2OD:~/kind-python-keydb$ kubectl get svc
NAME              TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                         AGE
haproxy-service   LoadBalancer   10.0.168.218   EDITED   8080:31623/TCP,6379:32570/TCP   15h
keydb             ClusterIP      10.0.137.22    <none>          6379/TCP,9121/TCP               42h
keydb-headless    ClusterIP      None           <none>          6379/TCP                        42h
kubernetes        ClusterIP      10.0.0.1       <none>          443/TCP                         20d

Template Configmap for HAProxy

We're going to define a new ConfigMap like the original one... but with something easily replaceable in place of the original password.

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: haproxy-config-template
  namespace: default
data:
  haproxy.cfg: |
    global
        daemon
        maxconn 256

    defaults
        mode tcp
        timeout connect 5000ms
        timeout client 50000ms
        timeout server 50000ms

    frontend http
        bind :8080
        default_backend stats    

    backend stats
        mode http
        stats enable

        stats enable
        stats uri /
        stats refresh 1s
        stats show-legends
        stats admin if TRUE

    resolvers k8s
      parse-resolv-conf
      hold other           10s
      hold refused         10s
      hold nx              10s
      hold timeout         10s
      hold valid           10s
      hold obsolete        10s

    frontend redis-write
        bind *:6379
        default_backend redis-online

    backend redis-online
        mode tcp
        balance roundrobin
        option tcp-check
        tcp-check send AUTH\ ##REPLACETHIS##\r\n
        tcp-check expect string +OK
        tcp-check send PING\r\n
        tcp-check expect string +PONG
      tcp-check send info\ replication\r\n
        tcp-check expect string role:active-replica
      tcp-check send QUIT\r\n
      tcp-check expect string +OK
      server-template keydb 3 server._tcp.keydb-headless.default.svc.cluster.local:6379 check inter 1s resolvers k8s init-addr none

Redeploy HAProxy

After having applied this configuration, we're going to redeploy HAProxy to make use of the template.

apiVersion: /v1
kind: Service
metadata:
  name: haproxy-service
  namespace: default
spec:
  type: LoadBalancer
  ports:
    - name: dashboard
      port: 8080
      targetPort: 8080
    - name: redis-write
      port: 6379
      targetPort: 6379
  selector:
    app: haproxy
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: haproxy-deployment
  namespace: default
  labels:
    app: haproxy
spec:
  replicas: 2
  selector:
    matchLabels:
      app: haproxy
  template:
    metadata:
      name: haproxy-pod
      labels:
        app: haproxy
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                    - haproxy
              topologyKey: "kubernetes.io/hostname"
      initContainers:
        - name: configeditor
          image: busybox:1.35.0
          env:
          - name: KEYDBPWD
            valueFrom:
              secretKeyRef:
                name: keydb-password
                key: password
          command:
          - sh
          - '-c'
          - sed "s/##REPLACETHIS##/${KEYDBPWD}/" /tmp/haproxy.cfg > /tmp2/haproxy.cfg  
          volumeMounts:
          - name: config-template
            mountPath: /tmp/haproxy.cfg
            subPath: haproxy.cfg
            readOnly: true
          - name: init-volume
            mountPath: /tmp2
      containers:
        - name: haproxy
          image: haproxy:2.3
          ports:
            - containerPort: 8080
            - containerPort: 6379
          volumeMounts:
          - name: init-volume
            mountPath: /usr/local/etc/haproxy/haproxy.cfg
            subPath: haproxy.cfg
            readOnly: true
      restartPolicy: Always
      volumes:
      - name: config-template
        configMap:
          name: haproxy-config-template
      - name: init-volume
        emptyDir:

Basic Kubernetes techniques here: - setting an environment variable's value from a Secret - using an initContainer to prepare the environment for the real worker - using EmptyDir volumes

Et voilà!!!

links

social