What is it about?

Lately, while trying to import a Mariadb backup into database (hosted on Kubernetes and using Bitnami Mariadb) I was hitting issues like following:

$ mysql -u root -h <edited-ip> -p <edited-dbname> < backup.sql
Enter password:
ERROR 2013 (HY000) at line 1302: Lost connection to MySQL server during query

Investigation

So, after I started the process of import, it was disrupted. I have verified Mariadb server, which was running in a k8s pod, and noticed it was restarted.

A mariadb restart would definitely cause above client error ("Lost connection to MySQL server during query").

Question is - why was it restarted? So I dug into k8s events and saw that Liveness probe failed.

Liveness probe in this Bitnami/Mariadb chart is a simple mysqladmin command:

          livenessProbe: {{- omit .Values.primary.livenessProbe "enabled" | toYaml | nindent 12 }}
            exec:
              command:
                - /bin/bash
                - -ec
                - |
                  password_aux="${MARIADB_ROOT_PASSWORD:-}"
                  if [[ -f "${MARIADB_ROOT_PASSWORD_FILE:-}" ]]; then
                      password_aux=$(cat "$MARIADB_ROOT_PASSWORD_FILE")
                  fi
                  mysqladmin status -uroot -p"${password_aux}"
          {{- else if .Values.primary.customLivenessProbe }}
          livenessProbe: {{- include "common.tplvalues.render" (dict "value" .Values.primary.customLivenessProbe "context" $) | nindent 12 }}
          {{- end }}
          {{- if .Values.primary.readinessProbe.enabled }}

Why would it stop working during an import process?

Resolution

The short answer is: import process creates quite huge performance pressure. Mariadb pod stops responding for anything, while this import is in progress. That's because I'm running a low - power, thin-client devices as k8s nodes.

So - livenessprobe fails, and k8s scheduler restarts pod, and everything gets back to normal. But database import is not finished and needs to be started all over again.

Solution was to actually change default timeouts in liveness probes for the time if the import process:

  livenessProbe:
    enabled: true
    initialDelaySeconds: 120
    periodSeconds: 30
    timeoutSeconds: 120
    failureThreshold: 30
    successThreshold: 1

And that was it. After import was successfull, I reverted above values to defaults.

Comments