Postgresql Backup With Pghoard Kubernetes (Incremental to S3 with AutoCallFlow)

TL;DR: Incremental PostgreSQL Backup With pghoard on Kubernetes (to S3) — plus operational monitoring

This tutorial shows how to set up incremental PostgreSQL backups using pghoard in a Kubernetes environment, uploading data to object storage (S3). The goal is to avoid “backup hunting” and get you to a working baseline quickly.

pghoard is a PostgreSQL backup daemon that performs incremental backups by capturing WAL and shipping backup segments to object storage such as Amazon S3, Google Cloud Storage, etc. In this guide, we’ll configure pghoard to upload PostgreSQL backup data to S3 and run it as a Kubernetes-managed pod so it restarts automatically.

Why pghoard + Kubernetes for PostgreSQL backup?

In many PostgreSQL deployments, the hard part isn’t starting backups—it’s keeping them incremental, repeatable, and operationally safe as infrastructure changes.

What pghoard does (in practical terms)

• Incremental backup flow: pghoard continuously captures PostgreSQL WAL changes and stores them in object storage.
• Base backups + WAL stitching: it creates base backups at configured intervals and pairs them with WAL to reconstruct consistent point-in-time recovery.
• Object storage friendly: store backups in S3 (or compatible storage), which is resilient and scalable.

What Kubernetes adds

• Automation: your backup daemon runs as a pod.
• Self-healing: if the pod crashes, Kubernetes can restart it.
• Configuration via environment: inject credentials and settings per environment (dev/staging/prod).

Operational add-on with AutoCallFlow: once you have backups running, the next real problem is knowing whether they’re actually succeeding. AutoCallFlow can help you route backup status and alerts into an internal support workflow (for example, paging the right on-call responder via your existing channels), so recovery isn’t delayed by manual checking.

Architecture overview

Here’s the mental model you should keep while setting this up:

• PostgreSQL primary (your database): emits WAL and supports replication for the backup process.
• pghoard container (backup daemon): connects to PostgreSQL using a replica user, then performs base backup + WAL incremental shipping.
• Object storage (S3): receives backup artifacts.
• Kubernetes orchestration: runs pghoard reliably and restarts it when needed.

Minimum moving parts

1. Create a container image with pghoard + its dependencies.
2. Provide a pghoard.json configuration that points to PostgreSQL and S3.
3. Use a launch script to inject credentials at runtime.
4. Deploy pghoard in Kubernetes so it restarts automatically.
5. (Optional) Add operational monitoring + alerting workflow using AutoCallFlow.

Step 1: Build a Docker image for pghoard

Start by building an image that contains pghoard and its dependencies. The reference approach uses alpine for a small footprint, then installs Python tooling and pghoard dependencies.

Example Dockerfile (mirrored and rebranded)

FROM alpine:3.4

ENV REPLICA_USER "replica"
ENV REPLICA_PASSWORD "replica"

RUN apk add --no-cache \
  bash \
  build-base \
  python3 \
  python3-dev \
  ca-certificates \
  postgresql \
  postgresql-dev \
  libffi-dev \
  snappy-dev

RUN python3 -m ensurepip && \
    rm -r /usr/lib/python*/ensurepip && \
    pip3 install --upgrade pip setuptools && \
    rm -r /root/.cache && \
    pip3 install boto pghoard

COPY pghoard.json /pghoard.json.template
COPY pghoard.sh /CMD /pghoard.sh

Why these ENV variables? They’re placeholders for your PostgreSQL replication credentials. In Kubernetes, you’ll inject real values via environment variables.

What to replace

• REPLICA_USER and REPLICA_PASSWORD — injected later by Kubernetes.
• pghoard.json.template — the config includes placeholders that the launch script will rewrite.

Step 2: Create pghoard.json (PostgreSQL + S3 configuration)

Your pghoard.json drives everything: how pghoard connects to PostgreSQL, what it backs up, where it stores backups, and how it exposes its own status endpoint.

Reference configuration (mirrored)

{
  "backup_location": "/data",
  "backup_sites": {
    "default": {
      "active_backup_mode": "pg_receivexlog",
      "basebackup_count": 2,
      "basebackup_interval_hours": 24,
      "nodes": [
        {
          "host": "YOUR-PG-HOST",
          "port": 5432,
          "user": "replica",
          "password": "replica",
          "application_name": "pghoard"
        }
      ],
      "object_storage": {
        "aws_access_key_id": "REPLACE",
        "aws_secret_access_key": "REPLACE",
        "bucket_name": "REPLACE",
        "region": "us-east-1",
        "storage_type": "s3"
      },
      "pg_bin_directory": "/usr/bin"
    }
  },
  "http_address": "127.0.0.1",
  "http_port": 16000,
  "log_level": "INFO",
  "syslog": false,
  "syslog_address": "/dev/log",
  "syslog_facility": "local2"
}

Key settings explained

• backup_location: local filesystem path where pghoard keeps temporary/working data (e.g., mount a persistent volume for large DBs).
• active_backup_mode: pg_receivexlog indicates WAL capture.
• basebackup_count: number of base backups to keep (helps with restore efficiency and retention strategy).
• basebackup_interval_hours: how often base backups are generated.
• nodes[]: the PostgreSQL connection info for replication/backup.
• object_storage: S3 credentials, bucket, and region.
• http_port: optional HTTP status endpoint (binds to 127.0.0.1 in this example).

Important: ensure you have enough space under /data. If your database is large, use a Persistent Volume for the /data directory.

Step 3: Use a launch script to inject credentials and start pghoard

In Kubernetes, it’s best practice to avoid hardcoding secrets inside the image. Instead, you template pghoard.json and use a startup script to replace placeholder values.

Example launch script (mirrored)

#!/usr/bin/env bash

set -e

if [ -n "$TESTING" ]; then
  echo "Not running backup when testing"
  exit 0
fi

cat /pghoard.json.template | \
  sed "s/\"password\": \"replica\"/\"password\": \"${REPLICA_PASSWORD}\"/" | \
  sed "s/\"user\": \"replica\"/\"password\": \"${REPLICA_USER}\"/" \
  > /pghoard.json

pghoard --config /pghoard.json

How this works:

• Rewrites placeholders in /pghoard.json.template using environment variables.
• Starts pghoard with the generated config.

Note: the sed replacements in reference scripts can be sensitive to exact placeholder values. Keep placeholders consistent (e.g., if your template uses different usernames/password markers, update the sed patterns accordingly).

Step 4: Deploy pghoard in Kubernetes with an auto-restarting controller

The big operational point: don’t run pghoard as a one-off pod that can die silently. If the daemon stops, incremental backups stop too—often unnoticed until restore time.

The reference approach uses a ReplicationController to ensure the pod restarts when it fails. You can translate this concept into modern Kubernetes constructs (e.g., Deployments), but the key principle remains: use a controller with restart behavior.

Example ReplicationController manifest (mirrored)

apiVersion: v1
kind: ReplicationController
metadata:
  name: pghoard
spec:
  replicas: 1
  selector:
    app: pghoard
  template:
    metadata:
      labels:
        app: pghoard
    spec:
      containers:
      - name: pghoard
        env:
        - name: REPLICA_USER
          value: "replicant"
        - name: REPLICA_PASSWORD
          value: "The tortoise lays on its back, its belly baking in the hot sun, beating its legs trying to turn itself over. But it can't. Not with out your help. But you're not helping."
        image: gcr.io/your-project/pghoard:latest

Why replicas: 1 still matters

Even a single backup pod is fine if it restarts reliably. The “replicas” setting mainly ensures the controller will recreate the pod when it fails.

What you should consider adding next

• Persistent Volume: mount a PVC at /data for larger databases.
• Secrets: store S3 and replication credentials as Kubernetes Secrets (instead of plain env vars).
• Resource requests/limits: ensure stable performance during base backups/WAL shipping.

Validation checklist: prove your backups are incremental and actually working

Once deployed, you need confidence that pghoard is doing work, not merely “running.” Use a simple validation sequence.

1) Confirm connectivity to PostgreSQL

• Replica user auth: verify pghoard can authenticate to the PostgreSQL node.
• Replication permissions: ensure the replication role has the required privileges.
• application_name: confirm it appears in PostgreSQL if you inspect views/logs.

2) Confirm it’s writing to local /data

• Check that pghoard backup_location is writable.
• If using a PVC, ensure it’s mounted correctly and has capacity.
• Look for ongoing activity (not just initial startup files).

3) Confirm WAL shipping to S3

• Check S3 bucket for new objects/updates after database activity.
• Validate that the region matches your S3 configuration.
• Ensure IAM credentials used by pghoard have permission to write to the bucket.

4) Confirm restore readiness (smoke test)

At minimum, periodically validate you can reconstruct a restore set (base backup + WAL) for a recent timeframe. Incremental setups are only valuable if you can actually use them under pressure.

Operational best practice: tie these validation signals into an alerting workflow. AutoCallFlow can help you turn “backup health” into an actionable response loop for engineers or on-call teams (e.g., route status updates to the right responder, reduce manual checking, and accelerate recovery coordination).

Monitoring & alerting: when pghoard says “INFO” but you still need proof

The reference post explicitly calls out “Monitoring” as a future item: are your backups actually done? In real production, log output alone isn’t enough. You need confirmation that backups are advancing and that restore points exist.

What to monitor for pghoard setups

• Daemon health: pghoard is running and restarting correctly.
• WAL progress: WAL segments are being received and uploaded.
• Base backup schedule: base backups occur at expected intervals (e.g., every 24 hours).
• Storage success: uploads succeed; no repeated S3 permission/region failures.

How AutoCallFlow fits (without changing your backup design)

AutoCallFlow is a workflow automation platform you can use to operationalize backup status and incident response for your engineering/support team. The key idea is not to replace pghoard, but to make sure backup failures are noticed and handled immediately.

Typical workflow pattern:

1. Collect backup status from your environment (logs/metrics/scripts).
2. When a backup is late or stalled, send an event into your operational workflow.
3. AutoCallFlow routes the event to the correct responder and ensures the right next step happens quickly.

This keeps your backups “boring” during normal operation and “fast to respond” during incidents.

Security considerations: protect credentials and (optionally) encrypt backups

Backups contain sensitive data. You should treat your pghoard configuration, PostgreSQL replication credentials, and S3 permissions as critical security controls.

Minimum security steps

• Use Kubernetes Secrets for S3 access keys and PostgreSQL replication passwords.
• Least privilege IAM: grant only the permissions needed to write backup objects.
• Network access: restrict access from pghoard pod to PostgreSQL and S3 endpoints.

Encryption of backups (optional)

The reference notes encryption “locally and then uploaded to the cloud (supported by pghoard).” If you need encryption, align your approach with your compliance requirements and test restore compatibility end-to-end.

Practical reminder: encryption changes operational details—make sure your restore procedure still works with encrypted artifacts, and validate regularly.

Common gotchas (and how to avoid them)

• Backups stop because the pod died: avoid standalone pods; use a controller (replication/controller/deployment) so it restarts automatically.
• Not enough disk in /data: WAL buffering and temp artifacts can consume space; use PVC for large DBs.
• S3 region or bucket mismatch: verify region, bucket_name, and storage_type are correct.
• Replication user misconfiguration: authentication failures look like “no backup progress.” Confirm privileges and credentials.
• Misleading logs: running is not the same as progressing. Monitor incremental progress and verify restore readiness.

Tip: build a runbook that includes “What to check first when backups appear stalled.” Then use AutoCallFlow to keep that runbook actionable by routing incidents to the right people quickly.