MongoDB Backup Manager Script

This script automates the process of backing up a MongoDB database running in a Docker container. It also uploads the backups to an AWS S3 bucket for safe storage and retention.

---

Features

• Creates compressed backups of your MongoDB database using mongodump, optionally limited to a subset of collections.
• Stores backups locally in a specified directory.
• Uploads backups to an AWS S3 bucket.
• Automatically cleans up old backups from the local directory (optional).
• Restores a backup to the MongoDB database.
• Verifies count and full index-spec parity between two namespaces after a restore/migration.
• Resolves the target container by exact name or a stable name prefix, so it survives orchestrators (e.g. Coolify) that regenerate the container name on every redeploy.
• Alerts on failure (Discord/Telegram) for unattended runs, naming the stage that failed (mongodump vs S3 upload).

---

Prerequisites

1. MongoDB in Docker

• Ensure Docker is installed on the host machine and The MongoDB instance must be running in a Docker container.

2. AWS CLI

• Install the AWS CLI by following the instructions here.
• Configure the AWS CLI with your AWS credentials:

  aws configure

  Provide:
  • AWS Access Key ID
  • AWS Secret Access Key
  • Default region (e.g., us-east-1)
  • Default output format (e.g., json)

3. Permissions

Ensure the IAM user associated with the AWS credentials has the correct permission, the following JSON policy can be attached to the IAM user:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetBucketLocation",
                "s3:ListBucket"
            ],
            "Resource": "arn:aws:s3:::BUCKET"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject"
            ],
            "Resource": "arn:aws:s3:::BUCKET/mongodb-backups/*"
        }
    ]
}

4. Misc installations

• sudo apt install dotenv

---

Environment Variables.

Create a .env file in the same directory as the script and add the following environment variables. A ready-to-copy template lives in .env-example:

cp .env-example .env   # then edit in your real values

# Set variables
CONTAINER_NAME="sodax-stateful-mongo"   # exact name OR a stable name prefix (see below)
S3_BUCKET_NAME="name-of-your-s3-bucket"
MONGO_PORT="27017"
MONGO_USER="user"
MONGO_PASSWORD="password"
MONGO_DB_NAME="test"
USE_CREDENTIALS="true" # set to false if the DB has no auth
USE_REMOTE="false"     # set to true to enable S3 upload/download/list/prune
USE_ALERTS="false"     # set to true to alert (Discord/Telegram) when a run fails

Variable	Description
CONTAINER_NAME	The MongoDB container: either its exact name or a stable name prefix. A prefix is resolved at run time (via docker ps, running containers only) to the single live match, and must resolve to exactly one — 0 or >1 matches is a hard error. See "Dynamic container-name resolution" below.
S3_BUCKET_NAME	Name of the AWS S3 bucket where backups will be stored (required only when USE_REMOTE=true).
MONGO_PORT	Port number of the MongoDB instance.
MONGO_USER	MongoDB username (required only when USE_CREDENTIALS=true).
MONGO_PASSWORD	MongoDB password (required only when USE_CREDENTIALS=true).
MONGO_DB_NAME	Name of the MongoDB database to backup.
USE_CREDENTIALS	When false, skips MongoDB username/password in dump/restore commands. Defaults to true.
USE_REMOTE	When true, enables S3 upload/download/list/prune. Defaults to false (local-only); set true to use S3.
S3_RETENTION_DAYS	Days after which S3 backups are pruned. Defaults to 7. 0 skips the S3 prune entirely so an S3 lifecycle rule owns retention. Must be a non-negative integer (validated when USE_REMOTE=true). Only affects S3 — the local disk prune is always 7 days.
USE_ALERTS	When true, enables failure alerting (see "Failure alerting" below). Defaults to false. Requires at least one destination below; validated in health_check.
DISCORD_WEBHOOK_URL	Discord webhook URL to post a failure alert to (required for Discord alerts when USE_ALERTS=true). Secret — never commit.
TELEGRAM_BOT_TOKEN	Telegram bot token from @BotFather (required, together with TELEGRAM_CHAT_ID, for Telegram alerts when USE_ALERTS=true). Secret — never commit.
TELEGRAM_CHAT_ID	Telegram chat/channel id to send the alert to (required together with TELEGRAM_BOT_TOKEN).

Dynamic container-name resolution

CONTAINER_NAME may be an exact container name (backward-compatible — it's the sole match) or a stable name prefix. This matters on platforms like Coolify, which regenerate the container's full name (e.g. sodax-stateful-mongo-<stackhash>-<timestamp>) on every redeploy — a hard-coded full name would silently break every docker exec/docker cp after the first redeploy. Set CONTAINER_NAME to the stable prefix (e.g. sodax-stateful-mongo) and the script resolves it to the one running container at run time (once per run, shared by backup/restore/verify). If the prefix matches no running container (nothing up) or more than one (ambiguous), the run aborts with a clear error instead of guessing.

If your .env is saved with Windows (CRLF) line endings, the script strips the trailing carriage return from these values automatically, so a stray \r won't break the container name, port, or S3 path. Only the CR is removed — other whitespace (e.g. inside a password) is preserved as-is.

---

How to Use

The script can be run manually or scheduled to run at regular intervals using cron jobs.

The following commands can be used to run the script manually:

First Make the script executable:

chmod +x mongo_backup_manager.sh

The help command can be used to display the script's usage information:

./mongo_backup_manager.sh help

The backup command can be used to create a backup of the MongoDB database:

./mongo_backup_manager.sh backup

To back up only a subset of collections, pass a comma-separated list (no spaces around the commas):

./mongo_backup_manager.sh backup users,tasks

A single collection is dumped directly with mongodump --collection and needs no mongosh; if the name is misspelled the script detects mongodump's "does not exist" report and errors instead of writing an empty archive. Backing up more than one collection requires mongosh inside the container — mongodump can't include multiple specific collections in one pass, so the script enumerates the database's collections and dumps the whole database minus everything you didn't request (names not present are skipped with a warning, and it errors if none of the requested collections exist).

The list_backups_local command can be used to list all backups stored locally:

./mongo_backup_manager.sh list_backups_local

The list_backups_s3 command can be used to list all backups stored in the S3 bucket:

./mongo_backup_manager.sh list_backups_s3

S3 is off by default (USE_REMOTE=false), so backups stay local. To upload to (and download/list/prune from) S3, set USE_REMOTE=true in .env and provide S3_BUCKET_NAME/AWS_PROFILE.

Downloading a Backup from S3

If you have a backup in the S3 bucket and you want to download it to your local machine, you can use the download_backup command by specifying the backup file name (the backup file name can be obtained from the list_backups_s3 command).

For example, first list the backups in the S3 bucket:

./mongo_backup_manager.sh list_backups_s3
Performing health check...
Health check passed! All required variables are set.
Listing MongoDB backups in S3...
2024-11-20 13:29:01       1383 mongodb-backups/mongo_backup_2024-11-20_13-28-59.gz

Then download the backup file:

./mongo_backup_manager.sh download_backup mongo_backup_2024-11-20_13-28-59.gz
Performing health check...
Health check passed! All required variables are set.
Downloading MongoDB backup from S3...
download: s3://hana-rewards-db-backups/mongodb-backups/mongo_backup_2024-11-20_13-28-59.gz to backups_temp/mongo_backup_2024-11-20_13-28-59.gz
Backup downloaded from S3 successfully.
Download completed successfully.

The backup file will be downloaded to the backups_temp directory.

Restoring a Backup

To restore a backup, you can use the restore_backup command by specifying the backup file name, this backup can either be in the folder with the local backups or if you want to restore a backup from S3, you can download the backup file first and then restore it.

The following command showcases how to restore a backup:

./mongo_backup_manager.sh restore backups_temp/mongo_backup_2024-11-20_13-28-59.gz

To restore only a subset of collections (into MONGO_DB_NAME), pass a comma-separated list (no spaces around the commas) as the single extra argument:

./mongo_backup_manager.sh restore backups_temp/mongo_backup_2024-11-20_13-28-59.gz users,tasks

Only the listed collections are restored from the archive (via mongorestore --nsInclude); --drop only affects the collections being restored, so the rest of the database is left untouched. A single name (e.g. users) restores just that one collection in place.

You can optionally remap a namespace by specifying source database and collection, then destination database and collection (all four arguments are required together):

./mongo_backup_manager.sh restore backups_temp/mongo_backup_2024-11-20_13-28-59.gz new-world users my-app stateful_users

This restores the collection new-world.users from the archive into my-app.stateful_users on the destination. The script will show: Remapping namespace: new-world.users -> my-app.stateful_users. The MongoDB user in .env must have readWrite (or at least listCollections and insert) on the destination database (e.g. my-app), or you will get "Command listCollections requires authentication".

Before copying or restoring anything, restore runs a preflight auth check: it uses mongosh to confirm the configured credentials can reach the destination database (the remap target, or MONGO_DB_NAME for a plain restore) and aborts with a clear message if they can't — so a permissions problem fails fast instead of partway through mongorestore. The preflight is skipped automatically if mongosh isn't available inside the container.

Example output:

Performing health check...
Health check passed! All required variables are set.
Copying MongoDB backup file to the container...
Successfully copied 3.07kB to mongodb-prod:/tmp/mongo_backup_2024-11-20_13-28-59.gz
Restoring MongoDB backup from backups_temp/mongo_backup_2024-11-20_13-28-59.gz...
2024-11-20T13:59:11.468+0000	The --db and --collection flags are deprecated for this use-case; please use --nsInclude instead, i.e. with --nsInclude=${DATABASE}.${COLLECTION}
2024-11-20T13:59:11.485+0000	preparing collections to restore from
2024-11-20T13:59:11.497+0000	reading metadata for test.user_tasks from archive '/tmp/mongo_backup_2024-11-20_13-28-59.gz'
....
....
2024-11-20T14:00:19.170+0000	8 document(s) restored successfully. 0 document(s) failed to restore.
MongoDB restore completed successfully.
Cleaning up temporary backup file in the container...
Restore completed successfully.

Verifying a Restore

After a restore — especially a namespace remap during a migration — use the verify command to prove that the target namespace matches the source rather than trusting that mongorestore recreated everything correctly:

./mongo_backup_manager.sh verify <source_db> <source_collection> <dest_db> <dest_collection>

The argument order mirrors the restore remap, so you can run the same source/dest pair you just restored:

./mongo_backup_manager.sh verify sodax-registration users new-world stateful_users

verify connects to the running MongoDB via mongosh inside the container and compares:

• Document count (countDocuments) of both collections.
• The full index spec of every index — name, key, unique, sparse, partialFilterExpression, collation, and TTL (expireAfterSeconds). The cosmetic v (index version) and ns (namespace, which legitimately differs across DBs) fields are stripped before comparison, and field ordering is normalized so only meaningful differences are reported.

This catches mismatches a names-only check would miss — for example a unique index on stateful_partner_naming.name that lost its sparse: true, which would otherwise pass review and later throw duplicate-key errors on rows whose name is null.

The command exits non-zero on any mismatch, so it can gate a migration script or run in CI. Example output on success:

Performing health check...
Health check passed! All required variables are set.
Verifying sodax-registration.users -> new-world.stateful_users...
source sodax-registration.users: count=8 indexes=3
target new-world.stateful_users: count=8 indexes=3
VERIFY OK: document counts and full index specs match.
Verification passed.

And on a mismatch (note identical counts and index names — only the spec differs):

target new-world.stateful_users: count=8 indexes=3
VERIFY FAILED:
  - index on source missing/differs on target: {"key":{"name":1},"name":"name_1","sparse":true,"unique":true}
  - index on target absent/differs on source: {"key":{"name":1},"name":"name_1","unique":true}
Verification failed.

The MongoDB user in .env needs read access (e.g. read) on both the source and destination databases. verify requires mongosh to be available inside the container (included in the official mongo images from 5.0+).

Scheduling automated backups

The backup command runs to completion and exits non-zero on failure, so it schedules cleanly. Two options; a systemd timer is recommended on Linux hosts.

Option A — systemd timer (recommended)

Ready-to-edit unit files live in systemd/. They run backup on a schedule, order after Docker and the network, and send output to the journal.

1. Put the script somewhere stable (e.g. /opt/mongo-backup) with its .env beside it, then edit the marked lines in systemd/mongo-backup.service — User/Group, WorkingDirectory, and ExecStart — so they point at that location and at a user who can reach Docker and your AWS profile.
2. Install and enable. Copy the alert unit too and edit its placeholders — mongo-backup.service references it via OnFailure=, so if it isn't installed (or is left with the placeholder User=youruser/paths) a failed backup adds a confusing secondary error. Once it's installed and configured it's a no-op until you set USE_ALERTS=true (see Failure alerting). If you don't want alerting at all, remove the OnFailure= line from mongo-backup.service instead.

   sudo cp systemd/mongo-backup.{service,timer} systemd/mongo-backup-alert.service /etc/systemd/system/
   sudo systemctl daemon-reload
   sudo systemctl enable --now mongo-backup.timer

3. Verify:

   systemctl list-timers mongo-backup.timer   # next / last run
   sudo systemctl start mongo-backup.service  # run one backup right now
   journalctl -u mongo-backup.service -e      # read its output

The default schedule is daily at 03:00 (OnCalendar in mongo-backup.timer); change it to hourly or any OnCalendar expression. Persistent=true means a run missed while the machine was off fires at the next boot.

Option B — cron

crontab -e

# docker/aws/mongosh are not on cron's minimal PATH — set one explicitly.
PATH=/usr/local/bin:/usr/bin:/bin
# Daily at 03:00. Pass the `backup` command (the whole point) and redirect output
# so a failure is captured somewhere you can read it.
0 3 * * * /path/to/mongo_backup_manager.sh backup >> /var/log/mongo_backup.log 2>&1

The cron user must be able to run docker (member of the docker group, or root) and — since the script always calls aws with --profile "$AWS_PROFILE" (and health_check requires AWS_PROFILE) — must own the ~/.aws profile named by AWS_PROFILE in .env. A named profile is required even on EC2: to use an instance role, point the profile at it (credential_source = Ec2InstanceMetadata in ~/.aws/config) rather than omitting the profile.

Retention

Every backup run prunes old backups from both locations:

• Local — find -mtime +7 removes *.gz files older than 7 days under backups/ (fixed at 7 days).
• S3 — objects under the mongodb-backups/ prefix whose embedded timestamp is older than S3_RETENTION_DAYS (default 7) are deleted (requires s3:ListBucket + s3:DeleteObject, both in the IAM policy above). This is best-effort: a transient S3 error warns but does not fail an otherwise-successful backup.

For defence-in-depth you can also add an S3 lifecycle rule on the mongodb-backups/ prefix to expire (or transition to Glacier) old objects, so retention still happens even if a scheduled run is skipped for a long stretch.

Letting a lifecycle policy own S3 retention: set S3_RETENTION_DAYS=0 to skip the tool's S3 prune entirely and make an S3 lifecycle rule the single source of truth for how long backups live. Without this, the tool's prune competes with the lifecycle rule — e.g. a 30-day lifecycle rule never sees anything older than 7 days because the tool already deleted it on the previous run. Use 0 when you want longer (or tiered) S3 retention managed by the bucket, and keep the default 7 when you want the tool to handle it.

Failure alerting

An unattended backup runs under set -e and exits non-zero the moment either the mongodump or the aws s3 cp upload fails, so a single systemd OnFailure= hook catches both. The companion unit systemd/mongo-backup-alert.service is a oneshot that runs mongo_backup_manager.sh alert mongo-backup.service: it tails the failed run's journal — which carries >>> STAGE: … markers and a Run FAILED during stage: … line — and posts it to Discord and/or Telegram, so the alert tells you which stage died (mongodump vs the S3 upload vs a prune), not merely that the backup failed.

mongo-backup.service already ships with OnFailure=mongo-backup-alert.service enabled. Once the alert unit is installed and its placeholders are edited (step 1 below), alerting stays dormant — a backup failure starts the alert unit, which just logs "alerting disabled" and exits 0 — until you set USE_ALERTS=true. Note the no-op only holds in that installed-and-configured state: if the alert unit is missing, or copied but left with the placeholder User=youruser/paths, a backup failure will make OnFailure= itself error (secondary "unit not found" or a failed start) on top of the real failure. To opt in:

1. Install the alert unit next to the backup units:

   sudo cp systemd/mongo-backup-alert.service /etc/systemd/system/
   sudo systemctl daemon-reload

   Edit its WorkingDirectory/ExecStart/User/Group to match mongo-backup.service. It runs as the same unprivileged user as the backup (not root) — the script sources .env, so running as root while .env is writable by the backup user would be a local privilege-escalation vector. Reading the backup unit's journal as that non-root user requires adding it to the systemd-journal group: sudo usermod -aG systemd-journal youruser. (As a backstop, the script also refuses to source a non-root-owned or group/world-writable .env when it is run as root.)
2. Enable alerting in the same .env the backup uses (real values live only on the host — never commit them):

   USE_ALERTS=true
   DISCORD_WEBHOOK_URL=https://discord.com/api/webhooks/<id>/<token>   # and/or
   TELEGRAM_BOT_TOKEN=123456:AA...
   TELEGRAM_CHAT_ID=-1001234567890

   Configure Discord, Telegram, or both. health_check validates the config when USE_ALERTS=true: it rejects a half-configured Telegram (only one of the two vars) and rejects USE_ALERTS=true with no destination at all, so a typo can't silently leave you with no safety net.
3. Test it (posts a "no journal output" style message when the backup unit hasn't actually failed):

   sudo systemctl start mongo-backup-alert.service
   journalctl -u mongo-backup-alert.service -e

Alerting is best-effort: a failed webhook/API call warns but never changes the run's exit status or masks the real backup failure (which systemd has already recorded), and each destination is attempted independently so one being down doesn't suppress the other. To opt out entirely, leave USE_ALERTS=false (the default) or remove the OnFailure= line from mongo-backup.service.

Out of scope: this only alerts on a run that ran and failed. A run that never fired (cron/box down) can't alert on itself — that "did it even run?" freshness check belongs in an external watcher, tracked separately.