Bash snippets | infrascribbles

Defaults:

Start every script with set -euo pipefail.
Quote every variable and declare function variables local.
Set a timeout on every network call.
Clean up with a trap ... EXIT, and make scripts safe to run twice.
Lint every script with shellcheck in CI.
Move to Python or Go once a script needs data structures, real error handling or tests.

Safe script header

#!/usr/bin/env bash
set -euo pipefail

Flag	Effect
`-e`	Stop on the first failing command.
`-u`	Treat an unset variable as an error (catches typos like `$NAEM`).
`-o pipefail`	Fail a pipeline if any part fails, not just the last.

set -e does not trigger inside if conditions or with ||. Use them to handle an expected failure.

Script template

Start real scripts from this, not from a blank file: usage text, option parsing, timestamped logs on stderr, a dry-run mode, input validation and guaranteed cleanup.

#!/usr/bin/env bash
# rotate-backups.sh - delete backups older than N days from a directory.
set -euo pipefail

readonly SCRIPT=${0##*/}
DRY_RUN=0
VERBOSE=0
DAYS=30
TMP_DIR=""

# Global, not local: the EXIT trap runs after main has returned
cleanup() {
  local rc=$?
  if [[ -n $TMP_DIR ]]; then rm -rf "$TMP_DIR" || true; fi   # set -e still applies here
  exit "$rc"   # keep the script's exit code, whatever cleanup did
}
trap cleanup EXIT   # runs on success, failure and Ctrl+C

usage() {
  cat <<USAGE
Usage: $SCRIPT [-n] [-v] [-d DAYS] DIR

Delete *.tar.gz files older than DAYS (default $DAYS) from DIR.

  -n       dry run: print what would happen, change nothing
  -v       verbose logging
  -d DAYS  age threshold in days
  -h       show this help
USAGE
}

log()  { printf '%s %s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$*" >&2; }
vlog() { if [[ $VERBOSE -eq 1 ]]; then log "$@"; fi; }
die()  { log "error: $*"; exit 1; }

# run prints the command in dry-run mode, otherwise executes it
run() {
  if [[ $DRY_RUN -eq 1 ]]; then
    log "dry-run: $*"
  else
    "$@"
  fi
}

main() {
  while getopts ':nvd:h' opt; do
    case $opt in
      n) DRY_RUN=1 ;;
      v) VERBOSE=1 ;;
      d) DAYS=$OPTARG ;;
      h) usage; exit 0 ;;
      :) usage >&2; die "-$OPTARG needs a value" ;;
      *) usage >&2; die "unknown option -$OPTARG" ;;
    esac
  done
  shift $((OPTIND - 1))

  [[ $# -eq 1 ]] || { usage >&2; exit 2; }
  local dir=$1
  [[ -d $dir ]] || die "not a directory: $dir"
  [[ $DAYS =~ ^[0-9]+$ ]] || die "-d must be a whole number, got: $DAYS"

  TMP_DIR=$(mktemp -d)
  find "$dir" -maxdepth 1 -name '*.tar.gz' -mtime "+$DAYS" -print0 > "$TMP_DIR/old"
  local count=0
  while IFS= read -r -d '' f; do
    vlog "deleting $f"
    run rm -f -- "$f"
    count=$((count + 1))
  done < "$TMP_DIR/old"
  log "done: $count file(s) older than $DAYS days"
}

main "$@"

The EXIT trap runs after main returns, so anything it touches must be global - a local variable is out of scope and set -u fails the run.
set -e still applies inside the trap. Guard cleanup commands with || true, and end with exit "$rc" so cleanup can’t change the script’s exit code.
Exit codes: 0 success, 1 failure, 2 bad usage.
find -print0 with read -d '' handles any filename, including spaces and newlines.

Log one-liners

For lines like 2026-10-13T14:02:11Z service=payments status=200 latency_ms=134:

# Requests per service, most first
grep -o 'service=[a-z]*' app.log | sort | uniq -c | sort -rn

# 5xx errors per service (the numeric check skips status=abc)
awk '{for(i=1;i<=NF;i++){split($i,a,"="); f[a[1]]=a[2]}
      if (f["status"] ~ /^[0-9]+$/ && f["status"]+0 >= 500) c[f["service"]]++; delete f}
     END{for(s in c) print c[s], s}' app.log

# Three slowest requests
grep -o 'latency_ms=[0-9]*' app.log | cut -d= -f2 | sort -n | tail -3

# Follow a log live, errors only
tail -f app.log | grep --line-buffered 'status=5'

Retry with exponential backoff

Copy the structure: local variables, quoting, a usage message with exit code 2, errors to stderr, a network timeout.

#!/usr/bin/env bash
set -euo pipefail

retry() {
  local max=$1; shift
  local delay=1
  for ((i = 1; i <= max; i++)); do
    if "$@"; then
      return 0
    fi
    echo "attempt $i/$max failed: $*" >&2
    sleep "$delay"
    delay=$((delay * 2))
  done
  return 1
}

check() {
  local url=$1
  local code
  code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 5 "$url") || code=000
  [[ $code =~ ^[23] ]]
}

main() {
  if [[ $# -lt 1 ]]; then
    echo "usage: $0 URL..." >&2
    exit 2
  fi
  local failed=0
  for url in "$@"; do
    if retry 3 check "$url"; then
      echo "OK   $url"
    else
      echo "FAIL $url"
      failed=1
    fi
  done
  exit "$failed"
}

main "$@"

Wait until something is ready

Never sleep 30 and hope. Poll with a deadline, and fail loudly when it passes.

# wait_for SECONDS COMMAND... - retry COMMAND every 2s until it succeeds or time runs out
wait_for() {
  local deadline=$((SECONDS + $1)); shift
  until "$@" >/dev/null 2>&1; do
    if ((SECONDS >= deadline)); then
      echo "timed out waiting for: $*" >&2
      return 1
    fi
    sleep 2
  done
}

# TCP port open? Uses bash's /dev/tcp, no nc needed
port_open() { timeout 3 bash -c "</dev/tcp/$1/$2" 2>/dev/null; }

wait_for 120 curl -fsS http://localhost:8080/healthz
wait_for 60 port_open db.internal 5432

Run commands in parallel

xargs -P caps concurrency without a job-control loop. Export the function so the child shells can see it.

check_url() {
  local code
  code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 5 "$1") || code=000
  echo "$code $1"
}
export -f check_url

xargs -P 8 -I{} bash -c 'check_url "$1"' _ {} < urls.txt | sort

Pass the value as $1 (the _ fills $0), never by pasting {} into the command string - that is a shell injection waiting for a URL with a quote in it.

Write files atomically

Write to a temp file in the same directory, then mv. A rename on the same filesystem is atomic: readers see the old file or the new one, never half a file.

write_atomic() {
  local dest=$1 tmp
  tmp=$(mktemp "$(dirname "$dest")/.${dest##*/}.XXXXXX")  # same filesystem as dest
  if cat > "$tmp"; then
    mv -f "$tmp" "$dest"   # rename is atomic: readers see old or new, never half
  else
    rm -f "$tmp"
    return 1
  fi
}

render_config | write_atomic /etc/app/config.yaml

Prevent overlapping runs

A cron job that runs longer than its interval will overlap itself. Take a lock and skip if it’s held. flock is Linux (util-linux).

exec 9>/var/lock/rotate-backups.lock
if ! flock -n 9; then
  echo "already running, skipping" >&2
  exit 0
fi

Or from crontab, without touching the script:

*/15 * * * * flock -n /var/lock/backup.lock /usr/local/bin/backup.sh

jq for kubectl output

Pipe kubectl get ... -o json into these. @base64d needs jq 1.6 or newer.

# Pods not Running, or with a container not ready
kubectl get pods -A -o json | jq -r '.items[] | select(.status.phase != "Running" or any(.status.containerStatuses[]?; .ready | not)) | "\(.metadata.namespace)/\(.metadata.name) \(.status.phase)"'

# Containers restarted more than 3 times, with the last termination reason
kubectl get pods -A -o json | jq -r '.items[] | .metadata as $m | .status.containerStatuses[]? | select(.restartCount > 3) | "\($m.namespace)/\($m.name) \(.name) restarts=\(.restartCount) last=\(.lastState.terminated.reason // "-")"'

# Every image running in the cluster
kubectl get pods -A -o json | jq -r '[.items[].spec.containers[].image] | unique[]'

# Containers without a memory limit
kubectl get pods -A -o json | jq -r '.items[] | .metadata as $m | .spec.containers[] | select(.resources.limits.memory == null) | "\($m.namespace)/\($m.name) \(.name)"'

# Pods per node
kubectl get pods -A -o json | jq -r '.items | group_by(.spec.nodeName)[] | "\(length) \(.[0].spec.nodeName)"'

# Decode a whole Secret
kubectl get secret app-db -o json | jq '.data | map_values(@base64d)'

Decoding a Secret prints credentials to your terminal and scrollback. Don’t do it on a shared screen, and never paste the output into tickets.

kubectl one-liners

kubectl rollout status deploy/api --timeout=5m        # wait for a rollout; non-zero if it fails
kubectl rollout undo deploy/api                       # back to the previous ReplicaSet
kubectl rollout restart deploy/api                    # restart pods without changing the spec
kubectl logs -l app=api --prefix --tail=50            # recent lines from every matching pod
kubectl logs api-7c9f8-xk2lp --previous               # logs from the crashed container
kubectl get events -n pay --field-selector involvedObject.name=api-7c9f8-xk2lp
kubectl top pods -A --sort-by=memory | head           # needs metrics-server
kubectl debug -it api-7c9f8-xk2lp --image=nicolaka/netshoot --target=app   # debug container sharing the app's processes
kubectl port-forward svc/api 8080:80                  # reach a Service from your laptop
kubectl auth can-i create deployments -n pay --as=system:serviceaccount:pay:ci   # test RBAC as someone else
kubectl get pods -A -o wide --field-selector spec.nodeName=NODE                  # what runs on a node
kubectl drain NODE --ignore-daemonsets --delete-emptydir-data                    # empty a node for maintenance

Disk and process triage

GNU/Linux flags; on macOS use du -d 1 and drop --sort.

df -h -x tmpfs -x overlay                               # real filesystems only
du -xh --max-depth=1 / 2>/dev/null | sort -h | tail     # biggest directories, one filesystem
find /var -xdev -type f -size +500M -exec ls -lh {} + 2>/dev/null   # files over 500 MB
lsof +L1                                                # deleted files still held open
ps aux --sort=-%mem | head                              # top memory users
dmesg -T | grep -i -E 'killed process|out of memory'    # kernel OOM kills
ss -tan state established '( dport = :5432 )' | wc -l   # open connections to Postgres

df says the disk is full but du can’t find the files? A process still has deleted files open. lsof +L1 shows which one; the space comes back only when it closes them or restarts.

Certificate and port checks

# Subject and expiry date
echo | openssl s_client -connect example.com:443 -servername example.com 2>/dev/null \
  | openssl x509 -noout -subject -enddate

# Exit non-zero if it expires within 30 days - drop into cron or CI
echo | openssl s_client -connect example.com:443 -servername example.com 2>/dev/null \
  | openssl x509 -noout -checkend $((30 * 24 * 3600))

# Hit one backend directly, bypassing DNS, with the right SNI and Host
curl -sv --resolve api.example.com:443:10.0.3.17 https://api.example.com/healthz

# Is a TCP port open? No nc needed
timeout 3 bash -c '</dev/tcp/db.internal/5432' && echo open

Always pass -servername to s_client. Without it, servers hosting many certificates return their default one, and you check the wrong certificate.

Troubleshooting commands

ss -tlnp                                                     # what's listening, and which process
dig +short example.com                                       # DNS
curl -sv https://example.com                                 # TCP, TLS and HTTP in one
curl -s -o /dev/null -w '%{http_code} %{time_total}\n' URL   # status and timing only
openssl s_client -connect host:443 -servername host          # certificate chain
df -h; du -sh /var/log/*                                     # disk
free -m                                                      # memory
journalctl -u nginx --since '10 min ago'                     # service logs
kubectl get pods -A | grep -v Running                        # pods that aren't running
kubectl get events -A --sort-by=.lastTimestamp | tail        # recent cluster events

Mistakes to avoid

Unquoted variables. rm -rf $DIR/ with an empty DIR becomes rm -rf /. Quote everything, and use ${DIR:?} to fail when it is empty.
Parsing ls. Its output breaks on filenames with spaces. Use globs (for f in *.log) or find -print0 | xargs -0.
Secrets as arguments. Anyone running ps can see them. Pass them through environment variables or files.
No timeouts. curl without --max-time can hang forever. Always pass --max-time.
local x=$(cmd) hides failures. local returns its own status, so set -e never sees cmd fail. Declare first, assign second: local x; x=$(cmd).
Piping into while read loses variables. The loop runs in a subshell, so counters reset when it ends. Feed it with done < file or done < <(cmd) instead.
read without -r mangles backslashes. Always while IFS= read -r line.
Pasting {} into bash -c strings. Input containing quotes becomes code. Pass values as arguments: bash -c 'cmd "$1"' _ {}.