Backup jobs fail in two very different ways. They fail loudly — the job errors, you get a notification, you fix it. Or they fail quietly: the job exits 0, writes a file, and the file is useless. The second kind is the one that ends careers.
Exit codes tell you that a command completed. They do not tell you that the archive opens, that the database dump replays, or that the snapshot contains the volume you thought it contained.
# verify rather than assume tar -tzf backup.tar.gz > /dev/null || echo "archive is corrupt" pg_restore --list dump.pgc > /dev/null || echo "dump is unreadable"
The practice that actually holds up is a scheduled restore into a throwaway namespace, followed by a smoke query against the restored data. If that runs weekly and pages you when it fails, your backups are evidence rather than hope.
Retention is a separate conversation. Keeping fourteen daily copies is a backup policy; deciding that you can survive losing the last six hours is a recovery policy. Write both down, because during an incident nobody wants to be deriving them from first principles.