This method involves an incremental backup bash script that runs on a production server. The aim is to automatically build a local backup which can then be synchronised with a remote backup server.
Production Server: Backup User
Open a command prompt and type rclone config. We will start by creating a “backblaze” container pointing to our B2 bucket. Select ‘n’ for a new remote and name it “backblaze”. Use the bucket applicationKeyId as the account and the Application Key itself as the key. The Backblaze Personal Backup client was started in January of 2007 as a ‘C’ and C source tree that compiled on Windows, Macintosh, and Linux (originally CentOS but later when we standardized on Debian for the server side we now compile the client on Debian).
Create a backup user - without sudo privileges:
Production Server: Source Directory
Create a directory called
/home/backupuser/source. This will be used as the source for the local rsync, so it forms a target for:
- a MySQL backup - mysqldump dumps database copies to
- Symlinks to config files (in
- Symlinks to site files (from the Apache root directory -
Set up directories and add symlinks to
Production Server: Incremental Backup
Incremental backup script:
- Add the incremental backup script to
- Make executable
- Set up cronjob
The delete option causes files that are not in source to be deleted from the target - and ONLY the target:
This tells rsync to delete extraneous files from the receiving side (ones that aren’t on the sending side), but only for the directories that are being synchronized. You must have asked rsync to send the whole directory … Files that are excluded from the transfer are also excluded from being deleted unless you use the –delete-excluded option or mark the rules as only matching on the sending side
I originally used the
--delete flag on this script - thinking that it would be necessary to keep the latest incremental backup synchronised with the current source filesystem.
Backblaze B2 Rsync
--delete has no effect when syncing into an empty directory, which is basically what happens when using the
--link-dest method of rsync. The directory specified by
--link-dest is used as a reference point, and any files in source that are unchanged relative to this reference point are hardlinked to their exisiting inode in the reference directory.
Files that are not in the source directory (because they have been deleted) are therefore not hardlinked in the backup directory.
However there is a possibility that rsync is backing up to an incomplete directory that is left over from a previous failed run, and may contain things to delete.
--delete is probably superfluous when using
--link-dest. The backup directory accurately reflects the source directory. However, we leave it in place in case of backing up to an incomplete directory.
Note: Be careful deleting old snapshots. The old directory in it’s entirety should be properly archived and a new full backup snapshot should be taken to kick off the next round of incremental backups.
This warning is not necessary as using the
--link-dest option creates hard links. Thanks to aselinux for pointing this out in the comments.
This will result in a backup directory on the production server,
/home/backupuser/backup, that contains dated incremental backups.
The Production backup directory can then be targeted by a Backup server, again via rsync. Typically, this might involve:
- Access to the backup directory by means of SSH public/private key pair
- rrsync access for the backupuser - only rsync can run on the given SSH key, restricted to read-only
- Setup Secure rsync link between servers
- Run a cronjob on the backup server to read and rsync the backup directory on the Production server
- Time machine/rsync with good illustration of hard links - good sample “instant backup” script - backup every minute!
- Link Dest behaviour explation: answer in the rsync mailing list archive
- Link Dest and deleting old snapshots