How I back up my servers using restic and Wasabi object storage

ansiblebackupfree-softwarehomelablinuxsysadmin
How I back up my servers using restic and Wasabi object storage
Table of Contents

For the past few months, I have been using Borg to backup my servers. It was working great and was pretty reliable, but a bit complicated.

My previous setup: SSH + Rsync + Borg#

Here's the setup:

  • The backup server rsync files to its local drive
  • The backup server makes databases dumps over SSH to its local drive
  • Then it uses Borg via borgmatic to store all the files in a tidy way.

This is quite simple, but it took me a lot of work to get good bash scripts and YAML borgmatic files (example). Borg is great, it handles deduplication, compression, encryption...

I could've used Borg in push mode (uploading to the backup server via Borg), but its append-only mode does not make any sense, so I decided to use a pull mode. Thus, I did not use encryption since the key would be on the backup server anyway.

My backup server was a dedicated server from Kimsufi, which costed me about 10€/month for a 2 TB hard drive. It was a good deal but the Atom CPU was very weak and Borg does not handle multi-threading so it was quite slow. Also, the downside of having one, big hard drive is the remaining unused space and the lack of redundancy. For the latter, I rsynced the borg repositories to one of my computer at home, but...

Anyway, this setup was not so bad, but in terms of speed and redundancy, I could find better.

The choice of the backup tool#

I decided that I would backup my servers to an S3-compatible storage provider, in push mode, so I began to search for Borg alternatives that supported this kind of backend.

I think the best one is Duplicacy, a software written in Go, because it's the fastest and more mature solution. They have a detailed README explaining why they're the best.

However it has 2 downsides for me: first, it's open-source, but not under a Free license. I'm okay with this, but it's worth to be noted. Second, it does not handle sdtin backup (for dumps) - at least I couldn't find how.

This is a deal breaker for me, since I want to use a push mode. The best alternative I found was restic, also written in Go. It's nearly as fast a Duplicacy and support stdin backups. The biggest downside is that it doesn't support compression yet, but it will do for now.

See gilbertchen/benchmarking for some detailed performance benchmarks.

restic handles encryption very easily and this is a very important aspect because, when I had my backup server, I could have used some encryption but at leat I had control over my server. Using an Object Storage Provider means I don't know how my data will be handled so encryption is mandatory.

The choice of the storage provider#

A few weeks ago, I was moving my Mastodon media files to Wasabi, a cheap AWS S3-like service. I really like the service so I began considering using it for my backups.

Since all my backup can be contained in less than 1 TB, I will still be paying $5/month along with my mastodon stuff. That's awesome!

wasabi pricing

Wasabi is not only cheap but fast. Though, I'm in Europe and their datacenters are in the US. But I know they are planning to get in Europe this year, so I can only look forward to it!

Spaces is the fastest but Wasabi is not far behind

Setting up Wasabi#

Now that I have you convinced that this is best combo ever, let's get started by setting up Wasabi.

Create an account#

You can sign up for free on Wasabi and get a 1 TB free trial for 30 days.

Then you're going to need 2 things:

  • A bucket
  • A user

Creating a bucket#

You should be able to do that by yourself. Be aware that you have to choose a unique bucket name.

Creating a user with a correct policy#

Then, create a user, name it however you want, and get API keys. I recommend to store them in your password manager.

Then we'll need to create and attach a policy for this user in order to give it full access to the bucket while restraining it to this bucket only.
Here's the one I use, my-backup-bucket being the name of my bucket.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::my-backup-bucket"
    },
    {
      "Effect": "Allow",
      "Action": ["s3:PutObject", "s3:GetObject", "s3:DeleteObject"],
      "Resource": "arn:aws:s3:::my-backup-bucket/*"
    }
  ]
}

Then go to the "Permissions" tab for your user and attach the policy.

We're good to go!

Setting up restic#

Installing restic#

See the documentation for installing restic. All my servers are running Debian stretch but the version in the repos is quite old.

In order to get the latest version, I install it from the testing/sid repository. It doesn't cause any issue since the only dependency of restic is libc.

First, add the sid repo in /etc/apt/sources.list.d/sid.list :

deb http://deb.debian.org/debian sid main

Then add the pin in /etc/apt/preferences.d/restic :

Package: *
Pin: release n=stretch
Pin-Priority: 990

Package: restic
Pin: release n=sid
Pin-Priority: 1000

Package: *
Pin: release n=sid
Pin-Priority: -1

Run apt-update and you should get:

root@server:~# apt-cache policy restic
restic:
  Installed: (none)
  Candidate = 0.9.1+ds-1
  Version table:
     0.9.1+ds-1 1000
         -1 http://deb.debian.org/debian sid/main amd64 Packages
     0.3.3-1+b2 990
        990 http://http.us.debian.org/debian stretch/main amd64 Packages

You can now proceed to install the package.

Setting up the keys#

I have a ~/.restic-keys on all my servers containing:

export AWS_ACCESS_KEY_ID=
export AWS_SECRET_ACCESS_KEY=
export RESTIC_PASSWORD=<some long random password>

The first two variables correspond to your Wasabi Access and Secret keys. The RESTIC_PASSWORD will be used to encrypt the backups.

You can now do a source ~/.restic-keys to export the keys to your shell.

Create a repository#

root@server:~# restic init --repo s3:s3.wasabisys.com/my-backup-bucket
created restic repository e9ed581d94 at s3:s3.wasabisys.com/my-backup-bucket

Please note that knowledge of your password is required to access
the repository. Losing your password means that your data is
irrecoverably lost.

Backup a folder#

root@server:~# restic -r s3:s3.wasabisys.com/my-backup-bucket backup /etc/apt/
repository e9ed581d opened successfully, password is correct

Files: 19 new, 0 changed, 0 unmodified
Dirs: 1 new, 0 changed, 0 unmodified
Added: 40.609 KiB

processed 19 files, 39.960 KiB in 0:04
snapshot bb3030dc saved

See the deduplication at work:

root@server:~# restic -r s3:s3.wasabisys.com/my-backup-bucket backup /etc/apt/
repository e9ed581d opened successfully, password is correct

Files: 0 new, 0 changed, 19 unmodified
Dirs: 0 new, 0 changed, 1 unmodified
Added: 0 B

processed 19 files, 39.960 KiB in 0:03
snapshot c577544a saved

List snapshots#

root@server:~# restic -r s3:s3.wasabisys.com/my-backup-bucket snapshots
repository e9ed581d opened successfully, password is correct
ID Date Host Tags Directory
----------------------------------------------------------------------
bb3030dc 2018-07-29 19:01:48 server.guest /etc/apt
c577544a 2018-07-29 19:02:16 server.guest /etc/apt
----------------------------------------------------------------------
2 snapshots

Do many others things#

I won't cover every feature here so please take a look at the docs!

Some interesting stuff:

By the way, deduplication does work with stdin backups!

Automatise the backups using bash and cron#

On my servers, I have a backup script that looks like this:

#!/bin/bash

source .restic-keys
export RESTIC_REPOSITORY="s3:s3.wasabisys.com/{{ ansible_hostname }}-backup"

echo -e "\n`date` - Starting backup...\n"

restic backup /etc
restic backup /root --exclude .cache --exclude .local
restic backup /home/stanislas --exclude .cache --exclude .local
restic backup /var/log
restic backup /srv/some-website

mysqldump database | restic backup --stdin --stdin-filename database.sql

echo -e "\n`date` - Running forget and prune...\n"

restic forget --prune --keep-daily 7 --keep-weekly 4 --keep-monthly 12

echo -e "\n`date` - Backup finished.\n"

Easy right?

I execute the script at night with some nice and ionice:

0 4 * * * ionice -c2 -n7 nice -n19 bash /root/backup.sh > /var/log/backup.log 2>&1

And we're done!

Enjoy#

I think I have covered everything you need to know about my current backup setup. restic is an awesome software and Wasabi is fast and reliable.

Here are some numbers: one server with ~3 GB of files and databases is backed up in one minute, and another server with ~50 GB in about 10 minutes. I just wish there was compression!

Here is the improvements I can think of for this setup :

Stanislas
Author
Stanislas
I like building things with code and computers

Related Posts

Monitoring with Prometheus
Monitoring with Prometheus
Using a Hetzner Storage Box as a restic backend for backups
Using a Hetzner Storage Box as a restic backend for backups
Object storage: migrating from Wasabi to Scaleway with rclone

Comments

14Atom feed
Markdown supported
  1. @kicou

    Nice, I was looking for a way to back up my personal NextCloud server (~1TB of personal files, family pictures, videos) and I think this is a solution worth investigating.

    Found the blog via Mastodon BTW :)

    1. Anonymous

      Hi! Any update on backing up NextCloud to Wasabi?

    3. Anonymous

      Hey! Thanks! Any further hints beyond this article? Is there a direct writeup somewhere?

    4. @kicou

      For those interested, I have been running this for 4 months now and I am very happy with it.

      The VPS that runs Nextcloud only has 2 GB of RAM, and because we are dealing with many, many files, restic would randomly die because it would have exhausted RAM. To work around this issue I run restic by prepending GOGC=20 to the restic command in order to force earlier garbage collection. This seems to have fixed my memory issues and things have been running smoothly.

  2. @taoeffect

    Hmm, very cool! Thank you so much! That was very informative.

    Here are some thoughts:

    • I really like how in your original setup you were doing pull-based backups. This way if any of your non-backup servers get compromised, you don't have to worry about your backups being deleted.

    • According to the table in Duplicacy's README, restic doesn't support deletion..? Is that not correct (since your script has pruning? Do you know what they're talking about?).

    • Is stdin support really so critical? Can't you mysqldump to a file and have Duplicacy backup that file?

    • I'm not an expert on open source licenses, but Duplicacy's license seems pretty "free"/"libre" to me, and I wouldn't be surprised if it passed the "open source" definition. They give you full access to the source, and only ask that you pay if you use it commercially.

    Duplicacy's docs say this about restic:

    restic is a more recent addition. It uses a format similar to the git packfile format.
    Multiple clients backing up to the same storage are still guarded by locks, and because a chunk database is used, deduplication isn't real-time (different clients sharing the same files will upload different copies of the same chunks). A prune operation will completely block all other clients connected to the storage from doing their regular backups.
    Moreover, since most cloud storage services do not provide a locking service, the best effort is to use some basic file operations to simulate a lock, but distributed locking is known to be a hard problem and it is unclear how reliable restic's lock implementation is. A faulty implementation may cause a prune operation to accidentally delete data still in use, resulting in unrecoverable data loss. This is the exact problem that we avoided by taking the lock-free approach.

    So I think I will give Duplicacy a try. :)

    1. @taoeffect

      Another thought: even if Duplicacy can't do pull-based backups, you can use it to push to a backup server, and then backup the backup server with a pull-based backup :D

    2. StanislasAuthor
      • My new setup allows me to do push-based backup without worrying about the backup server being compromised. More info about this in a future blog post :)
      • That's weird indeed... The restic forget can delete a snapshot (if you prune afterwards), so I'm not sure what else they want to delete.
      • Yes, I can't live without this. A few years ago, I was dumping the DBs to disk and they I backed them up. It was fine when I had DBs in the size of megabytes, but for instance the mstdn.io DB is 40 GB. I don't have the space on the VM to do that and it would really affect I/O on the server.
      • Yeah, I don't really mind about their license.
  3. Tom

    How about borg with rclone?

  4. nexar

    Stanislas thanks very much for a very detailed article which explains things very clearly.

    However your [Edit 2020 .... right at the beginning of the article is worrying. I am still within my trial period with Wasabi so can back out.

    You however don't say what the 'bad surprises' are. Please would you provide some extra information.

    Thanks

    1. StanislasAuthor

      Wasabi will bill you 3 months for each object. Restic overwrites objects a lot, so you'll end up billed primarily for deleted storage.

  5. nexar

    @Stanislas I posted yesterday regarding your [Edit 2020.... comment at the start of this article. However I don't see it appearing here. Perhaps a Medium issue.

    I just wanted to say that I enjoyed your detailed article and was planning on starting with Wasabi. However your comment makes me cautious and I would be grateful if you would perhaps explain what sort of 'bad surprises' you got. Thanks

  6. Kristin

    Thanks for the great article!

    May you please explain this? Edit (2020): I highly discourage using Wasabi. They have a very misleading pricing policy and you will end up with bad surprises on your invoices at the end of the month.

    I was thinking about using Wasabi because I hoped they had a simpler pricing than AWS, Azure, Google Cloud or DigitalOcean but now I am not sure anymore...

↑