Everybody knows that backups are essential, but getting them set up properly often seems like a tedious task. Well, the good news is that it doesn't have to be! In a cloud environment like cloud.ca, you have all the building blocks to make it a simple process.In this post we are going to explain how to use the open-source Duplicity tool to reliably back up and restore data folders to and from your object storage environments. This is a cost-effective option because object storage is our lowest-cost storage, yet it provides the high-redundancy you are looking for when preserving important data.
1) Create the destination for your backup
I like to create a dedicated object storage environment for my backups — it keeps things tidy and organized. Click on Configuration > Environments, then click on Add Environment and complete this form:
Then, from cloud.ca's Services tab, select your newly created Backups environment and add a bucket that will receive your backup data. I suggest creating a distinct bucket for each server you want to backup, but that is not mandatory.
2) Install and configure Duplicity
Then we will proceed with the installation of the Duplicity backup software.
On CentOS 7:
sudo su
rpm -iUvh http://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-5.noarch.rpm
yum -y install python-pip python-devel gnupg2 duplicity gcc
pip install python-swiftclient python-keystoneclient
On Ubuntu 14.04:
sudo su
add-apt-repository ppa:duplicity-team/ppa
apt-get update
apt-get install duplicity python-pip gnupg2 python-dev
pip install python-swiftclient python-keystoneclient==1.8.1 --upgrade retrying
The next step is to configure the environment variables that tells Duplicity how to connect with cloud.ca's Swift-based object storage. You can retrieve the required information from cloud.ca's API Keys option in the pull-down menu under your username:
Then create the following environment variables from this information:
export SWIFT_USERNAME="marcv-backups:marcv-marc-1234"
export SWIFT_PASSWORD="dummyPassword"
export SWIFT_AUTHURL="https://auth-east.cloud.ca/v2.0"
export SWIFT_AUTHVERSION="2"
3) Backup process
Now that everything is properly configured, you are ready to backup any desired source folder, for example the content of /var/log :
duplicity --no-encryption /var/log swift://server1-backup
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: none
No signatures found, switching to full backup.
--------------[ Backup Statistics ]--------------
StartTime 1449178551.72 (Thu Dec 3 21:35:51 2015)
EndTime 1449178558.14 (Thu Dec 3 21:35:58 2015)
ElapsedTime 6.42 (6.42 seconds)
SourceFiles 42
SourceFileSize 285464347 (272 MB)
NewFiles 42
NewFileSize 285462938 (272 MB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 42
RawDeltaSize 285439771 (272 MB)
TotalDestinationSizeChange 10609994 (10.1 MB)
Errors 0
-------------------------------------------------
That's all there is to it! You can also list the files included in your backup:
duplicity list-current-files swift://server1-backup
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Thu Dec 3 21:35:51 2015
Thu Dec 3 21:20:34 2015 .
Tue Mar 3 21:11:18 2015 anaconda
Thu Dec 3 19:08:20 2015 audit
Thu Dec 3 21:35:50 2015 audit/audit.log
Thu Dec 3 19:08:20 2015 audit/audit.log.1
...
This initial "full" backup can be followed by multiple incremental backups. After waiting for a few minutes, I repeated the same backup command to update my backup with the modified and new log files.
duplicity --no-encryption /var/log swift://server1-backup
Last full backup date: Thu Dec 3 21:35:51 2015
--------------[ Backup Statistics ]--------------
StartTime 1449180554.69 (Thu Dec 3 22:09:14 2015)
EndTime 1449180557.92 (Thu Dec 3 22:09:17 2015)
ElapsedTime 3.23 (3.23 seconds)
SourceFiles 42
SourceFileSize 281780139 (269 MB)
NewFiles 1
NewFileSize 4096 (4.00 KB)
DeletedFiles 0
ChangedFiles 11
ChangedFileSize 202769837 (193 MB)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 12
RawDeltaSize 27248281 (26.0 MB)
TotalDestinationSizeChange 1159452 (1.11 MB)
Errors 0
-------------------------------------------------
If you are curious, you can go to your cloud.ca Services tab and browse the content of your destination bucket, which looks like this after the above example:
4) Restore process
When disaster hits and you need to retrieve data from your latest backup, issue a restore command similar to this:
duplicity restore --no-encryption swift://server1-backup /tmp/log
Duplicity supports various ways to restore data from your backups. For example, you can retrieve only a few selected files instead of the whole archive, or target a specific point in time. See the man page for more details.
Wrapping up
This short intro should be enough to get you started. The next logical steps would be to encrypt those backups for additional security, and put in place an automated schedule for full and/or incremental backups. We will cover these in upcoming blog posts.
Original photo credit: Got Credit
About the author
 |
With 20 years of IT consulting and software development experience, Marc has a proven track record in delivering innovative products. He has worked at IBM, CGI, Coradiant and BMC Software. He is the product lead for cloud.ca. |
