What You Need

You’ll need PHP, bash, CRON access and python to get this working. And given the bash/cron requirements, that also means it assumes you’re running a *nix based server. It’s probably possible to get the requirements down, but I just wanted to get something done quick and it’s what I came up with.

The Files

backup.sh

First up is the bash script (based on this one) that does most of the work. It will tar+gzip all your site files, export your database(s), and remove old backup files (+5 days old) when necessary. You should only really have to edit the top part of the script. Make sure you remember what you set your BACKUP_DIR and FILES_DIR to – you’ll need them in other scripts later on.

#!/bin/sh

# EDIT THIS PART ------------------
# where all these backup scripts are stored
BACKUP_DIR=/home/theootz/backups

# where you want to keep all the actual backup folders
FILES_DIR=/home/theootz/backups/files

# where all your html files reside
HTML_DIR=/home/theootz/public_html

# your MySQL database name, username and password
SQL_USER1=myusername1
SQL_PASS1=mypass1
SQL_DB1=mydb1

# a second database
# Uncomment these lines if you want to back up a second database.
# Also uncomment the line specified below!
# You can also add more databases and follow the same format.
#SQL_USER2=myusername2
#SQL_PASS2=mypass2
#SQL_DB2=mydb2
# STOP EDITING ------------------

THEDATE=`date +%Y-%m-%d_%H-%M-%S`

mkdir -p $BACKUP_DIR/files

mysqldump -u${SQL_USER1} -p${SQL_PASS1} ${SQL_DB1} > ${FILES_DIR}/${THEDATE}_db1backup.sql

# Uncomment the line below if have a second database to back up
#mysqldump -u${SQL_USER2} -p${SQL_PASS2} ${SQL_DB2} > ${FILES_DIR}/${THEDATE}_db2backup.sql

tar -cf ${FILES_DIR}/${THEDATE}_site.tar ${HTML_DIR}
gzip ${FILES_DIR}/${THEDATE}_site.tar

find ${FILES_DIR}/*site* -mtime +5 -exec rm {} \;
find ${FILES_DIR}/*db* -mtime +5 -exec rm {} \;

backup_upload.py

Next is the Python script to upload the files. It uses a PHP library that I’ll give more information on below.

import sys
import os

########## Make sure these values match the ones you set in your shell script!
BACKUP_DIR = "/home/theootz/backups"
FILES_DIR = "/home/theootz/backups/files"

# this needs to be changed to reflect your settings on your s3 account
S3_BUCKET = "theootz"
S3_FOLDER = "mysite/backups"
##########

j = os.path.join
fname = os.path.basename(sys.argv[1])
cmdstr = j(BACKUP_DIR, "s3.php") + " put " + \
    j(FILES_DIR, fname) + " " + \
    j(S3_BUCKET, j(S3_FOLDER, fname.split("_")[-1]))

os.system(cmdstr)

PHP S3

The python script uses a PHP library to do all the S3 uploading work. The original is available here.

I had to edit a few paths in the php library to get it to work, so if you want to download my version it’s available here:

Regardless of which one you pick, make sure you open and edit the s3.php file! You’ll need to put your S3 keys in there (near the top of the file). It will look something like this:

// S3_CONF file
$aws_key = null;
$aws_secret = null;

You’ll want to put your access and secret keys there as strings, as follows:

// S3_CONF file
$aws_key = "my access key here";
$aws_secret = "my secret key here";

How To Get It All Working?

Ok, so you’ve got the files. Now upload them to your web server to your BACKUP_DIR folder (whatever you choose it as, I picked /home/theootz/backups). Finally, take the files from the the PHP S3 library and put them in your BACKUP_DIR as well. This should leave you with the following files in a single folder:

• backup.sh
• backup_upload.py
• s3sync.php
• s3.php
• libs3.php
• libaws.php

I’m pretty sure most of the files from the S3 library aren’t needed but once again not something I bothered to check.

Now you’ll want to enable execution for all our scripts. You can do this by running the following commands while in your BACKUP_DIR folder:

chmod u+x backup.sh
chmod u+x backup_upload.py
chmod u+x s3.php

At this point, you can go ahead and run backup.sh and see if everything worked. If it did, then we can set up the cron jobs to have this all work automatically for us

You’ll want to add two entries to cron. The first entry will run our backup.sh on a daily basis for us. This will give us 5 days worth of backups at any one point. The second script will upload our latest backup to the S3 service, replacing whatever is already there.

1       1       *       *       *       /path/to/BACKUP_DIR/backup.sh
1       1       *       *       *       find /path/to/FILES_DIR/* -mtime 0 -exec python /path/to/BACKUP_DIR/backup_upload.py '{}' \;

Make sure you edit the paths to reflect the way you have it configured for your system! For example, my cron jobs look like this:

0       0       *       *       *       /home/theootz/backups/backup.sh
0       0       *       *       0       find /home/theootz/backups/files/* -mtime 0 -exec python /home/theootz/backups/backup_upload.py '{}' \;

Done!

And that’s it Your website along with databases and other content will now be backed up daily to the server itself, and uploaded to your S3 account weekly. If you wanna be super extra paranoid you can download the backups from the FILES_DIR yourself or use rsync or something similar to automate that as well. One idea is perhaps sending it to your e-mail – use that g-mail space for something right?

Hope that’s some help to someone

No related posts.

Anyways, long story short – un-installing NOD32 and installing Microsoft Security Essentials (which is more than good enough anyways) seems to let me sync things. Jungle Disk is still giving odd errors but I’ll see if I can work through them.

Not ready to ditch Dropbox yet — but it does keep reminding me I’m getting close to running out of space :/

Related posts:

1. Jungle Disk – great idea, but…

Problem with dropbox’s paid service however, was that I was using only 2gb. It seemed like overkill to upgrade to a 50gb account for $10/month. I then remembered from a while back when I was first looking into cloud backup/sync software of one called Jungle Disk. It sounded great – uses Amazon S2 (as does dropbox) but with an open source client that gave you far more controla. Only reason I went with dropbox was because it was free and suited my needs at the time. So, it was time to give Jungle Disk another shot.

The sign up, payment and install processes went without a hitch. I decided to go with the “Jungle Disk Desktop” version, and hey turned out that with the base monthly fee they decided to include 5gb storage too – great! Payment was processed instantly and away I went!

On installing the program, it presents you with a wizard to help you set up your first ‘drive’. I figured I’d just enable all options and go from there – whether that’s a good thing or not I don’t know. To begin with, I wanted to sync my “My Dropbox” folder and if all goes well, I’d start with other folders too maybe – who knows? Yea, never got that far though. I set it to my drop box folder and let it sync over night. Waking up in the morning and expecting it to be complete I instead see several errors complaining it wasn’t able to upload one file or another and failed. It didn’t just skip the file (which from what I can tell, it’s supposed to!) but kept trying that same file over and over again until it finally gave up. So I figured, yea sure first time with the software – maybe it’s conflicting with dropbox already running in the background? So I gave that a close, and tried again. It got to the same file and stopped. No errors this time though (I was only able to check because of the logs I turned on from curiosity). Made changes to files? NEVER updated. I tried viewing what was saved on my disk via the virtual network drive – it only got bits of 2 main folders from about a dozen. Horrible!

Essentially, I’ve spent the last two days fighting with a horrible piece of software. It’s been constantly crashing on me whenever I try to make changes to configuration. And I’ve been making config changes trying to get it to sync successfully, if only just once! But every time it starts syncing, it gets to the same files and goes into an endless loop. Click cancel on the upload for that file? And it crashes! One of the files it was hanging up on was insignificant, so I decided to delete it but sure enough it finds another file to hang on.

I got frustrated to the point where I’ve decided to delete the virtual disk and try starting again. See where that leads me. If it -still- fails then I’ll contact support and see if the money I’m paying is worth it at all. If not, then I guess I’ll find something else. I’ve recently learned of another service called Mozy that seems to offer unlimited backup for around the same price as Jungle Disk’s base price but unlimited hosting of anything always makes me weary – more research would definitely be required. I’ve also been toying around with using Microsoft Mesh but the software is still a bit sluggish and doesn’t always sync properly. It also has no revision history which I’ve found can be VERY helpful!

Anyone else have any experience with Jungle Disk or other online backup/sync solutions? Any recommendations?

Related posts:

1. Jungle Disk – continued
2. Your site, saved and backed up on the S3 cloud!
3. Da Stank Bank “Uploader”