What You Need

You’ll need PHP, bash, CRON access and python to get this working. And given the bash/cron requirements, that also means it assumes you’re running a *nix based server. It’s probably possible to get the requirements down, but I just wanted to get something done quick and it’s what I came up with.

The Files

backup.sh

First up is the bash script (based on this one) that does most of the work. It will tar+gzip all your site files, export your database(s), and remove old backup files (+5 days old) when necessary. You should only really have to edit the top part of the script. Make sure you remember what you set your BACKUP_DIR and FILES_DIR to – you’ll need them in other scripts later on.

#!/bin/sh

# EDIT THIS PART ------------------
# where all these backup scripts are stored
BACKUP_DIR=/home/theootz/backups

# where you want to keep all the actual backup folders
FILES_DIR=/home/theootz/backups/files

# where all your html files reside
HTML_DIR=/home/theootz/public_html

# your MySQL database name, username and password
SQL_USER1=myusername1
SQL_PASS1=mypass1
SQL_DB1=mydb1

# a second database
# Uncomment these lines if you want to back up a second database.
# Also uncomment the line specified below!
# You can also add more databases and follow the same format.
#SQL_USER2=myusername2
#SQL_PASS2=mypass2
#SQL_DB2=mydb2
# STOP EDITING ------------------

THEDATE=`date +%Y-%m-%d_%H-%M-%S`

mkdir -p $BACKUP_DIR/files

mysqldump -u${SQL_USER1} -p${SQL_PASS1} ${SQL_DB1} > ${FILES_DIR}/${THEDATE}_db1backup.sql

# Uncomment the line below if have a second database to back up
#mysqldump -u${SQL_USER2} -p${SQL_PASS2} ${SQL_DB2} > ${FILES_DIR}/${THEDATE}_db2backup.sql

tar -cf ${FILES_DIR}/${THEDATE}_site.tar ${HTML_DIR}
gzip ${FILES_DIR}/${THEDATE}_site.tar

find ${FILES_DIR}/*site* -mtime +5 -exec rm {} \;
find ${FILES_DIR}/*db* -mtime +5 -exec rm {} \;

backup_upload.py

Next is the Python script to upload the files. It uses a PHP library that I’ll give more information on below.

import sys
import os

########## Make sure these values match the ones you set in your shell script!
BACKUP_DIR = "/home/theootz/backups"
FILES_DIR = "/home/theootz/backups/files"

# this needs to be changed to reflect your settings on your s3 account
S3_BUCKET = "theootz"
S3_FOLDER = "mysite/backups"
##########

j = os.path.join
fname = os.path.basename(sys.argv[1])
cmdstr = j(BACKUP_DIR, "s3.php") + " put " + \
    j(FILES_DIR, fname) + " " + \
    j(S3_BUCKET, j(S3_FOLDER, fname.split("_")[-1]))

os.system(cmdstr)

PHP S3

The python script uses a PHP library to do all the S3 uploading work. The original is available here.

I had to edit a few paths in the php library to get it to work, so if you want to download my version it’s available here:

Regardless of which one you pick, make sure you open and edit the s3.php file! You’ll need to put your S3 keys in there (near the top of the file). It will look something like this:

// S3_CONF file
$aws_key = null;
$aws_secret = null;

You’ll want to put your access and secret keys there as strings, as follows:

// S3_CONF file
$aws_key = "my access key here";
$aws_secret = "my secret key here";

How To Get It All Working?

Ok, so you’ve got the files. Now upload them to your web server to your BACKUP_DIR folder (whatever you choose it as, I picked /home/theootz/backups). Finally, take the files from the the PHP S3 library and put them in your BACKUP_DIR as well. This should leave you with the following files in a single folder:

• backup.sh
• backup_upload.py
• s3sync.php
• s3.php
• libs3.php
• libaws.php

I’m pretty sure most of the files from the S3 library aren’t needed but once again not something I bothered to check.

Now you’ll want to enable execution for all our scripts. You can do this by running the following commands while in your BACKUP_DIR folder:

chmod u+x backup.sh
chmod u+x backup_upload.py
chmod u+x s3.php

At this point, you can go ahead and run backup.sh and see if everything worked. If it did, then we can set up the cron jobs to have this all work automatically for us

You’ll want to add two entries to cron. The first entry will run our backup.sh on a daily basis for us. This will give us 5 days worth of backups at any one point. The second script will upload our latest backup to the S3 service, replacing whatever is already there.

1       1       *       *       *       /path/to/BACKUP_DIR/backup.sh
1       1       *       *       *       find /path/to/FILES_DIR/* -mtime 0 -exec python /path/to/BACKUP_DIR/backup_upload.py '{}' \;

Make sure you edit the paths to reflect the way you have it configured for your system! For example, my cron jobs look like this:

0       0       *       *       *       /home/theootz/backups/backup.sh
0       0       *       *       0       find /home/theootz/backups/files/* -mtime 0 -exec python /home/theootz/backups/backup_upload.py '{}' \;

Done!

And that’s it Your website along with databases and other content will now be backed up daily to the server itself, and uploaded to your S3 account weekly. If you wanna be super extra paranoid you can download the backups from the FILES_DIR yourself or use rsync or something similar to automate that as well. One idea is perhaps sending it to your e-mail – use that g-mail space for something right?

Hope that’s some help to someone

No related posts.

With the release of the Windows Phone 7 Series (I really really hate that name so I’m gonna just use winmo7) SDK, I decided to give it a shot and see how hard (or even easy?) creating an application on the platform would be.

I should probably mention I have very little experience coding in SilverLight and only dabbled with it a while ago, and I’ve been coding in C# for a good half year or so now but once again not quite that much experience in the language. On top of that, most C# applications I’ve written use Windows Forms and not WPF. I’ve also decided that my first application will be a simple browser for Amazon S3 (considering how much I’ve been fiddling with it recently), of whose API I have zero experience with as well. So from what I can tell, I have quite a learning curve ahead of me

I’m not going to go over any problems I have over the IDE, APIs, language, etc… but I’m going to try to focus on the experience of creating an actual application for the platform.

Initial Setup

The download and install of the SDK was relatively painless. It did take quite a while though because I had the beta of Visual Studio 2010 still installed and the SDK requires either the RC or it will install Visual Studio 2010 Express on its own. I didn’t really feel like downloading and installing Visual Studio 2010 again, so I had to uninstall VS2010 along with the .Net 4 beta. After which, the SDK installer downloaded and installed the RC for me. It was kind of annoying how it didn’t really tell me why the beta simply HAD to be uninstalled and I had to search through the depths of their forums to figure it out, but I guess most shouldn’t have this problem.

One thing to remember if you’re using the Express version of VS2010 though, I don’t think it’s possible to have any VCS integration so you’ll have to do it outside the IDE.

My First App

Opening up VS2010, I’m presented with a few nice big buttons on pre-set winmo7 specific projects. You can create a Windows Phone App, a Windows Phone List App (not sure what this is exactly but it’s somehow specific to “list and navigation controls”) and a Windows Phone Library.

I figured I want an application and I’ll code in a simple library for working with S3 inside that.

So I went ahead and choose the first option and it creates a nice looking UI preview for me with the phone chrome and all so I can see exactly how it will look. Not bad. Running the program launches the emulator (which takes quite a while :/) and launches my app which currently does nothing. Good enough of a Hello World for me!

Also of interest, the emulator need only be loaded ONCE. It can then be used for multiple debug/run sessions, very nice considering how long it took to load up!

Designing the S3 Browser

So at this point I need to figure out how I want the program to look and behave. I’ve decided to use this desktop S3 Browser by NetSDK Software as a basis on functionality. At the bare minimum, I’m going to need a way for users to enter in their keys and save them (because typing them in every time on a mobile device will DEFINITELY be annoying). They then need to be able to view buckets, and then be able to explore the contents of each bucket. Perhaps even adding in the ability to upload, download, delete and rename files if possible. Though at this point, I’m not even sure if winmo7 has a concept of a user accessible file system or if it’s abstracted/sandboxed like the iPhone.

Login Screen Design

Screens will probably be something akin to:

(1) Login/Key Manager -> (2) Buckets -> (3) Browse

(To the left you can see my first stab at the login screen using the built in controls.)

I’m going to have screen (1) act as a login screen for now, no saving of multiple accounts. That can be added later. Screen (2) will simply list buckets, and then later allow creation/deletion of them as well possibly. And finally screen (3) will list all your files and other functionality to be added later.

The actual design of each screen may be somewhat trivial, but I want the look and feel of application to be as if its a part of the operating system. A quick google search shows that there ARE a set of guidelines so I’m going to have to read them over and figure it out. From what I can tell though, as long as I’m using the Controls that are given to me then most of it should follow the look/feel anyways.

Initial Thoughts

So far, it seems that Microsoft has gone out of their to make it very easy to develop for the platform – leveraging technologies that developers should already be familiar with: Visual Studio, .Net, Expression Blend, SilverLight, etc…

I suspect that the SDK will become better as time goes on. For example, being based on .Net it wouldn’t surprise me if they allow programmers to use languages other than C#. Also from what I’ve been reading, a lot of people have had success porting over their existing SilverLight applications relatively easily and quickly to the new platform.

What’s Next?

Well, I’ve gotta make a library of some sort to connect to S3. I’ve gotta figure out how to use WPF but that shouldn’t be too difficult. Then I’ve gotta get it all working in the emulator. I’ll post updates as I have them, for now my project is available below along with a repository (SVN where it also turns out I managed to misspell theootz as theooz :/) link – feel free to check it out.

Back to list

Related posts:

1. Swan-dive into Android development!

There’s the new “Projects” section which is powered by a nice little back-end script to let me easily manage links and data to all the projects I’m working on, or have worked on.

I added a “LifeStream” – based on the same concept that facebook uses to aggregate many social sites into one ‘stream’. Ironically, I disabled facebook from showing up on the stream because it was posting way too much. Though the upside is, it gave me an idea for getting my whole cross-posting idea to work that I’m going to explore later. Originally I was trying to use my blog to push data to all the other services I use, but I figured I could also use facebook to push to my blog and from my blog to the rest of the services. I’d have to work out how I could make my blog still push unique posts to facebook and not push those specific posts but I haven’t really looked at any code yet so I’ll figure it out then.

Next, I finally figured out how to set up my blog as an OpenID provider So instead of remembering that convoluted verisign address that I *ALWAYS* forget, I can just use my blog address. Nice and simple.

I’ve added a massive caching system and found another use for my Amazon S3 account! I decided to create a new s3 bucket and linked it to Amazon’s Cloud Front. All media, most javascript, css, etc… is now served from local amazon end points (@ cloud.theootz.com) instead of my hosting provider I haven’t tested to see if this has resulted in any real speed improvement, but at least at a glance it seems to help a lot.

And to end it off, I found a wordpress plug-in for managing resume’s that I’m looking at using for posting my CV up on the site. Haven’t fiddled with it much yet but it seems to hold promise. Also another plug-in to find and display ‘related posts’ at the bottom of each post I make. Gotta love wordpress plug-ins

Related posts:

1. video+cross post test, second attempt
2. Da Stank Bank “Uploader”