(continued from part 1)
In the first part of this tutorial, we fired up a dreamhost account and got our home folder synced up with it. Let’s check where we are in the master plan:
The Plan
Signup for a Dreamhost AccountCreate a new user with SSH loginUsing shared keys, enable SSH access to the server without having to supply a passwordRun rsync from the terminal to create a back-up of your home folder on the server- Create an AppleScript that runs rsync automatically
- Use iCal to run the AppleScript backup daily, without doing a thing!
In this second part of the tutorial, I’ll show you how to create an AppleScript that does the backup for you automatically, and then we’ll set iCal to run that script automatically. Ready? No? Well, it’s a blog, so you can leave it and then come back later when you are. No duh.
5. Create an AppleScript that runs rsync automatically
One of the very cool things that Mac OS X has included is a programming language called AppleScript. It lets you write computer programs in a very clean and simple environment – the AppleScript programming language resembles the English language, which makes it easy to use.
We’re going to write a simple computer program. It only has one command, which is my favorite kind of computer program. Inside your Applications folder, and you’ll find a folder called “AppleScript”. Inside is an application called Script Editor. Launch it, and copy the following text into the blank window without any line breaks (again, don’t worry if it breaks the lines for you, just don’t hit enter yourself between the lines):
do shell script "rsync -avz --delete-after ~/Documents/ USER@SERVER:~/laptop.bak/Documents/"
Remember to replace USER and SERVER with your own info. Click the hammer icon labeled “Compile”, and if you entered everything correctly, it should change the colors of the text to blue and black. This script should look very familiar; it’s just like the command we entered in the terminal the first time we synced everything up. The only thing that’s been added is the term “delete-after”. This tells the rscync program to delete anything on the remote server that no longer exists on your local computer. This will keep you from accumulating a bunch of trash in your remote server.
Save the script someplace where you will remember it. Name it “dreamhost backup.scpt”. If you want to go poking around, there’s a folder already named “Scripts” inside the Library folder (/Library/Scripts), which would be a fine place to save the script. It’s not important where you put it, as long as you can remember where it is.
6. Use iCal to run the AppleScript backup daily, without doing a thing!
The last hurdle we need to eliminate is the human factor – a backup plan that requires you to constantly remember to backup your data is bound to fail. Instead, let’s make the automatic computrix machine remember for us!
Apple’s iCal software has a very cool little feature that we can use to automate our backups. iCal allows you to set alarms to remind when an event is coming up in a few minutes. Instead of the alarm making a sound, we can set the alarm to run an AppleScript. You see where this is heading?
- Launch iCal
- Create a new calendar called “Scripting Cal”
- Create a new event called “Dreamhost Backup”
- You can set the time to whenever you like. I have mine set for 11 PM, since I’ve usually finished working on my projects by then, but my computer is still awake and online.
- Set the alarm to “run script”
- A new line will appear, asking you to point to the script you want to use (you may have to select “other” first). Navigate to “dreamhost backup.scpt” and click “select”
- Set it to run 1 minute before the event
- Set the event to repeat daily
There are a few caveats here. In order for this to work, your computer needs to be awake, connected to the internet, and iCal has to be running. Otherwise, the backup won’t happen. If you want to run the backup in the middle of the night, you’ll have to go into your system preferences to adjust when the computer wakes up.
Conclusion
So, in conclusion, that’s why I think Zombie Nixon will run, and win, in 2008. Oh wait, that’s a different post. If you have any questions, please feel free to leave a comment. If you’re a level 23 Unix Mage, and would like to correct or amend anything in this tutorial, please, feel free!
(think this tutorial is useful? take a second to Digg it!)
Popularity: 7% [?]
CLIdiot is not a professional geek. He's just a slightly geeky guy who thought it would be neat to setup his own server. These are the rants, tirades, and occasional insights of an normal person trying to figure out linux.
[...] « emo kids How To: Backup your Mac to a Dreamhost Server (Part 2 of 2) [...]
Um…Mike? Are there people out there in the readership whom you believe will actually FOLLOW these posts??? Maybe you need another month off, sweety…
You might be surprised, my dear. The road house is home to all manner of nerds and geeks, even if they don’t yet know it themselves.
Just wanted to say a quick thank you for the effort you put into this, helped me out a bundle. No longer do I need to worry about what to do with all the spare space on my DreamHost account :)
Thanks for the writeup. This helped a bundle.
HA! take that, grammy.
I found this post linked from TUAW, and it’s a great help. I’ve been looking to find a good way to backup to my dreamhost account, but I haven’t invested enough time to figure out how I wanted to do it. Your method seems like it is best for catastrophe recovery, but it doesn’t allow for alternate backup sets. What if, for example, I deleted a file accedentally two weeks ago? Rsync would have deleted that file on the server too, and I’d be SOL.
Also, is there a way to exclude directories from the rsync? For example, what if I wanted to back up my documents, desktop, and pictures, but not my music folders?
Finally, in addition to using iCal to schedule backup events, it would also be quite simple to use cron to run the simple rsync command you use.
Thanks agaiN!
I set something like this up a few months ago. It works well. It’s worth mentioning, though, that when you do this, you’re leaving all your data on a server with lots of other users.
Now, the other users on the server don’t have read access to your directory, but Dreamhost allows you to create other users on your account (for instance, for if you set up a friend’s website and want to give them shell access) and those users DO have access to your backup unless you explicitly change perms on your backup directory.
(I wish there was a way to encrypt the remote backup in a way that was still compatible with rsync)
Mike, I would add two notes:
1) Rather than crufting up the user’s iCal, it would be grand to put the script into the user’s crontab. This does increase the geeky factor but using an app like Cronnix mitigates this some.
2) Rather than having the script in /Library, it would be better to put it in ~/Library so that it gets backed up with the user’s home folder.
Thanks for such a complete writeup!
Great tutorial. I found it a few weeks ago and am now doing the same thing, with two additions.
First, I’ve enabled SSHFS and MacFuse so that I can mount my DreamHost account as a network disk (works like WebDAV, but more secure.
Second, there is a lot of discussion online about the problem of resource forks and extended attributes when backing up macs to non-Apple servers.
I found a patched version of rsync which I like better than the Apple version. It preserves extended attributes and resource forks by using the ._filename format. That’s the same way the Finder accomplishes this task when moving files to a non-HFS disk. So when I mount my backup disk, all custom icons and finder metadata are intact. Here’s the link to the alternate rsync:
http://www.onthenet.com.au/~q/rsync/
Mac Backup To Dreamhost…
When I posted a brief comparison to Amazon S3 and Dreamhost one thing I didn’t do was suggest how one could actually use Dreamhost in a similar way as S3, like backups. For S3 Jungle Disk is one option for backups. Now there is a how to on back…
For security reasons, couldn’t you just make an encrypted dmg to upload? I understand you couldn’t then use rysnc, but for those concerned with security it would work. Also, if you used the dmg approach why not use webdav like a dot Mac account? I am going to try personally to implement this strategy once I figure out how.
[quote comment="68389"]For security reasons, couldn’t you just make an encrypted dmg to upload? I understand you couldn’t then use rysnc, but for those concerned with security it would work. Also, if you used the dmg approach why not use webdav like a dot Mac account? I am going to try personally to implement this strategy once I figure out how.[/quote]
Because .Mac doesn’t give you 170GB to work with nor TB’s of bandwidth.
I tried doing something similar using WebDAV, but it was dog slow for some reason. Trying out this method overnight. Thanks for the effort!
Grammy, you owe me a beer.
Yeah, this is pretty classic, mom. :) This is, I’m sure, already the single most linked to article in the short history of this blog.
If you’re up for something more integrated than an iCal/Applescript combo, I’d highly recommend using Lingon and Launchd from Tiger to handle this. Quick and dirty setup process (Once you have lingon):
1. Create a new User Agent
2. Label it what you like (”com.myname.dhbackup”)
3. In the ProgramArguments box, enter
"rsync -avz --delete-after ~/Documents/ USER@SERVER:~/laptop.bak/Documents/"(Inserting your details, of course). Where there is a space in the command, add a new parameter (Line one is rsync, line two is -avz, etc.)4. Give it a nice ServiceDescription
5. Set a WatchPath to your Documents folder (Under the “Paths” menu”)
Any time you modify your Documents folder now, it will be rsync-ed to your server. If you don’t have an internet connection at the time, it will wait until you have one. Sine I can’t guarantee my own connection (Roaming between work/school/home on wireless) I have it run on startup and once ever 10 minutes or so to keep my backups current.
Hope that helps! This is a great how-to as it is, launchd makes it a little more powerful.
[...] Part 2 of the tutorial shows you how to create an AppleScript that runs rsync automatically and how to setup iCal to run the AppleScript on a daily basis. [...]
Ok, Michael, I signed up for 2 years with Dreamhost…credited to you so I could get some technical support :)
I have been looking for this EXACT type of solution, so thanks for being so thorough in your post. I have four Macs (and a PC) I have been looking to backup in the cloud.
So I got the dreamhost account. Here are a few questions:
1) I suspect I need a separate RSA private key for all four Macs. If so, how does that affect the Passwordless login steps?
2) Like another commenter, above, terminal barfs when I try to run the rsync command. I get “rsync: mkdir “/home/USER/mbp.bak/Documents” failed: No such file or directory”. I used Dreamhost’s webftp to manually add a directory and file /Documents/mbp.bak and run rsync again. No luck. Same barfing. (USER in caps, means I know that would be my user ID for this account).
3) I also get these errors:
rsync error: error in file IO (code 11) at main.c(420)
rsync: connection unexpectedly closed (8 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at /SourceCache/rsync/rsync-24.1/rsync/io.c(359)
Thoughts?
[quote comment="69267"]
1) I suspect I need a separate RSA private key for all four Macs. If so, how does that affect the Passwordless login steps?[/quote]
That’s correct – just run the steps in the Dreamhost article from each mac you’re using. The
.ssh/authorized_keysfile on your server will simply add each new key into the file, and each of them will be able to login without using a password. I have 3 macs in different places all keyed to login to the same user account without password.[quote]2) Like another commenter, above, terminal barfs when I try to run the rsync command. I get “rsync: mkdir “/home/USER/mbp.bak/Documents” failed: No such file or directory”. I used Dreamhost’s webftp to manually add a directory and file /Documents/mbp.bak and run rsync again. No luck. Same barfing. (USER in caps, means I know that would be my user ID for this account).[/quote]
Be sure to include the trailing slash
mbp.bak/Documents/when you reference both the SOURCE and DESTINATION in your rsync command.Also, it sounds like you might be trying to create the directory on your local computer. Did you insert an extra space somewhere in the DESTINATION address?
I’ll look more into this later – I’m just on my way out the door.
Ok, I used webftp to manually create the directories powerbook_15.bak and powerbook_15.bak/Documents and all is good there. I have it doing its first round of backups. Hooray. Heading to dinner, too. Will see what they look like when I get back :)
Ok, so now I have 3 of the 4 computers backing-up at the same time (probably not a great idea, but I am not going to be home, so they can have at it, I guess). I am very interested in some of the “excludes” mentioned by another poster. Can you show me an example of what one of those rsync statements might look like? Where do I add the exceptions?
rsync is not really a backup solution, merely a file/directory copy/sync tool. What you want is rdiff-backup, which keeps incremental backups so you can restore files to their state as of any date that is still in the backup set–30-60 days works well. It can use rsync to transmit the data, so it works quickly.
The problem with that is your data is sent and stored in the clear. If you don’t mind Dreamhost–and whoever can read your network traffic–being able to read all your personal backup data, that’s fine. (Not that they’d care, but it’s the principle.) So the solution to that is Duplicity, which uses rdiff-backup, which uses rsync. Duplicity compresses and encrypts your data before transmitting it, so the data never leaves your computer in the clear. It remains encrypted on the server, so no one without your GPG key–or your passphrase if you choose symmetric encryption–can read your data.
I’ve been using Duplicity to backup to my Dreamhost account for over a year. It works wonderfully. I made a small shell script to back up a list of directories, in order, so the most important data gets backed up first. I rate-limited the SSH connection and niced down the process, so it can run in the background without affecting my connection or my local responsiveness. You can find the script here:
http://alphapapa.net/debian/generic-backup-with-duplicity-to-dreamhost.sh
By the way, it’d probably be better to use cron (or anacron or fcron) to schedule your backups. Anacron would be good, because then you don’t have to have the computer on at a certain time every day for the backups to run.
I think anacron is for total losers, myself. I wouldn’t be caught dead using it. Fcron is acceptable, in a pinch, if you absolutely must use it, and only if you have not committed a sin (omissional or commissional) in the past 48 hours.
To be honest, I think this whole conversation is a little pedestrian for those of us who know about all the crons and stuff.
Corey, Grammy? Who’s with me?
Debian User, rdiff-backup and duplicity are not trivial to get working on OSX. Netiher is cron, for that matter. For an entry level tutorial, which this is intended to be, I tried to minimize the amount of geeking out that was required by the user in order to get the system up off the ground.
I don’t know about you, but I think any tutorial that includes the phrase “now simply invoke GCC and compile the source code into your usr/bin/ folder with the following switches” has ceased to be an entry level tutorial.
[...] na segunda parte amosa como crear un applescript para facer esa copia de seguridade de xeito automático e botar man [...]
[quote comment="69339"]
The problem with that is your data is sent and stored in the clear.[/quote]
Stored in the clear, sent? Not so much. Rsync uses ssh and that is encrypted. If someone hacks into your Dreamhost account or someone from dreamhost decides to check out your stuff, that is another story. However, I doubt they have time to poke around your data. They are too busy crashing their servers. :-) I’ll have to look into your solution though. It sounds more secure, and in my opinion, more secure is definitely better. You solution may beyond the scope of this blog entry, but for people who aren’t afraid of GCC, it could be a nice tool.
Cheers,
Tom
I realize it’s been a while, but anyway…
It’s too bad that rdiff-backup and duplicity aren’t available easily on OS X. They’re both just Python apps; I don’t see why GCC should be involved, unless Python doesn’t come with OS X. Are you sure they aren’t in Fink or something like that?
Anyway, this problem would be moot if you used Debian… ;)
[quote comment="76266"]Anyway, this problem would be moot if you used Debian… ;)[/quote]
True, but then I’d be stuck trying to port Logic and Pro Tools over to Debian. Open Source isn’t quite ready for primetime yet.
I dunno, installing duplicity was pretty easy for me on my Mac:
$ sudo port install duplicity
;-)
Ok, ok, using MacPorts was a cheap way to get around the hard work, but there are plenty of good references for how to setup/install MacPorts so if you reference one of those, you could get duplicity set up and installed easily on a mac. I’m off to go play with it on my DH account and see how well it works for backup. Thanks for the pointer to duplicity, DebianUser, the security angle never really made me all that happy, though I suppose I could also use macfuse, sshfs, and a mounted encrypted image on top of that to achieve the same effect with more work.
you should really use duplicity (http://www.nongnu.org/duplicity/) since it allows you to use rsync & encrypt your files. I’m not sure I want all my stuff on a shared server. I’ve been doing this for over a year this way.
I think I might actually switch to duplicity. This looks like a good solution.
DREAM HOST TOS “The customer agrees to make use of DreamHost Web Hosting servers primarily for the purpose of hosting a website, and associated email functions.” THERE FOR USING IT AS DATA STORAGE IS A BREACH OF TOS. you people… silly. READ YOUR CONTRACTS
Your moral outrage and righteous indignation wouldn’t have anything to do with the fact that you work for an off-site data storage company, would it now Bryan?
A Debian User:
You’re script looks absolutely perfect for my needs. Unfortunately for me I’m a designer and only 1/3 of a geek (I understand your script but I’m not equipped to get it up and running on my system). Is there any chance you (or any of you other kind souls) might consider doing a tutorial on:
1. Installing Dublicity
2. Getting A Dubian User’s script up and running (include explanations/examples of what a passphrase is, what GPG keys are, how to format the backup.list etc.)
3. How to set it up as a cron with Anacron.
Would be forever grateful – even willing to donate $ to the cause!
Thanks!
Hey, Jeff. The geek of Addison Road is currently suffering for Jesus in Rome, and I’m pretty sure the rest of us have a snowman’s chance in hell of fulfilling your requests. But I sneaked a peek at the comment archives, and I think you can contact A Debian User directly at adam [at] alphapapa [dot] net.
Hope that helps…
Any questions or comments regarding….
Sorta-Deep Theology, Will Ferrell, Tina Fey, C.S. Lewis, Mute Math, Church Politics, Art Criticism (talk to June), Summer Movies, Why We Hate CCM, Exposed Oily Man-Junk, NPR, Hot Christian Sex, Parenting (a result of Hot Christian Sex), Weight Loss, Weight Gain, Depression, Fighting Out of Depression, Dallas Willard, New Jobs, Bono Worship, Justin Timberlake Worship, Good Coffee, or why Knocked Up is the best movie of the summer…
….are totally still cool.
Just an FYI.
Thanks for the replies.
The giving nature of internet peoples never ceases to amaze me. I’ll drop Adam an email. I wonder if Jesus were a geek would he be A Debian User?
If I get this up and running I’ll post the how to myself. It really would be the perfect set-up for all mac users.
Hey there,
If you use Duplicity to store your data on a remote server, are you able to access the data remotely, via a web front-end that decrypts the data for you?
Thanks,
DSL
Thanks for posting this. I recently got a MacBook Pro after having used a PC desktop for many years. It’s great, but I can’t put in more hard drives as backup disks. This tutorial helps me use up the 230GB of space I have on my dreamhost account (it’s 26 months old) and keeps my worry-free about my files. Great job!
Inspiring stuff and well wrtitten tutorial. Here’s what I did:
I first created a WebDAV directory named “backups” on one of my (Dreamhost) domains. I can easily connect to that directory, which now behaves as a virtual disc, from the Finder. Once the disc is mounted I can configure Apple’s Backup app to do the rest; works like a charm. I guess other backup applications also could be used to do this.
Which makes me think it could also be possible to get some of the build in .mac features working, since that also uses the WebDAV protocol.
Thanks for this excellent tutorial – I set it all up and have been using it for a few backups and it works like a charm. I’m a little concerned about the data not being encrypted but most of it is not very sensitive. I was thinking of trying the duplicity idea, however I don’t have the skills necessary to do it without at least general instructions. Then I found this post with details on how to setup duplicity on a mac.
It’s not for the faint of heart, but anybody that has read this far down is probably able to handle it. :) Enjoy!
This is all great, unless your data includes resource forks. They will not be copied by rsync. This can, and often will, lead to useless data when you try to restore. I have been there and lost vital data for my Parallels emulation of Windows. When the original became corrupted, I discovered that the backup was unrecognisable as a Windows disk. An expensive lesson. Unless you are sure that this will not apply to you, look elsewhere for a backup solution. Consider installing MacFuse (from Google) and sshfs to mount a Dreamhost filesystem locally on your machine. Use a tool like Carbon Copy Cloner to make an incremental backup to an encrypted disk image on the mounted filesystem. In that way, you will be sure to preserve all vital Macintosh specific data.
Carbon Copy Cloner has a version of rsync that has been designed to preserve the key information.
[quote comment="130493"]This is all great, unless your data includes resource forks. They will not be copied by rsync. This can, and often will, lead to useless data when you try to restore. I have been there and lost vital data for my Parallels emulation of Windows. When the original became corrupted, I discovered that the backup was unrecognisable as a Windows disk. An expensive lesson. Unless you are sure that this will not apply to you, look elsewhere for a backup solution. Consider installing MacFuse (from Google) and sshfs to mount a Dreamhost filesystem locally on your machine. Use a tool like Carbon Copy Cloner to make an incremental backup to an encrypted disk image on the mounted filesystem. In that way, you will be sure to preserve all vital Macintosh specific data.
Carbon Copy Cloner has a version of rsync that has been designed to preserve the key information.[/quote]
Actually, I believe the version of rsync that ships with MacOS 10.4 is resfork compatible. There’s also rsyncX, which is, if I remember right, free, unlike CarbonCopyCloner.
[...] How To: Backup your Mac to a Dreamhost Server (Part 2 of 2) [...]
[...] (continue reading part 2) [...]