27
Apr 08

Bash it! Bop it! Script it!

I would rather spend 3 hours writing a program to do a task than have to spend 3 minutes doing it myself more than once. Seriously. I’m that lazy. The only task I like to do over and over again is opening a Corona, taking a sip, filling it back up with tequila, and passing out in a pool of my own vomit. Or, as my kids like to call it, “Thursday.”

That kind of laziness means I use a lot of bash scripts to do regular tasks on my server. I have scripts to do automated backups, scripts to setup new virtual domains, scripts to prop up my fragile ego with repeated compliments, scripts to do just about every repeating task that goes into maintaining a barely functional webserver.

There are several tutorials out there for how to start writing scripts, but none of them have my flair for drunken bravado and outrageous sexual innuendo. So, I proudly present a very basic starter guide to writing a bash script:

How to Write A Bash Script

You know all those fancy commands you keep typing into the command line? Things like

cp -rv secretpr0nstash/*.avi /var/www/churchhats.com/

Well, those same commands can be stored in a file, and can be executed whenever you need to, by invoking the name of the file. Let’s start with a very basic file. It just has 3 lines:

#! /bin/bash
# sexy robot script
echo "You are the sexiest robot"

The first line tells the system which shell to use when interpreting the commands that follow. The second line is a comment, to remind me in 6 months what the point of the script is. You can place these throughout the script to remind yourself why you did what you did when you wrote the thing. Finally, the third line is the actual command. It tells the system to output the given text back to the shell.

If you’re looking to learn more about how to write complex scripts, I highly recommend these two guides:
Advanced Bash-Scripting Guide
Writing Shell Scripts

So, now what?

Where to Put It

Now, you need a place to put the script. I keep all of mine in a folder called “bin” inside of my user folder. To create the folder, type:
mkdir ~/bin
cd ~/bin

now, invoke your favorite text editor to open a new file, and start entering in the code:

nano sexrobot

Enter your code, save, exit, and TADA! you have your very own script to prop up your fragile ego with repeated compliments.

How to Make Go Go

Except, it still won’t run. Try it – type “sexrobot” into your command line. What happened? It mocked you, didn’t it. It told you that your crazy dreams of a sexy robot compliment did not exist.

Before we can invoke the command, we have to make it executable.

chmod 755 sexrobot

Now, I can execute the code. Try it again, with the whole location:

~/bin/sexrobot

Now that it works, we can do the final step – add a line to our .bash_profile file, so that every time we log in, the shell goes into our bin folder to look for commands.
nano ~/.bash_profile

Go to the end of the file, and add the following line:
export PATH="$PATH:~/bin"

Reload the bash_profile settings:
. ~/.bash_profile

If we did everything right, we should be able to execute our new script from any folder, just by typing the name of the file. Let’s try it:
cd /tmp
sexrobot

In Conclusions

Now that I don’t have to do all these repetitive server maintenance tasks, I can focus on more important things, like repetitive binge-drinking to drown my own sorrow.

Ha Ha! Alcoholism is funny!

Popularity: 18% [?]


29
Mar 08

Housekeeping, or, How To Obsessively Blow Up The Files Left Behind

I install, and then uninstall, lots of different software packages on Debian. I’m trying to learn how this thing works, so I’ll often install something, poke around, and then remove it. But, I get paranoid about the package manager leaving things behind on my system. Cron jobs, config files, libraries, all that jazz.

So, I do some housekeeping to make sure that everything, absolutely everything, gets blown up when I remove a piece of software. Is it necessary? Probably not. Does it satisfy my deeply rooted obsessive control issues? It does.

I’ll assume you’ve already removed the software using the package manager of your choice. For me, that’s aptitude. Throughout this little rundown, replace the word PACKAGE with the name of whatever you’re uninstalling.

sudo aptitude purge PACKAGE

Then, we need to hunt down and kill every remaining bread crumb that this thing left behind. We’ll be using the “locate” tool. It’s a fast search tool that refers to an existing database of files. Start by kicking into super user mode:

sudo su

Then, update the database that locate uses. This rebuilds an updated list of the files on your system.

updatedb

Next, we want locate to find all of the files that were left behind.

locate PACKAGE

This will print them all to the screen. See them there, smiling back at you? You thought you were clean, you dirty bird, but there they all are. Filthy. Filthy files. Dirty. Dirty.

Well, looking at them on the screen isn’t all that useful for us. What we really need is some way to generate a list of these files that we can then process for deletion, one at a time. Go go command line, go!

locate PACKAGE > /tmp/PACKAGE_files.txt

head to your /tmp folder and take a look – there should be a .txt file there that lists each of the files left behind. Check it out:

cat /tmp/PACKAGE_files.txt

It should look identical to the output of the locate command we used earlier.

Now, we use a little Command Line Magic©, and no, I’m not talking about cocaine. Sweet, sweet cocaine, gives us the go-go to make everything clean. Mmmm, clean. So clean.

We want a command that will pass each line of our new text file into the “rm” command, and give us the option to delete each one. Why, hello there xargs, so nice to see you.

cat /tmp/PACKAGE_files.txt | xargs -l1 -p rm -rf

What’s going on here? The “cat” command will output the content of our txt file. The pipe passes it along to the next command. “xargs” lets us input that data into a new command, in this case “rm”. The “-l1″ switch on xargs (that’s the letter “L” and the number “one”) executes the command on one line at a time, instead of the whole batch. The “-p” switch will ask us for permission on each file, so that we have some control over what we want to delete. The “-rf” switch on the “rm” command will remove files recursively, and will force the issue if there are warnings or errors.

If we did everything right, it should start ripping through the files, asking you if you want to delete each one. Y for yes, N for no, and Tada! you have a system clean enough to avoid those awkward OCD freakouts at 2am.

If you want to double check the process, run the locate command again:

updatedb && locate PACKAGE

It should come back clean. Not dirty. Clean. Like pure white linen sheets on a bed of unicorn feathers. So clean. So nice and clean. And now, the sleeping.

Popularity: 5% [?]


02
Feb 08

zip zip

It’s the simplest things. Really. I do the same dumb thing 50 times without ever stopping to think if there might be a better way. And then, when I stumble into it, I smack myself in the forehead.

Zips. I work with WordPress, and on my development install I’m constantly downloading new themes or plugins to try them out. I’d rather use the terminal than an FTP client, so I’m usually using wget to pull the zip file into the plugins folder, unzipping it, then deleting the original zip file.

It took me 6 months of doing this before I realized that I could add one single line to my bash_profile to ease all of the pain.

alias zz="unzip *.zip && rm *.zip"

It seems small, and it is, but my zen-like happiness quotient has been skyrocketing ever since.

Share and enjoy!

Popularity: 4% [?]


23
Oct 07

apt-get remove dovecot spamassassin postfix

Sweet! Google Apps / Gmail just got IMAP. There goes my final reason for trying to figure out email server configuration on my own. Now I can focus on the things that really matter – porting NES classic games to the ncurses interfaces. You haven’t played zelda until you’ve played ASCII graphics zelda!

Popularity: 12% [?]


20
Oct 07

Denial of (advertised) Service Attack!

Last spring, I posted this tutorial on how to use rsync to upload data to a Dreamhost server. It’s still the most popular article ever posted on that blog, even more popular than my liturgical poetry, if you can imagine.

The result was 25 new dreamhost customers who signed up through the link in that article.

Dreamhost then altered their terms of service to prevent this sort of thing, and made a big stink about it on their support site. I was willing to suffer through some of the downtime, the server chokes, the high load times and stick it out with them because of the huge bandwidth pipe and large dump truck they were offering.

Now, they’re limiting how we can use that bandwidth and data. Apparently, they didn’t really sell me a chunk of storage and a chunk of bandwidth, what they sold me was the ability to do one specific thing (host a website) up to a certain size and distribution load. What a load.

The new Terms of Service (TOS) now specify that you may not use Dreamhost for anything that’s not primarily related to one of your hosted websites. Here are some of the things I use my Dreamhost account for that, apparently, are no longer considered acceptable usage.

  1. Off-site backup of my studio computer. Ok, I get it. This is a pretty big strain on your system. Even though it was one of my reasons for maintaining such a large account, I’ll quit doing this and find another solution.
  2. WebDAV transfers between my students and I. I teach at a University, and I have a WebDAV folder setup for them to use to pass projects back and forth to me. It’s unclear to me why Dreamhost even offers WebDAV under the new TOS, since there’s no way to access that folder from an HTTP or PHP request. It is, by definition, unrelated to web hosting.
  3. WebDAV synchronization of my iCal calendars between my wife and I. Our lives would be a mess if we couldn’t sync calendars. Again, not related to web-hosting, now an illegal use of Dreamhost.
  4. FTP transfer of media files between my clients and I. I record and produce music, frequently music to video, which means I rely on the ability to transfer large media files to and from clients. One of the reasons I bought a Dreamhost account was to setup an FTP transfer account for this. Again, not related to web-hosting, not legal under their TOS.
  5. Mirrored backups of sites hosted elsewhere. I host many, many websites with other hosting providers, in addition to my sites with Dreamhost. I do regular rsync backups of these sites to Dreamhost, complete with intact folder structures and database archives. If one of those other hosts goes down, I can switch to Dreamhost and take that backup live. Is this a violation? I’m honestly not sure. Until I actually make the DNS switch and serve the data from Dreamhost, the files sitting on the server are, strictly speaking, just backup data, the kind of thing explicitly prohibited by their TOS.
  6. File Server for podcast files. I host a podcast with about 2,500 daily subscribers, and about 10,000 downloads per episode – I need it to be up 24/7, and Dreamhost seems to be able to provide about 22 hours at a stretch, before their central networking router goes down. So, I host the actual website at slicehost.com. There I have my own dedicated virtual server, no downtime, snappy response, it’s basically perfect. But the podcast uses about a terabyte of bandwidth a month, and that would be prohibitively expensive through slicehost. Instead, I host the actual podcast files on Dreamhost, and I reference them from the site on slicehost. Is this illegal? It sure seems to be, since I’m not hosting the site itself on Dreamhost, but I don’t know.

I think what the new Terms of Service boil down to is this: Dreamhost is no longer interested in working with power users. If you need to host a simple LOLcats homage site, by all means, come sign up with us. If you’re looking to actually USE the services we advertise though, like rsync between servers, WebDAV file sharing, FTP services, any backend sharing of data that doesn’t relate to your LOLcats site, then don’t bother. We’re not interested.

For those who feel unduly constrained by those limitations, let me be the first to invite you to Slicehost. Get your own dedicated bandwidth, your own chunk of harddrive, and then do whatever you want with it. Check out their Terms of Service: as long as it’s not illegal, hackerish, or spam-a-licious, they don’t care what you do with your server.

Come live the dream.

Popularity: 16% [?]


19
Oct 07

ubuntu upgrade rush

Note to self: don’t even attempt to use apt-get or aptitude during the week of a major distro release from Ubuntu. It’s not worth the pain of 8KB/s download times over a massive 1MB/s pipe.

Popularity: 11% [?]