$tail -f findings.out

Lost in a sea of terminals

2008-10-02T11:39:00.000-07:00

Usually I enjoy my screen being covered with a variety of terminals across multiple servers. But on occasion, I type utterly the wrong thing, in utterly the wrong place, and calamity ensues.

So to provide a slight buffer against this, I made a change in my bash prompt. It clues me in if I am on a remote server, a non-personal machine if you will, on which I need to exercise more caution before beating the enter key:
Just something noticeable, but not overly intrusive. Won't stop idiocy, of course, but it can't hurt.

To make it work:


# Prompt with easier to read coloring. Define and check for safe boxes:
declare -a LOCALBOXES[0]="MyBox1" LOCALBOXES[1]="MyBox2"
CURRENTBOX=`hostname`
for box in $LOCALBOXES
do
   if [[ $CURRENTBOX = $box ]] ; then
       PS1="\[\033[0;31m\][\T]\[\033[0;36m\]\u@\[\033[1;33m\]\H \[\033[1;34m\]\w: \[\033[0m\]"
   else
       PS1="\[\033[1;36m\]**REMOTE**\[\033[0;31m\][\T]\[\033[0;36m\]\u@\[\033[1;33m\]\H \[\033[1;34m\]\w: \[\033[0m\]"
   fi
done

ack: grep but better!

2008-09-22T10:57:00.000-07:00

I came across ack today, and now grep is sleeping outside. It's very much like grep, except it assumes all the little things that you always wanted grep to remember, but that it never did. It actually left the light on for you, and put the toilet seat down.

Anyway, there were some rough bits with installation. On Ubuntu, it is installed by sudo apt-get install ack-grep. If you think, hey, I will take the obvious path and install "ack"! You will be thinking with wrongitude. For ack is, in fact, a Kanji code converter, which I discovered after not a little anger. On RHEL, CentOS, and the like, you have to install with cpan -i App:Ack.

Once installed, just try some ack-grep STRING.
Want to ignore .svn dirs? It already did. Want to recursively search? It already did. And it brought over a bag of chips. Read the Top 10 for more goodies.

Oh, and don't forget to add ack-grep --thpppt to exit your shell scripts:

Wordpress contact forms and mailing made less complicated

2008-08-11T21:15:00.000-07:00

After setting up some Wordpress blogs on my own server, I ran into the problem of wanting a basic contact form that would email myself and users. I in no way wanted to bother setting up my own mail server. I also would prefer not buying SMTP mail service. After a wretched chain of annoying plugins, I came across a salutary trio that met my needs: Cimy Swift SMTP, Contact Form 7, and a Gmail surprise.

Cimy Swift SMTP allows you to setup SMTP settings, which is perfect if you want to use an external service and not use PHP mail() function. Apparently, Google will let you send mail to any address from any domain using SMTP. Handy! So you just install Cimy, go to its Settings page, put in "smtp.gmail.com" as SMTP server, port 465 (it tells you this is for Gmail), your Gmail user and pass, and that you want TLS (it also notes this is for Gmail. Save, put in a test address, and it should just work. At least it did for me, where others failed.

Now you are all dressed up with SMTP, but where to go? I tried a number of contact form plugins that looked great, had lots of features, etc. And none of them bothered using the SMTP settings I put in. Contact Form 7 did, however. It's simple, and takes a little customization to make it look decent. But still, it is a basic contact form that works, and that is all I needed.

Needs more cow...say!

2008-07-16T08:58:00.000-07:00

I stumbled across a wonderful program today: cowsay! It is just as cool as you would imagine:

cowsay "Text"

gives you an ASCII-art cow saying your Text.

I think I will use it as the welcome banner on all my servers:

Handy Network Utilities: nast, fping

2008-07-10T07:57:00.000-07:00

I discovered a wonderful utility last night: nast. I wanted to scan my local network, just to see what devices were on it, what their addresses were, etc. I could ping my entire subnet, but what if I had a machine that rejected ICMP packets? (Not that I do very often, but it can happen in larger environments). I thought using a protocol like ARP would be much more robust, since it is pretty much guaranteed to be available. Then I discovered nast.

nast can do a lot of neat things. The first I found was mapping a subnet of course, which you can do by running

sudo nast -i eth1 -m

adjusting the interface to fit. It will return something like:

Nast V. 0.2.0

Mapping the Lan for 255.255.255.0 subnet ... please wait

MAC address             Ip address (hostname)
===========================================================
00:19:D2:92:20:CE       192.168.1.1 (MyGateway)
00:1B:C0:B7:86:CB       192.168.1.2 (MyBox) (*)
00:14:38:E5:76:10       192.168.1.5 (SomeOtherBox)

nast can also try to find out if there are any nodes on your subnet acting in promiscuous mode, which I think is pretty hot. Check out the main nast page for the full feature set.

In my searching, I also came across fping. Its main improvements over regular old ping are:

More than one host can be passed (as well as a file containing hosts)
Its output is very simple and easy to parse, making it ideal for scripting
It tries each host and moves on if there is no response, making the whole process faster

Instead of

ping -c 1 myhost1 ; ping -c 1 myhost2

you can just run

fping myhost1 myhost2

which just returns

myhost1 is alive
myhost2 is alive

Both nast and fping are available in the Ubuntu repos.

Crashing Application Greeter

2008-07-09T18:52:00.000-07:00

I was playing around with themes and the like in Hardy Heron recently, and I ended up selecting a new login screen. Apparently, something was wrong with the one I installed, however, since the next time I went to login, I was presented with a black screen and the message "The greeter application appears to be crashing. Attempting to use a different one." Hitting Ok on this merely cycled through X starting up and led to the same message.

To fix this, I went into another tty with Alt + F2, logged in, and ran "sudo vim /etc/gdm/gdm.conf". I commented out the line "Greeter=/usr/lib/gdm/gdmgreeter" and added "Greeter=/usr/lib/gdm/gdmlogin". After restarting, a simple Gnome login screen appeared fine, which allowed me to get in and change the login screen to a know safe default.

Additional Tips for GNOME-Do

2008-07-08T21:29:00.000-07:00

After having played with GNOME-Do for a few days, experiencing some pain and finding more fun things, I have some suggestions that may make your adoption of it smoother.

Get to 0.5. 0.4 will likely be in your repo, so beware! It is a paltry substitute. If you have installed that, remove it and delete the plugins dir. Then add the repos to your "/etc/apt/sources.list" as described here, then run "sudo apt-get install gnome-do". Also, make sure you kill GNOME-Do if it is running, BEFORE you remove the old version and try to install.
0.5 has a lot of great features, mainly the built-in Preferences window, which lets you just check plugins you want. There is also a new Open With action, which comes in very handy. Get to 0.5.
Very neat plugins: Skype (lets you chat or call with Skype contacts, like the Pidgin plugin), Upload to Flickr, Locate Files, Rhythmbox.
Install "xclip" (is in repos). This will let you select any text, invoke GNOME-Do, and act on that text.

Once I got to 0.5 and got those plugins running, GNOME-Do started becoming even handier!

Killer App: GNOME-Do!

2008-07-03T13:26:00.000-07:00

I started to doubt my beloved Ubuntu after a co-worker showed me the various virtues of Quicksilver on the Mac. But, I have... Beagle? Yuck. Lunchbox was cute, but never seemed to work right. But today, I take that doubt back, great Linux gods. Behold: GNOME Do!

If you know what Quicksilver is, then you are basically there. If you don't, just imagine a pretty app launched by a simple key combo that lets you do anything from open files, to play songs, to email people with just a few easy keystrokes.

David Siegel's beautiful site has lots of documentation, plugins, and more. Just be on Gutsy or Heron, add a repo to your sources, install. In case you don't know what "super-space" is, as I didn't, super is the meta key, so the one with a Windows Logo on it on most keyboards.

Some additional plugins I installed and found handy:

Del.icio.us - Search your bookmarks and public ones
Rhythmbox - Play songs in your library, control volume, and more
Pidgin - IM people in your Pidgin lists

Handy script: LAMP stack version printer

2008-07-02T21:51:00.000-07:00

Often when troubleshooting or setting up servers, I need to know the installed versions of applications commonly grouped into the "LAMP" stack. I dislike having to remember the slight differences between them when trying to coerce each to divulge their version. So I created a script that does that, and even prints it all in easy-to-read colors!

The code:

#!/bin/bash
###################################
#
# Written by Samuel Huckins
#
# July 2007
#
# Prints out version info for
# things in the LAMP stack.
#
###################################
#
echo -e "\e[1;34mThis machine's LAMP stack:\e[0m"
echo ""
# Linux:
LINUX=`cat /etc/issue`
echo -e -n " * \e[31mL\e[1;33minux: \e[0m"
echo "$LINUX"
# Apache:
echo -e -n " * \e[31mA\e[1;33mpache: \e[0m"
if [ -e /usr/sbin/httpd ]
then
echo "`/usr/sbin/httpd -v| head -n 1 | awk '{print $3}'`"
elif [ -e /usr/sbin/apache2 ]
then
echo "`/usr/sbin/apache2 -v| head -n 1 | awk '{print $3}'`"
else
echo -e "\e[37mNot present\e[0m"
fi
# MySQL:
echo -e -n " * \e[31mM\e[1;33mySQL: \e[0m"
if [ -e /usr/bin/mysql ]
then
echo "`/usr/bin/mysql --version | awk '/Ver/ {print $2, $3, $4, $5}' | sed 's/,//'`"
else
echo -e "\e[37mNot present\e[0m"
fi
# PHP:
echo -e -n " * \e[31mP\e[1;33mHP: \e[0m"
if [ -e /usr/bin/php ]
then
echo "`php -v`"
else
echo -e "\e[37mNot present\e[0m"
fi
# Perl:
echo -e -n " * \e[31mP\e[1;33merl: \e[0m"
if [ -e /usr/bin/perl ]
then
echo "`perl -v | awk '/This is perl/ {print $4}' | sed 's/v//'`"
else
echo -e "\e[37mNot present\e[0m"
fi
# Python:
echo -e -n " * \e[31mP\e[1;33mython: \e[0m"
if [ -e /usr/bin/python ]
then
echo "`/usr/bin/env python -V 2>&1 | awk '/Python/ {print $2}'`"
else
echo -e "\e[37mNot present\e[0m"
fi
# done
echo ""

The result:

Call it "lamp-version-printer.sh", and add something like "alias lamp="~/my/code/lamp-version-printer.sh". Couldn't be easier.

Minor caveat: While the script does check each program for existence, I only check the most common place. It could be better expanded to cover less used locations for applications, conventions on more OSes, etc.

Reference script for CLI color codes

2007-10-31T11:08:00.000-07:00

I have been experimenting with adding more color to my bash prompts of late. I find it easier to read when the fields are in distinct colors. The problem I always have is remembering what those wonderfully obscure ANSI escape sequences represent. I always have to look at a table to remind myself that "light red" maps to "1;31".

In an interesting guide to configuring your bash prompt that I have been going through, there was a script listed that makes this much easier:

#!/bin/bash
#
#   This file echoes a bunch of color codes to the
#   terminal to demonstrate what's available.  Each
#   line is the color code of one forground color,
#   out of 17 (default + 16 escapes), followed by a
#   test use of that color on all nine background
#   colors (default + 8 escapes).
#

T='gYw'   # The test text

echo -e "\n                 40m     41m     42m     43m\
44m     45m     46m     47m";

for FGs in '    m' '   1m' '  30m' '1;30m' '  31m' '1;31m' '  32m' \
     '1;32m' '  33m' '1;33m' '  34m' '1;34m' '  35m' '1;35m' \
     '  36m' '1;36m' '  37m' '1;37m';
do FG=${FGs// /}
echo -en " $FGs \033[$FG  $T  "
for BG in 40m 41m 42m 43m 44m 45m 46m 47m;
do echo -en "$EINS \033[$FG\033[$BG  $T  \033[0m";
done
echo;
done
echo

This is from the Bash Prompt HOWTO, from a script by Daniel Crisman. It produces the following when run:

Just add an alias like alias colors="~/dir/of/handy/scripts/print_shell_colors.sh", and you can be reminded whenever you like.

I am still experimenting with my prompt colors, but differentiating the fields I find useful has made things a lot easier:

View compressed log files easily

2007-10-30T12:20:00.000-07:00

Any linux distribution around these days will have compression and rotation in place for files in /var/log (or wherever else they happen to go). So if you look in there, you will see one or two log files (current and the last one) for each process logged, as well as 3-7 compressed files for the same process. These then get rotated out.

In any case, say you want to look inside one of those compressed log files. You could write out an untar command and then view the file. Then have to delete the temporary file. Messy. There is a better way.

The first I found was interesting (from PuppyLinux): Simply run man ./COMPRESSEDFILE. You view the contents of the file with the man file viewer. The annoying thing is that is does funny things with line breaks.

A much better way: zcat COMPRESSEDFILE | less. You view the file in less, and when done, there is no temp file to delete. To make it faster, just add a simple alias: alias viewlog="zcat $1 | less". This makes viewing those compressed logfiles as painless as viewing the current ones.

Loops in the CLI: Warm and Fuzzy

2007-10-23T19:55:00.000-07:00

It seems nearly every week, I find something amazingly handy in bash that I had not used before. This time: Loops. I would bet nearly everyone who has written more than a handful of shell scripts over a few lines in length has used one or more of the loops bash has to offer. But, what I realized (not sure why it did not bash me sooner) is that loops are also very effective timesavers as one-off commands in everyday shell usage.

I suppose the association came from first using the shell in a very simple way, of just giving commands singly, and only using constructs like loops in scripts I wrote out in files. But recently, I had several identical operations to perform on a series of files with consistent names, and a loop came to mind. The files were all mp4 videos, and I needed to convert them with ffmpeg, and send them through flvtool2. Instead of 2 commands typed out for each file, I ran:

for FILE in $(find . -maxdepth 1 -type f -iname \*large.mp4); do ffmpeg -sameq -i $FILE -s 480x270 -ar 44100 -r 10 $FILE.flv; done

You can use loops as like in any shell script, just write the whole loop on one line, separating the conditions, parts in the suite, and the termination with semicolons (these are mostly optional when using line breaks in scripts in files).

Post that inspired me, with lots of practical tips for loop usage.
More advanced loop documentation from TLDP.

Handy Command: Fuser

2007-10-23T07:48:00.000-07:00

Fuser is a very handy command when you are trying to investigate what is listening on a box. Consider the following case. You, being a diligent systems administrator, have been performing regular nmap scans against your boxes from remote hosts. You discover that something is listening on port 587 on a server.

What you immediately need to know is: what program is actually listening on that port? The quickest way to find out is to simply run sudo fuser 587/tcp on the box in question. This queries the kernel for what PID is listening on the specified port and reveals almost what you need:

587/tcp:              8102

The first column is the port you specified, the second is the PID using that port currently. This can be combined with ps to give you the desired output, such as via echo `sudo /sbin/fuser 587/tcp` | cut -d' ' -f 2 | xargs ps:

 PID TTY      STAT   TIME COMMAND
8102 ?        S     14:13 /usr/libexec/postfix/master

I used echo in this case because I was unable to decipher the delimiter used between the two columns in the default output. The whole thing should be aliased such that you run the alias and pass a port, and the ps output is produced.

NOTE: fuser is generally found in /sbin or /usr/sbin, which you may have to add to your path.

Easier Sharing: SSHFS

2007-10-20T20:21:00.000-07:00

My last post was on how I used Samba to share files from my central media server to other machines on my local network. Seemed great at first, but having used it for a little while, I was rather disappointed. I had Rhythmbox looking at the mounted drive as my library. It started scanning the files once the directory was mounted. I could play them, but after a few minutes, Rhythmbox would freeze scanning the files. In addition, I could not even list the directory that contained the mounted directory. Something was getting frozen. This kept happening more and more, and was quite frustrating.

I then tried of SSHFS for the same purpose, and it has been working much better. If you can SSH to the machine with the files you want to share, then you don't have to make any changes to that computer. Just a few steps on the machine that will be accessing the remote store, and you are ready to go. From the top:

Create the dir which to which the remote dir will be mounted (e.g. /home/myuser/Music)
Make sure you have ssh access on remote machine from the local machine
On the local machine:

sudo apt-get install sshfs

sudo modprobe fuse

sudo usermod -G fuse -a youruser

sudo chgrp fuse /dev/fuse
Logout of myuser on the local machine, then back in
sshfs 192.168.1.6:/dir/on/remote/server LOCALDIR

sudo modprobe fuse will make sure you have that kernel module loaded. You have to be in the fuse group to perform the mount, sudo or not. I am not exactly sure why you have to change the group of the fuse device, but you definitely have to to get it to work.

At this point, you should be able to view all the files in /dir/on/remote/server from LOCALDIR. Performance wise, I had great success. Pointing Rhythmbox to the dir I mounted, it scanned all the files in a timely manner, and I was able to play them as if they were on the local machine without issue.

If you wish to unmount the share, run sudo fusermount -u LOCALDIR. One problem that has been suggested for which I do not have a fix yet is how to handle if the remote machine has a problem, reboots, or the network connection is otherwise dropped. I am not sure how this would be handled, since the mount would no longer work. fuse being a kernel module, this may cause freezing. If I find this addressed somewhere, I will post it.

Additional information and links:

Quick and Handy Samba Setup

2007-10-19T20:39:00.000-07:00

I just never got around to setting up and using Samba on my home network. I had my files in SVN or would login to the box that had what I needed, or used some other mechanism. But a certain case kept popping up that forced me to get it in place.

I have one server that has all of my movies, music, etc on my home network. I wanted to be able to access all my music from other machines. Using gnump3d was, aside from annoying for long term listening, a waste of bandwidth. Instead, I created a share on my media server accessible without password only on my LAN, and mounted this as my music library on my other machines.

To set up the first part, I followed this handy tutorial. He has all the details, but basically you just installed samba on the machine you want to share from, edit the conf file to create a share with no password access locally, and restart samba.

Then, on the machine you want to access the files from, you install smbfs. After that, create a dir you want to mount the shared files from (in my case ~/Music), then run
sudo smbmount //myserver/myshare ~/mnt. After that, you should be able to browse files you are sharing from the mountpoint you specified. Easy!

More details on setting up Samba, as well as having a shared directory mount on boot, may be found here. One last thing. smbclient allows you to find information on local shares from the machine you are sharing from. For example, smbclient -L SERVERNAME allows you to see all available shares on SERVERNAME.

More Tasty Tips for DenyHosts

2007-10-16T15:57:00.000-07:00

After living with DenyHosts for some time, I have made some additional configuration changes that are quite helpful.

Whitelist known good IPs
I found after not very long that my home IP address had been added to my webserver's hosts.deny file by DenyHosts. This was a result of having messed up my own login a few times. This could easily be an issue if you are regularly connecting to the box running DenyHosts (from work, home, etc).
A simple solution: Add your IP to an allowed-hosts file in your DenyHosts working directory. Once that is in place, you might also need to find and remove the entry for the IP in question from your hosts.deny file (if it has already been added). It would be a wise choice to add known good IPs to allowed-hosts when you first setup DenyHosts. Otherwise, you might find yourself blocked from your server with no way to get access!
For more details, see the FAQ.
Blacklist bad users
It is also a good idea to add users you know are not allowed in a restricted-usernames file in your DenyHosts working directory. This keeps users that should never be allowed to login from trying, even once. There are two handy scripts provided to help in generating this list (located in the scripts dir of your DenyHosts working dir).
The first, restricted_from_passwd.py, scans your /etc/passwd file and outputs users who have nologin set (print daemons, etc). You can redirect this to your restricted-usernames file.
The second script is only handy after you have been running for some time. restricted_from_invalid.py, when passed your working dir, prints out the users which bad SSH attempts use most often. By default, it prints the top 10. You could also pass it a large number, say 10000, and it would print all the bad users ever attempted since DenyHosts was first started. I take this list, remove the users I have in place and know to be good, and use it as my restricted-usernames list.
Set the purge threshold and purge rate
In your denyhosts,cfg file, set PURGE_DENY to something reasonable, 1 - 2 weeks perhaps. Otherwise, your hosts.deny might good huge.
Then, set the PURGE_THRESHOLD to 2 -3. This means that while entries older than your PURGE_DENY will be removed from the deny list, if it occurs more than your PURGE_THRESHOLD, they will not be purged. This is more likely to stop the entries most often attempted.
Rotate log files
Assuming you have logrotate installed (and why don't you?), the FAQ has a great example file for adding a rotate entry for DenyHosts.
Sync with central DenyHosts database
One feature which really makes DenyHosts shine is the ability to share known rogue agents among DenyHosts instances. It is off by default, but to turn it on, edit denyhosts.cfg in your working dir. To enable synchronization, simply uncomment SYNC_SERVER. The value it is set to should be correct. By default, once synchronization is one, every hour the daemon will check with the server specified. It will upload hosts that you have denied, and download hosts that at least 3 others have denied. If you wish, you can tweak the timing of this process, just to upload, the threshold for downloading, etc.
For more details, see the FAQ.

After these changes are made, be sure to restart DenyHosts via sudo /usr/share/denyhosts/daemon-control restart.

DenyHosts: Watches your SSH log for you

2007-10-15T18:47:00.000-07:00

While I knew that brute-force attacks on SSH servers are very common, I had not taken the time to look at the connection attempt logs on my home servers until recently (to do that, by the way, on Ubuntu, try sudo tail -n 100 /var/log/auth.log). I was seeing attempts every few seconds for some periods, mostly on non-standard ports!

So far as I knew, no one had gotten through, but why risk the worry. Instead I installed DenyHosts. DenyHosts is a Python script that watches your auth.log, and adds IPs that repeatedly try and fail to connect to the /etc/hosts.deny list, effectively denying then future access.

It is rather easy to install. There is a package in the repos, but I was unable to get this to work on my servers for some reason (it is still in testing). I instead followed this handy tutorial. It worked flawlessly, with one exception. I had to run sudo touch /etc/hosts.deny right before starting the service. Otherwise it threw an error that the file did not exist and closed. With the touch, all went fine. That fix was listed in this bug report.

A few other notes:

If you have not done so, be SURE to change this line in /etc/ssh/sshd_config:

PermitRootLogin no

While editing /usr/share/denyhosts/denyhosts.cfg according to the tutorial, I recommend (following others posting this tip) to also change this line:

BLOCK_SERVICE = ALL. Also, of course, comment the line: BLOCK_SERVICE = sshd. This blocks access on all ports to IPs that get denied. And really, if you want to block a potentially malicious IP from SSH access, why give them access to other services?

Topping top: htop

2007-10-13T08:33:00.000-07:00

Top is a very handy and common application. But there is always room for improvement, even in common and simple programs. In this case, meet htop. It performs a similar function to top, viz. showing you what is using your system's CPU, memory, or other resources. However, htop presents this information is a much friendlier graphical way.

On Ubuntu, install is a snap. Make sure you have the universe repos enabled, and then run sudo apt-get install htop.

And no more obscure commands to remember for configuring layout and appearance. There is a handy list of commands at the bottom of the display. In addition, that display is in a wide range of handy colors (also configurable of course). The whole presentation is easier to read, and a lot easier to customize.

As mentioned, there are a lot of options for htop. Once you configure things and exit htop, your changes are saved to a .htoprc in your home directory. To see the basic options I have set, you can check out my .htoprc.

I like to have top (and now htop) running in the top left of my dual monitor setup. Using fluxbox, this is rather easy. See my .fluxbox startup and apps files.

Run vim without being there...

2007-10-08T12:02:00.000-07:00

Say you have a text file. You need to alter it in some regular way before sending it on somewhere else. Instead of editing by hand, there is a neat option you can use to edit the file with familiar vim commands (instead of awk's lovely syntax), without having to open vim interactively.

The principle is simple: You make a file containing each of the commands you want to run, one per line. They will be the same form as if you typed them in a vim session, starting with ":". Make sure that ":wq" is the last one. Then you run vim with:

vim -s FILEOFCOMMANDS.txt FILETOEDIT.txt

Example case. My file to edit is tester.txt, contents being:

64.114.33.23,www.example.com
64.114.33.24,www.example.co.uk
64.114.33.25,www.example.co.jp
64.114.33.26,www.example.co.fr
64.114.33.27,www.example.co.cz

Then I have a commands.txt file containing:

:%s/^/("
:%s/,/","
:%s/$/"),
:wq

Finally I ran it as vim -s commands.txt tester.txt. As a result, cat tester.txt:

("64.114.33.23","www.zenoss.com.co.uk"),
("64.114.33.24","www.zenoss.com.co.de"),
("64.114.33.25","www.zenoss.com.co.fr"),
("64.114.33.26","www.zenoss.com.co.cp"),
("64.114.33.27","www.zenoss.com.co.jp"),

You could put something like this in a cron job, making transitions between data manipulation steps quick and easy, without having to mess with the files by hand each time.

Handy Alias: Grab latest svn log

2007-10-07T12:33:00.000-07:00

I keep having the occasion to view the log comments for the current revision in my SVN repository. This means:

Knowing the number of the current revision,
Passing this to svn log -r

Instead of doing that by hand, I made an alias to do it for me:

alias svnlastlog="svn info | grep 'Last Changed Rev' | cut -d ' ' -f 4 | xargs -I mystr svn log -r mystr"

So if you are in a directory that is versioned, you just run that command, and (after being prompted to authenticate to your repo, if applicable), out comes the log entry for the current revision. Perhaps there is a built in command to do this already, but I could not find it via Google and the SVN Book.

What are all the wxPython Events?

2007-10-01T20:44:00.000-07:00

On several occasions recently I found did not know what event to specify to trigger something in my wxPython apps. I had a hard time finding anything close to a comprehensive list of all events available, so I was limited to posting questions on the wxPython users mailing list. And, while this is a wonderful and active list, it would be nice to have a guide even closer at hand.

Cody Precord on the list provided me with a handy way to get this:

import wx

for x in dir(wx):
  if x.startswith('EVT_'):
      print x

Run that, and out comes a list of all the EVT types. Of course, in some cases you may not know what the event is just from the name, but they are mostly suitable self-explanatory. See for yourself:

EVT_ACTIVATE
EVT_ACTIVATE_APP
EVT_BUTTON
EVT_CALCULATE_LAYOUT
EVT_CHAR
EVT_CHAR_HOOK
EVT_CHECKBOX

Clean your code: Pylint

2007-09-26T20:48:00.000-07:00

pylint is a tool that "analyzes Python source code looking for bugs and signs of poor quality". Now, that can mean optimizing and cleaning your code at a fairly advanced level at which, if you are a beginning programmer like me, you will not get much out of.

However, it can help with lots of small checking too, and improve the form of your Python code in a quick and convenient manner. It will show you problems you might not even know were problems! I have not gotten into too many of the features myself, but some quick examples should show you its utility. Getting it is pretty simple. On Ubuntu for example, installing is only a matter of sudo apt-get install pylint. To run it, just do pylint YOURAPP.py.

It can take a little while to run, and then it spits out several blocks of results. Some examples:

C: 68: Line too long (154/80)

One of the lines in the file has way too many characters. It is recommended to keep each line at 79 characters or less. To learn why, check out PEP 8, under "Code Lay-out", "Maximum Line Length".

C: 5: Operator not preceded by a space __version__="0.1"
W: 40:Frame.__init__: Unused variable 'statusbar'

No need for this variable anymore, just taking up space.

C: 93:App.OnInit: Invalid name "OnInit" (should match [a-z_][a-z0-9_]{2,30}$)

As can be seen from these examples, the alerts are pretty easy to read. pylint also shows you how many lines are duplicated, metrics such as how many lines of code versus docstrings, how many modules, and more. In the end, it gives your code an overall score. If you fix some errors and re-run it, it will show you your current and previous scores.

To start using the more advanced features, peruse the man page. Now, go clean your code.

Tips: Toolbar icons in wxPython

2007-09-24T19:13:00.000-07:00

I have started playing around with wxPython, a great toolkit for making clean, cross-platform, flexible GUIs in Python. While there are a lot of great tutorials out there on how to get started making apps with wxPython, having The Book has been a definite boon.

While going through the basic exercises and starting to make my own GUIs, I kept running into an issue that got more and more annoying: toolbar icons. I wanted to have a toolbar, with some basic icons, for actions like New, Open, etc. In the book, Rappin and Dunn give an example of using toolbar icons (p49). The pertinent part:

import wx
import images
[SNIP]
toolbar =  self.CreateToolbar()
toolbar.AddSimpleTool(wx.NewId(), images.getNewBitMap(), "New", "Long help for 'new'")
toolbar.Realize()

This in no way works. When I ran it on Ubuntu (7.04), I got:

ImportError: No module named images

After some Googling, I found this is a common problem for beginners. Lots of related example code will be similar to the above, but specify something like "stock_new.png" for the image. To get all these to work, you actually need to download some images, and specify those. Seems obvious after the fact, but it seemed quite possible to me starting out there there would be a built-in facility to grab OS-specific icons for basic tasks such as New and Load.

A great source of good-looking free icons for desktop applications is the Tango Desktop Project. Sometimes that page is down or takes a very long time to load. If so, you can download the .zip on this page. Every icon you will likely want for a toolbar may be found in "actions", once you extract the Tango folder and select an icon size to use. I found it helpful to place the icons I wanted in a "resources" folder in the folder holding my application, and use them with code such as:

    toolbar = self.CreateToolBar()
   toolbar.AddSimpleTool(wx.NewId(), wx.Image('resources/document-new.png',
       wx.BITMAP_TYPE_PNG).ConvertToBitmap(), 'New', 'Long help for New')
   toolbar.AddSimpleTool(wx.NewId(), wx.Image('resources/document-open.png',
       wx.BITMAP_TYPE_PNG).ConvertToBitmap(), 'Open', 'Long help for Open')
   toolbar.AddSimpleTool(wx.NewId(), wx.Image('resources/document-save.png',
       wx.BITMAP_TYPE_PNG).ConvertToBitmap(), 'Save', 'Long help for Save')
   toolbar.Realize()

That will result in something like the following (Menubar added for positional context):

Quad terminal laser!

2007-08-25T20:43:00.001-07:00

I find, not too infrequently, that I need to perform operations across several directories, or across several servers, at once. To best facilitate this, I usually open 3-5 terminals, and then move them around so that I can see them all at once. Why not automate this, I thought. Instead of hard coding sizes and positions for the terminals and assigning the sequence to a keypress, I thought I would do something more useful and write a Python script to find out this information based on the current display, and launch the terminals appropriately.

This has been completed, but the script still has a lot to be desired. You still have to set as a variable whether dual monitors are being used (could not find a way to determine this programatically yet), and it only works with single monitor or TwinView in Linux. You can also set what sort of terminal you want to use (gnome-terminal is default), what side you want them to open on for dual monitor setups (right is default), and what directory they should be in (user's home dir is default). It gets your screen's dimensions, calculates where the terminals should be and how they should be separated, and launches them.

I then added this to my .fluxbox/keys file:

Mod1 q :ExecCommand python /home/sam/code_homerepo/code/userextensions/quadterm.py

So now I just press "Alt + q", and four gnome-terminals launch perfectly spaced on my right monitor!

The code is quite ugly right now, but as it stands:

#/usr/bin/python
#####################################################
# Creates 4 terminals evenly quartering a screen
#
# Linux only right now. Map it to a keystroke for
# maximum convenience.
#
# If you don't want to use gnome-terminal, change the
# next variable. Warning: The width and height
# specified are in characters and rows for gnome-
# terminal; they might have to be in pixels for other
# apps.
#####################################################
terminal = "gnome-terminal"
#####################################################
# If you are not using TwinView, toggle the following:
#####################################################
twinview = True
#####################################################
# Which monitor (if Dual) you want them on:
#####################################################
side = "right"
#####################################################
# Dir the terminals will be in:
#####################################################
workingdir = "~"
#####################################################
import commands
# Get the screen dimensions:
dimeninfo = commands.getoutput("xdpyinfo | grep dimensions")
# Read out the needed numbers:
dimeninfosplit = dimeninfo.split()
dimensions = dimeninfosplit[1]
prex = dimensions.split("x")
x = int(prex[0])
prey = dimensions.split("x")
y = int(prey[1])
# Take some visual buffers into account:
widthsansborder = x - 50
heightsansborder = y - 50
# Calculate dimensions in pixels:
termwidthpixels = widthsansborder / 2
termheightpixels =  heightsansborder / 2
# Convert to characters/rows for most apps (like gnome-terminal).
# No idea what the conversion rate should be...
termwidth = termwidthpixels / 16
termheight = termwidthpixels / 60
# Constant width/height for all 4.
t1width = t2width = t3width = t4width = termwidth
t1height = t2height = t3height = t4height = termheight
# Find the offset position for appropriate pairs (in pixels):
t1posx = t3posx = 50
t2posx = t4posx = (50 * 2) + termwidthpixels
t1posy = t2posy = 50
t3posy = t4posy = 50 + termheightpixels
# If TwinView is being used, we need to halve some offsets:
if twinview is True:
   t2posx = (((50 * 2) + termwidthpixels) / 2)
   t4posx = (((50 * 2) + termwidthpixels) / 2)
   if side == "right":
       t1posx = 50 + (x / 2)
       t2posx = x * .75
       t3posx = 50 + (x / 2)
       t4posx = x * .75
# Spawn the 4 terminals, with needed positions and sizes, then exit quietly:
commands.getoutput("%s --geometry=%dx%d+%d+%d --working-directory=%s" % \
   (terminal, t1width, t1height, t1posx, t1posy, workingdir))
commands.getoutput("%s --geometry=%dx%d+%d+%d --working-directory=%s" % \
   (terminal, t2width, t2height, t2posx, t2posy, workingdir))
commands.getoutput("%s --geometry=%dx%d+%d+%d --working-directory=%s" % \
   (terminal, t3width, t3height, t3posx, t3posy, workingdir))
commands.getoutput("%s --geometry=%dx%d+%d+%d --working-directory=%s" % \
   (terminal, t4width, t4height, t4posx, t4posy, workingdir))
# For debugging, print what we have calculated:
#print "%s at: %d x %d + %d + %d" % (terminal, t1width, t1height, t1posx, t1posy)
#print "%s at: %d x %d + %d + %d" % (terminal, t2width, t2height, t2posx, t2posy)
#print "%s at: %d x %d + %d + %d" % (terminal, t3width, t3height, t3posx, t3posy)
#print "%s at: %d x %d + %d + %d" % (terminal, t4width, t4height, t4posx, t4posy)

All your base...

2007-08-23T11:21:00.001-07:00

Master Ian (whom I would link to, if his blog were not continually DOWN), showed me a very simple yet useful command: basename.

You give it the path to a file, it gives you the filename and extension. If you also give it the extension, it just gives you the filename. Thus:

sam@ZenSam:~$ basename unique-ip-list.csv
unique-ip-list.csv
sam@ZenSam:~$ basename unique-ip-list.csv .csv
unique-ip-list

This could be quite useful in shell scripting, to grab a list of desired for various purposes. One implementation I thought of right off: If you wanted to throw an error message for the usage of a command in a script, instead of having it display "See /usr/bin/zip for details" or something similar, you could run it through basename, and get a better looking name.

Domainspotter Project Homepage Launched!

2007-08-12T00:52:00.000-07:00

I have spent some time thinking about what to do with my Domainspotter program. I believe that it can be further developed into something quite interesting, and somewhat useful. As a result, I decided to create a site to house its documentation, feature roadmap, source code, and other related items. This can now be viewed at domainspotter.org.

There is not a whole lot there right now, but I hope to keep adding to it. This will minimally include the feature map as I have it planned now, and access to the SVN code base. I am still working on the content for the site, but it will mainly link to my roadmap in Trac for the project, as well as the code browser. It will be the home for all things related to Domainspotter.

In the longer term, I will also create a frontend to view results from domainspotter, updated weekly. In addition, I will also be adding various statistics and graphing displays to show the results in different ways.

So check there soon, hopefully things of interest will be found!

Fun with Python: Domainspotter

2007-08-04T06:57:00.000-07:00

In an effort to learn Python better, I will be writing a number of relatively small scripts to perform set tasks. I find that giving myself some goal and having to code towards it is more effective in learning syntax than going through coding exercises in a tutorial.

For the first such project, I wanted to determine how many words in a current English dictionary are not taken as domain names. This could have become a rather complicated exercise, but I limited it to the following:

single English words, using the AGID-4 dictionary
.com and .net TLDs only

There are many domains made with acronyms, combinations of names, made up words, and many other forms. But I was curious just how many of the more common terms in the language were still available.

The code as it stands is at the end of this post.

Before you look at how I did it, however (which is far from optimal anyway), I would encourage you to try out the exercise on your own, given the basic requirements stated above. I found that this program was perfect for learning more Python, since it required learning how to perform a number of disparate activities, such as reading and parsing files, using external modules, performing web lookups, and displaying desired data. These are, of course, some of the most general tasks most programs of use must perform.

A few notes. The whois lookups was the only section of the program that I assumed would require something outside of core Python. I came across rwhois.py, a module for performing whois lookups. I still have the code to implement this listed in the program as it stands for illustration, but found it did not meet my needs. It could not locate many domains I confirmed were taken using the simple command whois. So, I decided to fall back on something I knew: the command whois! This seemed to work fine for my needs.

Additionally, while I know the code could stand for optimization all over the place, I was curious how long the web lookups would take. To do this, I modified the lookup function to the following:


import time
self.totalchecked = 0
self.availdomains = []
times = []
for potdomain in self.potdomains:
start = time.time()
runthis = "whois %s" % potdomain
check = commands.getoutput(runthis).split('\n')
if check[8] == 'No match for "%s".' % potdomain.upper():
self.availdomains.append(potdomain)
else:
pass
self.totalchecked+=1
end = time.time()
timeforquery = end - start
times.append(timeforquery)
import pdb
pdb.set_trace()

I ran this against a much shortened dictionary file (the top 10 lines of the real one). Once the debugger launched, I did the following:

(Pdb) print sum(times) / len(times)
0.884103870392

This gave me the average time per web query. Now, note that I found this out after I started the final run for the first time... And since the actual dictionary file has 112,505 entries, I estimated the entire series should be done in around 27.6 hours. So I started my final run at around 7:32pm Saturday evening, and expected it to run until around 11pm Sunday evening. But instead I got an error during the night:

  File "domainspotter.py", line 125, in lookup
if check[8] == 'No match for "%s".' % potdomain.upper():
IndexError: list index out of range

This, I am pretty certain, was due to my internet connection dropping for a moment. This not being an unlikely event, I should add in handling for such situations, and perhaps write to the results output in chunks, instead of all at once! So the complete list will have to wait. But, running against the beginning of the list showed there are some exciting domains still out there:

Domains that seem to be available:

AMusD.net
Abelmoschus.net
Aberdonian.net
Abkhas.com
Abkhas.net
Abkhasian.com
Abkhasian.net
Abkhazian.net
Abnaki.net
Abramis.net

One last note you might already be asking yourself about: The AGID-4 dictionary contains proper and common nouns. I left the proper ones capitalized. It could be argued I should lower() them first, but I don't think one is able to buy a domain name with merely differing capitalization.

Current code of domainspotter.py:

[EDIT] The code base of domainspotter has changed around a lot since I posted this. Instead of having a big block of code here that is static, check out the latest form on my Trac site.

Beginner Python Debugging

2007-08-02T20:57:00.000-07:00

While I won't go into all the details and best practices of Python debugging here (because I do not know them), Chris Blunck shared a very easy and useful method with me that can be used in many situations. First, the code:

import pdb
pdb.set_trace()

That's it. All you have to do is put those two lines just before a line where, for example, an error is being thrown, or where you are not sure what the state of things is. pdb is the "Python Debugger", appropriately enough. There are several ways to invoke and use this module, other than what I listed above. The format I gave is most useful for setting a breakpoint. When your code runs and reaches where you have placed those two lines, you enter the debugging command line. You can check variable values as they exist at the breakpoint, execute code, whatever you want to try to understand the error you are examining.

An example:

>>> list = [1,2,3,4,5,6]
>>> import pdb
>>> pdb.set_trace()
--Return--
> (1)()->None
(Pdb) print list[2]
3

As can be easily seen in this very simple case, once the breakpoint is reached and one enters the debugging command line, it is trivially easy to, for example, verify variable values as compared to what you were expecting them to be. If desired, you can also continue executing the code after the breakpoint by pressing "c" on the pdb command line.

For details on other ways to use pdb, see the official doc. This separate guide has very useful information for Python debugging, specifically with pdb, for beginners.

An awesome xargs option and cleaning up SVN accidents

2007-08-01T06:31:00.000-07:00

I started using a wonderfully helpful option for xargs recently, after Master Ian showed me its use in a particularly thorny case. Before seeing the cool option, I will give the case as an example of its use:

I am still learning about the best (or even proper) way to use some aspects of SVN. In this particular case, I need to take all the files in a directory that was in the repo, move them into a new directory in the same directory, add another new directory, and copy new files into that.

To better explain the example, here is the structure I had in place, initially:

Directory 1 (In SVN)

FilesetA

And this is what I wanted in the end:

Directory 1

Directory1.1

FilesetA

Directory1.2

FilesetB

What I should have done is: made the 2 new directories (1.1 and 1.2) within the first one, copied the new files into one (FilesetB), add and commit both new directories and their contents. Then I should have run svn mv on the files in the top directory (FilesetA) into the second new directory.

What I actually did was: created the two new directories, simply mv'ed FilesetA into one, copied in FilesetB into the other, and then svn add'ed the two new directories and their contents. Then I had a problem. I still had FilesetA in Directory 1 in the repo, but not locally. So I needed to svn rm those, but I could not since they did not exist locally any more!

In the end, I copied the FilesetA out into a temp directory, when up to the directory above my repository, check the entire thing out again, svn rm'ed the FilesetA in Directory 1, and then needed to add FilesetA back into Directory 1.1. This was why I moved it to a temp directory. The problem was that that set had several nested directories with .svn directories in them. I had to remove these before adding them into the SVN'ed directories, else mass confusion would result. So I ran:

find . -type d | xargs -I % rm -rf %/.svn

This gem found all directories from the current directory down, and passed them to rm -rf. However, I could not just pipe the results and use xargs, as I needed to append "/.svn" to specify the right directories to remove. This is where -I comes in handy. This option replaces all instances of the string following it with the results of the first command (before the pipe). In this case the string was "%", for similarity to Python syntax, but any string could be used. The man page uses "replace-str". So in this particular case, if FilesetA included Directory 1.1.1 and 1.1.2, the command would resolve to: "rm -rf Directory1.1.1/.svn Directory1.1.2/.svn", which is precisely what was needed!

For some reason, I had a hard time understanding the option from the man page, but Ian translated it to Python for me:

for dir in [find . -type d]:
  cmd = "rm -rf %s/.svn" % dir
  os.popen(cmd)

This was much clearer to me. What it basically comes down to is:

OPERATION1 | xargs -I replaceme ANOTHEROPERATION replaceme OTHERTHINGS

translates into

ANOTHEROPERATION OPERATIO1RESULTS OTHERTHINGS

Chained No More, or, How I got WoW running perfectly in linux

2007-07-31T17:49:00.000-07:00

The single largest reason I had a Windows (XP) installation on my main home PC was to play games. Everything else I needed or wanted to do I could do just as well or better in linux. I did not try making the final switch with Wine, Cedega, and related solutions for gaming until recently, however. Windows is annoying, slow, prone to crashes, and generally unproductive. So why be stuck with it for fun?

After deleting my Gordian knot of a Windows installation in a moment of rage and adding only Feisty Fawn back, I was forced to face just that question. Getting Wine installed is very simple.

sudo apt-get install wine

works just fine, in fact. All you have to do after that is run winecfg once, make sure OSS is set under Audio, tweak your resolution under Graphics, and that's about it. I recommend that you check the emulate a virtual desktop option, and uncheck the allow window manager to control windows option. Before I made the first change, I could not even see the Wine window when it launched.

Something else to note: After installation, there will be a .wine hidden folder in your home directory. In this, there is a folder called "drive_c". This is where Wine installs programs (in "Program Files", intuitively enough).

To install WoW, pop in the first CD, navigate to your CD drive, and run wine Installer.exe. This will load the familiar installer, and run as normal. When it prompts for CD 2, just switch the disks and press OK and likewise until done. After install is complete, start up the game. To do that, you must navigate to where WoW.exe lives, and run it with wine. You should also pass the "--opengl" option, as using DirectX is generally problematic, and not necessary. I made myself a handy alias:

alias wow='~/.wine/drive_c/Program\ Files/World\ of\ Warcraft/WoW.exe -opengl'

Faster than a double click! You might be prompted to allow ActiveX controls when the Updater loads, which you should allow. (Wine installs it all for you, you just accept the option).

Now, depending on a whole legion of options, such as your graphics card, driver, the alignment of the celestial spheres, etc, you may encounter more and less graphical, sound, and performance issues when you first play. In my case, sound with OSS was fine, the app ran fine, and graphics were ok. Once I got playing, however, the graphics got slower, and were quite jerky in high traffic areas. There are several tutorials on what to tweak, but I found this one quite helpful. Once I performed the tweak with regedit, I was getting 80-90 FPS no problem (on a GeForce 6600 GT card no less), and 45-55 FPS with all the options on the highest setting. This is better than what I got in Windows!

Aside from running better, the other thing I love about running WoW in linux is that I can switch apps much faster. For example, if I have a browser open in my second monitor, all I have to do is... move my cursor over onto it. WoW does not mind, in fact does not even blink. I can immediately interact with the rest of my desktop. In Windows, I could not move my cursor out of WoW at all. I had to minimize it, then do whatever it is I wanted to do. No more chains!

[EDIT]: By popular demand, a few screenshots are in order!

WoW running on my Ubuntu Desktop, dual LCD setup, with fluxbox:

WoW in game shot from same system:

These were taken with my favorite screengrabber, ksnapshot. The last image is of my main, Temeluchus.

The scripts are speaking to me...

2007-07-29T16:00:00.000-07:00

WARNING: Reading about the following program may induce long periods of time wasting and excessive enjoyment.

Alright, now that that is out of the way... I started playing with a program called Festival some time ago, but have been unable to write about it until now. I am still trying to think out how it might actually be useful to the average linux fanatic, but it is unequivocally fun. While it is actually a very advanced linguistic program, the basic function that gives it greatness is: you give it text, it reads it to you. And not just in a Stephen Hawking sort of tone (while I love him too), but in a wide range of voices, from distinguished British gentlemen to American women.

While very neat to play around with, I had a few ideas on how it might be put to more practical use in interacting with a linux system.

Alerting: One immediate use that came to mind is to create a script that would monitor various statistics and warn you if they reached dangerous pre-defined levels. The warning could then be sent to Festival. Having my computer warn me of something out-loud would be much more effective than having to remember to check the metric myself.
Testing/Stats: I think it might be quite handy to have Festival read you the levels of certain resources or the progression of certain tasks while performing testing. While testing a program, it could read out available memory at the time, or the percent completed of a task being performed. This way you could keep track of needed statistics while monitoring others at the same time, or watching the program output.
News: This idea was proposed in one of the articles I read on Festival, and I think it might be quite useful. In short, you create a script to get the latest news items, likely from one or more RSS feeds you subscribe to, and send these to Festival. That way you could be read the latest news while working on other tasks.

There is an Ubuntu package for Festival, which makes the basic install rather trivial. Source may be downloaded here. It will compile on any *nix machine, OSX, and apparently Windows machines as well.

As mentioned, it is a rather advanced program with a wide range of features I will not go into here. Some helpful sites for further information:

Getting X display info easily

2007-07-27T22:22:00.000-07:00

When editing my X display configuration, I sometimes want to know what resolution is actually being displayed, as compared to what I have set in xorg.conf. An easy command to do this: xdpyinfo. This command is designed to provide X display information, just what I want!

The most common things I need are resolution and DPI. Here are examples of these:


$ xdpyinfo | grep resolution
resolution:    90x88 dots per inch
$ xdpyinfo | grep dimensions
dimensions:    3360x1050 pixels (948x303 millimeters)

This tells me just what I need to see to confirm if what I am trying to set in the X config is manifesting or not.

Of course, if you want to do it the hard way, you can use the Pythagorean theorem, and divide the diagonal pixel resolution of your screen ( equal to the square root of the width squared plus the height squared) by diagonal size of your screen in inches... I'll take the command.

Find out why webpages load slowly

2007-07-27T08:37:00.001-07:00

I came across a very helpful utility for web development today: YSlow. It integrates into everyone's favorite Firefox plugin, Firebug, and reports what is slowing the page the most, based on some standards for high performance sites.

Once you are on the page to check, you just have to click the YSlow button (sits beside the Firebug icon), it gets the info it needs, and displays something like the following:

You can expand each category to see details:
Check it out!

File renaming made simpler

2007-07-07T12:38:00.001-07:00

Some people have difficulty renaming lots of files at once on the command line. I used to experience this issue a lot, especially since "mv" was the command recommended. While I can appreciate why mv makes sense for renaming, it still was not intuitive for me. The original file is, in a sense, gone. Renaming a file can be seen as making a new file.

However, there is another command, nearly as available, which can rename things quite easily. You will never guess its name. Something simple, and descriptive of the action of renaming files...: rename! It syntax is easy:

rename 'PERL REGEX' FILELIST

The FILELIST can use regex as well. So if I wanted to renamed all of my Mahler "Symphony No. 1" files (they all started with "_- Mahler"), I would simply run this in the folder the files were in:

rename 's/^\_\-\ Mahler/Mahler/' *.mp3

Voilà! Now all my file names have been fixed in one fell swoop. An explanation of regex is not something I want or can do right now, but internet tutorials and articles on it are legion.

Generate Random Passwords on the Command Line

2007-07-02T20:22:00.000-07:00

Not too infrequently, I have the need to generate random strings on the command line, mostly for password creation. Usually, I go to a password generation site. However, this is slow, requires I have an internet connection at the time, and may be insecure as the site can record the password and my IP.

A far simpler way would be to have a script to do the generating for me. There are several such scripts for BASH floating around on the internet. But most of them are rather simple and do not let you easily select how long of a string you want. The following works nicely:

#!/bin/bash
###################################
# Created on July 2, 2007 by Samuel Huckins
# Adopted from the basic script floating
# around the internet.
#
# Generates random password. Of length equal to number given after
# command, otherwise 10.
#
# Uses /dev/urandom and not /dev/random since the latter
# can block without enough hardware entropy.
###################################
#
# Checks to see if user entered something after invocation.
# If so, use that to generate right amount of random bytes,
# and display that number.
# Only works up to 25 characters, not sure why yet.
if [ -n "$1" ] ; then
     let "RANDOMTOGRAB=$1+20"
     echo `head -c $RANDOMTOGRAB /dev/urandom |uuencode -m - |tail -n 2 |head -c $1`
# If they entered nothing, give them a 10 character random string.
elif [ -z "$1" ] ; then
     echo `head -c 20 /dev/urandom |uuencode -m - |tail -n 2|head -c 10`
exit 0
fi

I am not sure why strings longer than 25 characters are not working just yet. Passing a larger number displays a far shorter string than requested, and one which ends in several consecutive equal signs.

I need to solve this and improve its performance, but this basic script still work nicely for most tasks.

[EDIT, 10/09/07]: I just noticed that I forgot to mention originally that you will need to install uuencode. On Ubuntu, the package for this will not be obvious, since searching for uuencode in aptitude will not turn it up. You need to install sharutils. This includes uudecode and a few others in addition.

Use of split in VIM

2007-07-02T11:08:00.000-07:00

split is a super handy command to use in VIM (and vi, for those so inclined). It allows you to open a second (and third, and fourth, as many as you can fit!) window in your vim instance. These windows are not separate vim instances, but buffers within the original instance. They make it very easy to compare two files, a task which is common, but rather difficult on the command line with most tools. You can also move between them easily, which makes copying and pasting simple.

Ok, so how to do it. It is actually quite simple. If you open one file in vim, simply run ":split /second/file/to/open". This will split the screen horizontally and open the second file. You can do the same with "vsplit" to split vertically. To switch between them, press CTRL+w and an arrow key in the direction of the window you want to move into. Same for moving back.

If you simply run "split" a second window will open with the original file in it as well. Once it is open, you can move into it, and run ":e /second/file/to/open", and the second file will open. You can close the split by the usual ":q" or ":wq" to save and close the file, thus closing the split window.

If you find you lack the ability to sudo...

2007-06-30T21:12:00.000-07:00

While trying to setup Subversion, I needed to add myself to the subversion group. What I ran was "sudo usermod -G subversion sam". That was bad. Not, oh there is no more milk kind of bad. Bad like you just ran over a family member sort of bad. A section of the usermod man page will explain:

-G groups Supplementary groups given by name or number in a comma-separated list with no whitespace. The user will be removed from any groups to which they currently belong that are not included in groups.

Notice that second sentence? I didn't. I continued on my merry way until I needed to sudo. I ran sudo, and nothing happened. No error. Nothing. Being the only user on the system, I frantically tried to reverse the situation, since I would be able to do little of import without this. In the end, I found a related tutorial that let me fix it. The steps:

Restart the computer, and be at the console.
Press Esc when GRUB comes up.
Hover over the latest kernel (what you would normally boot) and press e (to edit it).
Go down to the line starting with "kernel /boot/vmlinuz" and press e.
At the end of that line, press space and then type "single".
Press b to boot into that kernel.
Once the system loads, you will be in single user mode, as root.

Of course, as root, you can change your password, your normal user's password, or (as in my case), add yourself back to the admin group.

After you login, you will be able to sudo, but you may notice some other tasks not working. In my case, I was unable to play any music or video files. I was baffled. I checked the permissions, they seemed fine. It turns out there are a lot of groups your user is put in by default, e.g. audio, cdrom, etc, which let you do things like that. If you use Gnome, go to System, Administration, Users and Groups. Find your user, view Properties, and then User Privileges. Check all those boxes, and all should be fine. Alternatively, you could make a new user and run id on it, comparing their groups to yours. This would work on default installs, without any custom user creation scripts.

As a side note, this method creates an extreme security flaw in certain systems. It is possible to turn off interactive GRUB editing if desired. For personal systems without sensitive data, it is best to leave it on, for eventualities like the one I encountered.

Personal Settings in SVN Increases Happiness by 125%!

2007-06-30T19:17:00.000-07:00

This may sound a little silly, but when I first started using aliases in Linux (and loving their potential utility), I was put off by the realization that once I got used to using them, I would end up being on another machine without them. This annoyance made me resistant to using them for some time. Eventually, I started using them so much, that I copied my .bashrc (and .vimrc and other such files) to all the remote servers I would need to interact with regularly.

This helped quite a bit, but it introduced another level of annoyance. Whenever I made a change, improved a command, added a new alias, I would have to add that to all the copies I had floating around. Dumb. Dumb and annoying.

So, I thought, why not set up a repo? While I use SVN regularly, I had never setup one on my own before, so it seemed like a win/win situation. In sum, the following is what I did to get it all to work.

The first thing to decide is how the repository will be accessed, either via SSH or Apache. The use of Apache is very common, and there are many tutorials on how to do it (like this one). I chose not to use Apache for 2 reasons: I did not want to have to be concerned with securing Apache on the server in question unless necessary, and secondly any computer on which I would want to access the repositories would have SSH. If you want to be able to see your repo in a browser, use Apache (or Trac).

Initial steps:

sudo apt-get install subversion # As of now on Feisty, this installs 1.4.3
sudo mkdir /home/svn/configfiles #For the repo, easy place to remember

You can replace "configfiles" with whatever you want to call the repo. For this case, it does not matter a lot, since I will be checking out the files in many various places, not all in one directory.

Now for permissions and such:

sudo addgroup subversion
sudo usermod -a -G subversion YOURUSER

And the real magic:

sudo svnadmin create /home/svn/configfiles/
sudo chown -R subversion configfiles # make sure perms are right after repo creation
sudo chmod g+rws configfiles # Ditto
sudo svn import /your/initial/files file:///home/svn/configfiles/repo -m "initial import"
sudo apt-get install xinetd

Now run

sudo vim /etc/xinetd.conf

and add:

defaults
{
}

service svn
{
port = 3690
socket_type = stream
protocol = tcp
wait = no
user = www-data
server = /usr/bin/svnserve
server_args = -i -r /home/svn
}

Now go into the conf dir in your repo and

sudo vim svnserve.conf

and edit what you like. I uncommented "anon-access = read" and "password-db = passwd".

You are done! The test: "svn co svn+ssh://SERVERWITHREPO/home/svn/configfiles". You will be prompted for as password 3 times, do not worry. If all is well, you should see "Checked out revision 1.", and the file(s) in the repo will not be in the dir where you ran the above command.

For more info, see the SVN Book. Thanks to this guide for getting me through the main steps.

[NOTE:] After some experimenting, I ended up creating the repo like this:

sudo svnadmin create /home/svn/configfiles/trunk
sudo svn import ~/.bashrc file:///home/svn/configfiles/trunk/bashrc -m "Initial import of .bashrc"

Then I checked it out via:

svn co svn+ssh://SERVERWITHREPO/home/svn/configfiles/trunk ~/DIR/TO/CHECKOUT/IN

Also, the way to delete a repo completely (as I did during testing) is simply to remove the entire dir (e.g. "sudo rm -rf /home/svn").

Exporting a MySQL DB to .csv

2007-06-29T12:21:00.000-07:00

I thought the operation would be simple. Short, no casualties, 100% chance of success. No such luck.

I needed to export the data of a MySQL database into a .csv format file. I did not want it in a huge XML file, and I did not need the actual .sql statements. A common need, one would think. The command to use is:

mysqldump -u root --fields-terminated-by=, --tab=/tmp --tables DATABASE

Now, this might seem simple. But what I tried first was having the " > dump.sql" file at the end, and had the path after "--tab" as my home directory. Now the first choice was fine, as you can separate where you send the data files versus where you send the table syntax. But, you will get an error:

mysqldump: Got error: 1: Can't create/write to file 'dumpdir/tablename.txt' (Errcode: 13) when executing 'SELECT INTO OUTFILE'

A post on the MySQL forums clued me in:
It's because the mysql user doesn't have the right permissions to write to that dir.
a simple chmod should fix the problem

So that led me to make a dir and chown it to the mysql user, which worked. However, if you write it to /tmp, you do not need to chown. So this version writes all the files to one dir without permission woes:

mysqldump -u root --fields-terminated-by=, --tab=/tmp --tables DATABASE

New and Helpful Aliases

2007-06-29T10:45:00.000-07:00

Some aliases I have added of late and found to be useful:

Displays a calendar and then your locale's 12-hour clock time:

alias now='cal;date +%r'

For those who often like to edit X display config:

alias editxorg='sudo vim /etc/X11/xorg.conf'

For those use window managers without a shutdown button:

alias turnoff='sudo shutdown -h now'

Does all you need to do for updating:

alias updateme='sudo aptitude update && sudo aptitude upgrade && sudo aptitude
dist-upgrade && sudo aptitude autoclean'

I am also working on something that I hope to works as follows. "help CMD" will run "CMD -h" or "--help", checking the results. If that fails, it runs "info CMD", then "man CMD". The idea being that you can use one command to display information on a command, and it will get you to the first that has something to show. This would be helpful when one runs into programs that do not handle -h, or only have a man page, etc.

Summary Execution

2007-06-29T09:30:00.000-07:00

I find myself coming across strange files on occasion. Crowded folders, shady thrice nested system dirs and the like are filled with them. Usually system files, files whose nature and purpose I might not know. To get to know such individuals, there are several built-in tools I might employ. I could run file on them, cat them, tail them, head them, run ls -lh in the folder, et al. But this takes too many steps. I want one command. And if I can get what I want in 5 steps, I can get it in one. It is the Linux way.

Thus I created the "summarize" function. It's a simple script that basically just grabs data using the methods I listed above, and formats them into an easy-to-read summary. Right now, it consists of:

#!/bin/bash
# A command to provide lots of
# info about a file at glance!
FILE=`file $1`
echo "$FILE has `wc -l $1 | cut -d" " -f1` lines." &&
echo -e "\e[1;34mOwned by: \e[0m`ls -l $1 | cut -d" " -f3`" &&
echo -e "\e[1;34mGroup: \e[0m`ls -l $1 | cut -d" " -f4`" &&
echo -e "\e[1;34mSize: \e[0m`ls -lh $1 | cut -d" " -f5`" &&
echo "**********************************"
echo -e "\e[1;34mTop bits: \n \e[0m" &&amp;amp;amp;amp;amp;amp;amp;
head -n 5 $1 &&
echo "" &&
echo -e "\e[1;34mBottom bits: \n \e[0m" &&
tail -n 5 $1
exit 0

One handy thing I learned while writing this was how to display text in chosen colors in bash, using the "echo -e" syntax. I find this makes it easier for my eye to quickly move to different fields.

You can also add something like this to your .bashrc:

alias summarize='/location/of/script/summarizer.sh'

Example output on a test .txt file (screenshot to show colors. Your base text color will be different):

[EDIT, 07/03/07]: It is also useful to add contextual lines to the head and tail output. This can be done via:

head -n 5 $1 | nl -b a
nl -b a $1 | tail -n 5

This prints the first and last five lines, numbered according to the number of lines in the entire file. I find this makes it much easier to immediately see how long the file is. (Additional note: without the "-b a" option, nl will print lines for each textual line, skipping blank ones. I myself would rather the blank lines also be included, as this is standard practice.)

Awesome App: Icepodder

2007-04-17T22:23:00.000-07:00

I have started listening to a lot of podcasts lately. I started with just the Linux Action Show. Then it spread, like an addiction. Now I keep up with 6-15 a week, some even in video! I find they are a great way to fill my morning walk to work with new information about all the latest releases, vulnerabilities, and just general linux/open source/tech stuff. In addition, they are, on the whole, far more amusing than tech news sites.

General praise for podcasts aside: The only issue really is how to track them, get the latest episodes, etc. I started with iTunes, but this very very quickly became an annoyance, since I did not want to have to boot into Windows daily, just for podcasts. So I looked around for a Linux podcast client. And, what a surprise, I quickly found a great open source alternative: Icepodder.

Icepodder is:

[...] a podcatcher (RSS client) for Linux conceived as a replacement for CastPodder. IcePodder is written in python, and requires python 2.4 or later, and pyGTK. The goal of this project is to provide Linux users with a dedicated client to download podcasts, with an emphasis on reliability and stability.

That sums it up nicely. There was a .deb, so install was a breeze. I started it, it looked nice, all was well. Then for the fun. I opened my handy dandy list of Linux podcasts, clicked Add New podcast in Icepodder, and dragged in the RSS URL of the feed I wanted. In a flash, it had all the information about the show and all the latest episodes, with the very latest selected to be downloaded. Much faster feed loading than iTunes. Of course, downloading can still be slow, but that is limited by your connection speed and the remote server.

Icepodder has some other neat features as well. It can keep running and checking for content in the background. You can configure it to keep or delete content based on its age. The list of features goes on. Check it out.

[EDIT, Thu, Apr 19, 2007 03:19:09]: I found after a while that Icepodder was indeed great for doing all sorts of things with podcasts, but that it has no facility for copying/syncing them to my iPod itself, which was the real goal for me. Perhaps a different integrated app would be better for this.

Putting Windows in its place: Virtual Machines

2007-04-14T18:29:00.000-07:00

I was getting tired of having to reboot my machine and go into my Windows XP install to run certain programs. Data was not a problem any more, since I can read from and write to NTFS drives from linux now. But running certain programs only available for Windows came down to using Wine, or rebooting. Wine is ok for some things, but it has its problems still, especially for demanding programs.

A better solution I am trying: turn my Windows installation into a virtual machine, and run that from linux. A basic howto on this process can be found here. A guide to install vmware server on Ubuntu can be found here. Installing VMware server is not as simple as one might wish in Ubuntu. Following the above instructions, all will go well, until you try to launch. You may get an error like:

/usr/lib/vmware/bin/vmware: /usr/lib/vmware/lib/libpng12.so.0/libpng12.so.0:
no version information available (required by /usr/lib/libcairo.so.2)

I found a fix for this:

cd /usr/lib/vmware/lib
sudo mv libpng12.so.0 libpng12.so.0.bak
sudo ln -s /usr/lib/libpng.so libpng12.so.0

However, that led this to happen when I tried to launch vmware:

/usr/lib/vmware/bin/vmware: error while loading shared libraries: libsexy.so.1: cannot open shared object file: No such file or directory

because Edgy had upgraded to libsexy.so.2.

However! I discovered at this point that I did not need to run "vmware". I needed to install and run the VMware Server Client for Linux. To do this, download the binary tar.gz called "VMware-server-linux-client-1.0.0-CURRENTBUILD" from the VMware downloads page. Then go to where you downloaded the file, and extract it to /tmp. Lastly do:

cd vmware-server-distrib
./vmware-install.pl

Follow the prompts, use the default values. Once it successfully installs, run it by "vmware-server-console". I ran it, and got this error:

vmware-server-console
/usr/lib/vmware-server-console/bin/vmware-server-console: /usr/lib/vmware-server-console/lib/libpng12.so.0/libpng12.so.0: no version information available (required by /usr/lib/libcairo.so.2)

But the console still opened fine. I selected Local Host, and imported the VM I had made of my Windows machine. For more details on importing, see this.

An alternate guide on the whole thing.

An important note: When you run your Windows VM, you will see and be able to run everything like normal if you were logged into Windows. You will also be able to change and add files. However, making changes here will not result in the changes persisting when you actually log into Windows. They are completely separate. This is a difficult problem, for which I have not found a solution yet.

[EDIT, Thu, Apr 19, 2007 03:16:27]: One additional problem I had that others may run into. When I made the VM from my Windows install, I did not think how large the virtual disk for it would be. It was as large as my C:\ drive, of course, ~70GB. The problem is, although I can see this from linux on the NTFS drive I made it on, I would need to copy it to a non-NTFS drive to be able to write to it, because of its permissions. In other words, be conscious of where you put the image initially with an eye to the final use.

Operatic Experimentation

2007-04-14T17:56:00.000-07:00

I love Firefox. I have used it for years, to the exclusion of other browsers. But, of late, I have been getting a little tired of its speed, or lack thereof. Initial loading, tabbing, and other features, were getting sluggish. I was tired of my browser being slower at opening things than my OS. I was tired of restarting the app just to use a plug-in. What is this, Windows?

So I decided to try Opera, as I had heard of it being like Firefox in many ways, but faster and with additional features. I have spent much less time with it than Firefox, but I will say that so far, it has been great. It is faster, it has more useful features, and it is more intuitive.

Things I have loved from the start:

Changing skins, adding skins, and adding widgets can all be done from sidebars or windows, not new sites. Also, they do not require a restart to use. This has been great. I HATE having to restart Firefox when I have 500 tabs open, just to start using a new plug-in or theme.
There are lots of neat display options and informative displays. The Info sidebar, for example, or the advanced loading display.
Built-in BitTorrent support, no external app needed.
You can get the tab bar to wrap to multiple lines. This is great if, like me, you like to have 500.8 million tabs open, and hate having to scroll right and left to get to them all.
It's faster. Did I mention that?

So, minimally, give it a try. It's neat.

System Tray in Flux and fbpager Config

2007-04-08T09:08:00.000-07:00

I needed a way to display system tray icons for applications like Gaim and Tomboy in fluxbox. One way that was easy to implement and works well is a dockapp called docker. After installing it (it's in the repos), just add "docker -wmaker" to your .fluxbox/startup file. Then when you, for instance, launch Gaim, the Gaim system tray icon will appear as a 24x24 (configurable size) icon in the docker box. Check docker --help for more options.

fbpager is a handy app for displaying what desktops you have windows open in, and allow you to go to them quickly. Its default display layout is a bit ugly, though. There is an easy way to configure the way it appears, however. First, make a file ~/.fluxbox/fbpager.conf, and put this in it:

fbpager.workspace.width: 32
fbpager.workspace.height: 32
fbpager.workspacesPerRow: 2
fbpager.alpha: 64

Then add this to ~/.fluxbox/startup:

fbpager -rc ~/.fluxbox/fbpager.conf &

This will run fbpager, and look at the specified config file. You can play with the values to your liking. See the linked home page for more options.

NTFS Read AND Write Access in Ubuntu, is Easy

2007-04-07T08:31:00.000-07:00

Write access to NTFS type HDDs was a little sketchy for some time in linux. This is a big problem for people like me that dual boot linux and Windows. Now, it is not only reliable, but extremely easy. This howto has all the steps you will need to take. Basically, I just added a line to my sources list, updated my system, ran the config, checked a box in the interface, and that was it.

One extra step I had to take was to boot into Windows, shut down fully, then go into Ubuntu and run the config. This was necessitated because my last Windows shutdown was not clean. This was caused by yet another ridiculous Windows error that impelled me to shut down via the power button instead of the ever-cheerful Start button.

After taking such simple steps, I decided to test write access out. I decided the best choice was to make a file on the Desktop of my Windows user called "Linux_Made_This.txt" containing:
"HAHAHAHHA
You have no chance to survive make your time."

Of course, you may find your own way to test as well.

Utility Assistance: dos2unix

2007-04-01T20:15:00.000-07:00

I had known of the utility "dos2unix" for some time, but I never had occasion to use it until recently. I have been working on a script to assist in system administration, and had put in Yahoo! Briefcase so that I could access the single file on various systems. In the process, it got Windows formatting added in (open such a file in vim, for example, and you will see "^M" characters peppering each line). This causes no end of errors when the script is run on a linux system.

So, I thought I would try the dos2unix command I had read about. Command not found... sudo apt-get? Nope. Not in the repos. It turns out that the utility is called "tofrodos" (that's right, no "m") in the repos, with the added explanatory note "alias tofromdos". So I installed this, and tried to run "tofromdos". Not found.

After some searching, I found that once tofrodos is installed, you can run "dos2unix [filename]", and it runs as you would expect. I am still not sure why, but at least the utility is useable.

Finding the time...

2007-03-31T08:58:00.000-07:00

After discovering that the clocks on several of the servers I work with now were out of sync, I decided to check my own system's accuracy. The simple command "ntpdate [server]" (on Ubuntu by default) will sync your clock with whatever server you select. "ntp.ubuntu.com" or any of the North American NTP pool servers (e.g., 0.north-america.pool.ntp.org) will work fine.

However, before you think of the clever idea of adding this command to a cron job or startup script, know that there is a better way: ntpd. This is a daemon that is designed to bring your clock in sync with time from the atomic clock time servers optimally. It adjusts the time to a synchronized state in small increments so as not to harm programs that rely on a regular passage of time, and so that your logs don't appear to jump. It works very well, and keeps your clock as accurate as possible all the time.

One catch: ntpd is not installed on Ubuntu by default, as they did not want any network listening apps installed on the desktop version by default. And good for them to improve security. The problem is that ntpd will not be found in the repos. It goes under the guises "ntp-simple" and "ntp-refclock". The latter is really only when you want to set up your own time server, and have the means to do so. The former is what should be used to keep "simple" (single, personal) systems synced. So install that.

There are a few more steps left. Run ntpdate to get the current time:

sudo ntpdate ntp.ubuntu.com

Then edit /etc/ntp.conf:

sudo vim /etc/ntp.conf

Add the lines after these 2 (should be there already):

# You do need to talk to an NTP server or two (or three).
server ntp.ubuntu.com
server 0.north-america.pool.ntp.org
server 1.north-america.pool.ntp.org
server 2.north-america.pool.ntp.org
server 3.north-america.pool.ntp.org

Lastly start up the ntp server:

sudo /etc/init.d/ntp-server start

To see the results:

sudo ntpd -p

Neat Firefox Shortcut and More Handy Display Config Candy

2007-03-30T06:49:00.000-07:00

I accidentally discovered an awesome shortcut in Firefox today. I wanted to switch to a workspace in fluxbox, but my sausage finger excuses for fingers pressed Alt+4, not F4, of their own volition. Serendipitously, Firefox had focus at the moment, and I was surprised to see my current tab switch... With confusion and adumbrated glee, I tried my mistake again. It turns out that Alt+NUMBER switches to that tab in the current Firefox window, handily similar to workspace switching. It is a great shortcut.

On an unrelated note, I was still having problems getting my fonts to look how I wanted in fluxbox. A friend directed me to try running "gnome-settings-daemon" (add "gnome-settings-daemon &" somewhere in .fluxbox/startup) at boot. Upon doing this, my fonts automagically looked as they should. I don't know what it does exactly, but run it.

Linux Display Config... Can be Simple!

2007-03-30T06:21:00.000-07:00

On rare occasions, one encounters a tool or application which makes blithely simple a task which was once annoying/hard/cumbersome/painful. Such a thing have I seen today: nvidia-settings. It has long been the case that having an NVIDIA graphics card has made using a GUI much simpler in Linux (although support from other vendors has been increasing). However, trying something as crazy and far-fetched as... dual monitors... has been somewhat of a dreadful task, involving much editing of xorg.conf, tears on the keyboard when startx reveals another error, et cetera ad nauseum. No, I don't want to see detailed output. I want a simple feature of having 2 monitors to work.

NO MORE! Simply type "sudo nvidia-settings" (in some window manager of course, gnome, kde, etc. Or fluxbox, if you are that cool), and a pretty little interface comes up, where you can do all the obscenely obvious things that one would want to do in configuring a graphical interface, like set up 2 monitors, configure refresh rate, and much much more.

It's amazing. In a few clicks, saving to my xorg file, and a restart of X, I had everything configured how I wanted it. No pain. No errors.

Dude, is Linux made of leprechauns? Cause it is awesome!

The Sound of Linux

2007-03-17T19:33:00.000-07:00

Recently, I came across some rather neat, if obscenely geeky, things to do with sound in linux. Actually, the first can be EXTREMELY useful. Often, using new systems, I do something that results in the PC speaker squawking. If it is trying to use tab-complete in a dir with too many matches available, or a bad VIM command, that tiny speaker decides to notify me of the error of my ways. After this happens one or two times, I am sorely tempted to put my fist through the computer, just to stop that wretched sound. There is a better way, viz., "modprobe -r pcspkr". This effectively kills the PC speaker, for your current session. No more squawking, no more pain. Once you logout, the speaker works fine, it just temporarily saves you the annoyance.

Another thing I have been experimenting with is much cooler. Apparently, you can take any file and send it to your speaker. To try this, get into the CLI, and first do "wget www.cnn.com". Then do "cat index.html > /dev/dsp". Of course, you can dispense with downloading cnn.com's homepage and use any file you want, of any type. Even "ls /etc > /dev/dsp" will do. As long as your sound is enabled, you will get some static. I believe what is actually happening is that you are sending the binary content of whatever you pass to your sound card, and it is playing through the speaker. Images, I find, are particularly interesting. An amusing experiment I have not finished: Make a recording, hopefully a comical one, encode it, and turn its binary content into some text. Embed this in a hidden div in a web page, and get someone to try the above commands on the page. I don't know anything about encoding, or much of anything at this level, but it is fun to play with.

Eterm Oddity

2007-03-17T18:46:00.000-07:00

While configuring my Fluxbox setup, I decided I wanted transparent terminals tailing various logs, or running top, from startup. I described getting most of this setup in a previous post. Since then, I ran into and solved two additional problems. The first is a result of specifying the positioning and size of the eterm windows via their title bar, using "Remember", which writes to .fluxbox/apps. Doing this for a transparent window can cause a black bar to appear at the bottom of said window. This is because of the second issue I ran into. When specifying the geometry of eterm through the command line option "-g", the format is "-g NUMOFCOLUMNSxNUMOFROWS+XOffset+YOffset ". The Offset items are in pixels, but the column and row specifications are in, of course, columns and rows, i.e. character widths and rows. So a setting of "-g 80x50+0+500" would make a window holding 80 columns, 50 rows, and it would be placed flush with the left side of the screen, and 500 pixels from the top.

This difference in units is what causes the black bar to appear when using Remember to set eterm's position and dimensions. This is because the Remember setting is in pixels, and a given set of pixel dimensions will not usually correspond to the exact number needed for a whole number of rows. I believe this difference causes the black bar.

Top to the Top

2007-03-10T20:10:00.000-08:00

I have used the top command many times to find out, for example, what errent process was eating up all my memory or CPU. Recently I configured fluxbox to start top in a transparent, borderless, bar-less window at startup, so I could always see what was up, no pun intended.

What I had not done before, however, was read man top. It turns out top is much more than a simple command. There is an entire interactive interface! Just to get a quick idea, type "top" in a terminal. Once it starts, press "f". Up comes a menu to pick which columns you want to see displayed! Just press the letter displayed by each item (its case toggles as you toggle each option), and it will display (or not) once you go back to top (press any non-mapped key, like Tab).

Another neat one: press "o" while top is running. This lets you order the columns that display. I found I liked to have them ordered thus: "Command, PID, S, User, %CPU, %MEM, TIME+". And those are all I really needed, so I disabled the rest. Now top is far less cluttered and much easier to read. Along this same line, press "B" (yes, capital). This bolds important fields and values. It also bolds processes when they are in a Running state, again very useful.

An additional informative note and two warnings: For those woefully uninformed of the coolness of top, it can do quite a bit more than I have mentioned here. You can do a lot with it interactively. But you must be careful as well. Pressing "k", for instance allows you to kill a process after specifying its PID. In addition, after you have configured the columns, order, and anything else to your liking, be sure to press "W". This will write your current setup to ~/.toprc. Otherwise, it's all lost once top is stopped.

For more information on top's abilities in a form shorter than the gargantuan man page for it, see here.

Dockapps for Flux's Slit Reviewed

2007-03-10T12:16:00.000-08:00

I have tried out quite a few dockapps over the last week after starting to use fluxbox. I have found some that are helpful, some that are neat, some that are dumb/ugly, and a few that are just plain broken.

To save others the heartache, I shall mention the best in show thus far:

wmcube - A neat looking dockapp that shows CPU use as a percentage, as well as via as a randomly selected rotating 3D object. You can change the object by clicking on it. Keeping abreast of CPU usage can now be done from the corner of your eye, since the objects' rotation increases accordingly.
wmtop - This shows the 3 items highest on the list provided by top. It can let you know if the app you think should be starting via a keyboard shortcut is taking a long time to start, or isn't starting at all, not to mention if a random thing is hogging your resources. There are 5 different skins for it, so check "wmtop --help" and find one you like.
wmusic - This is a tiny remote for the XMMS player. I looked and looked, but basically all of the dockapps for controlling music are for XMMS. This saddened me, until I tried XMMS and got hooked on it as well! It's basically WinAmp, but SO fast. Really. I added 10,000 songs in 2 minutes. They were immediately searchable. It responds immediately to whatever you try do to it. I was amazed, after having tried more music players in linux than I can count with both hands. Aside from these things, there are a ton of skins and plugins for it, greatly expanding functionality, and making it look like whatever you want.
wmcalclock - There are a lot of dockapps to show you the time and date, but I liked this the best because of the large, bright, easy-to-read display. Just what you need, nothing else.
asmem - The problem I had with the myriad system monitoring applets was that I had to click them to see CPU versus memory usage. I wanted dockapps that were always in a useful state. And I already had 2 neat dockapps for monitoring CPU usage. asmem only monitors memory usage, and it neatly displays RAM and swap use, in MB (configurable) and percentage free (or used). Be sure to pass this "-withdrawn", so that it doesn't have a menu bar and plays nicely with the other slit apps.
wmfire - This does the same thing as wmcube, but I like to have them both for eye candy. It shows CPU usage as a fire, increasing in intensity as usage goes up. It's pretty spiffy.
wmnet - Providing a quick eye to network activity, this highly configurable dockapp can be quite useful. You can change the refresh rate, all the colors, interfaces, and more. Be sure to pass this "-w" as well, so that it actually fits in the slit.
wmwifi - This applet displays what wireless network you are connected to, with scrolling SSID, a bar as well as percentage display of signal strength, and an antennae that indicates connection or lack thereof. A handy thing to have for those that often switch between wireless networks.
wmnd - I actually started using this in place of wmnet, since I found it looked much better. It displays the interface being watched, a blinking arrow indicator of upload/download activity, a numeric indicator of data transferred, up and down, as well as a nice graph of activity. It displays much more useful information than wmnet for the real estate used. I found that "wmnd -c red -C blue" works well. You can also lower the -s value to increase scroll rate ( I think 5 is default). Be careful with this. Setting it at 3 made my fluxbox just load the background, and nothing else. After playing with everything in .fluxbox/startup and restarting gdm on the CLI, I found this was the culprit. -c is the download traffic line, -C is for upload.

I thought of something usefully clever to add to wmnet that I have not gotten to work, as a result of my stupidity. You can pass it "-e COMMAND" so that it will launch COMMAND when you click the window. I wanted to view the results of netstat upon clicking it, so that if I saw odd spikes of network activity, I could immediately see the cause. BUT, doing this "-e gnome-terminal -e netstat" will not work. You will get this error: "wmnet: duplicate --execute", since wmnet sees two of the same option.

My proposed solution was to make an alias that I would call "netstat-term", simply being "gnome-terminal -e netstat", so that I could run my wmnet as "wmnet -l -d 60000 -t red -r cyan -e netstat-term", and get what I wanted, while not confusing wmnet. Two problems remain: wmnet does not have access to aliases set in .bashrc, and when I run "gnome-terminal -e netstat", the new terminal opens, runs netstat, and closes. I have tried several ways, but I can't manage to make it stay open as a new process. Hopefully I can solve both soon...

How Flux got his Groove back

2007-03-10T11:59:00.000-08:00

One thing you may (or may not) miss upon switching to Fluxbox in Ubuntu is the happy startup jingle. But fear not! You can get it back, via the instructions listed in the Ubuntu guide:

sudo apt-get install sox
gedit ~/.fluxbox/startup

Find this line:

exec /usr/local/bin/fluxbox

Put this above it somewhere:

play /usr/share/sounds/login.wav > /dev/null 2>&1 &

I did this, exited flux, logged back in, and the happy sound greeted me.

Transparent Terminals on the Desktop

2007-03-08T22:41:00.000-08:00

There are a number of cases in which you might want a terminal window always present on your desktop, perhaps tailing an important log file, or looking at the output of top. Whatever your reason, here are the steps to take if you just want magically dynamic text appear on your desktop.

First, assuming you have Eterm, start up an instance of it and Toggle Transparency in Background. You might also want to play around with the text color, to optimize it for your background. Then add an entry to start it when fluxbox starts in .fluxbox/apps.

Now for the window title bar, button bar, scroll bar, border, and positioning. All of these tweaks can be done via start options. Here's what I have for my Eterm entry in .fluxbox/apps:
[startup] {Eterm -g 60x50+80+5 -O -x --no-cursor -w 0 --scrollbar 0 --buttonbar 0 -e tail -f /var/log/messages}
Going in order: the "-g" options allows you to specify the geometry of the window. You pass it Width x Height + X offset + Y offset. So in my example, I have the window as 60x50, and it appears 80 from the left side of the screen and 5 from the top (this is just past where I have fbpager, which seemed like a convenient place). The -O option takes the background from the root window, assuring transparency. The -x option removes the window's border, -w sets the border width (0 in my case), and the scrollbar and buttonbar as 0 removes those as well.

The -e tells eterm to execute a command. I chose to follow tail /var/log/messages, but you can set it to whatever log file or process you want. Top is a good one to implement like this as well.

So with just that one line addition to .fluxbox/apps, you can have a permanent transparent log output to your desktop. It's useful, and looks awesome.

Fluxbox and Thunar

2007-03-07T21:42:00.000-08:00

I have switched over to using Fluxbox as my window manager with Thunar as the file manager. After some configuration fiddling, I find it much faster and more efficient than Gnome with Metacity and Nautilus. It runs a LOT faster. A couple things that might throw you off of using it at first: When you first use it after install, it is very plain. It needs help getting dressed. Also, configuration is done through text files. This should not be a problem, but it bothers some people.

Trying it is easy, at least with Ubuntu. Just install it, and log out of Gnome. Then select Options in the lower left, and select session as fluxbox. You can try it out once, or set it as default. There are a lot of tutorials out there on how to configure basic things like the background and menus and shortcuts, and they are mostly very helpful. Not to mention the fluxbox documentation itself, which is great.

Some suggestions:

Use Thunar for file management. I find Thunar to be just what I need in all cases thus far, and nothing else. It is fast and efficient. Here are some handy tips for using Thunar.
Unless you really want icons on the desktop, Rox can be more trouble than its worth. Case in point: It refuses to be transparent. Say you set a background via fbsetbg, then keep this constant by setting the appropriate rootcommand in ./fluxbox/init. Then you start Rox. Your background turns gunmetal, how peculiar! Of course, you can set a background in Rox, which works fine. Until you want to have an application, e.g. Eterm, be transparent. It will look to the original background, not the Rox one. And there is no way to make Rox itself transparent with respect to background, at least in the current version.
Move fluxbox tab placement into the title bar of windows. To me, that wart called a tab on every window got annoying very fast. Positioning them into the title bar means you can still use fluxbox tab functionality, without the wart.
The slit is a great thing. You can put lots of handy little dockapps in it to do everything from showing an LCD clock to watching log files. There are a lot in the Ubuntu repos, so run "sudo apt-cache search wm" first. Install what looks good, then add an entry for each in .fluxbox/apps. Restart fluxbox and see what you like. If you decide any of the dockapps aren't you, remove them from the apps file. If you want them in a particular order, you can put them in ./fluxbox/slitlist.

As a side note, it seems that .xpm files need to be used for the icons in the fluxbox menu. I found this out after trying to use .png files, a common format for icons. It's easy enough to convert them: Just open your favorite one per app in the Gimp, Save As, and xpm as the format. Usually, they are stored in /usr/share/pixmaps. Then just update the references to the icons in .fluxbox/menu, and voila, nice icons in your menu.

Middle Click to the Rescue!

2007-03-06T21:54:00.000-08:00

Yet another extremely simple but well-executed method to increase speed and efficiency present in Linux: Middle mouse button pasting. Open an application, anything with text, a terminal, browser, anything. Select some text. Any text you want. Now go into something else, say a terminal, and press your middle mouse button (or, for the 3 people left without a middle mouse button, press the left and right buttons at the same time (if you have 3 button emulation turned on)).

That's right, your selection just got pasted in. No CTRL+c, no CTRL+v, just select and middle button. Very quick and satisfying.

Some caveats: This only works for text, not for files. So you can't use it in all cases you would use CTRL+c and v, but it is still very useful for text. I find this especially so in a terminal where, if you are using gnome-terminal, for example, you have to copy and paste with Shift+CTRL+c and v. Also, you have to leave the application you selected text in open until you paste. This is because you are not really copying to a clipboard and pasting, but "moving" the selected text.

Fun Times in the CLI

2007-03-02T10:57:00.000-08:00

I discovered a few things in the last day that I found to be preponderously fun. Many familiar with the CLI may know them, but I did not, and so I shall share.

The first thing I still don't completely understand, but it works. If you run a script that has lots of requests for input with defaults that you want to accept, there is a better way to run it than "./MYCOOLSCRIPT.sh" and hitting enter a lot. Imagine if it was a script you wanted to run on 20 remote machines to configure something. We need a better way. And it comes in the form of linux's own black hole: /dev/null.

If you redirect from /dev/null to a running script (and to other things, although this is all I have tried), it somehow supplies an equivalent to pressing the Enter key. So you can start the script like "./MYCOOLSCRIPT.sh < /dev/null", and any prompts will be answered with the defaults. I have not been able to get a good explanation of how this works, aside from something about it providing STDIN to the thing it pipes to, but it does work. Beautiful Black Magic.

Another trick is less spine-tingling, but arguably more helpful. Sometimes after typing in a long command, I decide I need to do something else, change the middle of the command, etc. Basically, sometimes I want what I have typed gone, and fast. Usually, I press the Up arrow, hoping to Linus that I entered ls recently, and thus only have to press backspace twice. If I am unlucky, I must resort to holding backspace. Dumb. But, again, there is a better way! Namely, press CTRL+a, then k. You can either hold CTRL+a+k, or hold CTRL+a, release a, and press k with CTRL held. If you release CTRL and a, k will just be entered as a character, so don't do that. Another, shorter, method is CTRL+u. This does the same thing as CTRL+a, k on most systems. I have found that it does not work on certain older distros, so knowing the CTRL+a, k is useful as well. In addition, I found that CTRL+w erases the last word, as defined by a block of characters without a space or tab. This can be helpful if you just botched the last part of a command.

One last mot(e) of coolness. "-A" is a VERY helpful option for grep. If you invoke grep with "-A SOMENUMBER", it will print not only the line matching your text you are grepping for, but also the next SOMENUMBER lines. If there are multiple matches, each match and the specified number of context lines will be separated by a "--" line. I found this so helpful because in a lot of things grep, I don't just want the line the matched string is on, but a little context as well. Options to the rescue.

Ubuntu and the Search for a Better Font

2007-02-28T21:13:00.000-08:00

There are a few things about Ubuntu that I found lacking from the start. One of them was the lack of good fonts. Since I have to stare at the form I choose all day, I would like one that looks nice. In addition, there are some fonts that are well suited to normal application use, but that do not work on the CLI at all.

On the last point, this is a great guide on available monospace/fixed width fonts suitable for programmer's use, viz. in the CLI. It is OS independent and has links to download all the fonts mentioned, and samples of text in each one along with descriptions.

Regarding good looking fonts for GUI use, I will keep my preferences to myself. But, on how to get a variety of fonts to try:
sudo apt-get install gsfonts-x11 msttcorefonts
will give you lots of Microsoft fonts. Many look great. You have to give it to them, Microsoft does put millions of dollars into font design, and it shows. There are also lots of international options available, such as
sudo apt-get install xfonts-intl-arabic

You need to run this when you are done:
sudo fc-cache -f -v
Alternatively, you can restart X (usually mapped to CTRL+ALT+BACKSPACE).

Another good source in the repos:
deb http://www.elisanet.fi/mlind/ubuntu edgy fonts
deb-src http://www.elisanet.fi/mlind/ubuntu edgy fonts
Just add those 2 to the end of your /etc/apt/sources.list, then do sudo apt-get dist-upgrade. After that, run dpkg-reconfigure fontconfig-config again. Of course, update "edgy" to whatever release you are using.

This guide does a great job explaining how to install fonts in Ubuntu by hand and with a handy program called kfontview, which lets you find and install fonts in the GUI.

Aside from these sources and methods, you can put any new TrueType fonts you get in /usr/share/fonts/truetype, and restart X.

Another way to increase your eye candy variable is:
sudo dpkg-reconfigure fontconfig-config
Once this runs, select Autohinter, Always, No to the questions.
This turns on neat features to emulate Mac OSX font rendering. It will make you eyes happy. Highly recommended. When you finish running the reconfigure, be sure to restart X. One note: When I made this change, everything else worked fine, except for my System Monitor applet on the Gnome Panel. I had to change its width to 35 pixels to display full height. I found this rather odd, and you may not experience the problem.

Thanks to Master Ian for mad tips.

Handy *nix Commands, Part 1: Finding Processes

2007-02-26T11:46:00.000-08:00

On countless occasions, I have needed to find the PID of a process, in order to watch it or, more often, to kill it. For quite some time, I would simply run "ps ax | grep" followed by part of the process' name that I wanted to find. (Side note: it helps to also add " | grep -v grep" to this to eliminate the grep itself from the results.)

This works, but it is slow to use. I had to read through the results of ps to get the PID, when that is often all I wanted. One could use awk to grab just the PID column, but there are faster solutions:

pgrep (man) - Looks up processes based on options and displays matching PID. Helpful options are -f, finds text match of process name, -o finds oldest matching process.
pidof (man) - finds PID of a matching program name.

In both cases, you just pass the command the name (or part of it) of the process you want to find, and you get all matches. You can use pkill, or pipe to kill, to initiate the demise of the process once you find it.