Generating Mock Userdata

I often find myself needing to generate dummy data when doing software projects. The rig command-line utility makes it easy to do so.

# apt-get install rig

$ rig
Maureen Cummings
614 Spring County Blvd
Phoenix, AZ  85026
(602) xxx-xxxx

Cheap Process Monitoring: Echo, Watch, and Netcat

I frequently work with memcached, gearman, and a variety of other services that allow you to retrieve useful stats by connecting via telnet and issuing a status command. In a lot of cases, retrieving these stats in realtime is beneficial for debugging and diagnosing data pipeline issues.

Fortunately, a few basic tools found on almost any Linux system can make this easy.

The watch utility runs whatever command you provide it at the interval you specify. Here's the world's cheapest realtime clock.

$ watch -n 1 'date'

Netcat is too versatile to cover in depth in this post, but one of its most basic capabilities is accepting data via standard input, sending that data to a host / port combination of your choosing, and then dumping the result to standard output.

$ printf "HEAD / HTTP/1.0\r\n\r\n" | nc google.com 80 | grep Server
Server: gws

The above command takes the HTTP HEAD request from printf, sends it to netcat, which then passes it on to google.com on port 80. The results are printed to standard output, and the grep command limits the result output to the server returned (in this case gws).

Combining these two commands provides a ton of utility. Here are two useful examples that I use regularly.

1) Monitoring Gearman queues in realtime

watch -n 1 '(echo status; sleep 0.1) | nc localhost 4730'

publishMessage  4       0       1
consumeMessage  7       0       8
archiveMessage  28      0       12
.

2) Monitoring memcached get/set commands in realtime

watch -n 1 '(echo stats; sleep 0.1) | nc localhost 11211 | grep cmd'

STAT cmd_get 506
STAT cmd_set 8
STAT cmd_flush 0
STAT cmd_touch 0
STAT auth_cmds 0

Sort: Human Numeric

The sort command can accept input data formatted in human readable form. The du -h command is a longtime favorite for finding file sizes in a format that's easy to read with the naked eye. Coupled with sort, you can easily find and display the largest files in a directory tree.

$ du -h | sort -r --human-numeric | head

Update: I should have mentioned that --human-numeric is a recent addition to GNU coreutils, so this won't work on OS X or older distros. Here's a more generic solution (borrowed from here).

for i in G M K; do du -ah | grep [0-9]$i | sort -nr -k 1; done | head -n 11

Handy Shell Pattern

When piping commands, you can often benefit from breaking your data into multiple lines. A combination of sed and xargs can make this very easy to do. Take the following commands for example:

cd /tmp && mkdir blah && cd blah;
touch file-1 file-2 file-3;
ls | sed 'p; s/-//' | xargs -n2

Breaking this down, we create a directory and touch three empty files. From there, we use sed to print two lines. The first line is the original file name, the second line is the filename without the dash. From here, we can use the -n argument in conjunction with xargs to merge the two lines back into a single line. The resulting output would be:

file-1 file1
file-2 file2
file-3 file3

Once you have the lines joined, you could do something like:

... | while read a b; do mv $a $b; done

Bonus tip: the "read" bash builtin takes a line and associates each word with a given variable. Type "help read" from the shell for the complete documentation.

In summary the full command-chain for this tip would be:

cd /tmp && mkdir blah && cd blah;
touch file-1 file-2 file-3;
ls | sed 'p; s/-//' | xargs -n2 | while read a b; do mv $a $b; done

Thanks to Chris Sutter for contributing this handy shell pattern, and thanks to Andeers for the coffee!

bashreduce

I stumbled across this project recently. It's a program that allows you to apply standard shell tools in a map reduce style fashion. Large memory hungry tasks like sorting can be distributed across machines to maximize throughput. As a bonus, it's written in bash, so it's highly portable and works with a combination of vanilla shell utilities and ssh shared-key authentication.

The bashreduce project.

Bash, Re-writing a Line

There are instances when writing a shell script that you want to write to the same line over and over. This is mainly used for progress meters that update the same screen position over and over. Most languages represent a carriage return without newline as the \r escape sequence. In shell, it's no different, but there is one small caveat. The echo command requires two arguments.

#!/bin/bash

# simplest progress meter
items=100000
for ((i=0; i<$items; i+=1)); do
    echo -n -e "Processed $i/$items\r"
done

The -n argument to echo tells it to suppress the implicit newline that it usually includes. The -e argument tells it to respect escape sequences such as \r.

Preserving History

If you use more than one terminal during your workday, you've probably been annoyed that your shell history always seems incomplete. This is due to most shells overwriting existing history data in lieu of their own history when a session is closed. Fortunately, both bash and zsh support variables to avoid this behavior.

For bash, you should be able to just add the following to your bashrc:

shopt -s histappend
PROMPT_COMMAND='history -a'

For zsh, I use the following history related options:

setopt appendhistory histignoredups
setopt histignorespace extended_history
setopt inc_append_history share_history

Bash Substring Manipulation

I recently completed a project for work to automate pulling specific subversion revisions of code and publishing them to various servers. While relatively simple in practice, a number of operational dependencies had to be considered, and it had to be written in bash. I've done a bit of shell programming in the past, but never really written a "system" in shell. Not surprisingly, I learned a few things in the process, which I'll be sharing on this blog as they come to mind. One thing I learned, which will come in handy down the road, is that bash has a number of builtin substring operations. As an example, I had to trim a number of paths to insure that there wasn't a trailing forward-slash at the end. Obviously, there are a few ways of doing this, but I found a very convenient and simple idiom that I'll be adding to my toolbox.

${string%substring}

removes shortest match of substring from end of string

so, if mydir="/home/travis/"

echo "${mydir%/}" will produce /home/travis
echo "${mydir%/*/}" will produce /home

What's nice as opposed to just chopping the last character off the string is that if the string is already formatted without a trailing slash, nothing will change.

mydir="/home/travis"
echo "${mydir%/}" will produce /home/travis

History

Most people know you can use the Unix history command to see what's been done on the command-line recently.

travis@travis-desktop:/home/travis% history
9985 ls
9986 grep -ri pidgin
9987 grep -ri pidgin *
9988 cd ..
9989 ls
9990 cd .gnome2
9991 grep -ri pidgin *
9992 cd ..
9993 cd .purple
9994 ls
9995 vim prefs.xml

What some people don't know, is you can execute anything in the history list very easily. Just use !number where number is the item number on the left-hand side of the history output.

!9995 would execute `vim prefs.xml'.

Quickly Truncate a File

Quick shell tip:

If you want to truncate a file to zero bytes from the command-line, here's an easy way.

user@host [~]$ > filename

I used to use:

user@host [~]$ cp /dev/null filename

But the first approach is less typing.