Sort: Human Numeric

The sort command can accept input data formatted in human readable form. The du -h command is a longtime favorite for finding file sizes in a format that's easy to read with the naked eye. Coupled with sort, you can easily find and display the largest files in a directory tree.

$ du -h | sort -r --human-numeric | head

Update: I should have mentioned that --human-numeric is a recent addition to GNU coreutils, so this won't work on OS X or older distros. Here's a more generic solution (borrowed from here).

for i in G M K; do du -ah | grep [0-9]$i | sort -nr -k 1; done | head -n 11

Sorting Files By Size

In case anybody was wondering, I'm still alive. I've been inordinately busy lately and have been spending the majority of my free time online re-learning Ruby on Rails. I've also been digging into multi-master and circular replication schemes for MySQL and MySQL proxy. I plan on sharing some of the things I've learned sometime in the near future. Anyway, here's a handy tip that I just used this morning.

If you want to sort all the files in a directory by size, you can do the following:

ls -lsR | sort -nr

The sort command will order by the number of blocks in each file. The biggest files are at the top.

Sort + De-dupe? Easy.

Sometimes when dumping data, it makes more sense from a performance perspective to not worry about removing duplicate data when constructing a SQL query and do it after the fact instead. This is accomplished really easy on the shell as follows:

cat filename.csv | sort --buffer-size=32M | uniq > filename_uniq.csv

You can omit the buffer-size argument to sort in favor of the default size or set it to whatever you want.

Filtering

Sometimes it's helpful to filter your current editing session through an external command. Vim makes this easy to accomplish with the following ex command, :%!command. To filter a file containing strictly numbers through the Unix sort command using numerical comparison, you would :%!sort -u