Powerhouse Programs of Linux

Last Updated: Wed, Apr 29, 2009

rsync

There are hundreds of different backup applications out there. Rsync is one of those programs what once you learn how to harness its power you will never even think of backing up using anything else. Rsync is a staple utility that is used in almost all professional backup solutions. Not only is it the preferred method of backing up large amounts of data over a slow network, but also so efficient that it is the preferred way for transferring files to Amazon’s S3 service. When you pay for bandwidth by the bit you cannot afford anything but rsync.

Common Uses

rsync /path/to/source/ /path/to/destination

For a more detailed guide on how to actually put rsync to good use be sure to check out, Use Rsync for Daily, Weekly and Full Monthly Backups

dd

Not only is dd a court recognized bit for bit hard drive forensic imaging utility, it is also a great tool for making your own backups. Dd has proven itself time and time again. You can use it to copy CDs and other media. You can also use it to move data to another hard drive or cat out strings stored in RAM.

Common Uses

Clone one hard drive to another.

dd if=/dev/sda2 of=/dev/sdb2 bs=4096 conv=notrunc,noerror

Detailed dd examples and a guide to Rick Roll your Hard Drive with dd.

grep

Grep, Global Regular Expression Print, is an incredibly useful utility for finding strings of text in a folder full of files. Getting its roots directly from Unix this command has some tricks up its sleeves. Grep is often times paired with other commands and connected using the pipe, ‘|’.

Common Uses

Search history for the ‘mount’ command

history | grep mount

Find the process named, ‘firefox‘

ps -A | grep firefox

This searches all files and folders in the current directory recursively looking for, ‘some string’.

grep -r “some string” *

Find out what service uses port 25

cat /etc/services | grep 25

find

Find is an awesome utility for finding files. By itself it isn’t really that great but when you pipe the file output to other commands you can do amazing things. This is one of those commands that can increase your productivity 10 fold.

Common Uses

Find all TXT files and change gzip them.

find . -name *.txt -exec gzip {} \;

Find all PNG images resize and add a drop shadow at 90 degrees.

find . -name “*.png” -exec convert {} ( +clone -background black -shadow 60x5+0+5 ) +swap -background white -layers merge +repage {} \;

Real World Find Usage Find is a beautiful tool

sed

sed - stream editor for filtering and transforming text

Common Uses

Change all occurrences of red and replace with blue

sed ’s/#FF0000/#0000FF/g’ main.css

If you are new to sed definteily check out Eric Wendelin’s Get sed savvy Part 1, 2, and 3 Sed - UNIX Stream Editor - Cheat Sheet

awk

Awk is a very powerful text manipulation programming language that can be called from the command line.

Common Uses

Double space a file

awk ‘1; { print “” }’

Update on Famous Awk One-Liners Explained awk is a beautiful tool

xargs

Build and execute command lines from standard input. Xargs is a tool in which you a lot of complex commands are based on. You can use xargs to execute a specific command using multiple file names. Xargs is often used with the find command.

Common Uses

Delete more than one month old thumbnails from home directory

find ~/.thumbnails/ -type f -atime +30 -print0 | xargs -0 rm

history

History of commands

Common Uses

Replace a string in the previous command

^typo^correction^

Search previous commands for strings that match ‘mount‘

history | grep mount

alias

Make your own commands. Alias is a great way to take an otherwise long command and map it down to something of your own. Oftentimes SSH connections are mapped to smaller commands as well as others.

Common Uses

alias ls=‘ls –color=auto’

alias mybox=‘ssh [email protected]’

Using aliases and command-line functions for speed

gzip

Gzip is a very fast compression that is often used to compress HTML and CSS data on the web. It is also used to compress files on a Linux machine most notably log files. Take a peak in ‘/var/log’ to see what i mean.

Common Uses

Taking the dd command that we talked about earlier, we can backup an entire hard drive into a gzip file.

dd if=/dev/hdz1 | gzip -9 > /mnt/backup.gz

ssh

SSH is a secure remote login program and is absolutely awesome. I use ssh to login to remote servers every day. You can also use it to create encrypted tunnels for transferring data and also NAT traversal using reverse tunnels.

Common Uses

Login to a remote host

ssh [email protected]

Create a reverse tunnel

ssh -R 10002:localhost:22 middleuser@middle

scp

Secure file copy. I am usually using ssh to do most of my work and I use scp every time I need to move files remotely. No longer do I use the insecure protocol, FTP.

Common Uses

scp user@remote:/path/to/remote /path/to/local

tail

Output the last parts (tail) of a file(s)

Common Uses

View the last kernel messages (helpful to detect boot problems)

dmesg | tail

head

Like tail this ouputs part of a file. Head outputs the beginning of a file.

Common Uses

Print the first few lines of ‘filename.txt’

head filename.txt

Print the last modified file

ls -t1 | head -n1

ln

Make Links between files

Common Uses

Create a symbolic link

ln -s file/name linkName

curl

cURL seems to be the defacto standard application for integrating online web service APIs. I have personally used cURL to do some pretty cool stuff with my own PHP code as well as local bash/perl scripts. It is often used to emulate a web browser in automation scripts.

You can use cURL to:

• Interact with Google data services

• Get Your Twitter Updates with Curl

• Track UPS Packages with cURL and PHP

• Calculate UPS Shipping Rates with cURL and PHP

wget

wget is like the command line version of cURL. There is no library for wget and is usually not integrated into programming languages directly like cURL but is definitely a powerful tool to have in your arsenal.

Common Uses

Download Ubuntu from the command line

wget http://ubuntu.cs.utah.edu/releases/jaunty/ubuntu-9.04-desktop-i386.iso

Use wget to Download Youtube Videos

wc

wc is a simple app that packs a handy feature. It has the ability to output counts for: bytes, chars, lines, longest line, words. You can even give it a list of files to check by using the, ‘–files0-from’ flag.

Common Uses

Do a word/line count on ‘myfile.txt‘

wc myfile.txt

netcat

The swiss army knife of networking and was ranked #4 for top security programs. This program can be integrated into scripts or used all by itself to do some amazing things.

You can use netcat to:

• Transfer a dd hard drive image over the network
• Transfer files
• Create a chat environment
• Pipe audio over a lan

Common Uses

Make the bash process a service

nc -l -p 12345 -e /bin/bash

Create a makeshift webserver

while netcat -lp 8080 -c 'echo HTTP/1.0 200 OK';echo;cat file;do;done

A Unix Utility You Should Know About: Netcat