Thoughts on Programming

August 8, 2011

Unix commands: sed & awk

Filed under: Linux — shadiyya @ 5:20 pm


It is a stream editor is used to perform basic text transformations on an input stream. The sed utility is a stream editor that reads one or more text files, makes editing changes according to a script of editing commands, and writes the results to standard output.

The pattern to match is typically included between a pair of slashes // and quoted. For example, to print lines containing the string “1024”, we may use:

cat filename | sed -n ‘/1024/p’

Here, sed filters the output from the cat command. The option “-n” tells sed to block all the incoming lines but those explicitly matching the expression.
Here is another example, this time for deleting selected lines:

cat filename | sed ‘/.*o$/d’ > new_file

In this example, lines ending with an “o” will be deleted. It uses a regular expression for matching any string followed by an “o” and the end of the line. The output (i.e., all lines but those ending with “o”) is directed to new_file.

To search and replace, use the sed ‘s’ action, which comes in front of two expressions:

sed ‘s/string_old/string_new/’ filename > newfile


The awk is mostly used for pattern scanning and processing. It searches one or more files to see if they contain lines that matches with the specified patterns and then perform associated actions. Awk breaks each line of input passed to it into fields. By default, a field is a string of consecutive characters delimited by whitespace, though there are options for changing this.

Awk runs through a text file by reading and processing one record at time. Its commands are written with the intention that they act repetitively on each record as it is read in to awk. A record that has been read by awk is broken into separate fields, and actions can be performed on the separate fields as well as on the whole record.

As awk processes each line of the input file, each word on the line is assigned to variables named $1 (the first word), $2 (the second word), and so on.

Let’s start with a file, words.txt, that contains these lines:

nail hammer wood
pedal foot car
clown pie circus

Now we’ll use the print function in awk to plug the words from each input line into a template, like this:

awk ‘{print “Hit the”,$1,”with your”,$2}’ words.txt

Then the output will be:

Hit the nail with your hammer
Hit the pedal with your foot
Hit the clown with your pie


Leave a Comment »

No comments yet.

RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

Create a free website or blog at

%d bloggers like this: