Stream editor (sed) edits text files without opening them.
Most often, the editing would be a replacement of one character or word by another. This way a delimiter of a csv file can be changed from comma to semicolon, name of a person can be replaced throughout the text. But that is just a beggining; sed has much more to offer:

  1. The “word” can be a regular expression making it super flexible
  2. The lines where this replacement will be done, can be specified by their line number, range, or content

Taken togehter, sed helps to delete, replace, and append characters, words, lines or whole files.

Useful oneliners

Change semicolon to comma in csv file

sed 's/;/,/g' file.csv > changed-file.csv

Make a new column out of last two digits before .tif extension which are preceded by an underscore

sed 's/_\([0-9]\{2\}\).tif/,\1/' file.csv > new-file.csv

How sed operates in general

sed operates on individual lines of the input

  1. reads first line - puts into a pattern space
  2. excutes commands
  3. prints what is in pattern to output
  4. clears the pattern space
  5. reads next line

So if you supply no commands to sed sed '' file it just outputs the content of the file. It is the same as running cat file

The -n switch prevents default printing

There is an important switch -n going ahead of the commands. When -n flag is present it negates the action of a 3rd step: negates content of pattern space to output.

Example

sed -n '' file`

runs and prints nothing

Telling the sed what to do with which lines

The ’ ’ structure is used to define range of lines on which to operate specific commands.
The general structure is: sed 'ranges commands'.

More commands can be included inside the ' ' to run consequtively when seprated by semicolon: sed 'ranges commands1; commands2'

Ranges

Ranges are defined by line numbers or pattern matching

Examples:

'1,2' first two lines
'1,+N' first line plus N more lines
'/pattern1/,/pattern2/' on lines from appearance of first pattern to the first occurence of the second pattern

Commands print, delete, change, append

These commands tell sed what to do with the content of the pattern space.

Caution:

Remember that the default action of sed is to print everyting which is in the pattern space after the commands on the line are executed.
This will lead to suprising doubling of lines when using print.
Use -n flag to supress this behaviour.

print

Examples:

sed '1,2p' file

Will print the first lines twice each then the rest as normal.

sed -n '1,2p' file

Will print the first two lines, does not print the rest

delete

sed '1,2d' file deletes first two lines sed '/pattern1/,/pattern2/d file deletes lines from pattern1 containing line to pattern2 containing line

change

sed '1,2cXXXX file' will replace the first two lines with a single line containing XXXX In case you weant to have a multiline replacement use ‘\n’ inside the string

read

sed '1,2rfile2' file1 will put:

  • first line of file1
  • whole file2
  • second line
  • whole file2
  • the rest of the file

substitute s/pattern1/pattern2/gI

To replace a pattern1 with a pattern2.

Examples:

  • sed 's/hello/goodbye/' file will replace the first occurence of hello with goodbye on all lines.\
  • sed 's/hello/goodbye/g' file will replace all occurences (g goes for global).\
  • sed 's/hello/goodbye/gI' file will replace all occurences of hello case independently (Hello, heLLo and so on).\

To delete pattern use sed 's/hello//' file which will delete first occurence of hello.

To delete a pattern in specific range sed '1,2s/hello//g' file will replace all hello occurences on first two lines.

Using regular expression groups in substitute command

  1. It is possible to use regular expressions for the patterns, like [a-Z][0-9],
  2. It is possible to group the regular expressions and use them for the substitution

Example:

sed 's/_\([0-9]\)/_a\1/' file will insert “a” before a first digit following an underscore.