A page to collect together various bash tricks I’ve needed.

Recursive Search and Replace

Edit files in place, applying a simple search and replace:

find [1,2]* -type f -exec sed -i.bak 's|foo|baz|' {} +

Note the + at the end, which allows find to pass multiple files to the command to be exec-ed. This reduces the number of invocations of sed.

Insert a line (if it’s not there already)

The scenario: I have some YAML meta data on a blog post, and I want to insert a permalink key after the author key, but only if the file doesn’t already have a permalink entry.

Solution:

find . -type f ! -exec grep -q 'permalink:' {} \; -exec gawk -i inplace '/author:/ { print; print "permalink: /blog/:title"; next}1' {} \;

This requires GNU Awk for the in-place editing of a file.

Batch Rename

Adding a prefix:

for f in *.png; do mv "$f" "prefix-${f}"; done

Removing a prefix:

or f in *.png; do mv "$f" "${f#prefix}"; done

Thank you:

• Unix Stackexchange 1136
• Parameter Expansion

Stomp on the Content of a Set of Files

For example: you want to find ever build.properties file and set the content to be sbt.version=1.0.1:

find . -name build.properties -exec bash -c 'echo "sbt.version=1.0.1" > {}' \;

Note the need to escape the echo command because otherwise the > part is executed early by Bash. Thanks to SO 8828974.

Remove Recent Docker Container

This is handy if, for example, you’re running a database in Docker, and you want to totally blow it away.

Solution:

Stop the docker container, and then go check for the ID:

docker ps -a | head -2

That will give you the CONTAINER ID along with other useful sanity-checking information. E.g.,

CONTAINER ID        IMAGE                              COMMAND                  CREATED             STATUS                      PORTS               NAMES
9b30fe9e0da6        postgres:latest                    "docker-entrypoint..."

Next, remove the container:

docker rm 9b30fe9e0da6 

…or whatever the container id should be.

Remove Multiple Docker Containers

You try to remove a Docker image but discover it’s referenced by containers. You want to remove those containers.

Solution:

docker ps -a | grep /book | awk '{print $1}' | xargs  docker rm

Here we’re grep-ing for /book which happens to be an image name I’m interested in.

Email on failure

Use exit 0 for success, and anything else for failure. Then…

some_command > /home/richard/logs/some_command.log 2>&1
if [ $? -ne 0 ]; then
  mail user@example.com -s "Some command failed" < /home/richard/logs/some_command.log
fi

Cleaning up from a script

See: http://redsymbol.net/articles/bash-exit-traps/

For example (from that post):

#!/bin/bash
scratch=$(mktemp -d -t tmp.XXXXXXXXXX)
function finish {
  rm -rf "$scratch"
}
trap finish EXIT

Scatter plots

Here I’m using rg (faster grep) and cut to pick out x and y values I want to plot:

$ rg ':E:' atari.log | cut -d' ' -f 5,6 | head
9 235.00000
18 235.00000
27 300.00000
...

Then using GNU Plot to plot to the terminal and to a PNG:

$ rg ':E:' atari.log | cut -d' ' -f 5,6 | gnuplot -e 'set terminal "dumb"' -e "plot '< cat' notitle with lines"

$ rg ':E:' atari.log | cut -d' ' -f 5,6 | gnuplot -e 'set xlabel "Evaluations"' -e 'set ylabel "Score"' -e 'set terminal "png"' -e 'set output "progress.png" ' -e "plot '< cat' notitle with lines"

In a loop running a full terminal plot every 10 minutes:

while true; do clear ; grep  ':E:' atari.log | cut -d' ' -f 5,6 | gnuplot -e 'set terminal "dumb" size `tput cols` `tput lines`' -e "set yrange [0:1800]"  -e "plot '< cat' notitle with lines" ; sleep 600 ; done

Two y-axis in Gnuplot

gnuplot> plot "foo.txt" u 1:2 with lines axis x1y1, "foo.txt" u 1:3 axis x1y2

Adding line numbers to a file

$ cat -n foo.txt > numbered-foo.txt

Converting SVN repositories to GIT

With an uncompressed svndump archive called foo.svndump, import it to an SVN repository called old:

$ svncreate old
$ svnadmin load ./old < foo.svndump

Take a look at the available repositories:

$ svnlook tree -N old

Then convert one, perhaps the wibble repository:

$ git svn clone file:///Users/richard/Downloads/old/wibble/

…which will create a git repository in the wibble folder.

Using cut with tabs

$ cut -f1 -d$'\t' data.tsv > column1.txt

This $'\t syntax is Bash ANSI-C quoting.

Image resizing from the terminal

I used Imagemagik for this, but SO 28489793 is even easier:

$ sips -Z 640 foo.jpg