Essential command line tools – A quick and dirty toolbox

Nowadays it’s all about the GUI.  Many people have never used the command line,  or are fearful of it. It’s that thing that nerds and geeks use isn’t it?  I don’t want to touch that,  it’s way too hard and I’m sure I’ll break something?  And the like.  Can’t you just download me a program that’ll do what I want?

It’s normal behaviour to avoid what we don’t know,  to be fearful of what we don’t understand,  to want to stay within our comfort zone,  to give up control and let someone else do it all for us.  But it needn’t be this way.  I’m going to outline below a small set of commands that will get you well on your way to being more productive in a short period of time.

Now,  the command line isn’t the best tool to solve every problem.  But for many,  especially text based ones,  it can be very powerful.  The other day a good friend of mine wanted to extract some data out of an XML file.  The problem was that the specific data was in the attributes of the tags,  not the content.  They had downloaded an XML manipulation program but had failed to get it to achieve what they wanted.  So they asked me for help.  Rather than trying to learn a new program the first thing I did was look at the file then open up a terminal.  A quick pipe of the data through grep and awk (I probably could have just used awk with a bit of creativity but grep was easy) and they had the data in their hot little hands.

This got me thinking,  what would a basic command line toolkit consist of.  I’ve decided on the following list which I’ve outlined below:

  • ls – used for listing the files in a given directory.  Find out more
  • cat – great for viewing the contents of text files and joining several together.  I like to use it when putting the contents of a file through a pipe also as it keeps the flow on the screen better than using redirection.  Find out more
  • head – useful for obtaining the top X number of lines of a file.  Find out more
  • tail – useful for obtaining the bottom X number of lines of a file.  Find out more
  • sort – sort data into alphabetical order.  Find out more
  • grep – is a very powerful tool for identifying files or lines within files that meet a certain criteria.  Find out more
  • sed – good for modifying data based on a certain criteria.  Find out more
  • awk – useful when looking at data that is split into fields.  With a bit of creativity you’ll find that a lot of content can actually be split up that doesn’t immediately look to be.  Find out more
  • piping – using the “|” character to feed the output from one command in as the input for another command.  This allows you to put the previous commands together in creative ways like building blocks.  It is where the command line get’s a lot of it’s power from and is important to understand.  Find out more
  • redirection – in it’s basic sense it good for saving the output of your pipeline of commands to a text file.  Use “>” at the end of your last command,  followed by the name of a file,  to save the output to a file.  It can do more than this and if you’re interested you can read about it here.
  • regular expressions – not really essential but a very powerful addition to your toolbox.  Regular expressions are a means for defining a pattern.  Find out more
  • man – manual pages,  you’re going to want to refer to these as it’s silly to try and remember how all the commands work.  Find out more

There are many other commands that are available to you however as a basic toolbox I reckon the above list,  mashed together with a bit of creativity,  will allow you to acheive a great many tasks you may need to perform.  Let me run through a few examples to illustrate.

  • find all the files in a given directory that make a reference to UTS:grep -l ‘UTS’ *
  • tell me what the 5th line in the file tax_return.txt is:cat tax_return.txt | head -5 | tail -1
  • process a csv file and print only the 2nd and 5th columns and save to another filecat mydata.csv | awk -F, ‘{if ($3 > 1000) print $2″,”$5}’ | sort > processed.csv
  • Mary has just gotten married and her last name has changed from Smith to Jacobs.  You have to update her profile to change every instance of her name.cat profile.txt | sed ‘s/Mary Smith/Mary Jacobs/g’ > profile_updated.txt
  • You are considering 2 people for a promotion and wish to find how many times they are listed in publications,  you have downloaded text based versions of several publications.cat * | grep -c ‘Greg Stevenson’

The command line may seem awkward at first.  If you think about it,  so was touch typing whan you first started to learn it.  But would you consider going back now?  Well the command line is similar.  It will be a bit hard at first but soon enough you’ll wonder how you ever made do without it.  Best of luck and have fun.

Advertisements

0 Responses to “Essential command line tools – A quick and dirty toolbox”



  1. Leave a Comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s





%d bloggers like this: