Creating complex commands is very common while administering a server or working with linux platforms and applications, because you could need to manage a large amount of files, folders, and scripting will become a must.
Before landing in the scripting territory, it is possible to accomplish less complex tasks with the help of basic functions of the shell. After the description of redirecting and streams (Linux streams, what? STDIN, STDOUT, STDERR ) we will try to face the so called ‘piping‘.
Exactly like real life pipes, in linux a pipe ( | ) is an operator that creates a sort of connection between two (or more) consecutive commands, allowing the output of the left-hand command to ‘flow’ and become available for further elaboration to the right-hand command(s).
A quick example to manipulate output with pipe
Let’s take a directory full of files and folders and let’s list them (if you want more details about listing files check out How to list files in linux):
This is the output for your listing, for any reason you want to just get information about items that contain “dir” in their name, we can use the ‘grep‘ command to refine our output.
Keep in mind that when you use the ‘|‘ (pipe) operator, ALL output is processed from the next command, so be careful choosing the right output to process, we will see why.
It is possible to see how the output is processed:
As a result, only entries with the indicated pattern are shown (if you want more options about grep, checkout this great cheatsheet from b0rk@twitter)
Let’s add some spice!
I can see you right now… it seems like you want to complicate things, and I will satisfy your needs! Come and see what happens if we want to sort out only the items containing a ‘9‘ (keep in mind that the FULL output of ls is processed!)
Exactly the same output, beacause as you can see, the digit we are looking for is also present in the size of the items (4096), so what can we do to refine our search?
Yeah! Let’s pipe the output into something magic that will get out of our way all the unneeded information! We will use awk command that is a powerful language processing tool and we will ask it to please read the output coming from the first grep, tokenize it using a blank as a separator (implicit) and then print only the last field of each line ($NF)
Eventually, we will be able to find what we were looking for, the great entry with a ‘9’ in its name!
To infinity and beyond!
Yep, you can potentially pipe infinite commands to manipulate the output in linux, and you’ll probably do when you will be managing a server and/or working with logfiles, configurations and scripts, but I hope that now the concept of piping is a little bit more clear to you.
If you liked the article please rate it and share with your friends/colleagues!