Pipes and Text Processing commands in Linux

Linux's command-line prowess is amplified by the ability to pipe commands together, allowing you to chain operations and manipulate data efficiently. In this post, we'll explore the power of pipes (|) and the text processing commands cut, paste, sort, and tee.

Understanding Pipes (|)

Pipes allow the output of one command to be used as the input for another command. This creates a powerful workflow for processing data.

Syntax: command1 | command2 | command3

Essential Text Processing Commands:

• cut: Extracts specific columns or fields from text.

  • -d specifies the delimiter.

  • -f specifies the field(s) to extract.

  • Example: cut -d ',' -f 1,3 data.csv (extracts the first and third columns from a comma-separated file)

• paste: Merges lines of files together.

  • -d specifies the delimiter.

  • Example: paste file1.txt file2.txt (merges corresponding lines of file1.txt and file2.txt)

• sort: Sorts lines of text.

  • -n sorts numerically.

  • -r sorts in reverse order.

  • Example: sort numbers.txt (sorts lines in numbers.txt)

• tee: Reads from standard input and writes to both standard output and one or more files.

  • Example: command | tee output.txt (displays the output and saves it to output.txt)

Practical Examples:

1. Extracting Fields:

$ cat data.csv | cut -d ',' -f 2 (extracts the second column from data.csv)

1. Merging Files:

$ paste names.txt ages.txt (merges names and ages)

1. Sorting Output:

$ cat numbers.txt | sort -n (sorts numbers numerically)

1. Saving and Displaying Output:

$ ls -l | tee file_list.txt (lists files and saves the list to file_list.txt)

Practice Session 1: With Example Commands

1. Create sample data:

$ echo "John,25,New York" > data.csv

$ echo "Alice,30,London" >> data.csv

$ echo "Bob,22,Paris" >> data.csv

$ echo "10" > numbers.txt

$ echo "5" >> numbers.txt

$ echo "15" >> numbers.txt

$ echo "Name1" > names.txt

$ echo "Name2" > names.txt

$ echo "20" > ages.txt

$ echo "30" > ages.txt

• Extract names from data.csv:

$ cat data.csv | cut -d ',' -f 1

• Sort numbers in numbers.txt:

$ cat numbers.txt | sort -n

• Merge names.txt and ages.txt:

$ paste names.txt ages.txt

• List files and save the output to "files.txt":

$ ls -l | tee files.txt

Practice Session 2: Without Example Commands

1. Create a CSV file named "users.csv" with columns "username", "age", and "city". Add a few lines of data.

2. Extract the "age" column from "users.csv" and display it.

3. Create a file named "list1.txt" and another named "list2.txt". Add a few lines of text to each file.

4. Merge "list1.txt" and "list2.txt" and display the result.

5. Create a file named "unsorted.txt" with lines of text in random order. Sort the lines and display the sorted output.

6. List all files in the current directory, display the list, and save it to a file named "dir_list.txt".

7. Extract the city column from users.csv, sort it and display the result.

8. Extract the username and age column from users.csv, and save the result to a file named "user_ages.csv".

By mastering piping and these text processing commands, you can efficiently manipulate data and streamline your Linux workflow.