DEV Community

Cover image for Processing Text with Bash Using Grep, Sed, and Awk Commands
Vultr profile image Sanskriti Harmukh
Sanskriti Harmukh for Vultr

Posted on with Aashish Chaurasiya Originally published at docs.vultr.com

Processing Text with Bash Using Grep, Sed, and Awk Commands

#bash #linux #cli #scripting

Text processing in Bash using the grep, sed, and awk commands lets you filter and manipulate input to perform specific tasks, such as searching log files, transforming content, and extracting structured data. You can combine these commands with loops, conditional statements, and expressions for more advanced processing. This guide walks through each tool individually and then shows how to combine them in real scripts. By the end, you'll have working Bash scripts that filter, rewrite, and report on a log file using grep, sed, and awk together.

1. Create a New Bash Script

1. Create a new log.txt file to store multiple lines of text:

$ nano log.txt
Enter fullscreen mode Exit fullscreen mode

2. Add the following contents to the log.txt file:

2024-10-25 08:15:23 INFO: Apache Web Server Application started successfully.
2024-10-25 09:00:01 ERROR: Unable to connect to the database on port 3306.
2024-10-25 09:30:45 WARNING: The Disk space running low.
2024-10-24 10:15:12 INFO: root User logged in.
2024-10-24 10:45:55 ERROR: File not found: www.yml.
2024-10-24 11:15:00 WARNING: High memory usage detected.
2024-10-23 14:25:01 INFO: A scheduled backup is complete.
2024-10-23 15:00:00 ERROR: Apache: The Network connection lost.
2024-10-23 16:20:30 INFO: root User logged out successfully.
2024-10-23 18:45:10 WARNING: CPU temperature exceeding normal level.
Enter fullscreen mode Exit fullscreen mode

Save and close the file.

3. Create a new processing.sh Bash script that loads the log.txt file into a variable:

$ nano processing.sh
Enter fullscreen mode Exit fullscreen mode

4. Add the following contents to the processing.sh file:

#!/bin/bash

LOG="log.txt"
Enter fullscreen mode Exit fullscreen mode

Save and close the file. The LOG variable holds log.txt as input — you'll process this variable with grep, sed, and awk in the next sections.

2. Use the grep Command to Process a Text File

The grep command searches for specific text patterns in files, such as the log.txt file you created, and outputs the results for further processing in a Bash script.

1. Open the processing.sh script:

$ nano processing.sh
Enter fullscreen mode Exit fullscreen mode

2. Add the following command at the end of the file:

grep -i "ERROR" "$LOG"
Enter fullscreen mode Exit fullscreen mode

Save and close the file. This grep command searches for the term ERROR in log.txt and displays all matching results.

3. Run the script using Bash:

$ bash processing.sh
Enter fullscreen mode Exit fullscreen mode

Output:

2024-10-25 09:00:01 ERROR: Unable to connect to the database on port 3306.
2024-10-24 10:45:55 ERROR: File not found: www.yml.
2024-10-23 15:00:00 ERROR: Apache: The Network connection lost.
Enter fullscreen mode Exit fullscreen mode

3. Use the awk Command to Process a Text File

The awk command analyzes structured data and searches for specific patterns in a file. You can use it to customize file contents, split lines into fields or columns for processing, and match specific patterns.

1. Open the processing.sh Bash script:

$ nano processing.sh
Enter fullscreen mode Exit fullscreen mode

2. Add the following contents to the file:

awk '{print $1, $3}' "$LOG"
Enter fullscreen mode Exit fullscreen mode

Save and close the file. This awk command prints the first and third columns in log.txt to display the date and log level.

3. Run the script using Bash:

$ bash processing.sh
Enter fullscreen mode Exit fullscreen mode

Output:

2024-10-25 INFO:
2024-10-25 ERROR:
2024-10-25 WARNING:
2024-10-24 INFO:
2024-10-24 ERROR:
2024-10-24 WARNING:
2024-10-23 INFO:
2024-10-23 ERROR:
2024-10-23 INFO:
2024-10-23 WARNING:
Enter fullscreen mode Exit fullscreen mode

4. Use the sed Command to Process a Text File

Stream Editor (sed) searches, modifies, and replaces content in standard input or a specific file.

1. Open the processing.sh Bash script:

$ nano processing.sh
Enter fullscreen mode Exit fullscreen mode

2. Add the following command at the end of the file:

sed -e 's/ERROR/New ERROR/gI' -e 's/WARNING/New WARNING/gI' "$LOG"
Enter fullscreen mode Exit fullscreen mode

Save and close the file. This sed command replaces ERROR with New ERROR and WARNING with New WARNING in log.txt.

3. Run the script using Bash:

$ bash processing.sh
Enter fullscreen mode Exit fullscreen mode

Output:

2024-10-25 08:15:23 INFO: Apache Web Server Application started successfully.
2024-10-25 09:00:01 New ERROR: Unable to connect to the database on port 3306.
2024-10-25 09:30:45 New WARNING: The Disk space running low.
2024-10-24 10:15:12 INFO: root User logged in.
2024-10-24 10:45:55 New ERROR: File not found: www.yml.
2024-10-24 11:15:00 New WARNING: High memory usage detected.
2024-10-23 14:25:01 INFO: A scheduled backup is complete.
2024-10-23 15:00:00 New ERROR: Apache: The Network connection lost.
2024-10-23 16:20:30 INFO: root User logged out successfully.
2024-10-23 18:45:10 New WARNING: CPU temperature exceeding normal level.
Enter fullscreen mode Exit fullscreen mode

5. Combine grep, sed, and awk Commands

You can combine grep, sed, and awk to analyze and manipulate large inputs with pipes.

1. Create a new combine.sh Bash script:

$ nano combine.sh
Enter fullscreen mode Exit fullscreen mode

2. Add the following contents to the file:

#!/bin/bash

LOG="log.txt"

echo "Processing the log file: $LOG"

echo "Date       Time        Log Level   Message"
echo "------------------------------------------"

grep -E 'ERROR|WARNING' "$LOG" | sed -e 's/ERROR/New ERROR/gI' -e 's/WARNING/New WARNING/gI' "$LOG" | awk '{print $1, $2, $3, $4, $5, $6, $7}'

echo -e "\nLog processing complete."
Enter fullscreen mode Exit fullscreen mode

Save and close the file. The combined grep, sed, and awk commands use pipes (|) to pass the output of each command to the next and display the modified contents of log.txt in a structured format.

3. Run the script using Bash:

$ bash combine.sh
Enter fullscreen mode Exit fullscreen mode

Output:

Processing the log file: log.txt
Date       Time        Log Level   Message
------------------------------------------
2024-10-25 08:15:23 INFO: Apache Web Server Application
2024-10-25 09:00:01 New ERROR: Unable to connect
2024-10-25 09:30:45 New WARNING: The Disk space
2024-10-24 10:15:12 INFO: root User logged in.
2024-10-24 10:45:55 New ERROR: File not found:
2024-10-24 11:15:00 New WARNING: High memory usage
2024-10-23 14:25:01 INFO: A scheduled backup is
2024-10-23 15:00:00 New ERROR: Apache: The Network
2024-10-23 16:20:30 INFO: root User logged out
2024-10-23 18:45:10 New WARNING: CPU temperature exceeding

Log processing complete
Enter fullscreen mode Exit fullscreen mode

6. Use grep, sed, and awk with Conditional Bash Statements

You can use grep, sed, and awk with if-else, case, and if-then statements to enhance output readability and handle errors.

1. Create a new conditional-processing.sh Bash script:

$ nano conditional-processing.sh
Enter fullscreen mode Exit fullscreen mode

2. Add the following contents to the conditional-processing.sh file:

#!/bin/bash

LOG="log.txt"

if [[ -f "$LOG" ]]; then
    echo "Processing the log file: $LOG"

    if grep -qE 'ERROR|WARNING' "$LOG"; then

        echo "Date       Time        Log Level   Message"
        echo "------------------------------------------"

        grep -E 'ERROR|WARNING' "$LOG" | sed -e 's/ERROR/New ERROR/gI' -e 's/WARNING/New WARNING/gI' "$LOG" | awk '{print $1, $2, $3, $4, $5, $6, $7}'

        echo -e "\nLog processing complete."
    else
        echo "No ERRORs or WARNINGs found in the log file."
    fi
else
    echo "The Log file: $LOG is not available"
fi

echo "...............Completed.............."
Enter fullscreen mode Exit fullscreen mode

Save and close the file. The outer if-else validates the input variable, and the inner if-else runs a grep condition to find ERROR or WARNING entries — the combined grep, sed, and awk commands run only when that condition is true.

3. Run the script using Bash:

$ bash conditional-processing.sh
Enter fullscreen mode Exit fullscreen mode

Output:

Processing the log file: log.txt
Date       Time        Log Level   Message
------------------------------------------
2024-10-25 08:15:23 INFO: Apache Web Server Application
2024-10-25 09:00:01 New ERROR: Unable to connect
2024-10-25 09:30:45 New WARNING: The Disk space
2024-10-24 10:15:12 INFO: root User logged in.
2024-10-24 10:45:55 New ERROR: File not found:
2024-10-24 11:15:00 New WARNING: High memory usage
2024-10-23 14:25:01 INFO: A scheduled backup is
2024-10-23 15:00:00 New ERROR: Apache: The Network
2024-10-23 16:20:30 INFO: root User logged out
2024-10-23 18:45:10 New WARNING: CPU temperature exceeding

Log processing complete.
...............Completed..............
Enter fullscreen mode Exit fullscreen mode

7. Use grep, sed, and awk with Bash Loops

Loops such as while, for, and until let you run advanced text processing tasks when using grep, sed, and awk.

1. Create a new loop-combined.sh script:

$ nano loop-combined.sh
Enter fullscreen mode Exit fullscreen mode

2. Add the following contents to the loop-combined.sh file:

#!/bin/bash

LOG="log.txt"

if [ -z "$1" ]; then
    echo "Enter a command option such as `ERROR` or `WARNING`. Example Usage: loop-combined.sh ERROR."
    exit 1
fi


LOG_LEVEL=$1
echo "Filtering '$LOG_LEVEL' logs..."

echo -e "Date       Time        Log Level   Message"
echo -e "------------------------------------------"


grep "$LOG_LEVEL" "$LOG" | 
sed "s/$LOG_LEVEL/${LOG_LEVEL^^}/g" |
awk '{print $1, $2, $3, $4, $5, $6}' |  

while read -r line; do
    echo "New entry: $line"
done
Enter fullscreen mode Exit fullscreen mode

Save and close the file. This script prompts for a log level option and runs a while loop that prints the contents of log.txt using the piped output of grep, sed, and awk one line at a time.

3. Run the script and pass ERROR as the option:

$ bash loop-combined.sh ERROR
Enter fullscreen mode Exit fullscreen mode

Output:

Filtering 'ERROR' logs...
Date       Time        Log Level   Message
------------------------------------------
New entry: 2024-10-25 09:00:01 ERROR: Unable to connect
New entry: 2024-10-24 10:45:55 ERROR: File not found:
New entry: 2024-10-23 15:00:00 ERROR: Apache: The Network
Enter fullscreen mode Exit fullscreen mode

4. Run the script again, passing WARNING as the option:

$ bash loop-combined.sh WARNING
Enter fullscreen mode Exit fullscreen mode

Output:

Filtering 'WARNING' logs...
Date       Time        Log Level   Message
------------------------------------------
New entry: 2024-10-25 09:30:45 WARNING: The Disk space
New entry: 2024-10-24 11:15:00 WARNING: High memory usage
New entry: 2024-10-23 18:45:10 WARNING: CPU temperature exceeding
Enter fullscreen mode Exit fullscreen mode

Next Steps

  • Adapt conditional-processing.sh to point LOG at a real application or system log instead of the sample file.
  • Extend the awk pipeline to write matched entries to a separate report file instead of just printing them.
  • Wrap loop-combined.sh in a cron job to generate a daily filtered summary of ERROR and WARNING entries.
  • Explore sed's in-place edit flag (-i) for rewriting files directly instead of only streaming output.

For the full guide with additional tips, visit the original article on Vultr Docs.

Top comments (0)