GNU Parallel | Cheatsheet
GNU Parallel is a powerful tool that allows for the concurrent execution of jobs, making efficient use of multiple CPU cores. It is widely used for various tasks, such as processing large data files and performing distributed computations across a network. This cheatsheet provides a comprehensive collection of practical examples to demonstrate the functionality of GNU Parallel.
Run multiple commands in parallel
echo -e "sleep 1\necho Hello" | parallel
Run commands from a file in parallel
parallel :::: commands.txt
Run commands from a script in parallel
parallel ./myscript.sh ::: arg1 arg2 arg3
Run commands with multiple arguments
parallel echo ::: A B C ::: 1 2 3
Run commands with multiple arguments from a file
parallel -a argfile.txt echo
Parallelize a shell loop
parallel 'echo processing file {}; mv {} {}.bak' ::: *.txt
Parallelize a find command
find . -name "*.txt" | parallel gzip {}
Parallelize a grep command
parallel grep pattern ::: file1 file2 file3
Control the number of jobs
parallel -j4 echo ::: A B C D E F G
Control the number of jobs based on CPU
parallel -j% echo ::: {1..100}
Set the maximum load
parallel --load 75% echo ::: {1..100}
Use a job slot with arguments
parallel -j2 'sleep {} && echo {}' ::: 1 2 3 4 5
Retrying Failed Jobs
parallel --retries 3 --timeout 200% faulty-command ::: inputs
Dry-run to check the commands
parallel --dry-run command ::: arg1 arg2
Batch Processing and Parallelism
Process groups of 10 jobs, count to 1000, and echo group parts
for i in {1..1000}; do
echo $i
if ((i % 10 == 0)); then
echo "Group part reached"
fi
done | parallel -j 10 'echo {}'
Run 10 parallel jobs and count to 10
NUM="10"
seq $NUM | parallel -n1 -P$NUM 'seq 10 | parallel -j10 echo {}'
Process groups of 4 jobs in parallel, count to 1000, and echo group parts
for i in {1..1000}; do
echo $i
if ((i % 5 == 0)); then
echo "Group part reached"
fi
done | parallel -j 4 'echo {}'
Handle a slow job example with 4 jobs in parallel
for i in {1..1000}; do
echo $i
if ((i % 5 == 0)); then
echo "This is a slow job"
sleep 1
fi
done | parallel -j 4 'echo {}'
Advanced line grouping and parallel processing
input_file="input.txt"
lines_per_group=4
counter=0
group=""
while IFS= read -r line; do
group+="$line"$'\n'
counter=$((counter + 1))
if ((counter % lines_per_group == 0)); then
echo -n "$group" | parallel -j4 'echo {}'
group=""
fi
done <"$input_file"
# Check if there are any lines left at the end of the file
if [ -n "$group" ]; then
echo -n "$group" | parallel -j4 'echo {}'
fi
Execute 4 parallel commands every second
for i in {1..1000}; do
echo $i
if ((i % 5 == 0)); then
sleep 1
fi
done | parallel -j 4 'echo {}'
Expand tab-separated variables in parallel
echo -e "Var1\tVar2\tVar3" | parallel --colsep '\t' 'echo {1} and {2} and {3}'
Expand line-separated variables in parallel
echo -e "Var1\nVar2\nVar3" | parallel 'echo {}'
Creating files in parallel
```bash
seq 100 | parallel 'touch file{}.txt'
```
Renaming files in parallel
```bash
ls *.txt | parallel 'mv {} {.}.bak'
```
Removing files in parallel
```bash
ls *.txt | parallel 'rm {}'
```
Fast and efficient file searching
```bash
find . -type f | parallel grep -l "search pattern"
```
Use GNU Parallel with rm -rf
```bash
find . -name "*.tmp" | parallel 'rm -rf {}'
```
Using the progress bar
```bash
seq 100 | parallel --progress 'sleep 0.1 && echo {}'
```
Displaying ETA (Estimated Time of Arrival)
```bash
seq 100 | parallel --eta 'sleep 0.1 && echo {}'
```
Pipe output to a file
```bash
seq 100 | parallel 'echo {}' > output.txt
```
Run command on multiple servers
```bash
echo -e "server1\nserver2\nserver3" | parallel --tag -j0 ssh {} hostname
```
Transfer and execute a script on multiple servers
```bash
echo -e "server1\nserver2\nserver3" | parallel -j0 --tag 'scp myscript.sh {}:/tmp/; ssh {} bash /tmp/myscript.sh'
```
Run different commands on different servers
```bash
echo -e "server1:uname\nserver2:hostname\nserver3:date" | parallel --colsep ':' ssh {1} {2}
```
Run a command on each file in a directory on multiple servers
```bash
echo -e "server1\nserver2\nserver3" | parallel -j0 --tag 'ssh {} "ls /path/to/directory | parallel grep pattern"'
```
Copy multiple files to multiple servers
```bash
parallel scp ::: file1 file2 file3 ::: server1:/path server2:/path server3:/path
```
Copy multiple directories to multiple servers
```bash
parallel scp -r ::: dir1 dir2 dir3 ::: server1:/path server2:/path server3:/path
```
Run a command on each line of file on multiple servers
```bash
echo -e "server1\nserver2\nserver3" | parallel -j0 --tag 'ssh {} "cat /path/to/file | parallel command"'
```
Backup a directory to multiple servers
```bash
echo -e "server1\nserver2\nserver3" | parallel -j0 --tag 'ssh {} "tar -czf /backup_dir/backup.tar.gz /path/to/directory"'
```
Update package on multiple servers
```bash
echo -e "server1\nserver2\nserver3" | parallel -j0 --tag ssh {} sudo apt-get update
```
Running Commands in Parallel
Run multiple commands in parallel
echo -e "sleep 1\necho Hello" | parallel
Example: - The commands "sleep 1" and "echo Hello" will be executed concurrently.
Run commands from a file in parallel
parallel :::: commands.txt
Example: - The commands listed in the file "commands.txt" will be executed in parallel.
Run commands from a script in parallel
parallel ./myscript.sh ::: arg1 arg2 arg3
Example: - The script "myscript.sh" will be executed with arguments "arg1," "arg2," and "arg3" in parallel.
Run commands with multiple arguments
parallel echo ::: A B C ::: 1 2 3
Example: - The command "echo" will be executed with combinations of arguments "A," "B," and "C" and "1," "2," and "3."
Run commands with multiple arguments from a file
parallel -a argfile.txt echo
Example: - The command "echo" will be executed with arguments read from the file "argfile.txt."
Parallelize a shell loop
parallel 'echo processing file {}; mv {} {}.bak' ::: *.txt
Example: - The commands "echo processing file {}" and "mv {} {}.bak" will be executed concurrently for each file in the current directory with the extension ".txt."
Parallelize a find command
find . -name "*.txt" | parallel gzip {}
Example: - The command "gzip" will be executed in parallel for each file in the current directory and its subdirectories with the extension ".txt."
Parallelize a grep command
parallel grep pattern ::: file1 file2 file3
Example: - The command "grep" will be executed in parallel to search for the specified pattern in "file1," "file2," and "file3."
Controlling the Number of Jobs
parallel -j4 echo ::: A B C D E F G
Example: - The command "echo" will be executed in parallel with a maximum of 4 jobs at a time, processing arguments "A," "B," "C," "D," "E," "F," and "G."
Controlling the Number of Jobs Based on CPU
parallel -j% echo ::: {1..100}
Example: - The command "echo" will be executed in parallel, with the number of jobs automatically determined based on the available CPU resources.
Setting the Maximum Load
parallel --load 75% echo ::: {1..100}
Example: - The command "echo" will be executed in parallel with a maximum load of 75%, ensuring optimal resource
utilization.
Using a Job Slot with Arguments
parallel -j2 'sleep {} && echo {}' ::: 1 2 3 4 5
Example: - The commands "sleep {}" and "echo {}" will be executed concurrently, where each "sleep" command has an argument and is followed by the corresponding "echo" command.
Retrying Failed Jobs
parallel --retries 3 --timeout 200% faulty-command ::: inputs
Example: - The command "faulty-command" will be executed with the arguments provided in the "inputs" file. Failed jobs will be retried up to three times with a timeout of 200% of the expected execution time.
Dry-Run to Check the Commands
parallel --dry-run command ::: arg1 arg2
Example: - The command "command" will not be executed, but a preview of the commands that would be executed is displayed for the given arguments "arg1" and "arg2."
Batch Processing and Parallelism
Process Groups of Jobs and Echo Group Parts
for i in {1..1000}; do
echo $i
if ((i % 10 == 0)); then
echo "Group part reached"
fi
done | parallel -j 10 'echo {}'
Run Parallel Jobs within Parallel Jobs
NUM="10"
seq $NUM | parallel -n1 -P$NUM 'seq 10 | parallel -j10 echo {}'
Process Groups of Jobs, Echo Group Parts, and Control Parallelism
for i in {1..1000}; do
echo $i
if ((i % 5 == 0)); then
echo "Group part reached"
fi
done | parallel -j 4 'echo {}'
Handling Slow Jobs with Parallelism
for i in {1..1000}; do
echo $i
if ((i % 5 == 0)); then
echo "This is a slow job"
sleep 1
fi
done | parallel -j 4 'echo {}'
Advanced Line Grouping and Parallel Processing
input_file="input.txt"
lines_per_group=4
counter=0
group=""
while IFS= read -r line; do
group+="$line"$'\n'
counter=$((counter + 1))
if ((counter % lines_per_group == 0)); then
echo -n "$group" | parallel -j4 'echo {}'
group=""
fi
done <"$input_file"
# Check if there are any lines left at the end of the file
if [ -n "$group" ]; then
echo -n "$group" | parallel -j
4 'echo {}'
fi
Example: - The lines from the file "input.txt" will be grouped into sets of 4 lines. Each group will be processed in parallel, with the "echo" command applied to each line in the group.
Execute Parallel Commands with Delay
for i in {1..1000}; do
echo $i
if ((i % 5 == 0)); then
sleep 1
fi
done | parallel -j 4 'echo {}'
Expand Tab-Separated Variables in Parallel
echo -e "Var1\tVar2\tVar3" | parallel --colsep '\t' 'echo {1} and {2} and {3}'
Expand Line-Separated Variables in Parallel
echo -e "Var1\nVar2\nVar3" | parallel 'echo {}'
Creating Files in Parallel
seq 100 | parallel 'touch file{}.txt'
Renaming Files in Parallel
ls *.txt | parallel 'mv {} {.}.bak'
Removing Files in Parallel
ls *.txt | parallel 'rm {}'
Fast and Efficient File Searching
find . -type f | parallel grep -l "search pattern"
Using GNU Parallel with rm -rf
find . -name "*.tmp" | parallel 'rm -rf {}'
Running a Command on Multiple Servers
echo -e "server1\nserver2\nserver3" | parallel --tag -j0 ssh {} hostname
Transferring and Executing a Script on Multiple Servers
echo -e "server1\nserver2\nserver3" | parallel -j0 --tag 'scp myscript.sh {}:/tmp/; ssh {} bash /tmp/myscript.sh'
Running Different Commands on Different Servers
echo -e "server1:uname\nserver2:hostname\nserver3:date" | parallel --colsep ':' ssh {1} {2}
Running a Command on Each File in a Directory on Multiple Servers
echo -e "server1\nserver2\nserver3" | parallel -j0 --tag 'ssh {} "ls /path/to/directory | parallel grep pattern"'
Copying Multiple Files to Multiple Servers
parallel scp ::: file1 file2 file3 ::: server1:/path server2:/path server3:/path
Copying Multiple Directories to Multiple Servers
parallel scp -r ::: dir1 dir2 dir3 ::: server1:/path server2:/path server3:/path
Running a Command on Each Line of a File on Multiple Servers
echo -e "server1\nserver2\nserver3" | parallel -j0 --tag 'ssh {} "cat /path/to/file | parallel command"'
Backing Up a Directory to Multiple Servers
echo -e "server1\nserver2\nserver3" | parallel -j0 --tag 'ssh {} "tar -czf /backup_dir/backup.tar.gz /path/to/directory"'
Updating a Package on Multiple Servers
echo -e "server1\nserver2\nserver3" | parallel -j0 --tag ssh {} sudo apt-get update