Running Bash Commands Inwards Parallel

Introduction

A modern server is typically multi-core, perchance fifty-fifty multi-CPU. That is enough of computing ability to unleash on a given job. However, unless yous piece of work a labor inward parallel, yous are non maximizing the purpose of all that power.

Below are to a greater extent than or less typical everyday operations nosotros tin speed upward using parallel computing:

1. Backup files from multiple origin directories to a removable disk.
   
2. Resize ikon files inward a directory.
   
3. Compress files inward a directory.
   

To execute a labor inward parallel, yous tin purpose whatever of the next commands:

• ppss
• pexec
• GNU parallel

This ship service focuses on the GNU parallel command.

Installation of GNU parallel

To install GNU parallel on a Debian/Ubuntu system, piece of work the next command:

$ sudo apt-get install parallel

General Usage

The GNU parallel programme provides many options which yous tin specify to customize its behavior. Interested readers tin read its man page to larn to a greater extent than nearly their usage. In this post, I volition narrow the execution of GNU parallel to the next scenario.

My objective is to piece of work a musical rhythm ascendancy inward parallel, exactly on the same multi-core machine. The ascendancy tin accept multiple options, exactly exclusively 1 is variable. Specifically, yous piece of work concurrent instances of the ascendancy past times providing a unlike value for that 1 variable option. The unlike values are fed, 1 per line, to GNU parallel via the criterion input.

The residual of this ship service shows how GNU parallel tin backup multiple origin directories past times running rsync inward parallel.

Parallel backup

The next ascendancy backs upward ii directories inward parallel: /home/peter together with /data.

$ echo -e '/home/peter\n/data' | parallel -j-2 -k --eta rsync -R -av {} /media/myBKUP

Standard input

The echo ascendancy assembles the ii origin directory locations, separated past times a newline grapheme (\n), together with pipes it to GNU parallel.

How many jobs?

By default, GNU parallel deploys 1 labor per core. You tin override the default usint the -j option.

-j specifies the maximum give away of parallel jobs that GNU parallel tin deploy. The maximum give away tin last specified inward 1 of several ways:

• -j followed past times a give away
  
  -j2 agency that upward to ii jobs tin piece of work inward parallel.

• -j+ followed past times a give away
  
  -j+2 agency that the maximum give away of jobs is the give away of cores addition 2.

• -j- followed past times a give away
  
  -j-2 agency that the maximum give away of jobs is the give away of cores minus 2.

If yous don't know how many cores the auto has, piece of work the ascendancy below:

$ parallel --number-of-cores
eight

Keeping output order

Each labor may output lines to the criterion output. When multiple jobs are piece of work inward parallel, the default behaviour is that a job's output is displayed equally presently equally the labor finishes. You may detect this confusing because the output fellowship may last unlike from the input order. The -k choice keeps the output sequence the same equally the input sequence.

Showing progress

The --eta choice reports progress piece GNU parallel executes, including the estimated remaining fourth dimension (in seconds).

Input place-holder

GNU parallel substitutes the {} parameter alongside the side past times side draw inward the criterion input.

Each input draw is a directory location, e.g., /home/peter. Instead of the sum location, yous tin specify other parameters inward fellowship to extract a component thereof - e.g., the directory name(/home) together with the basename (peter). Please refer to the man page for details.

Summary

GNU parallel is a tool that Linux administrators should add together to their repertoire. Running a labor inward parallel tin exclusively amend one's efficiency. If yous are already familiar alongside xargs, yous volition detect the syntax familiar. Even if yous are novel to the command, at that spot is a wealth of on-line assistance on the GNU parallel website.

Share this post