Linux gawk command
The gawk command is the interface for GAWK, a powerful pattern-matching and processing language. It is based on the language AWK.
Description
Working with text files often requires repeated tasks. You might want to extract certain lines and discard the rest. Or you may need to make changes wherever certain patterns appear, but leave the rest of the file alone. Writing single-use programs for these tasks in languages, such as C, C++, or Java, is time-consuming and inconvenient. Such jobs are often easier with awk. The awk utility interprets a special-purpose programming language that makes it easy to handle simple data-reformatting jobs.
The GNU (GNU's Not Unix) implementation of awk is called gawk; if you invoke it with the proper options or environment variables (see Options), it is fully compatible with the POSIX (portable operating system interface for Unix) specification of the awk language and with the Unix version of awk maintained by Brian Kernighan.
Using awk (or gawk) allows you to:
- Manage small, personal databases
- Generate reports
- Validate data
- Produce indexes and perform other document preparation tasks
- Experiment with algorithms you can adapt later to other computer languages
Also, gawk provides facilities that make it easy to:
Syntax
gawk [ POSIX or GNU style options ] -f program-file [ -- ] file ...
gawk [ POSIX or GNU style options ] [ -- ] program-text file ...
pgawk [ POSIX or GNU style options ] -f program-file [ -- ] file ...
pgawk [ POSIX or GNU style options ] [ -- ] program-text file ...
dgawk [ POSIX or GNU style options ] -f program-file [ -- ] file ...
Option format
gawk options may be either traditional POSIX-style one letter options, or GNU-style long options. POSIX options start with a single "-", while long options start with "--". Long options are provided for GNU-specific features and POSIX-mandated features.
gawk-specific options are used in long-option form. Arguments to long options are either joined with the option by an = sign, with no intervening spaces, or they may be provided in the next command line argument. Long options may be abbreviated, as long as the abbreviation remains unique.
Additionally, each long option has a corresponding short option, so that the option's functionality may be used from in #! executable scripts.
Options
| -f program-file, --file program-file |
Read the AWK program source from the file program-file, instead of from the first command line argument. Multiple -f (or --file) options may be used. |
| -F fs, --field-separator fs |
Use fs for the input field separator (the value of the FS predefined variable). |
| -v var=val, --assign var=val |
Assign the value val to the variable var, before execution of the program begins. Such variable values are available to the BEGIN block of an AWK program. |
| -b, --characters-as-bytes |
Treat all input data as single-byte characters. In other words, don't pay any attention to the locale information when attempting to process strings as multibyte characters. The --posix option overrides this option. |
| -c, --traditional | Run in compatibility mode. In compatibility mode, gawk behaves identically to UNIX awk; none of the GNU-specific extensions are recognized. |
| -C, --copyright | Print the short version of the GNU Copyright information message on the standard output and exit successfully. |
| -d[file], --dump-variables[=file] |
Print a sorted list of global variables, their types and final values to file. If no file is provided, gawk uses a file named awkvars.out in the current directory. Having a list of all the global variables is a good way to look for typographical errors in your programs. You would also use this option if you have a large program with a lot of functions, and you want to be sure your functions don't inadvertently use global variables you meant to be local. This is a particularly easy mistake to make with simple variable names like i, j, and similar. |
| -e program-text, --source program-text |
Use program-text as AWK program source code. This option allows the easy intermixing of library functions (used via the -f and --file options) with source code entered on the command programs used in shell scripts. |
| -E file, --exec file | Similar to -f, however, this is option is the last one processed. This should be used with #! scripts, particularly for CGI (common gateway interface) applications, to avoid passing in options or source code (!!!) on the command line from a URL (uniform resource locator). This option disables command-line variable assignments. |
| -g, --gen-pot | Scan and parse the AWK program, and generate a GNU .pot (Portable Object Template) format file on standard output with entries for all localizable strings in the program. The program itself is not executed. See the GNU gettext distribution for more information on .pot files. |
| -h, --help | Print a summary of the available options on the standard output. Per the GNU Coding Standards, these options cause an immediate, successful exit. |
| -L [value], --lint[=value] |
Provide warnings about constructs that are dubious or non-portable to other AWK implementations. With an optional argument of fatal, lint warnings become fatal errors. This may be drastic, but its use certainly encourages the development of cleaner AWK programs. With an optional argument of invalid, only warnings about things that are actually invalid are issued. Note: This is not fully implemented yet. |
| -n, --non-decimal-data | Recognize octal and hexadecimal values in input data. Use this option with great caution! |
| -N, --use-lc-numeric | This forces gawk to use the locale's decimal point character when parsing input data. Although the POSIX standard requires this behavior, and gawk does so when --posix is in effect, the default is to follow traditional behavior and use a period as the decimal point, even in locales where the period is not the decimal point character. This option overrides the default behavior, without the full draconian strictness of the --posix option. |
| -O, --optimize | Enable optimizations upon the internal representation of the program. Currently, this includes only simple constant-folding. The gawk maintainer hopes to add additional optimizations over time. |
| -p[prof_file], --profile[=prof_file] |
Send profiling data to prof_file. The default is awkprof.out. When run with gawk, the profile is a "pretty printed" version of the program. When run with pgawk, the profile contains execution counts of each statement in the program in the left margin and function call counts for each user-defined function. |
| -P, --posix | This turns on compatibility mode, with the following additional restrictions: \x escape sequences are not recognized. Only space and tab act as field separators when FS is set to a single space, newline does not. You cannot continue lines after ? and : The synonym func for the keyword function is not recognized. The operators ** and **= cannot be used in place of ^ and ^= The fflush() function is not available. |
| -r, --re-interval | Enable the use of interval expressions in regular expression matching. Interval expressions were not traditionally available in the AWK language. The POSIX standard added them, to make awk and egrep consistent. They are enabled by default, but this option remains for use with --traditional. |
| -R, --command file | Dgawk only. Read stored debugger commands from file. |
| -S, --sandbox | Runs gawk in sandbox mode, disabling the system() function, input redirection with getline, output redirection with print and printf, and loading dynamic extensions. Command execution (through pipelines) is also disabled. This effectively blocks a script from accessing local resources (except for the files specified on the command line). |
| -t, --lint-old | Provide warnings about constructs that are not portable to the original version of Unix awk. |
| -V, --version | Print version information for this particular copy of gawk on the standard output. This is useful mainly for knowing if the current copy of gawk on your system is up-to-date with respect to whatever the Free Software Foundation is distributing. This is also useful when reporting bugs. Per the GNU Coding Standards, these options cause an immediate, successful exit. |
| -- | Signal the end of options. This is useful to allow further arguments to the AWK program itself to start with a "-". This provides consistency with the argument parsing convention used by most other POSIX programs. |
In compatibility mode, any other options are flagged as invalid, but are otherwise ignored. In normal operation, as long as program text is supplied, unknown options are passed on to the AWK program in the ARGV array for processing. This is particularly useful for running AWK programs via the "#!" executable interpreter mechanism.
AWK program execution
An AWK program consists of a sequence of pattern-action statements and optional function definitions.
@include "file name" pattern { action statements }
function name(parameter list) { statements }
gawk first reads the program source from the program-file(s) if specified, from arguments to --source, or from the first non-option argument on the command line. The -f and --source options may be used multiple times on the command line. gawk reads the program text as if all the program files and command line source texts had been concatenated. This is useful for building libraries of AWK functions, without having to include them in each new AWK program that uses them. It also provides the ability to mix library functions with command line programs.
Also, lines beginning with @include can include other source files into your program, making library use even easier.
The environment variable AWKPATH specifies a search path to use when finding source files named with the -f option. If this variable does not exist, the default path is ".:/usr/local/share/awk". The actual directory may vary, depending on how gawk was built and installed. If a file name given to the -f option contains a "/" character, no path search is performed.
gawk executes AWK programs in the following order. First, all variable assignments specified via the -v option are performed. Next, gawk compiles the program into an internal form. Then, gawk executes the code in the BEGIN block(s) (if any), and then proceeds to read each file named in the ARGV array (up to ARGV[ARGC]). If there are no files named on the command line, gawk reads the standard input.
If a file name on the command line has the form var=val it is treated as a variable assignment. The variable var is assigned the value val. This happens after any BEGIN blocks are run. Command line variable assignment is most useful for dynamically assigning values to the variables AWK uses to control how input is broken into fields and records. It is also useful for controlling state if multiple passes are needed over a single data file.
If the value of a particular element of ARGV is empty (""), gawk skips over it.
For each input file, if a BEGINFILE rule exists, gawk executes the associated code before processing the file's contents. Similarly, gawk executes the code associated with ENDFILE after processing the file.
For each record in the input, gawk tests to see if it matches any pattern in the AWK program. For each pattern that the record matches, the associated action is executed. The patterns are tested in the order they occur in the program.
Finally, after all the input is exhausted, gawk executes the code in the END block(s) (if any).
Official gawk user's guide
To learn more about this incredibly powerful language, check out the GNU gawk user guide.
Examples
gawk '{ num_fields = num_fields + NF }
END { print num_fields }'
Print the total number of fields in all input lines.
gawk 'length($0) > 80'
Prints every line longer than 80 characters. The sole rule has a relational expression as its pattern, and has no action (so the default action, printing the record, is used).
ls -l files | awk '{ x += $4 } ; END { print "total bytes: " x }'
Prints the total number of bytes used by files.
Related Commands
awk — Interpreter for the AWK text processing programming language.
sed — A utility for filtering and transforming text.