Developing Command Line Applications with argparse and configparser

In this post, I’ll demonstrate an approach to command line application development that minimizes errors, provides ready access to configurable parameters and is straightforward to implement. As Python was originally created to serve as an improved scripting language and environment, in-built support for command line application development is top-notch. In particular, argparse simplifies the implementation of command line options, arguments and sub-commands, and configparser makes it possible to specify program inputs in a file independent of the companion script. The examples that follow were written and tested on Linux targeting the Python3 interpreter.

Application Description

The application, cmdapp.py, filters and subsets the Linux dictionary file, usually found at /usr/share/dict/words or /var/lib/dict/words. A copy of this file (with all words containing digits removed) can be obtained here.

cmdapp.py can accept command line options which result in subsetting the word list, and after filtering has been applied, the remaining subset of words can be (optionally) written to file. What follows is the full list of available command line options:

--words-file
      The file system location of words file

--output-directory
      Directory of exported file (if applicable)

--output-filename
      Name of exported file (if applicable)

--export
      Should matching words be written to file (False)?

--startswith=PATTERN
      Only return words starting with PATTERN

--endswith=PATTERN
      Only return words ending with PATTERN

--contains=PATTERN
      Only return words containing specified PATTERN

--suppress
      Should matching words be suppressed (False)?

Implementing these options with argparse is straightforward (Python’s documentation contains a high-level overview of argparse and sample use cases. It can be found here. We initialize an instance of the ArgumentParser class, and then proceed as follows: 

import argparse

parser = argparse.ArgumentParser()

# Specify command line arguments.
parser.add_argument(
    "--words-file", type=str,
    help="Absolute filepath of words(.txt) file",
    default=None
    )
parser.add_argument(
    "--output-directory", type=str,
    help="Directory of exported file, if applicable",
    default=None
    )
parser.add_argument(
    "--output-filename", type=str,
    help="Name of exported file, if applicable (matches.txt)",
    default="matches.txt"
    )
parser.add_argument(
    "--export",
    help="Should matching words be written to file (False)?",
    action="store_true"
    )
parser.add_argument(
    "--startswith", type=str,
    help="Only return words starting with specified char(s)",
    default=None
    )
parser.add_argument(
    "--endswith", type=str,
    help="Only return words ending with specified char(s)",
    default=None
    )
parser.add_argument(
    "--contains", type=str,
    help="Only return words containing specified char(s)",
    default=None
    )
parser.add_argument(
    "--suppress", help="Should matching words be suppressed (False)?",
    action="store_true"
    )
parser.add_argument(
    "-v", "--verbose", help="Should additional runtime details be displayed (False)?",
    action="store_true"
    )

# Parse command line arguments.
argvs = parser.parse_args()

# Bind reference to specified options.
argvs_words_file       = argvs.words_file
argvs_output_directory = argvs.output_directory
argvs_output_filename  = argvs.output_filename
argvs_suppress         = argvs.suppress
argvs_export           = argvs.export
argvs_startswith       = argvs.startswith
argvs_endswith         = argvs.endswith
argvs_contains         = argvs.contains

# Optionally view specified command line options and associated values.
print(argvs._get_kwargs())


add_argument‘s action parameter can be set to “store_true” or “store_false”. If set to “store_true”, the presence of the flag evaluates to True. If set to “store_false”, the presence of the flag evaluates to False. In the preceding example, the options --suppress and --export are logical indicators with action set to “store_true”.

All other options, --words-file, --output-directory, --output-filename, --startswith, --endswith and --contains, when specified require a string of text to follow the option (note that by default, argparse interprets all specified values as strings. If an option should be interpreted as something other than a string, specify the datatype using add_argument‘s type parameter). The string can follow the command line option separated by a space: 

$ python3 cmdapp.py --export --startswith "a" --endswith "e" --contains "p"

Or separated by an equal sign:

$ python3 cmdapp.py --export --startswith="a" --endswith="e" --contains="p"

When developing command line applications, a popular convention has command line flags overriding default values, with defaults obtained from environment variables or configuration files. configparser helps to facilitate the latter. We create a configuration file that consists of sections, each containing key-value pairs. The pairs are read into cmdapp.py and are set as the defaults for each modifiable option. The configuration file, defaults.cfg, contains the following: 

[SETTINGS]
WORDS_FILE=E:/Temp/dictionary.txt
OUTPUT_DIRECTORY=E:/Temp
OUTPUT_FILENAME=matches.txt

[PARAMETERS]
SUPPRESS=N
EXPORT=N
STARTSWITH=
ENDSWITH=
CONTAINS=

When specifying character strings in a configuration file, it is not necessary to surround the content in quotes. Key-value pairs can be separated by = or :. We next demonstrate how to read defaults.cfg into cmdapp.py and extract the content using an instance the ConfigParser class: 

import configparser

# Provide absolute path to defaults.cfg.
DEFAULTS = "defaults.cfg"

config = configparser.ConfigParser()

config.read(DEFAULTS)

def_words_file       = config["SETTINGS"]["WORDS_FILE"]
def_output_directory = config["SETTINGS"]["OUTPUT_DIRECTORY"]
def_output_filename  = config["SETTINGS"]["OUTPUT_FILENAME"]
def_suppress         = config["PARAMETERS"]["SUPPRESS"]
def_export           = config["PARAMETERS"]["EXPORT"]
def_startswith       = config["PARAMETERS"]["STARTSWITH"]
def_endswith         = config["PARAMETERS"]["ENDSWITH"]
def_contains         = config["PARAMETERS"]["CONTAINS"]


The next step is to determine the final parameter values for each option. We have the def_-prefixed values read from defaults.cfg and the argvs_-prefixed values containing command line overrides. We set the final_-prefixed variables to the command line overrides (if specified), otherwise fall back to the key-value pairs specified in defaults.cfg: 

# Bind reference to final parameter values.
final_output_directory = argvs_output_directory if argvs_output_directory else def_output_directory
final_output_filename  = argvs_output_filename if argvs_output_filename else def_output_filename
def_suppress_ind = True if def_suppress.upper()=="Y" else False
final_suppress   = argvs_suppress if argvs_suppress is not None else def_suppress_ind
def_export_ind   = True if def_export.upper()=="Y" else False
final_export     = argvs_export if argvs_export is not None else def_export_ind

# Determine patterns to match against, if applicable.
if argvs_startswith is not None:
    final_startswith = argvs_startswith
else:
    final_startswith = def_startswith if len(def_startswith)>0 else None
if argvs_endswith is not None:
    final_endswith = argvs_endswith
else:
    final_endswith = def_endswith if len(def_endswith)>0 else None
if argvs_contains is not None:
    final_contains = argvs_contains
else:
    final_contains = def_contains if len(def_contains)>0 else None


We now present the full implementation of cmdapp.py: 

import sys
import argparse
import configparser
import os.path


# Provide absolute path to defaults.cfg.
DEFAULTS = "defaults.cfg"


if __name__ == "__main__":

parser = argparse.ArgumentParser()
config = configparser.ConfigParser()

# Specify command line arguments.
parser.add_argument(
    "--words-file", type=str,
    help="Absolute filepath of words(.txt) file",
    default=None
    )
parser.add_argument(
    "--output-directory", type=str,
    help="Directory of exported file, if applicable",
    default=None
    )
parser.add_argument(
    "--output-filename", type=str,
    help="Name of exported file, if applicable (matches.txt)",
    default=None
    )
parser.add_argument(
    "--export",
    help="Should matching words be written to file (False)?",
    action="store_true"
    )
parser.add_argument(
    "--startswith", type=str,
    help="Only return words starting with specified char(s)",
    default=None
    )
parser.add_argument(
    "--endswith", type=str,
    help="Only return words ending with specified char(s)",
    default=None
    )
parser.add_argument(
    "--contains", type=str,
    help="Only return words containing specified char(s)",
    default=None
    )
parser.add_argument(
    "--suppress", help="Should matching words be suppressed (False)?",
    action="store_true"
    )


# Parse command line arguments.
argvs = parser.parse_args()

# Read in `settings.cfg` to obtain default configuration values.
config.read(DEFAULTS)
def_words_file       = config["SETTINGS"]["WORDS_FILE"]
def_output_directory = config["SETTINGS"]["OUTPUT_DIRECTORY"]
def_output_filename  = config["SETTINGS"]["OUTPUT_FILENAME"]
def_suppress         = config["PARAMETERS"]["SUPPRESS"]
def_export           = config["PARAMETERS"]["EXPORT"]
def_startswith       = config["PARAMETERS"]["STARTSWITH"]
def_endswith         = config["PARAMETERS"]["ENDSWITH"]
def_contains         = config["PARAMETERS"]["CONTAINS"]

# Parse parameter overrides specified as command line arguments.
argvs_words_file       = argvs.words_file
argvs_output_directory = argvs.output_directory
argvs_output_filename  = argvs.output_filename
argvs_suppress         = argvs.suppress
argvs_export           = argvs.export
argvs_startswith       = argvs.startswith
argvs_endswith         = argvs.endswith
argvs_contains         = argvs.contains
final_words_file       = argvs_words_file if argvs_words_file else def_words_file

# Verify existence of final_words_file.
if not os.path.isfile(final_words_file):
    raise FileNotFoundError(f"File not found: {final_words_file}")

# Bind reference to final parameter values.
final_output_directory = argvs_output_directory if argvs_output_directory else def_output_directory
final_output_filename  = argvs_output_filename if argvs_output_filename else def_output_filename
def_suppress_ind = True if def_suppress.upper()=="Y" else False
final_suppress   = argvs_suppress if argvs_suppress is not None else def_suppress_ind
def_export_ind   = True if def_export.upper()=="Y" else False
final_export     = argvs_export if argvs_export is not None else def_export_ind

# Determine patterns to match against, if applicable.
if argvs_startswith is not None:
    final_startswith = argvs_startswith
else:
    final_startswith = def_startswith if len(def_startswith)>0 else None
if argvs_endswith is not None:
    final_endswith = argvs_endswith
else:
    final_endswith = def_endswith if len(def_endswith)>0 else None
if argvs_contains is not None:
    final_contains = argvs_contains
else:
    final_contains = def_contains if len(def_contains)>0 else None

# Begin processing words list.
with open(final_words_file, "r") as f:
    ll_init = f.read().split("\n")

# Remove whitespace from words and convert to lowercase.
ll = [i.strip().lower() for i in ll_init]
len_words_init = len(ll)

# Filter ll by final_startswith.
if final_startswith is not None:
    ll = filter(lambda w: w.startswith(final_startswith), ll)

# Filter ll by final_endswith.
if final_endswith is not None:
    ll = filter(lambda w: w.endswith(final_endswith), ll)

# Filter ll by final_contains.
if final_contains is not None:
    ll = filter(lambda w: final_contains in w, ll)

ll_final = list(ll)
len_words_final = len(ll_final)
pct_retained    = round(100*(len_words_final/len_words_init),2)

if not final_suppress:
    print(f"Number of words in total : {len_words_init}")
    print(f"Number of words in subset: {len_words_final}")
    print(f"Percent of words retained: {pct_retained}%")
    print("")
    disp_list = list(enumerate(ll_final, start=1))
    for i in disp_list:
        print(f"{i[0]}. {i[1]}")
    print("")

# Construct filepath if `--export` present in sys.argv.
if final_export:
    # Check for trailing slash with final_output_directory.
    if final_output_directory.endswith("\\"):
        final_output_directory = final_output_directory.rstrip("\\")
    elif final_output_directory.endswith("/"):
        final_output_directory = final_output_directory.rstrip("/")
    export_fullpath = \
        final_output_directory + os.path.sep + final_output_filename

    with open(export_fullpath, "w") as fw:
        for i in ll_final: fw.write(f"{i}\n")


Another useful feature of argparse is the automatic creation of help menus. To demonstrate, we run cmdapp.py with the --help flag. Recall that we didn’t include any logic to produce a command line help menu: 

$ python3 --help
usage: cmdapp.py [-h] [--words-file WORDS_FILE]
             [--output-directory OUTPUT_DIRECTORY]
             [--output-filename OUTPUT_FILENAME] [--export]
             [--startswith STARTSWITH] [--endswith ENDSWITH]
             [--contains CONTAINS] [--display] [-v]

optional arguments:
  -h, --help            show this help message and exit
  --words-file WORDS_FILE
                        Absolute filepath of words(.txt) file
  --output-directory OUTPUT_DIRECTORY
                        Directory of exported file, if applicable
  --output-filename OUTPUT_FILENAME
                        Name of exported file, if applicable (matches.txt)
  --export              Should matching words be written to file (False)?
  --startswith STARTSWITH
                        Only return words starting with specified char(s)
  --endswith ENDSWITH   Only return words ending with specified char(s)
  --contains CONTAINS   Only return words containing specified char(s)
  --suppress            Should matching words be suppressed (False)?

Pretty slick! As long as you provide useful details pertaining to your application’s functionality and configurable parameters, argparse will generate a nicely formatted help menu that can benefit your users.

Let’s demonstrate cmdapp.py in action: Match all words in the dictionary file beginning with “s”, ending with “r” and containing “u”: 

$ python3 cmdapp.py --startswith="s" --endswith="r" --contains="u"
Number of words in total : 4319
Number of words in subset: 15
Percent of words retained: 0.35%

1. santour
2. sauncier
3. secutor
4. sejour
5. semicircular
6. sinistrocular
7. souder
8. spitefuller
9. squidder
10. strummer
11. stubborner
12. suffixer
13. sunbather
14. superoposterior
15. sur


If we include the --output-directory option, the results will be written to a file named matches.txt in that directory. Also, inclusion of --suppress prevents matches from being written to stdout: 

$ python3 cmdapp.py --startswith="s" --endswith="r" --contains="u" --export --output-directory="E:/Temp" --suppress

Conclusion

At least part of what makes Python great is the collection of really useful, practical and straightforward-to-implement utilities that make up the standard library. In this post, we demonstrated two of those utilities that you’ll find yourself going back to time and time again. I encourage you to check out the documentation for argparse and configparser, as they expose a good amount of fucntionality we didn’t cover here.

Until next time, happy coding!