Using genpairs.py

Overview

genpairs.py is a simple combinatorial test vector generator written in Python. It is not as full-featured or efficient as commercial systems like AETG, but it is free and requires only a Python interpreter to run.

genpairs.py accepts input specifications in a format based loosely on category partition test specifications as originally described by Ostrand and Balcer, and as presented in Pezzè & Young, chapter 11. Details of the syntax are described below. genpairs.py uses a heuristic method similar to that of other tools, including AETG. It produces test vectors, that is, sets of choices that together specify one or test cases. As the program name implies, genpairs.cp creates test vectors that cover all pairwise value combinations. The basic pairwise coverage can be modified by a variety of constraints inspired by the category partition test suite specification method. The test vector output is in a tabular form that can then be processed further for producing actual test cases.

Usage

Invoking genpairs.py

genpairs.py is a Python program that reads its input from the standard input, so the most basic invocation from the command line will look something like this (depending on your operating system):
% python genpairs.py < specfile.cp

Options

genpairs.py accepts several command-line options. The examples here are for Unix systems, including Linux and MacOS X.

• -h or --help : Print a short synopsis of options and quit.
• -d or --debug : Print a lot of debugging messages (useful mainly for debugging genpairs.py itself, not for using it).
• -l or --license : Print the (open-source) license terms and quit.
• -c or --csv or --comma-separated-values : Print output in comma-separated-values format, suitable for importing into an Excel spreadsheet and some other programs that accept CSV format.
• -v or --varying or --varying-columns-only : Print only the columns that vary within the test suite. If there is only one valid choice that is not marked "error" or "single", that column will be omitted. (Useful for viewing the combinations manually while developing a test specification.)
• -o or --omit-singles : Do not print combinations with "error" or "single" elements.
• -s or --singles-only : Print only combinations with "error" or "single" elements.
• -i INITIAL_SUITE or --initial=INITIAL_SUITE or --initial-suite=INITIAL_SUITE : Read initial test suite (in csv format). Often used together with -p.
• -p or --pairs --print-pairs : Report pairs not covered by initial test suites. (Useful only with --initial)

Input syntax overview

Input to genpairs.py consists of a sequence of categories, each containing one or more choices. Each choice may optionally be marked with one or more properties, one or more property qualifiers ("ifs"), and zero or one error or single qualifier.

Each line of input may include a comment following //. Except for this, white space (including line breaks) is not significant.

"if", "prop", "except", "single", and "error" are reserved words with special meanings. Category names and value names can be any other strings of non-blank characters, including punctuation symbols. A string ending in ":" is taken as a category name, and other strings are taken as names of choices.

Good practice: Although the rules for identifiers in the current version of genpairs.py is very permissive (e.g., ".38#dk2*&:" could be a legal category name), future versions may be stricter. It is a good idea to use the following conventions:

• Category names should follow the same conventions as identifiers in programming languages like Java and C, except for the terminating ":".
• Choice names should look like identifiers or like constant literals (e.g., integers or floating point numbers), even though genpairs.py treats them like strings. (Note however that blanks are not permitted in choice names, even within quotes.)

Example

Here is a very simple input specification for genpairs.py:

// col0, col1, and col2 are categories. 
col0: 
   e0 error  // e0 is an error entry.  
   e1 error  // So is e1.  They will each appear only once in output. 
   v0.0 prop v0  // v0.0 satisfies property v0
   v0.1 prop v1

col1:
   s0 single
   s1 single
   v1.0 if v0  // v1.0 must be paired with v0.0 (property v0)
   v1.1 if v1 //  v1.1 must be paired with v0.1 (property v1)

col2: 
   v2.0 if v0 if v1  // v2.0 requires properties v0 AND v1 (impossible)
   v2.1

Note that this specification is inconsistent. It says that, for category col2, choice v2.0 requires properties v0 and v1. However, properties v0 and v1 cannot both be true, because property v0 requires col0=v0.0, but property v1 requires col0=v0.1. Here is what we see when we run gencp.py on this input:

python genpairs.py <tests/smoke-tests/small-spec-exclude-3.cp 
Warning - No pair possible:  [ col1=v1.1 col2=v2.0 ]
Warning - No pair possible:  [ col1=v1.0 col2=v2.0 ]
Pairwise coverage: 2  test vectors

           col0            col1            col2 
____________________________________________________________
           v0.1            v1.1            v2.1 
           v0.0            v1.0            v2.1 

Single and error vectors: 4  test vectors

           col0            col1            col2 
____________________________________________________________
             e0            v1.1            v2.0 
             e1            v1.1            v2.0 
           v0.0              s0            v2.1 
           v0.0              s1            v2.1 

The first two test vectors are the normal (non-error, non-single) test cases. The second set of test vectors are error and single cases.