This is gregdoc.info, produced by Makeinfo version 3.12h from
gregdoc.texi.

START-INFO-DIR-ENTRY
* Greg: (greg).            Another GNU testing framework.
END-INFO-DIR-ENTRY

   Copyright (C) 1998,1999,2000 Free Software Foundation, Inc.

   Permission is granted to make and distribute verbatim copies of this
manual provided the copyright notice and this permission notice are
preserved on all copies.

   Permission is granted to copy and distribute modified versions of
this manual under the conditions for verbatim copying, provided also
that the entire resulting derived work is distributed under the terms
of a permission notice identical to this one.

   Permission is granted to copy and distribute translations of this
manual into another language, under the above conditions for modified
versions.


File: gregdoc.info,  Node: Top,  Next: Copying

Greg
****

   Greg is a framework for running test suites on software tools.  Greg
was developed for the _GNUstep_ project.

   This file describes version 1.2 of Greg.

* Menu:

* Copying::		GNU Library
* News::                New in this release
* Overview::            Greg in brief
* Installation::	Installing Greg
* Invoking Greg::	How to run tests
* Writing tests::	How to write a testsuite
* Index::               Index


File: gregdoc.info,  Node: Copying,  Next: News,  Prev: Top,  Up: Top

Copying
*******

   See the file `COPYING.LIB'.


File: gregdoc.info,  Node: News,  Next: Overview,  Prev: Copying,  Up: Top

New in this release
*******************

   New in 1.2

   * Permit the files in the `greg-files' variable to use the `.scm'
     extension or not.

   * Added `greg-paths' variable to permit direct running of named files
     rather than using the DejaGNU-like tools directory structure.

   * Alter the `greg' script to set the `greg-paths' variable to
     contain any files listed on the command-line.

   * Added `greg-expect-pass' and `greg-expect-fail' macros.

   New in 1.1

   * Changed from using GNU Library General Public License to the newer
     GNU Lesser General Public License.

   * Changed from using a compiled module (deprecated in guile) to
     using a dynamic library for pty support.

   New in 1.0

   * Nothing really - a few very minor fixes - no bug reports, which is
     why I decided to risk bumping the version number to 1.0 -
     presumably people will now find hundreds of bugs...

   * Updated to work with Guile and 1.3.4

   New in 0.7

   * Updated to work with Guile 1.3.2 and 1.3.3

   * Updated to work on GNU/Linux systems with pts rather than ptys

   * Improved configuration code

   New in 0.6

   * Converted makefiles to work without GNUstep

   * Improved configuration code

   * Improved documentation somewhat

   * Updated to work with guile version 1.3

   New in 0.5

   * greg-testcase now returns a useful result.

   * Fixed bug in creating child processes on some systems.

   New in 0.4

   * Allow an empty string as the program name for creating a child
     process - in which case, we fork without execing to another
     program.  Modified Greg to allow `(quit)' to exit from the test
     system.

   New in 0.3

   * Improved installation - now checks for location of guile shell.

   * Fixed a couple of minor errors in the documentation.

   New in 0.2

   * Ported to Unixware 2.1.3 - should work for any SysV4.2 based unix.

   General news

   This release of Greg provides a test framework much like that of
DejaGNU, but also provides Guile modules to permit `embedded' testing of
applications that use Guile as a scripting language and libraries that
are directly accessible to Guile.

   This release has been tested with Guile-1.3.4 and 1.3.3, it will
certainly not run on versions earlier than 1.3.1

   Apologies to Guile/Scheme/Lisp programmers out there - I came to
this from Objective-C programming and taught myself during the four
weeks that I wrote GNUstep-Guile and Greg - so my code is probably
really ugly - but it does seem to work.

   The code to run a child process in a pseudo-terminal works for
GNU/Linux and SysV4.2 systems - it probably needs work to make it more
portable - Please email bugfixes and comments to <bug-gnustep@gnu.org>
or directly to me <rfm@gnu.org>


File: gregdoc.info,  Node: Overview,  Next: Installation,  Prev: News,  Up: Top

Greg in brief
*************

   Greg is a framework for testing other programs and libraries.  Its
purpose is to provide a single front end for all tests and to be a
small, simple framework for writing tests.  Greg leverages off the
Guile language to provide all the power (and more) of other test
frameworks with greater simplicity and ease of use.

   The simplicity of the Greg framework makes it easy to write tests for
any program, but it was specifically written for use with GNUstep-Guile
to permit direct testing of the _GNUstep_ libraries without the
necessity to run a separate driver program.

   The core functionality of Greg is a Guile module which can be loaded
into any software with an embedded Guile interpreter.  Any program
which uses Guile as it's scripting language can therefore use Greg to
test itself directly!

   For testing external programs, Greg provides a compiled module that
may be dynamically linked into Guile to permit you to run an
application as a child process on a pseudo-terminal.  In conjunction
with the standard Guile `expect' module, this lets you test external
programs.

   Also provided is `greg' - a Guile script to invoke the Greg test
framework in much the same way that `runtest' is used in DejaGNU.

   All tests have the same output format (enforced by the
`greg-testcase' procedure).  Greg's output is designed to be both
readable and readily parsed by other software, so that it can be used
as input to customised testing processes.

   Greg provides most of the functionality of DejaGNU but is rather
simpler.  It omits specific support for cross-platform/remote testing
since this is really rather trivial to add where required and tends to
vary from site to site so much that an attempt at a generic solution is
pretty pointless.  What Greg does do, is provide hooks to let you
easily introduce site specific code for handling those sorts of
situations.

   The current version of Greg can normally be found on GNU ftp sites,
with documentation online at
_http://www.gnu.org/software/greg/gregdoc.html_

   or, for the bleeding edge - availably by anonymous cvs as part of the
GNUstep-Guile package in the GNUstep project -

   CVSROOT=":pserver:anoncvs@cvs.net-community.com:/gnustep"
export CVSROOT
cvs login (password is `anoncvs')
cvs -z3 checkout guile
* Menu:

* Running Tests::               How to run a Greg testsuite
* Sample Test::                 A trivial example of a testcase
* Design Goals::                Why Greg does what it does
* Posix::			Greg conforms to POSIX 1003.3


File: gregdoc.info,  Node: Running Tests,  Next: Sample Test,  Up: Overview

How to run a Greg testsuite
===========================

   To run tests from an existing collection, try running

     make check

   If the `check' target exists, it usually saves you some trouble--for
instance, it can set up any auxiliary programs or other files needed by
the tests.

   Alternatively, if you are in the top-level source directory of an
existing testsuite (ie. there are subdirectories containing files with
a `.scm' extension), you can get the `greg' script to test all the
tools in the directory by typing  -

     greg

   Finally, if you just want to run the tests that are in a specific
file (or files), you can get the `greg' script to run them simply by
listing the files on the command line.

     greg a-file-to-run another-file-to-run
     
     or, for verbose output -
     
     greg --verbose a-file-to-run another-file-to-run

   If you have a test suite that is intended to be used for `embedded'
testing - You need to start the application to be tested, gain access
to it's Guile command line (or other guile interface) and enter the
commands -

     (use-modules (ice-9 greg))
     (greg-test-all)


File: gregdoc.info,  Node: Sample Test,  Next: Design Goals,  Prev: Running Tests,  Up: Overview

A trivial example of a testcase
===============================

   Each Greg test is a Guile script; the tests vary widely in
complexity, depending on the nature of the tool and the feature tested.

     ;
     ; GNUstep-guile interface library test
     ;
     ; Create an object using the NSString [stringWithCString:] method and
     ; check that the resulting object is of the correct class.
     ;
     (greg-testcase "The `stringWithCString:' method creates an NSString object" #t
     (lambda ()
       (define obj ([] "NSString" stringWithCString: "Hello world"))
       (gstep-bool ([] obj isKindOfClass: ([] "NSString" class)))
     ))

   Though brief, this example is a complete test.  It illustrates some
of the main features of Greg test scripts:

   * The test case does not deal with accessing the library - the Guile
     interpreter in use is linked with the library.  If you were
     testing an external application which needed to be started up as a
     child process running on a pseudo-terminal, the process startup
     would have been performed in a separate `begin.grg' script.

   * You use the Guile programming language to write the script and, in
     this case, the builtin macros and procedures (such as `gstep-bool'
     and `[]' provided by the GNUstep-Guile library.

   * Test cases use the procedure `greg-testcase' to record the test
     outcome.  You pass this procedure a string (`assertion')
     describing the testcase, a boolean specifying the expected
     outcome, and a `thunk' (parameterless procedure) that performs the
     actual test.
     The `thunk' is normally a `lambda expression' - you can look these
     up in the Guile/Scheme documentation  - or just follow the example.

   * As much code as possible is placed inside the `thunk' passed to
     `greg-testcase' so that, in the event of an error, the
     `greg-testcase' procedure can trap it and report an unresolved
     test.
     The code - `(define obj ([] "NSString" stringWithCString: "Hello
     world"))' could have appeared outside the testcase, but that would
     have been less safe.


   Here is the same example in a slightly different form - using the
`greg-expect-pass' macro -

     ;
     ; GNUstep-guile interface library test
     ;
     ; Create an object using the NSString [stringWithCString:] method and
     ; check that the resulting object is of the correct class.
     ;
     (greg-expect-pass "The `stringWithCString:' method creates an NSString object"
       (define obj ([] "NSString" stringWithCString: "Hello world"))
       (gstep-bool ([] obj isKindOfClass: ([] "NSString" class)))
     )


File: gregdoc.info,  Node: Design Goals,  Next: Posix,  Prev: Sample Test,  Up: Overview

Why Greg does what it does
==========================

   Greg was written to support regression testing for the _GNUstep_
libraries.  It was inspired by an earlier test framework (by Ovidiu
Predescu) that used DejaGNU along with a `driver' program (to make the
calls to the library) and a suite of TcL scripts to control the driver.

   There were three main problems (inherent in the nature of DejaGNU)
with that approach -

   * Writing/Debugging tests was a pain - errors in the TcL code of a
     testcase could all too easily interfere with the TcL routines that
     interfaced to the driver program.
     Solution - use another language where errors in testcases are
     localised.

   * Writing/Debugging tests was a pain - you needed to attach gdb to
     the driver program that was running as a child of the DejaGNU
     process in order to debug errors in the driver/library under test.
     Solution - `embedded testing' make the test framework and the code
     under test part of the process, so it's easy to run the whole
     thing under gdb.

   * The driver program and TcL scripts required quite a bit of support
     when adding new tests.
     Solution - `embedded testing' we needed to remove the intermediary
     layers and test as directly as possible.


   So, something different was required, a test framework in a safer,
simpler language that made it easy to create thin interfaces to
libraries, so simplifying the task of producing testcases.

   Of course, the good points of DejaGNU needed to be retained - clear
output, Posix compliance, the ability to test separate programs as well
as libraries.

   A couple of additional goals seemed worthwhile -

   * Make the test framework flexible so that applications could easily
     use it for self-testing.

   * Give the framework a similar structure to, and a user interface
     like that of DejaGNU, so that people with a background in testing
     could easily adopt it.

   Hopefully, Greg meets all its goals.


File: gregdoc.info,  Node: Posix,  Prev: Design Goals,  Up: Overview

A POSIX conforming test framework
=================================

   This section is copied almost directly from the DejaGNU documentation
with minor modifications -

   Greg is believed to conform to the POSIX standard for test
frameworks.

   POSIX standard 1003.3 defines what a testing framework needs to
provide, in order to permit the creation of POSIX conformance test
suites. This standard is primarily oriented to running POSIX
conformance tests, but its requirements also support testing of features
not related to POSIX conformance.

   The POSIX documentation refers to "assertions".  An assertion is a
description of behavior.  For example, if a standard says "The sun
shall shine", a corresponding assertion might be "The sun is shining."
A test based on this assertion would pass or fail depending on whether
it is daytime or nighttime.  It is important to note that the standard
being tested is never 1003.3; the standard being tested is some other
standard, for which the assertions were written.

   As there is no test suite to test "testing frameworks" for POSIX
1003.3 conformance, verifying conformance to this standard is done by
repeatedly reading the standard and experimenting.  One of the main
things 1003.3 does specify is the set of allowed output messages, and
their definitions.  Four messages are supported for a required feature
of POSIX conforming systems, and a fifth for a conditional feature.
Greg supports the use of all five output messages; in this sense a test
suite that uses exactly these messages can be considered POSIX
conforming.  These definitions specify the output of a test case:

`PASS'
     A test has succeeded.  That is, it demonstrated that the assertion
     is true.

`UPASS'
     POSIX 1003.3 does not incorporate the notion of unexpected passes,
     so for strict POSIX, `PASS', instead of `UPASS', is returned for
     test cases which were not expected to pass but did.  This means
     that `PASS' is in some sense more ambiguous than if `UPASS' is also
     used.

`FAIL'
     A test _has_ produced the bug it was intended to capture.  That is,
     it has demonstrated that the assertion is false.  The `FAIL'
     message is based on the test case only.  Other messages are used to
     indicate a failure of the framework.

`XFAIL'
     POSIX 1003.3 does not incorporate the notion of expected failures,
     so for strict POSIX, `FAIL', instead of `XFAIL', is returned for
     test cases which were expected to fail and did not.  This means
     that `FAIL' is in some sense more ambiguous than if `XFAIL' is also
     used.

`UNRESOLVED'
     A test produced indeterminate results.  Usually, this means the
     test executed in an unexpected fashion; this outcome requires that
     a human being go over results, to determine if the test should
     have passed or failed.  This message is also used for any test
     that requires human intervention because it is beyond the
     abilities of the testing framework.  Any unresolved test should
     resolved to `PASS' or `FAIL' before a test run can be considered
     finished.

     Note that for POSIX, each assertion must produce a test result
     code.  If the test isn't actually run, it must produce `UNRESOLVED'
     rather than just leaving that test out of the output.  With Greg
     this is not a problem - any unexpected termination of a
     `greg-testcase' procedure will produce `UNRESOLVED'.

     Here are some of the ways a test may wind up `UNRESOLVED':

        * A test's execution is interrupted.

        * A test does not produce a clear result. This is usually
          because there was a Guile exception thrown while processing
          the test.  This usually requires a human being to examine the
          output to determine what really happened--and to improve the
          test case.

        * A test depends on a previous test, which fails.

        * The test was set up incorrectly and failed to produce a
          boolean value.

`UNTESTED'
     A test was not run.  This is a placeholder, used when there is no
     real test case yet.

The only remaining output message left is intended to test features that
are specified by the applicable POSIX standard as conditional:

`UNSUPPORTED'
     There is no support for the tested case.  This may mean that a
     conditional feature of an operating system, or of a compiler, is
     not implemented.  Greg uses the same output procedures to produce
these messages for all test suites, and these procedures are already
known to conform to POSIX 1003.3.  For a Greg test suite to conform to
POSIX 1003.3, you must set the variable `greg-posix' to be true (or run
the `greg' command with the `--posix' flag).  Doing this will ensure
that non-posix extensions are not used.


File: gregdoc.info,  Node: Installation,  Next: Invoking Greg,  Prev: Overview,  Up: Top

Installing Greg
***************

Requirements
------------

   Greg needs to have Guile installed.  It should work with Guile-1.3
or later.  You need to have the `guile' program in your path in order
for the installation process to determine the proper locations for
things.

   You can get Guile from any GNU ftp site.

   The current version of Greg can normally be found on GNU ftp sites,
with documentation online at
_http://www.gnu.org/software/greg/gregdoc.html_

   or, for the bleeding edge - availably by anonymous cvs from the
GNUstep project (_http://www.gnustep.org/_) -

   CVSROOT=":pserver:anoncvs@cvs.net-community.com:/gnustep"
export CVSROOT
cvs login (password is `anoncvs')
cvs -z3 checkout guile/Greg
Building
--------

To build Greg -

   Type `./configure' in the main Greg directory to configure for your
system.

   Once configuration is complete, go into the `Library' subdirectory
and type `make install' to build and install things.

   You should end up with -

   A dynamic library, which can be dynamically linked into Guile with `
(if (not (feature? 'greg-pty))     (dynamic-call "scm_init_greg"
(dynamic-link "libgreg.so"))) '

   a module defining Guile procedures and variables providing the main
test framework, which can be accessed using `(use-modules (ice-9
greg))',

   and a Guile script that you can use to run tests from the unix
command-line (`greg').

   You _MUST_ install Greg before you attempt to use it (or run it's
self-tests) because the Guile modules making it up must be in place in
the standard Guile directories before Greg can work.

   Once Greg is installed, you can type `make check' in the Tests
directory to get Greg to test itself.

   You can type `make' in the Documentation directory to build the
documentation in _info_, _html_ and _dvi_ formats.

   NB.
You must have the `makeinfo' program installed to build the
documentation in info format
You must have the `texi2html' program installed to build the
documentation in html format
You must have the `texi2dvi' program installed to build the
documentation in dvi format
Problems
--------

Greg is quite simple, so there is not much to go wrong with it.  Of
course, you must have a working copy of Guile installed, and you need
to make sure you ran the configure script to configure Greg for your
system, and installed Greg, but after that, most stuff should just work.

   The single area where you are most likely to encounter problems is
if you are using Greg to test external programs run in a child process
using the `(greg-child)' procedure.

   * If you install Greg in a non-standard location, it's possible that
     guile will not be able to find the `libgreg.so' dynamic library to
     load it.  In this case, you will need to change/set the
     `LD_LIBRARY_PATH' environment variable (or whatever other
     environment variable your operating system uses to list the places
     in which dynamic libraries may be found) to include the directory
     in which the library has been installed.

   * The C code for this procedure is quite system-dependent and has
     only been tested on a few operating systems.  If you have problems
     with this, look at `greg.c' in the Library directory.

   Please attempt to make a patch to fix things on your
operating-system and send it to me - <rfm@gnu.org> or to
<bug-gnustep@gnu.org>


File: gregdoc.info,  Node: Invoking Greg,  Next: Writing tests,  Prev: Installation,  Up: Top

Using `greg'
************

   The Greg framework is designed to be used in two ways - as an
embedded system from within any application which is linked with the
Guile library, or stand-alone using the command-line `greg' driver
script.  For both of these methods of usage the test cases are written
the same way and the expected output is the same.

* Menu:

* Output::			Expected output format from a Greg test
* Files and directories::	The layout of files and directories
* Embedded usage::		What does a Greg test case look like?
* Command-line usage::		Using the `greg' script.


File: gregdoc.info,  Node: Output,  Next: Files and directories,  Up: Invoking Greg

Output
======

   While Greg may produce more verbose output in response to various
settings, the basic output of a test run is a series of lines
describing the success/failure state of each testcase encountered,
followed by a summary of all test cases.

   In `normal' mode, only unexpected results are displayed, but in
`verbose' output mode, results for _all_ results are displayed.

   `greg' flags the outcome of each test case with a line consisting of
one of the following codes followed by a colon, a space, and then the
testcase description.

`PASS'
     The most desirable outcome: the test succeeded, and was expected to
     succeed.

`UPASS'
     A pleasant kind of failure: a test was expected to fail, but
     succeeded.  This may indicate progress; inspect the test case to
     determine whether you should amend it to stop expecting failure.

`FAIL'
     A test failed, although it was expected to succeed.  This may
     indicate regress; inspect the test case and the failing software
     to locate the bug.

`XFAIL'
     A test failed, but it was expected to fail.  This result indicates
     no change in a known bug.  If a test fails because the operating
     system where the test runs lacks some facility required by the
     test, the outcome is `UNSUPPORTED' instead.

`UNRESOLVED'
     Output from a test requires manual inspection; the test suite
     could not automatically determine the outcome.  For example, your
     tests can report this outcome is when a test does not complete as
     expected.

`UNTESTED'
     A test case is not yet complete, and in particular cannot yet
     produce a `PASS' or `FAIL'.  You can also use this outcome in dummy
     "tests" that note explicitly the absence of a real test case for a
     particular property.

`UNSUPPORTED'
     A test depends on a conditionally available feature that does not
     exist (in the configured testing environment).  For example, you
     can use this outcome to report on a test case that does not work
     on a particular target because its operating system support does
     not include a required subroutine.


File: gregdoc.info,  Node: Files and directories,  Next: Embedded usage,  Prev: Output,  Up: Invoking Greg

Files and directories
=====================

   A Greg test run expects to find files and directories in a certain
layout (modeled on that used by DejaGNU) - though it is possible to
override this DejaGNU compatibility feature and simply run the tests in
a list of files.

   The test source directory (normally your current directory) is
expected to contain one or more "tool directories".  Each tool directory
should contain one or more "test scripts".  In fact any file in a tool
directory which has a `.scm' extension is assumed to be a Guile test
script.

   When a normal Greg test run is done, Greg goes through each tool
directory in turn and loads each test script in turn.

   You may set the Guile variable `greg-tools' or use the `--tool ...'
command-line option to specify a list of tools directories to use
rather than assuming that _all_ subdirectories are tool directories.
If you do this, the tools are tested in the order in which they appear
in the list rather than the default order (ASCII sorted by name).

   You may set the Guile variable `greg-files' or use the `--file ...'
command-line option to specify a list of file names to use rather than
assuming that _all_ `.scm' files in each tool directory are test
scripts.  If you do this, the files are loaded in the order in which
they appear in the list.  You may omit the `.scm' extension from
filenames and Greg will supply it for you if necessary.

   You may set the Guile variable `greg-paths' to specify a list of
test files to be run directly, or simply list the files to be run on
the command-line.

   Doing this overrides the `greg-tools' and `greg-files' variables, and
simply runs the files you list in the order you list them.

   As a (minor) complication to this simple layout, Greg permits the
use of `begin.grg' and `end.grg' scripts in both the main source
directory and in each tool directory.  These scripts permit you to add
any initialisation and cleanup code you want.  Typically (for
non-embedded testing) you would use a `begin.grg' script to start the
application to be tested.

   If `begin.grg' exists in the main source directory, it will be loaded
before any tools are tested.

   If `end.grg' exists in the main source directory, it will be loaded
after all the tools are tested.

   If `begin.grg' exists in a tool directory, it will be loaded before
any test scripts in that directory are loaded.

   If `end.grg' exists in a tool directory, it will be loaded after all
the test scripts in that directory are loaded.

   NB. Even when you use the `greg-paths' variable to run one or more
test files directly, the `begin.grg' and `end.grg' files in your current
directory will be loaded.


File: gregdoc.info,  Node: Embedded usage,  Next: Command-line usage,  Prev: Files and directories,  Up: Invoking Greg

Embedded usage
==============

   Greg is designed primarily for embedded usage - any application that
uses Guile as it's scripting language should be able to use Greg to
test itself.

   To this end - Greg provides a Guile module containing definitions of
various procedures (used to run tests) and variables (used to modify the
behavior of a test run).

   Before trying to use any part of Greg, You need to load the Greg
module with `(use-modules (ice-9 greg))'

   The main procedure to run Greg tests is `(greg-test-run)'.  You can
use this to run tests in much the same way as the `greg' script is used
to run tests from the command-line.  The behavior of this procedure is
modified by setting the following top-level variables -

   * greg-debug The `greg-debug' variable is a boolean used to
     determine whether to output extra debug information.  The debug
     information is written to a `.dbg' file.  Test scripts may use the
     `greg-dlog' procedure to output debug information.

   * greg-files The `greg-files' variable is a list of strings used to
     specify the names of the test scripts to be run.  If this list is
     empty, all the files with the `.scm' extension in a tool
     subdirectory are loaded.  If the list contains names, the named
     scripts are loaded in the order in which they occur in the list
     rather than the default order (ASCII sorted by name).
     The exact names of scripts in this list will be used to match
     filenames, and if no match is found a `.scm' extension will be
     added automatically by Greg and a re-match attempted.

   * greg-obj-dir The `greg-obj-dir' variable is a string naming the
     directory in which Greg expects to find binaries to be run as
     child processes.
     By default, this is the current directory (`.').

   * greg-out-dir The `greg-out-dir' variable is a string naming the
     directory in which Greg will generate log and debug output files.
     By default, this is the current directory (`.').

   * greg-paths The `greg-paths' variable is a list of pathname of
     files to be run.  This is used where you don't want to use the
     normal tools hierarchy and need to test specific files explicitly.
     The filenames are used exactly as given.  If you use this
     variable, you should set the `greg-tools' variable to an empty
     list.

   * greg-posix The `greg-posix' variable is a boolean used to
     determine whether to produce strict posix output, or permit
     non-posix extensions.

   * greg-src-dir The `greg-src-dir' variable is a string naming the
     directory in which Greg will look for tool subdirectories and
     where it expects to find the main initialisation and cleanup
     scripts (`begin.grg' and `end.grg') for a test run.
     By default, this is the current directory (`.').

   * greg-tools The `greg-tools' variable is a list of strings used to
     specify the names of the tools to be tested (ie the names of the
     subdirectories of the main source directory).  If this list is
     empty, all the subdirectories of the main source directory
     (normally `.') are assumed to be valid tool directories.  If the
     list contains names, the named subdirectories are used in the
     order in which they occur in the list.
   * greg-verbose The `greg-verbose' variable is a number used to
     determine the level of detail produced in the log output.  At zero
     (the default), only minimal information is output (unexpected
     testcase results and a summary of the number of testcases passed).
     At one, the result of each testcase is reported and a more
     detailed summary is output.  At two, entry and exit of tool
     directories and test files is reported.



File: gregdoc.info,  Node: Command-line usage,  Prev: Embedded usage,  Up: Invoking Greg

Command-line usage
==================

   `greg' is the executable test driver for Greg.  This is a Guile
script that you can use to run tests from the command line.  The
command-line options that you can pass to `greg' are listed below.

   `greg' returns an exit code of `1' if any test has an unexpected
result; otherwise (if all tests pass or fail as expected) it returns
`0' as the exit code.

   This is the full set of command line options that `greg' recognizes.

     greg --tool TOOL ...
     [ --debug ]
     [ --file SCRIPT ... ]
     [ --help ]
     [ --objdir PATH ]
     [ --outdir PATH ]
     [ --posix ]
     [ --srcdir PATH ]
     [ -v | --verbose ]  [ -V | --version ]
     [ files to run ]

`--tool TOOL ...'
     TOOL specifies what set of tests to run.  It provides a list of
     subdirectories (each corresponding to a tool) in which test scripts
     can be found.  Initialisation code (including executable tool
     startup) for each tool may be placed in `begin.grg' in the
     appropriate tool subdirectory.  Cleanup code may be placed in
     `end.grg'.

     For example, including `--tool gcc' on the `greg' command line
     runs tests from the `gcc' subdirectory.

     The order in which the tools are tested will be the same as the
     order in which the tool names occur on the command line.

`--file SCRIPT ...'
     Specify the names of testsuites to run.  By default, `greg' runs
     all tests for the tool, but you can restrict it to particular
     testsuites by giving the names of the `.scm' Guile scripts that
     control them.  You do not need to supply the `.scm' file extension
     - it will be assumed.

     TESTSUITE may not include path information; use plain filenames.

     The order in which the test scripts are run will be the same as
     the order in which the script names occur on the command line.

`--debug'
     Turns on the `expect' internal debugging output.  Debugging output
     is displayed as part of the `greg' output, and logged to a file
     called `tests.dbg'.  The extra debugging output does _not_
     normally appear on standard output.

`--help'
     Prints out a short summary of the `greg' options, then exits (even
     if you also specify other options).

`--objdir PATH'
     Use PATH as the top directory containing any auxiliary compiled
     test code. This defaults to `.'.  Use this option to locate
     pre-compiled test code.  You can normally prepare any auxiliary
     files needed with `make'.

`--outdir PATH'
     Write output logs in directory PATH.  The default is `.', the
     directory where you start `greg'.  This option affects only the
     log and the debug files `TOOL.log' and `TOOL.dbg'.

`--srcdir PATH'
     Use PATH as the top directory for test scripts to run.  `greg'
     looks in this directory for any subdirectory whose name matches
     any toolname (specified with `--tool').

`--verbose'
`-v'
     By default, `greg' shows only the output of tests that produce
     unexpected results; that is, tests with status `FAIL' (unexpected
     failure), `UPASS' (unexpected success), or `ERROR' (a severe error
     in the test case itself).  Specifying `--verbose' to see output
     for tests with status `PASS' (success, as expected) and `XFAIL'
     (failure, as expected). It also causes a more detailed summary to
     be displayed.
     Specifying `--verbose' more than once causes more detail to be
     displayed.

`--version'
`-V'
     Prints out the version numbers of Greg, and Guile, then exits
     without running any tests.


File: gregdoc.info,  Node: Writing tests,  Next: Index,  Prev: Invoking Greg,  Up: Top

Writing tests
*************

getting started
---------------

   The simplest way to get started is to write a file (say `myTests')
and type `greg --verbose myTests' to run the tests in it.

   Your file might contain code like -

     ; A simple test that basic arithmetic works
     
     (greg-expect-pass "one plus one is two"
       (eq? (+ 1 1) 2)
     )
     
     (greg-expect-pass "one minus one is zero"
       (eq? (- 1 1) 0)
     )

   And would produce output like -
     PASS: one plus one is two
     PASS: one minus one is zero
     
     # of testcases attempted   2
     # of expected passes       2
     # of expected failures     0
     # of unexpected passes     0
     # of unexpected failures   0
     # of unresolved testcases  0
     # of unsupported testcases 0
     # of untested testcases    0

types of testsuite
------------------

   There are three types of situation where Greg may be used as a test
framework -

   * Embedded testing
     In this case, you have an application that uses Guile as it's
     scripting language - so you have provided an interface between the
     internals of your program and the Guile interpreter.

     Your testcases can use that interface to test your program
     directly, and you run the tests by loading the Greg framework and
     then doing `(greg-test-run)'

     A testsuite for "embedded" use does not normally need `begin.grg'
     or `end.grg' since the application to be tested will be running the
     tests on itself - so it doesn't need to control child processes.

   * Library testing
     When you want to test a library, you need to write a tiny program
     to start up the Guile command-line interpreter with the library to
     be tested inked in to the program via a via a set of stubs.  The
     resulting program can then be used to run tests on the directly,
     just like normal embedded testing.

     There is a tool called G-Wrap (by Chris Lee), which will generate
     stubs to import your library functions into Guile.  The home page
     of this tool is at _http://www.cs.cmu.edu/~chrislee/g-wrap/_

     For testing Objective-C libraries, you should use the GNUstep-Guile
     interface library (documented online at
     _http://www.tiptree.demon.co.uk/gstep/guile/gstep_guile_toc.html_)
     which provides a fairly complete interface between the Guile and
     Objective-C languages (using the GNU objective-C runtime).

     You can get hold of this as part of the GNUstep core package (from
     any GNU ftp site) or, for the bleeding edge - by anonymous cvs as
     the GNUstep-Guile package in the GNUstep project -

          CVSROOT=":pserver:anoncvs@cvs.net-community.com:/gnustep"

          export CVSROOT

          cvs login (password is `anoncvs')

          cvs -z3 checkout guile

   * External testing
     When you want to test an external program, you use Greg in much
     the same way that you would use DejaGNU.  That is - you run the
     program to be tested as a separate child process and your test
     cases consist of Guile code to send commands to the child process
     and wait for responses from it.

     The child process is normally started by a `begin.grg' script
     using the `(greg-child)' procedure.


testsuite file layout
---------------------

   A "testsuite" for a tool is a directory containing one or more test
script files and (optionally) `begin.grg' and `end.grg' files to handle
initialisation and cleanup.

   Each script file has a `.scm' extension and contains Guile (Scheme)
code, but you do not need to know much about the Guile programming
language to write most tests.

   A script file will contain one or more testcases - each of which
constitutes a test of a single well defined feature of the tool that
the script is meant to test.  A testcase is always written using the
`greg-testcase' procedure, though this procedure could be invoked by a
convenience macro.

greg-expect-pass
----------------

   The `greg-expect-pass' macro is a shorthand method of writing the
most usual sort of testcase - where a fragment of Guile code is run and
is expected to return a `true' result.  It passes an assertion and a
fragment of Guile code that performs a test to the `greg-testcase'
procedure -
     ; A simple test that basic arithmetic works
     
     (greg-expect-pass "one plus one is two" (eq? (+ 1 1) 2))
     
     is equivalent to -
     
     (greg-testcase "one plus one is two" #t
       (lambda ()
         (eq? (+ 1 1) 2)
       )
     )

greg-expect-fail
----------------

   The `greg-expect-fail' macro is a shorthand method of writing a
testcase to confirm that a known bug is still present in the code being
tested.  Once the bug is fixed, it would be altered to be a
`greg-expect-pass' testcase.
     ; A test that basic arithmetic DOESN'T work!
     
     (greg-expect-fail "one plus one is two" (eq? (+ 1 1) 2))
     
     is equivalent to -
     
     (greg-testcase "one plus one is two" #f
       (lambda ()
         (eq? (+ 1 1) 2)
       )
     )

greg-testcase
-------------

   The `greg-testcase' procedure takes three arguments -

   * The assertion
     This is a string describing the test to be performed.  It should
     take the form of an assertion that is expected to hold true.

   * The expected outcome
     This is a boolean value - `#t' if the test is expected to return
     `#t' (ie. the assertion is expected to be proved correct), and
     `#f' if the test is expected to return `#f' (ie. the assertion is
     expected to be proved to be incorrect).

   * The test itself
     This is a "thunk" (a Guile procedure that takes no arguments) and
     is normally a "lambda expression".  The return value of this
     procedure should be `#t' if the assertion is proved correct, `#f'
     if it is proved incorrect.

   The Guile programming language permits the thunk to return in four
ways -

   * Return the boolean value - `#t'
     In which case, the test has succeeded, the assertion has proved
     correct, and Greg will record either an expected success (PASS) or
     an unexpected success (UPASS) depending on the expected outcome
     passed to `greg-testcase'.

   * Return the boolean value - `#f'
     In which case, the test has failed, the assertion has proved
     incorrect, and Greg will record either an expected failure (XFAIL)
     or an unexpected failure (FAIL) depending on the expected outcome
     passed to `greg-testcase'.

   * Return a non-boolean value
     In which case, the test is unresolved (UNRESOLVED).

   * Throw an exception
     In which case Greg's behavior depends on the exception thrown -
        * 'pass
          Greg acts as if the test had returned the boolean `#t'

        * 'fail
          Greg acts as if the test had returned the boolean `#f'

        * 'unsupported
          Greg reports this case as unsupported (UNSUPPORTED)

        * 'untested
          Greg reports this case as untested (UNTESTED)

        * (quit)
          If the script calls the `(quit)' primitive - an exception is
          raised.  In this special case, the test is reported as
          unresolved (UNRESOLVED) and no further tests are executed.
          The testrun is terminated.

        * any other exception
          Greg reports this case as unresolved (UNRESOLVED)

   As there are no other ways in which the thunk may be exited, it is
impossible for a testcase to produce a result that doesn't fit into the
framework (unless your testcase manages either to crash Guile or enter
an infinite loop - in which case you won't get _any_ output).

   The value returned by the `greg-testcase' procedure is a boolean -
`#t' if the test resulted in an expected pass, `#f' otherwise.
You can use this return value to make the execution of subsequent
testcases dependent on the success of an earlier testcase.


     ;
     ; A testcase to check an instance of numeric addition
     ;
     (greg-testcase "One plus One is two" #t
     (lambda ()
       (eq? (+ 1 1 ) 2)
     ))
     
     ;
     ;  The above testcase will generate output -
     ;  `PASS: One plus One is two'
     ;

Multiple testcases
------------------

   It is normal to have more than one testcase in a file and this
produces no problems - the only thing to watch out for is communicating
information between testcases -

   The scope of variables defined in the thunk in a `greg-testcase'
procedure call is that thunk - the variable will _not_ be visible to
the next testcase.

   So - to pass information from one testcase to the next it is
necessary to define variables that can be seen in each testcase.  The
way to do this is normally to define these variables at the start of
the file and then use the `set!' procedure within each testcase to set
a value for a variable to be passed to the next testcase.


     (define arith-ok #f)
     
     ;
     ; A testcase to check an instance of numeric addition
     ;
     (greg-testcase "One plus One is two" #t
     (lambda ()
       (if (eq? (+ 1 1 ) 2)
         (begin (set! arith-ok #t) #t)
         #f)
     ))
     
     ;
     ; A testcase to check arithmetic - only supported if we have addition.
     ;
     (greg-testcase "X multiplied by 2 is X plus X" #t
     (lambda ()
       (if arith-ok
         (eq? (+ 1 1) (* 1 2))
         (throw 'unsupported))
     ))

   Of course, if (as above) the only information you want to pass from
a testcase is whether the test succeeded or not, you can use the return
value from the `greg-testcase' procedure directly -


     (if
       (greg-testcase "One plus One is two" #t
         (lambda ()
           (eq? (+ 1 1 ) 2)
         )
       )
       (greg-testcase "X multiplied by 2 is X plus X" #t
         (lambda ()
           (eq? (+ 1 1) (* 1 2))
         )
       )
       (greg-dlog "Arithmetic operations not supported\n")
     )

External tests
--------------

   When Greg is used to test an external application, you usually want
to run that application as a child process on a pseudo-terminal and
handle tests sending a sequence of commands to the application and
reading anticipated output from the application.

Starting a child process
........................

   Greg provides the `greg-child' procedure to start up a child process
on a pseudo-terminal.  You would usually call this procedure in the
`begin.grg' file in your tool directory, but you could call it at the
start of each script to get a new child process for each script.

   The `greg-child' procedure expects one argument (the name of the
program to be executed) followed by any number of additional arguments
which are the arguments to be passed to the child process.
If the program name does not begin with a slash, Greg will look in the
directory specified in `greg-obj-dir' to find it (by default the
current directory).
If you want your normal PATH to be searched for the program, you should
use -
`(greg-child "/bin/sh" "-c" "foo args")'
to get the shell to execute program `foo' with arguments `args'.

   The `greg-child' procedure will automatically close down the I/O
channels to any process previously started and wait for that process to
die.  If the old child process is badly behaved and will not die, this
can result in Greg hanging - in this case you may need to explicitly
kill the old child by another method before starting the new child
process (this is one of the uses of the `end.grg' script).

   As a special case, you can use an empty string as the program name -
if you do this, another copy of the guile process will be created as a
child and the value returned by `greg-child' in the child process will
be a list containing the single number 0 (in the parent it will be a
list containing the input port, output port and process id of the
child).  You can use this information to get the child copy of the
process to be the program under test.  This is useful for embedded
testing where you want to test the I/O capabilities of the program.

   NB. The `greg-child' procedure is implemented on top of the new
primitive `pty-child'.  This primitive is used to create a new child
process on the end of a pseudo-terminal.  Arguments and return values
are as for `greg-child'.

   * When successfully starting a new program a three item list is
     returned in the parent - input port to read from the child, output
     port to write to the child, process id of child.

   * When returning in a child process because the program name was an
     empty string a list of one item (zero) is returned.

   * After a failure to create a child process, an empty list is
     returned.

Sending to a child process
..........................

   The `greg-send' procedure is provided to send instructions to a
child process.  This procedure takes one or more string arguments and
sends them to the child process (if one exists).

Reading from a child process
............................

   The `greg-recv' macro is used to read data from a child process.
This procedure actually provides a simple front-end to the `expect'
module.  You can use the `expect' module facilities directly if you
want more control than is offered by `greg-recv'.

   The `greg-recv' macro expects one or more lists as arguments - each
list containing a string (a pattern to match) and a result to return on
a successful match.  The value returned by `greg-recv' is the result
for the pattern actually matched.
If no pattern is matched within the timeout period then an empty list
is returned (unless you use `(set! expect-timeout-proc xxx)' to
override Gregs timeout handler.
If no pattern is matched before an end-of-file is read, then an empty
list is returned (unless you use `(set! expect-eof-proc xxx)' to
override Gregs end-of-file handler.


File: gregdoc.info,  Node: Index,  Prev: Writing tests,  Up: Top

Index
*****

* Menu:

* --debug (greg option):                 Command-line usage.
* --help (greg option):                  Command-line usage.
* --objdir (greg option):                Command-line usage.
* --outdir (greg option):                Command-line usage.
* --srcdir (greg option):                Command-line usage.
* --tool (greg option):                  Command-line usage.
* --verbose (greg option):               Command-line usage.
* --version (greg option):               Command-line usage.
* -V (greg option):                      Command-line usage.
* -v (greg option):                      Command-line usage.
* FAIL <1>:                              Output.
* FAIL:                                  Posix.
* Greg test driver:                      Command-line usage.
* News:                                  News.
* PASS <1>:                              Output.
* PASS:                                  Posix.
* POSIX conformance:                     Posix.
* UNRESOLVED <1>:                        Output.
* UNRESOLVED:                            Posix.
* UNSUPPORTED <1>:                       Output.
* UNSUPPORTED:                           Posix.
* UNTESTED <1>:                          Output.
* UNTESTED:                              Posix.
* UPASS <1>:                             Output.
* UPASS:                                 Posix.
* UPASS, avoiding for POSIX:             Posix.
* XFAIL <1>:                             Output.
* XFAIL:                                 Posix.
* XFAIL, avoiding for POSIX:             Posix.
* ambiguity, required for POSIX:         Posix.
* auxiliary test programs:               Command-line usage.
* begin.grg:                             Files and directories.
* building Greg:                         Installation.
* check makefile target:                 Running Tests.
* cleanup:                               Files and directories.
* command line options:                  Command-line usage.
* debug log for test cases:              Command-line usage.
* debug variable:                        Embedded usage.
* design goals:                          Design Goals.
* directories and files:                 Files and directories.
* embedded usage:                        Embedded usage.
* end.grg:                               Files and directories.
* example:                               Sample Test.
* existing tests, running:               Running Tests.
* exit code from greg:                   Command-line usage.
* expected failure:                      Output.
* external tests:                        Writing tests.
* failing test, expected:                Output.
* failing test, unexpected:              Output.
* failure, POSIX definition:             Posix.
* files and directories:                 Files and directories.
* files variable:                        Embedded usage.
* getting started writing a test:        Writing tests.
* greg description:                      Command-line usage.
* greg exit code:                        Command-line usage.
* greg option list:                      Command-line usage.
* greg, listing options:                 Command-line usage.
* greg-child:                            Writing tests.
* greg-debug:                            Embedded usage.
* greg-expect-fail:                      Writing tests.
* greg-expect-pass:                      Writing tests.
* greg-files:                            Embedded usage.
* greg-obj-dir:                          Embedded usage.
* greg-out-dir:                          Embedded usage.
* greg-paths:                            Embedded usage.
* greg-posix:                            Embedded usage.
* greg-recv:                             Writing tests.
* greg-send:                             Writing tests.
* greg-src-dir:                          Embedded usage.
* greg-testcase:                         Writing tests.
* greg-tools:                            Embedded usage.
* greg-verbose:                          Embedded usage.
* help with greg:                        Command-line usage.
* initialisation:                        Files and directories.
* installing:                            Installation.
* invoking:                              Invoking Greg.
* log files, where to write:             Command-line usage.
* make check:                            Running Tests.
* module usage:                          Embedded usage.
* multiple testcases:                    Writing tests.
* naming tests to run:                   Command-line usage.
* obj-dir variable:                      Embedded usage.
* object directory:                      Command-line usage.
* option list, greg:                     Command-line usage.
* options:                               Command-line usage.
* out-dir variable:                      Embedded usage.
* output directory:                      Command-line usage.
* output, additional:                    Command-line usage.
* overview:                              Overview.
* paths variable:                        Embedded usage.
* posix variable:                        Embedded usage.
* problems:                              Installation.
* reading from a child process:          Writing tests.
* requirements:                          Installation.
* run test procedure:                    Embedded usage.
* running:                               Invoking Greg.
* running tests:                         Running Tests.
* selecting a range of tests:            Command-line usage.
* selecting tests for a tool:            Command-line usage.
* sending to a child process:            Writing tests.
* source directory:                      Command-line usage.
* src-dir variable:                      Embedded usage.
* standard conformance: POSIX 1003.3:    Posix.
* starting a child process:              Writing tests.
* success, POSIX definition:             Posix.
* successful test:                       Output.
* successful test, unexpected:           Output.
* test cases, debug log:                 Command-line usage.
* test programs, auxiliary:              Command-line usage.
* test, failing:                         Output.
* test, successful:                      Output.
* test, unresolved outcome:              Output.
* test, unsupported:                     Output.
* testcase:                              Writing tests.
* tests, running:                        Running Tests.
* tests, running specifically:           Command-line usage.
* tests.dbg file:                        Command-line usage.
* tools variable:                        Embedded usage.
* troubleshooting:                       Installation.
* turning on output:                     Command-line usage.
* unexpected success:                    Output.
* unsupported test:                      Output.
* untested properties:                   Output.
* variables:                             Embedded usage.
* verbose variable:                      Embedded usage.
* version numbers:                       Command-line usage.
* writing tests:                         Writing tests.



Tag Table:
Node: Top811
Node: Copying1289
Node: News1411
Node: Overview4255
Node: Running Tests6890
Node: Sample Test8106
Node: Design Goals10848
Node: Posix12941
Node: Installation17804
Node: Invoking Greg21258
Node: Output21937
Node: Files and directories24147
Node: Embedded usage26952
Node: Command-line usage30803
Node: Writing tests34459
Node: Index48299

End Tag Table
