This directory tree contains the Hugs testsuite.

The testsuite is used to detect differences between Hugs' documented
behaviour and Hugs' actual behaviour.  There are two ways of dealing
with failed tests: change the documentation to conform to Hugs or
change Hugs to conform to the documentation.  Generally the latter is
preferred.  

If you are not in a position to commit changes and/or are unable to
fix the problem, you should send a bug report to
hugs-bugs@haskell.org.  

If you "fix" a failed test by changing the documentation, it is probably
wise to send a message to hugs-bugs@haskell.org.

Running the testsuite
~~~~~~~~~~~~~~~~~~~~~

To run the tests:

1) Build Hugs in the usual way

     cd hugs98
     make

2) Run the test script in the tests directory.

     cd ../tests
     sh testScript static tcheck rts libs ffi demos

   You can vary the argument list to vary which tests are performed.
   Tests are somewhat approximately broken down into:

     static - static checks on well-formedness
     tcheck - type checking
     rts    - evaluating code
     libs   - libraries that come with Hugs
     ffi    - foreign function interface 
     demos  - demos that come with Hugs

3) Once you get bored of seeing all the successful tests (yahoo!), you
   can use grep to restrict output to the failures:

     sh testScript static tcheck rts libs ffi demos | grep -v '^--'

   A few of the tests generate different output on different platforms
   and so you might get false error reports.  These should all be 
   preceded by warnings of what to expect.  For example:
   
   -- On Windows, it may also produce output that looks like this:
   -- 
   --   ./hugs +q -pHugs:  ./system1.hs < ./system1.input
   --   expected stdout not matched by reality
   --   *** ./system1.output  Fri Jul 11 13:25:27 1997
   --   --- /tmp/runtest3584.3  Fri Jul 11 15:55:13 1997
   --   ***************
   --   *** 1,3 ****
   --     ...
   --   | Hugs:ExitFailure 1
   --   | Hugs:ExitFailure 2
   --     ...
   --   --- 1,3 ----
   --     ...
   --   | Hugs:ExitSuccess
   --   | Hugs:ExitSuccess
   --     ...
   -- 
   -- This reflects the sad fact that System.system always returns
   -- ExitSuccess on DOS machines.  This is a known bug in DOS.
   

Adding new tests
~~~~~~~~~~~~~~~~   

The core of the test harness is the program runstdtest - a hacked up
version of a similar script from the GHC distro.  This runs Hugs over
some input and compares the output against a file containing the
"correct" output.

We tell Hugs to be quiet (+q) and not list modules being loaded (-w),
but it still outputs a banner before generating the interesting part
of the output so we tell runstdtest to ignore the first N lines of
output using the -s<N> flag.  The first line after that should be
either an ERROR that happened while reading the initial script, or
"Type :? for help".  If you change the length of the banner you will
have to change this -s flag.

To add a new test, you:

1) Write a Haskell module, sample input and sample output files.
   (If any are empty, you can omit them.)

   Put a comment of the form 

     --!!! What is being tested

   at the start of the Haskell module.  runstdtest will print
   this string when it runs the test.

   If there is anything subtle about the test or if the test checks
   that an old bug has been fixed, it's a good idea to include that
   information in the test files as normal Haskell comments.

   Tests should be as small as possible and run in as little time
   as possible so that you can run the entire testsuite in a short
   (1-2 minutes) time.

2) Test

   perl runstdtest \
        ../src/hugs -P../lib:../lib/hugs:../lib/exts +q -w -pHugs: -s10 \
        "-Orts/arith.hs" "-irts/arith.input" "-o1rts/arith.output"

   The arguments here are:

     -P...     use the library in this copy of Hugs
     +q        reduces the amount of noise generated by Hugs
               (this makes Hugs output more consistent)
     -w        don't list the files read
               (this makes Hugs output more consistent)
     -pHugs:   makes Hugs use "Hugs:" as the prompt string
               (this makes Hugs output more consistent)
     -s10      ignore the first 10 lines of Hugs' output
               (this skips the Hugs banner)
     -O<prog>  program to load       (optional)
     -i<file>  input to feed to Hugs (/dev/null if unspecified) 
     -o1<file> expected output

3) Decide which category to put your test in: static, tcheck, rts, ...

   There is a separate subdirectory for files associated with each
   category.

4) Add the test command to testScript
   and add your scripts to the appropriate subdirectory
   
   To keep the test commands short, the script defines 

     HUGS=../src/hugs
     STDOPTS='+q -w -pHugs: -s10 -P../lib:../lib/hugs:../lib/exts'

   So you write just:

     perl runstdtest $HUGS $STDOPTS \
        "-Orts/arith.hs" "-irts/arith.input" "-o1rts/arith.output"


Enjoy!

--
Alastair Reid   
alastair@reid-consulting-uk.ltd.uk  
http://www.reid-consulting-uk.ltd.uk/alastair/