Running Acceptance Tests With TextTest

1. Download texttest from testtest.org.

2. Install by running install.py (non-standard installer; you need to indicate target installation directory)

3. Define 2 environment variables. In my case:

export MOS_HOME=/home/ggheo/proj/mail-onna-stick
export TEXTTEST_HOME=$MOS_HOME/tests/texttest

4. Go to texttest installation directory and run texttest.py

  • with no options, it will run all test suites it finds in TEXTTEST_HOME
  • to run a specific test suite, use the -a switch, e.g.:
    ./texttest.py -a mos-index-local
    ./texttest.py -a mos-index-pop
    ./texttest.py -a mos-search
    

Short intro to textest mode of operation

  • A directory under TEXTTEST_HOME (for example mos-search) is considered an application; its name is denoted as app in the discussion that follows
  • Inside the application directory, you need to create a configuration file called config.app. The only thing mandatory in the config file is a binary entry pointing to the AUT "executable" (AUT == Application Under Test). Example:
    binary:$MOS_HOME/bin/index-mail-and-search
    
  • Inside the application directory you create sub-directories which are considered test cases, for example search_local_mailbox_grig and search_local_mailbox_titus
  • Inside the application directory you also create a testsuite.app file containing the test cases you want to run in that test suite (basically the test case directory names, one per line)
  • Inside a test case directory, you need the following files:
    • an options.app file containing command-line arguments to be passed to the AUT executable
    • an output.app file containing the expected stdout emitted by the AUT when called with the given command-line arguments
    • an errors.app file containing the expected stderr emitted by the AUT when called with the given command-line arguments; this file needs to be empty if no stderr is emitted
  • If the AUT also writes to a log file, you need to add an entry to the config.app file of the form:
    [collate_file]
    log:$MOS_HOME/var/mos.log
    
    • You also need to put a copy of the expected log inside each test case directory and call it log.app (or whatever.app if you decide to replace log with whatever in the configuration file entry
  • The config file accepts a myriad of other options, among the most useful being filtering options which allow you to filter out certain lines or words in the output/error/log files so that they don't result in errors during the test run; other useful options, which I'm still investigating, are batch-mode run options, causing the results of the test run to be emailed to specified recipients

Sample output

[ggheo@concord texttest]$ ./texttest.py
Using Application MOS-INDEX-IMAP, checkout /home/ggheo/proj/mail-onna-stick/tests/texttest/mos-index-imap/./
Using Application MOS-INDEX-LOCAL, checkout /home/ggheo/proj/mail-onna-stick/tests/texttest/mos-index-local/./
Using Application MOS-INDEX-POP, checkout /home/ggheo/proj/mail-onna-stick/tests/texttest/mos-index-pop/./
Using Application MOS-INDEX-URL, checkout /home/ggheo/proj/mail-onna-stick/tests/texttest/mos-index-url/./
Using Application MOS-SEARCH, checkout /home/ggheo/proj/mail-onna-stick/tests/texttest/mos-search/./
Running MOS-INDEX-IMAP test-suite mos-index-imap
  Running MOS-INDEX-IMAP test-case index-dzur-mostest2
Comparing differences for MOS-INDEX-IMAP test-suite mos-index-imap
  Comparing differences for MOS-INDEX-IMAP test-case index-dzur-mostest2 - SUCCESS! (on errors.mos-index-imap,log.mos-index-imap,output.mos-index-imap)
Running MOS-INDEX-LOCAL test-suite mos-index-local
  Running MOS-INDEX-LOCAL test-case index-mbox-1msg
Comparing differences for MOS-INDEX-LOCAL test-suite mos-index-local
  Comparing differences for MOS-INDEX-LOCAL test-case index-mbox-1msg         - SUCCESS! (on errors.mos-index-local,log.mos-index-local,output.mos-index-local)
  Running MOS-INDEX-LOCAL test-case index-mbox-socalpiggies
  Comparing differences for MOS-INDEX-LOCAL test-case index-mbox-socalpiggies - SUCCESS! (on errors.mos-index-local,log.mos-index-local,output.mos-index-local)
  Running MOS-INDEX-LOCAL test-case index-two-mboxes
  Comparing differences for MOS-INDEX-LOCAL test-case index-two-mboxes        - SUCCESS! (on errors.mos-index-local,log.mos-index-local,output.mos-index-local)
Running MOS-INDEX-POP test-suite mos-index-pop
  Running MOS-INDEX-POP test-case index-dzur-mostest
Comparing differences for MOS-INDEX-POP test-suite mos-index-pop
  Comparing differences for MOS-INDEX-POP test-case index-dzur-mostest - SUCCESS! (on errors.mos-index-pop,log.mos-index-pop,output.mos-index-pop)
Running MOS-INDEX-URL test-suite mos-index-url
  Running MOS-INDEX-URL test-case index-alife-mbox
Comparing differences for MOS-INDEX-URL test-suite mos-index-url
  Comparing differences for MOS-INDEX-URL test-case index-alife-mbox - SUCCESS! (on errors.mos-index-url,log.mos-index-url,output.mos-index-url)
Running MOS-SEARCH test-suite mos-search
  Running MOS-SEARCH test-case search-grig
Comparing differences for MOS-SEARCH test-suite mos-search
  Comparing differences for MOS-SEARCH test-case search-grig       - SUCCESS! (on errors.mos-search,log.mos-search,output.mos-search)
  Running MOS-SEARCH test-case search-titus
  Comparing differences for MOS-SEARCH test-case search-titus      - SUCCESS! (on errors.mos-search,log.mos-search,output.mos-search)
  Running MOS-SEARCH test-case search-grig-titus
  Comparing differences for MOS-SEARCH test-case search-grig-titus - SUCCESS! (on errors.mos-search,log.mos-search,output.mos-search)