Aquila_info
Aquila - an automated test harness for AceDB
New!
Aquila can test either giface or tace - see
Latest Changes
Aquila is a perl script which is used to test acedb executables. I
have tested it with tace and giface.
It will perform two basic types of testing: you can provide it with
a script specifically designed to test changes you have made to the executable,
or a full suite of scripts to perform a full regression test. For example,
it runs in regression-test mode every night at the Sanger Institute, comparing
the experimental version of tace with the supported and development versions.
The fundamental concept is as follows: aquila is used to run two versions
of the binary using the same script(s). It is then run a third
time to compare the results of the first two runs.
The script may consist simply of the tests you want to perform, or it may
use the "include" command to include other script files at run-time, or a
mixture of the two. Included files may also contain the "include" command,
though I haven't tested more than a few layers of recursion.
Preamble
Obviously you're going to have to create an appropriate working environment
before you can run aquila. You're going to need the two binaries you
want to test, one or more scripts and a database, as well as the aquila source
code.
So first get the aquila source code. I haven't yet thought about allowing
downloading from the web site, so for now just get it out of CVS. The
default directory tree created by CVS may not be what you want.
cvs checkout build/aquila
This will create build/aquila/ immediately below where you are and put everything
in aquila/. If this is not what you want, use the -d flag as follows:
cvs co -d aquila build/aquila
This creates the aquila/ directory within your current working directory.
(The cvs checkout command can be abbreviated to cvs co.) As currently
set up, this gets you the aquila source code, all the scripts, database schemas,
data, etc. If we ever make this available over the net we may want to
revisit this.
Assuming you're working from wherever the aquila source code is located, you'll
need two directories immediately below there, one for each version of the
binary you're testing. I typically call them test_old and test_new,
but you can call them what you like, as you have to provide the name on the
command line when you run aquila. These are referred to as "version"
directories from here on. Within each of those, you must have a bin/ directory,
containing the binary to be tested. So I have ...aquila/test_old/bin/tace
and ...aquila/test_new/bin/tace. There is no code in aquila to allow
you to have your binary in a different location, it must be in a directory
called bin within the version directory you specify on the command line. This
doesn't seem like too much of a hardship to me, but if you feel differently,
email me and I'll consider changing it.
When you download aquila, you should find yourself with an aquila/config/
subdirectory. This is where the scripts, schemas and data live. Aquila
uses a default script name of "script" but you can specify a different one
using the -s flag (see Usage below.) The tests typically consist of
a database initialisation phase, using one of the wspec schemas, followed
by a parse statement to load data from .ace files, and finally a series of
commands to perform the actual tests.
For each test, the appropriate schema is copied into a local wspec directory
within the "version" directory, eg aquila/test_new/wspec, before the database
is reinitialised. Any missing directories are created automatically,
and a new, local passwd.wrm is created containing just the login id of the
person running the tests. This precludes the need to ensure you already
exist in any of the passwd.wrm files.
All script files, whether default or not, must live in the config directory.
At present there is no facility to allow them to be anywhere else. Once
again, this doesn't seem too much of a hardship, but I'm willing to listen.
Running Aquila
Usage:
aquila -h
print usage info
aquila [-d] [-f <field-list>] [-s <scriptfile>]
with no arguments, generates
a bug report
aquila [-d] [-q] [-n] [-k] [-c <binary>] [-s <scriptfile>]
<version>
with one argument, uses
file config/script to measure
the tace version <version>
and places results in <version>/result/.
aquila [-d] <version> <version2>
with two arguments, compares
the results previously placed in
<version>/result/ and <version2>/result/
-f <field-list> list of fields to report (e.g. "all",
"model,data")
-s <scriptfile> use this scriptfile instead of "script".
-q turns off trace messages
-d turns on verbose debugging output
-k keep the database at startup (by default, the database
is scratched)
-n do not produce the file <version>.oneresult
! options must be given in order shown above
I'm not convinced the option to generate a bug report works, but I haven't
spent much time on it.
NB If you run aquila with the perl -w flag to generate warning messages, be aware
that it produces lots of "Use of uninitialized variable" messages. I
did start looking at ways of eliminating these, but after a while decided
life was too short, given the program seems to work.
Simple testing.
Without bothering with special scripts, etc, the commonest way to run
aquila is first against the previous, working binary, then against the new
one, then specifying both to compare the results.
So in my example above, I'd say
aquila test_old
This tests the copy of tace sitting in test_old/bin, creates a database in
test_old/bin, copies the .wrm files to test_old/wspec and puts the results
in test_old/result. It would look in aquila/config for a file called
"script". The standard config/script consists of a series of include
statements, and aquila constructs a complete script file consisting of all
the actual tests which it then places in the results directory.
By default, the contents of all the individual result files are concatenated
into a single file called <version>.oneresult and placed in the current
directory. I don't think this is used by the comparison phase, but
it's sometimes useful to look at.
I would then run aquila against the second binary
aquila test_new
This would do exactly the same, but for test_new.
To compare the two sets of results, specify both version directories on the
command line:
aquila test_old test_new
This compares the two sets of results and produces a formatted report identifying
differences between them. Do not specify a script file for the compare
phase. If you do, Aquila will issue a warning message and ignore the
script you specified. Aquila uses the script file in the first version
results directory for the comparison. You may want to pipe the output
through more or redirect it into a file for later checking, depending on
how big your script is.
Special Scripts
When performing day-to-day development and maintenance tasks on acedb programmes,
you probably won't want to run a test script which takes more than a few seconds
if possible, so aquila allows you to specify your own script file instead
of the default, thus allowing you to create your own test script specifically
designed to test the changes you're making to acedb.
aquila -s my.script test_old
Your script file still needs to live in the config directory, but it can contain
include statements to incorporate other scripts if appropriate. They,
too, must be in config.
Once you've completed testing, if you want to add your script to the standard
set of tests, simply add an "include" statement to the standard script. the
syntax is about as simple as it's possible to get, ie include <filename>,
eg include script.rclass.
Notes
The top line of the oneresult output file gives the version of Aquila that
was used for the run. This version is now the CVS revision number,
and will automatically increment each time Aquila is committed back into
CVS. However, it does mean that there is a backward step in versions,
since the last hardcoded version number was 1.14 and the current (1st Nov
2002) one is 1.7.
Failure Notification
When it fails, aquila sends a rather terse email saying it's failed. By default
this is sent to whoever is running aquila, simply using the perl getlogin
function, but it can be overridden by specifying the -u option on the command
line.
If you want to notify extra people without resorting to a mailing list, you can
list them in a file which aquila interrogates. At present, this must be
called buildfailure.email. It must reside in the aquila/ directory,
and each addressee must be preceded by a -c flag, thus:
-c mieg@ncbi.nlm.nih.gov -c rd@sanger.ac.uk
I don't yet know about:
the debug report, including the -f <fieldlist> option
White space.
There are various reasons why you might examine the output files, but when you
do so, be aware that when running in comparison mode, aquila strips out all
white space from the two files being compared, and can leave the stripped-out
files in place. This does not mean that tace/giface dumped the data in this
format. The absence of whitespace is not a symptom of a problem, it's a red
herring.
Latest Changes
Aquila can now be used to test both giface and tace binaries, and you can
mix such tests, provided any one script is limited to testing one or other
of the two. In our automated overnight regression testing, for example,
most scripts specify tace as the binary, but the gff script specifies giface.
We have disabled the old -c flag (it now generates a warning but is otherwise
ignored) which told aquila which binary to test, and
altered aquila to allow it to determine the binary from the script on the fly. For
pragmatic reasons, it still requires a tace binary in the bin/ directory in
order to initialise the database, but once that's done, it uses the script
commands to determine the binary to test.
And obviously if you run aquila to test both tace and giface, you're going to
need both binaries in your bin/ directory.
White space.
There are various reasons why you might examine the output files, but when you
do so, be aware that when running in comparison mode, aquila strips out all
white space from the two files being compared, and can leave the stripped-out
files in place. This does not mean that tace/giface dumped the data in this
format. The absence of whitespace is not a symptom of a problem, it's a red
herring.
Back to top of page
