New!

Aquila has been changed to permit testing of giface as well as tace. See Latest Changes


Aquila tests are controlled by a simple scripting language. The language includes an include command, and the main test suite is organized as a master file that includes the individual test files. The master file is at /nfs/disk100/acedb/aquila/config/script_main/script

How to put your test into the test suite

  1. Log in as the user that owns the test directories, i.e. the AceDB build user
  2. Place your script in /nfs/disk100/acedb/aquila/config/script_main/ and put any files it refers to into place, which will normally be in /nfs/disk100/acedb/aquila/config/ for wspec directories, .ace files etc. (In your script, you can refer to this directory by the name $data.)
  3. Test your script by running Aquila against it. Please see the section on running Aquila for full details.
    Briefly: aquila -s scriptfile program_version_1
             aquila -s scriptfile program_version_2
             aquila program_version_1> program_version_2
  4. Add an include line to /nfs/disk100/acedb/aquila/config/script -- I suggest you put this at the end of the list unless there is an obvious reason to do otherwise.
  5. The output from the overnight Aquila run appears in ~acedb/aquila. There are plans to make it available on the web but this work hasn't been completed yet.
  6. The result format is described in How to interpret Aquila test results

You can view the collection of tests currently in use by looking in the latest test results which link through to a web copy of the test scripts used.

The scripting language

The Aquila language is a line-based format, in which each line contains a command and may contain a label.

Block structure and labels

Some commands form the start of a block of commands, and such blocks may be nested. The labels are used not as jump targets but in the generation of filenames for the stored results files on which the comparisons are done. (If there is no label set, an underscore is used in the file name at that point.) A label given for a command is used in the filenames throughout that command; the label of a block-starting command is used in all commands in that block. The label names for nested blocks are concatenated, with dot as the separator. (The last part of the filename is the name of the command that produced that file.)

Note that labels must be unique within a set of tests. This is because they are used to generate the names of the files used to store the results, and duplicate filenames would obviously cause problems. The next section should make that clear. The difficulty here is that as we've recently moved from a single gigantic script file to dozens of little ones, it's now much easier to accidentally end up with duplicate labels. I think that then the second set of tests is ignored.

Here is an example test script showing the labelling structure:

abnmqry: dbinit wspec.mini // angle bracket name queries
  tace
    parse: do parse $data/mini.angleb.ace
    abfind: do find M "<someth>"
    abfindw: do Query find M "<someth>"
    abfindesc: do find M \<someth\>
    abfindq: do find M <someth>

    abclear: do Query find M T & NOT T   // clear active keyset

    abfind2: do find M "Wolf D."
    abfind3: do Query >?M "Wolf D."

  exit

This produces the followning files in the test directory

  -rw-r--r--   1 acedb    acedb          0 Oct 11 10:28 abnmqry._
  -rw-r--r--   1 acedb    acedb         88 Oct 11 10:29 abnmqry._.abclear.do
  -rw-r--r--   1 acedb    acedb         54 Oct 11 10:29 abnmqry._.abfind.do
  -rw-r--r--   1 acedb    acedb         54 Oct 11 10:29 abnmqry._.abfind2.do
  -rw-r--r--   1 acedb    acedb         88 Oct 11 10:29 abnmqry._.abfind3.do
  -rw-r--r--   1 acedb    acedb         54 Oct 11 10:29 abnmqry._.abfindesc.do
  -rw-r--r--   1 acedb    acedb         54 Oct 11 10:29 abnmqry._.abfindq.do
  -rw-r--r--   1 acedb    acedb         88 Oct 11 10:29 abnmqry._.abfindw.do
  -rw-r--r--   1 acedb    acedb         89 Oct 11 10:29 abnmqry._.exit
  -rw-r--r--   1 acedb    acedb        252 Oct 11 10:29 abnmqry._.parse.do
  -rw-r--r--   1 acedb    acedb       1049 Oct 11 10:29 abnmqry.dbinit

It is possible to omit the labels from the script. then aquila assumes _: as label, as is done for the "tace" line in the script above.

If two commands would produce the same file name, at least one of them must be labelled to avoid a conflict.

tace

This starts a tace sessions, and contains a block of commands to be applied to that session. It ends with an "exit" command.

The "tace" command does not initialize the database -- you must used dbinit for that.

Note that recently we've altered aquila to permit testing of either tace or giface. We've disabled the -c command line option (gives a warning but is otherwise ignored) and now allow you to specify either tace or giface in the script.

For pragmatic reasons you still have to have tace available in your bin/ directory as it is still used for database initialisation even when you're testing giface.

exit

This ends the block started by a "tace" aquila command. The data recorded is the time taken by that tace session.

do tacecommand

Does a tace command from the script file. This must be used within a "tace" aquila command block, as must most of the following commands.

show

Tells the AceDB session to run a 'show' command, to output the current keyset.

write

Causes the AceDB session to execute a 'write' command.

list

Causes the AceDB session to execute a 'list' command.

mem

Records the memory used by the AceDB session at the time.

count

Records the size of the current keyset. This is often regarded as enough, in terms of checking whether the result of an operation is still OK.

dbinit

Prepares a fresh database for the test, with a given wspec directory which is copied to configure the new database. This is one of the commands that should be used outside the tace command block.

dbinit -f wspec deletes the database directory rather than simply removing the ACEDB.wrm file to force re-initialization.

setenv

Sets a runtime environment variable for the AceDB session (or any subprocess it starts) to see. This is one of the commands that should be used outside the tace command block.

Result files

The result files are meant to be human-readable and platform- independent, here are some examples:

_._ contains:
!! 26 objects read with 0 errors

_._._.mem contains:
sz 7659 KBytes
rss 5222 KBytes

_.exit contains:
aborted 0
exit_status 0
exit_msg  A bientot 
real_time 2 seconds
cpu_time 2.18 seconds

However, the main point of the result files is to compare them against those from a reference version, rather than to examine them directly.

Handling commands with long output

The result of an aquila run is meant to be a fingerprint of a specific ace version on a specific platform on a specific machine. Since there will probably be a set of such fingerprints around, they should be small in size, but keep all what is necessary to see deviations between versions. This sections show what aquila provides to keep fingerprints small.

consider this script file:

tace
  1: do find Sequence
        count
  2: do -e table sometable.def
        mem
exit

for examining the current keyset there are four commands -- to be used depending on the number of objects retrieved:

write and show -- to see full objects
list           -- to see only object names
count          -- to just count

if 'find Sequence' retrieves, say, 100,000 sequences, there is (usually) no point in listing all of them. just counting the number of objects is enough.

For the tablemaker output there is no correspondence, so aquila allows to limit the number of lines recorded. if sometable.def generated, say 100,000 output lines, do -e just records the 'ends' of the output, i.e. the first and the last line and a count of lines in between.

managing multiple ACEDB schemata

ACEDB is applied in a variety of environments using different database schemata. The schema of an ACEDB database is stored as a set of files in subdirectory called 'wspec'. These different wspecs are stored in the aquila/config/ directory, e.g

aquila/config/wspec.igd
aquila/config/wspec.ace4

this is used in the following way in a script:

default: dbinit
  tace
    mem      // check base memory requirement
    do classes
    mem      // and after loading lexiques
  exit

igd: dbinit wspec.igd
  tace
  load: do pparse longline.ace
  find: do find Sequence "gb:X561*"
           count
exit

ace4: dbinit wspec.ace4mieg
  sample: tace
	 do pparse sample.ace
	 mem
	 exit
  
  similar: tace
      do -e Table tables/similar2.def
      mem
      exit

At execution time, a symbolic link from <ace_version>/wspec to the specified config/wspec.x is made. dbinit without a wspec parameter is equivalent to dbinit wspec

Usually, it is enough to erase the file database/ACEDB.wrm to force initialization of the database. So, by default aquila does just this. If you suspect that this feature of ACEDB is bugged (as was in some ace33) use
dbinit -f
this erases all files in database/ at startup time.

Manipulating the environment:

the operation of some acedb commands are affected by environment variables. aquila can set those for the duration of one tace block, e.g.

setenv ACEDB_DUMP_ATTACH 1
t1: tace
  do find Sequence
     write
  exit
t2: tace
  do find Sequence
     write
  exit

then t1 should dump with @queries, since ACEDB_DUMP_ATTACH is set; the setenv is effective until the next exit statement. thus t2 executes without ACEDB_DUMP_ATTACH.

Combining Multiple Tests

It is not good to pack tests together into one database. Each test should start at a fresh database, even if this triples the execution time of the test suite.

What belongs to a complete test set?
the whole contents of the config directory, i.e.
- the driving script
- .ace, .ace.Z, .ace.gz data
- .cmd command batch files
- tables/*.def table defintions
- wspec* schemata

there are more examples in the distribution package, please see files config/script.example.*


Latest Changes

Aquila can now be used to test both giface and tace binaries, and you can mix such tests, provided any one script is limited to testing one or other of the two. In our automated overnight regression testing, for example, most scripts specify tace as the binary, but the gff script specifies giface.

We have disabled the old -c flag (it now generates a warning but is otherwise ignored) which told aquila which binary to test, and altered aquila to allow it to determine the binary from the script on the fly. For pragmatic reasons, it still requires a tace binary in the bin/ directory in order to initialise the database, but once that's done, it uses the script commands to determine the binary to test.

And obviously if you run aquila to test both tace and giface, you're going to need both binaries in your bin/ directory.

To test giface instead of tace, simply specify giface on the second line of your script. So, for instance, in the first example from the top of this page, the new script would start like this:

abnmqry: dbinit wspec.mini // angle bracket name queries
  giface
    parse: do parse $data/mini.angleb.ace

Back to top of page