@Part(using, Root="acats.msm")

@comment{$Source: e:\\cvsroot/ARM/ACATS/grade.mss,v $}
@comment{$Revision: 1.2 $ $Date: 2016/07/01 01:43:25 $}

@LabeledSection{ACATS Grading using the Grading Tool}

ACATS 4.1 introduces an optional tool to (mostly) automate grading of ACATS
tests.

When the ACATS was designed (as the ACVC in the early 1980s), the intention
always was that running it would give a simple and clear Pass or Fail result.
However, grading of tests (particularly of B and L Tests) is somewhat
subjective and very time-consuming. (Test grading by formal testers
typically involves poring over compiler listings of the entire ACATS with a
large highlighter.)

The grading tool greatly reduces this effort and enforces both the processing
rules (as outlined in @RefSecnum{Dependencies}) and expected results for a
test (see @RefSecNum{Grading Test Results}).

@Red{@b{Use of the grading tool is optional for ACATS 4.1.} Manual test grading
is acceptable for formal conformity assessments; whether to use the grading
tool will be left to implementers and their ACAL. The ACAA will use experience
with the tool to inform whether using the tool should be required for future
ACATS versions.}


@LabeledClause{Using the Grading Tool}

The ACATS Grading Tool, "Grade" takes three files as input and produces
a grading report, with details for each test followed by a summary and
an overall Passed or Failed result.@Defn{Grading Tool}

The three files are:

@begin{Indent}
@begin{Description}
Event Trace File@\Includes each interesting event that occurs during the
processing of an ACATS test. Each ACATS User (or their implementor) needs
to provide a method of producing an event trace file for their implementation
in order to use the Grading Tools. See clause @RefSecNum{Event Trace Files}
for more information.

Test Summary File@\A machine-readable distillation of one or more
ACATS tests. These are created by a tool included with the ACATS. See
clause @RefSecNum{Test Summary Files} for more information.

Manual Grading File@\A list of tests that may require manual grading. This
file can be created with any plain text editor, and may be empty.
See clause @RefSecNum{Manual Grading Request Files} for more information.
@end{Description}
@end{Indent}

An example using the grading tool is given in subclause
@RefSec{Annotated Grading Tool Example}.

@LabeledSubClause{Workflow using the Grading Tool}

The workflow using the Grading Tool is similar to using the ACATS manually. The
steps needed are outlined below.

@begin{Indent}
@begin{Hang1List}
1.@\Install and configure the ACATS in the normal way, as outlined in clauses
@RefSecNum{Installation of the ACATS Test Suite},
@RefSecNum{Tailoring the ACATS Test Suite}, and
@RefSecNum{Processing the Support Files}. It is particular important that
Macro Defs customization @RefSecNum{Macro Defs Customization} is accomplished
@i<before> generating any test summaries, as the summary program is unaware
of the macro syntax. Also, do not use the grading tool on the support tests in
the CZ directory, as some of these include intentional failure messages that the
grading tool is not prepared to handle.

2.@\Compile the Grading and Test Summary Tools, as described in
@RefSecNum{Compiling the Grading Tool and the Test Summary Tool}.

3.@\Determine how Event Traces are going to be constructed. If the
implementation provides direct writing of an event trace (as described in
@RefSecNum{Creating an Event Trace Directly by the Implementation}), then
go to step 4a. Otherwise, acquire or create a listing convertion tool as
described in @RefSecNum{Creating an Event Trace from Listings}, and go to
step 4b.

4a.@\Create command scripts (as described in
@RefSecNum{Establishing Command Scripts}) to process the ACATS. Include in
those the appropriate option to create event traces. Also, modify Report.A
so that the constant Generate_Event_Trace_File has the value True. When
complete, go to step 5.

@\The command scripts could generate one giant event trace for the entire
ACATS, but it probably is more manageable to create several smaller event
traces for portions of the ACATS. One obvious way to do that is to create
a single event trace for each subdirectory that contains ACATS tests in its
default delivery structure. (Such event traces can be combined later, if
desired.)

4b.@\Create command scripts (as described in
@RefSecNum{Establishing Command Scripts}) to process the ACATS. Include in
those use of the listing conversion tool to make event traces. Then go to
step 5.

5.@\Create test summaries for each grading segment. If, for instance, you
will grade each directory individually, then you will need a test summary file
for each directory. These files can be generated by running the Test Summary
tool on each source file in the directory using a single summary file as output.
On most operating systems, this is easily accomplished with a script command.
(Some possibilities are discussed in @RefSecNum{Test Summary Tool Reference}).

@\The Test Summary Files will only need to be regenerated if the ACATS tests
change in some way (typically, when an ACATS Modification List is issued).
It's probably easiest to make a script to regenerate the entire set of
summaries so that it can be used when the suite changes. Once the entire
set of test summaries has been created, move to step 6.

6.@\Create an empty manual grading request file. This is just an empty text
file. (See @RefSecNum{Manual Grading Request Files} for more information.)

7.@\Process the ACATS tests, creating event traces. The event traces should
contain the same tests as the test summary files. This process is described
in @RefSecNum{Processing ACATS Tests}.

8.@\Run the Grading Tool (GRADE) on the pairs of event traces and summaries,
using the current manual grading request file. Typically, the default options
are sufficient, but some implementations or event traces may need options. The
options are described in @RefSecNum{Grading Tool Reference}.

9.@\If all of the grading reports display Passed, you're done. But most likely,
some tests will be reported as failed. The grading tool will
report the first failure reason for each test, but there may be additional
failure reasons for each test.

@begin{Indent}
@begin{Hang1List}
A.@\If the failure reason is a process failure or a missing compilation, most
likely there is a problem with the scripts that process the ACATS. Make sure
that the test files are compiled in the appropriate order and no test files
are missing. A missing compilation might also mean that the test needs to be
split. See item B.

B.@\If the failure reason is extra or missing errors, grade the test manually
(see @RefSecNum{Grading Test Results}) to see if the problem is with the
implementation or the Grading Tool being too strict about error locations.
If manual grading indicates the test passed, add the test to your Manual
Grading Request file - again, see @RefSecNum{Manual Grading Request Files}
(preferably with a comment explaining why it was added). Note that it is not
necessary to remove tests from this list: if the grading tool determines that
the test grades as Passed or Not Applicable, it will not request manual grading
for the test even if it appears in this list.

@\If the manual grading indicates that the test needs to be split, do the
following. First, add the test to your Manual Grading Request file - the
ACATS requires processing the original test in this case. (Be sure to put in
a comment that the test is split, since it won't be necessary to manually
grade the original test in that case.) Then, split the
test following the guidelines in @RefSecNum{Allowed Test Modifications},
and add the split tests to a processing script and the test summary script.
The Test Summary tool can create summaries for split tests, so the grading tool
can be used to grade them.

C.@\For other failure reasons, most likely the implementation is at fault.
Fixing the implementation is likely the only way to meaningfully change
the result. In the unlikely event that there is a problem with a test,
the procedure for test challenges are outlined in
@RefSecNum{Deviation from Expected Results - Petition & Review}.
@end{Hang1List}
@end{Indent}

@\You'll also need to handle any special handling tests, including any tests
that require manual grading. (This is one good reason to keep the manual
grading list as short as possible.)

@\Then return to step 7 and repeat the test run.
@end{Hang1List}
@end{Indent}

Using this procedure, the vast majority of tests will not require hand grading.
Future ACATS updates may improve tests that are particulary difficult to grade
automatically. @Red{The ACAA is interested in which tests need manual grading
for your implementation - see @RefSecNum{Manual Grading Request Files}.}


@LabeledSubClause{Annotated Grading Tool Example}

Following is an example of using the grading tool on Chapter 5 C-Tests for an
implementation that has not yet implemented most of Ada 2012. For this example,
file ChapC5.csv is an event trace containing the events generated by processing
the 102 ACATS C-Tests for chapter 5. The file C5-Sum.csv is the test summary
file for the 102 ACATS C-Tests for chapter 5. Man.Txt is an empty file (there
are no manual grading requests for this run).

In the following, @i<annotations are given in italics>, and are not part of
the output of the Grading Tool.

@leading@;The command line used for the example is:

@begin{Example}
Grade ChapC5.csv C5-Sum.csv Man.Txt "C-Tests for Chapter 5" - Use_Timestamps -Check_All_Compiles -No_Positions
@end{Example}

@leading@;This gives the following expected results:

@begin{Example}
ACATS Grading Tool - version 1.0
  Compile Checks: CHECK_ALL
  Make checks of time stamps in event log
  Use only line numbers of error messages when grading
  Report verbosity: NORMAL
  779 event trace records read from ChapC5.csv
      @RI[The non-comment records in the event trace file.]
  117 test summary trace records read from C5-Sum.csv
      @RI[The non-comment records in the test summary file. All tests in this file are graded,]
      @RI[whether or not they appear in the event trace. Extra tests in the event trace are ignored.]
  0 manual grading requests read from New-Man.Txt
      @RI[The number of tests for which manual grading was requested, excluding comments.]
-- Test C51004A passed execution
      @RI[The result for an individual test. The passed results can be suppressed with the -quiet option rather than -normal.]
-- Test C52005A passed execution
-- Test C52005B passed execution
-- Test C52005C passed execution
-- Test C52005D passed execution
-- Test C52005E passed execution
-- Test C52005F passed execution
-- Test C52008A passed execution
-- Test C52008B passed execution
-- Test C52009A passed execution
-- Test C52009B passed execution
-- Test C52010A passed execution
-- Test C52011A passed execution
-- Test C52011B passed execution
-- Test C52101A passed execution
-- Test C52102A passed execution
-- Test C52102B passed execution
-- Test C52102C passed execution
-- Test C52102D passed execution
-- Test C52103A passed execution
-- Test C52103B passed execution
-- Test C52103C passed execution
-- Test C52103F passed execution
-- Test C52103G passed execution
-- Test C52103H passed execution
-- Test C52103K passed execution
-- Test C52103L passed execution
-- Test C52103M passed execution
-- Test C52103P passed execution
-- Test C52103Q passed execution
-- Test C52103R passed execution
-- Test C52103X passed execution
-- Test C52104A passed execution
-- Test C52104B passed execution
-- Test C52104C passed execution
-- Test C52104F passed execution
-- Test C52104G passed execution
-- Test C52104H passed execution
-- Test C52104K passed execution
-- Test C52104L passed execution
-- Test C52104M passed execution
-- Test C52104P passed execution
-- Test C52104Q passed execution
-- Test C52104R passed execution
-- Test C52104X passed execution
-- Test C52104Y passed execution
-- Test C53007A passed execution
** Test C540001 has unexpected compile error at line 144 in file C540001.A
   because: Feature not implemented
      @RI[The test uses an Ada 2012 feature not implemented in the test implementation. Thus, this test]
      @RI[(and several others) fail. The message is not part of the grading (and is suppressed if -quiet is]
      @RI[used), but should be helpful in determining the reason a test has failed.]
** Test C540002 has unexpected compile error at line 112 in file C540002.A
   because: Unable to resolve expression
** Test C540003 has unexpected compile error at line 69 in file C540003.A
   because: Feature not implemented
-- Test C54A03A passed execution
-- Test C54A04A passed execution
-- Test C54A07A passed execution
-- Test C54A13A passed execution
-- Test C54A13B passed execution
-- Test C54A13C passed execution
-- Test C54A13D passed execution
-- Test C54A22A passed execution
-- Test C54A23A passed execution
-- Test C54A24A passed execution
-- Test C54A24B passed execution
-- Test C54A42A passed execution
-- Test C54A42B passed execution
-- Test C54A42C passed execution
-- Test C54A42D passed execution
-- Test C54A42E passed execution
-- Test C54A42F passed execution
-- Test C54A42G passed execution
** Test C550001 has unexpected compile error at line 72 in file C550001.A
   because: Feature not implemented
** Test C552001 has unexpected compile error at line 150 in file C552001.A
   because: This must name a type
++ Test C552002 N/A because of expected error at line 59 in file C552002.A
   because: Feature not implemented
      @RI[Note that the text of error messages does not have any effect on test grading. This test is]
      @RI[Not Applicable because the first error occurred on a N/A => Error line, regardless of the message text.]
** Test C552A01 has unexpected compile error at line 73 in file C552A01.A
   because: F552A00_PRIME_NUMBERS; WITHed compilation unit not found
      @RI[Here, a foundation (which is not graded by itself) failed to compile. The failure to compile the]
      @RI[foundation (because of an unimplemented feature) caused this test to fail.]
** Test C552A02 has unexpected compile error at line 94 in file C552A02.A
   because: F552A00_SPARSE_ARRAYS; WITHed compilation unit not found
-- Test C55B03A passed execution
-- Test C55B04A passed execution
-- Test C55B05A passed execution
-- Test C55B06A passed execution
-- Test C55B06B passed execution
-- Test C55B07A passed execution
++ Test C55B07B N/A because of expected error at line 45 in file C55B07B.DEP
   because: Identifier is not defined
      @RI[This test is Not Applicable because the implementation in question does not define a type Short_Integer.]
-- Test C55B10A passed execution
-- Test C55B11A passed execution
-- Test C55B11B passed execution
-- Test C55B15A passed execution
-- Test C55B16A passed execution
-- Test C55C02A passed execution
-- Test C55C02B passed execution
-- Test C56002A passed execution
-- Test C57003A passed execution
-- Test C57004A passed execution
-- Test C57004B passed execution
-- Test C58004C passed execution
-- Test C58004D passed execution
-- Test C58004G passed execution
-- Test C58005A passed execution
-- Test C58005B passed execution
-- Test C58005H passed execution
-- Test C58006A passed execution
-- Test C58006B passed execution
-- Test C59002A passed execution
-- Test C59002B passed execution
-- Test C59002C passed execution


=======================================

Summary for C-Tests for Chapter 5

Result                        Overall  B-Tests  C-Tests  L-Tests  Other Tests

   Total Tests Graded           102        0      102        0        0
     @RI[The total tests graded. The numbers in each column reflect the number of tests of the appropriate type for]
     @RI[the category in question.]

** Failed (Process)               0        0        0        0        0
     @RI[A process failure is some problem with the test processing, such as running a test before it is compiled or]
     @RI[compiling units out of the required order. Usually, correcting the test processing is all that is needed to]
     @RI[eliminate this error.]
** Failed (Compile Missing)       0        0        0        0        0
     @RI[This means that some required unit was not compiled. This is a failure even if the test executed and passed.]
** Failed (Compiler Crash)        0        0        0        0        0
     @RI[This means that some compilation started but did not finish normally. This usually reflects some internal]
     @RI[compiler error.]
** Failed (Error in OK area)      0        0        0        0        0
     @RI[This means that an error was reported in the range of an OK tagged line. These are counted separately as these]
     @RI[typically indicate a significant compiler bug.]
** Failed (Unexpected Error)      7        0        7        0        0
     @RI[This means that an error was reported outside of the range of any test tag.]
** Failed (Missing Error)         0        0        0        0        0
     @RI[This means that no error was reported where one was required (for instance, for an error tag). This is only]
     @RI[likely for a B-Test.]
** Failed (Bind Missing)          0        0        0        0        0
     @RI[This means that no bind operation was found for the test (and no other failure occurred first).]
** Failed (Bind Crash)            0        0        0        0        0
     @RI[This means that bind started but did not finish normally.]
** Failed (Bind Error)            0        0        0        0        0
     @RI[This means that bind reported an error.]
** Failed (Run Missing)           0        0        0        0        0
     @RI[This means that no test execution was found (and no other error was detected first).]
** Failed (Run Crash)             0        0        0        0        0
     @RI[This means that test execution started but did not complete and report a result.]
** Failed (Runtime Message)       0        0        0        0        0
     @RI[This means that test execution explicitly reported Failed.]
++ Not-Applicable (Annex C)       0        0        0        0        0
     @RI[This means that the test did not meet an Annex C Requirement; if Annex C is being tested,]
     @RI[this should be considered a failure.]
++ Not-Applicable (Compile)       2        0        2        0        0
     @RI[This means that the test had an error in an N/A => Error range.]
++ Not-Applicable (Runtime)       0        0        0        0        0
     @RI[This means that the test execution reported that it was Not Applicable.]
!! Special Handling (Runtime)     0        0        0        0        0
     @RI[This means that the test execution reported that it required special handling.]
!! Manual Grading Requested       0        0        0        0        0
     @RI[This means that the test failed for some reason and manual test grading was requested. If the test grades]
     @RI[as Passed or Not Applicable, the manual grading request is ignored and the test will be counted in the]
     @RI[appropriate other category.]
== Passed (Expected Errors)       0        0        0        0        0
     @RI[This means that the test had compile-time errors as expected, and no extra errors.]
== Passed (Bind Error)            0        0        0        0        0
     @RI[This means that the test had bind errors as expected (this is most likely for an L-Test.]
== Passed (Runtime)              93        0       93        0        0
     @RI[This means that the test reported Passed when executed.]
=======================================

Overall result for C-Tests for Chapter 5 is **FAILED**
     @RI[The overall result is Failed unless all tests either pass, are not applicable, or had manual grading]
     @RI[requested. For this implementation, the result of failed is expected. The implementer could have requested]
     @RI[manual grading for the tests that were not expected to work, in order to change this result to Passed.]
@end{Example}

@LabeledSubClause{Compiling the Grading Tool and the Test Summary Tool}

@leading@;The Grading Tool and the Test Summary Tool are provided in six Ada source
files. Some of the files are shared by both tools, so we present them as a
single group. The six files are:

@begin{Indent}
@begin{Hang2List}
GRADE.A@\The main subprogram for the grading tool; contains option processing
and test grading.

GRD_DATA.A@\A package to store grading data; contains the code to read and
display all of the file information (events, test summaries, and manual grading
requests).

SPECIAL.A@\A package containing the special handling for the test summary
program. This package contains the data needed to handle optional units and
tests with severe syntax errors that cannot be processed by normal means.

SUMMARY.A@\The main subprogram for the test summary tool; contains lexical
and syntactic analysis for ACATS test files.

TRACE.A@\A package containing the data types that define an event trace.

TST_SUM.A@\A package containing the data types that define a test summary; it
also contains a routine to write an individual test summary record.
@end{Hang2List}
@end{Indent}

The source of the grading tools is written in Ada, and only uses Ada 95
features other than the following:
@begin{Itemize}
Raise statements with messages;

Ada.Containers.Generic_Array_Sort;

Ada.Calendar.Formatting.
@end{Itemize}

Uses of the first could be replaced by calls to Ada.Exceptions.Raise_Exception;
and the latter could be replaced by similar routines. We did not do this for
readability and to avoid re-inventing the wheel.

The source code should be compilable by any Ada 95 or later compiler that
supports the above three features; it certainly should be compilable by any
Ada 2012 compiler.

As each Ada implementation uses different commands for compiling, we can only
give the general direction that the six source files need to be compiled and
then bound into two execuable programs: Grade (the Grading Tool) and Summary
(the Test Summary Tool). Elsewhere in this documentation we assume that this
has been done.


@LabeledSubClause{Grading Tool Reference}

@leading@;The command line for the Grading Tool is:

@begin{Example}
  Grade <Event_Trace_File_Name> <Summary_of_Tests_File_Name> <Manual_Grading_Request_Name> <Quoted Report Title> [options]
@end{Example}

@begin{Indent}
@begin{Hang2List}
<Event_Trace_File_Name>@\The name of an event trace file (see @RefSecNum{Event Trace Files}).
This can use any file name acceptable to the implementation that compiled the
tool (in particular, full paths may be used on most implementations). The event
trace should contain traces of the processing of at least the tests to be
graded; it is acceptable to include traces for additional tests that are not
being graded.

<Summary_of_Tests_File_Name>@\The name of a test summary file (see
@RefSecNum{Test Summary Files}). This can use any file name acceptable to the
implementation that compiled the tool (in particular, full paths may be used on
most implementations). The test summary file should contain the summaries of
exactly the tests that need to be graded; all of the tests summarized in the
file will be graded regardless of the contents of the event trace. If the
event trace does not include a test that is in the summary file, the test
will be graded "Failed - Compile Missing".

<Manual_Grading_Request_Name>@\The name of a manual grading request file (see
@RefSecNum{Manual Grading Request Files}). This can use any file name acceptable to the
implementation that compiled the tool (in particular, full paths may be used on
most implementations).  This file may include names of tests not included in the
test summary file; such names will have no effect on grading.

<Quoted Report Title>@\A double quoted string containing the name of the test
report. This is primarily intended so that similar-looking reports can be
differentiated.

[options]@\Zero or more optional setting flags for the Grading Tool. These
are case-insensitive, and can be:
@end{Hang2List}
@end{Indent}

@begin{Indent}
@begin{Indent}
@begin{Hang2List}
-Specs_Optional@\Compiling of specifications is optional. Use for source-based
compilers (such as the commonly used GNAT compiler) that don't compile
specifications in normal operation. Compilation of bodies, instances, and so on
are checked.

-Check_All_Compiles@\All compilations are checked and must be present (unless
processing the unit is marked as optional). This is the default.

-No_Compile_Checks@\No compilation checks are made. This option is not allowed
for formal conformity assessments.

-Use_Time_Stamps@\Check event trace time stamp information as part of checking,
specifically, enforce that all units of a test are compiled and run in an
appropriate order and reasonably close in time. This is the default.

-No_Time_Stamps@\Do not make any check of event trace time stamps. Use this
option only if there is no meaningful timestamps in the event trace. This
option is not allowed for formal conformity assessments.

-Use_Positions@\Use position information when checking whether errors are
appropriately detected. This is the default.

-No_Positions@\Use only line information when checking whether errors are
appropriately detected. Use this option if the event trace doesn't have position
information. It is acceptable to use this option for formal conformity
assessments.

-Quiet@\Produce minimal information: a list of failed tests and the summary
report.

-Verbose@\Produce information about every test processed (including passed
tests), along with the summary report. In this mode, the grading tool also
produces a warning for multiple messages for one error tag.

-Normal@\Produce details about each failed test, along with the
summary report. This is the default.
@end{Hang2List}
@end{Indent}
@end{Indent}

Only one of the options -Specs_Optional, -Check_All_Compiles,
or -No_Compile_Checks can be given. Only one of the options -Use_Positions or
-No_Positions can be given. Only one of the options -Quiet, -Verbose, or
-Normal can be given. Only one of the options -Use_Time_Stamps or
-No_Time_Stamps can be given.

An annotated example using the grading tool is given in subclause
@RefSec{Annotated Grading Tool Example}. That example explains the output
of the grading tool.


@LabeledSubClause{Test Summary Tool Reference}

@leading@;The command line for the Test Summary Tool is:

@begin{Example}
  Summary <ACATS_Test_File_Name> <Summary_of_Tests_File_Name>
@end{Example}

@begin{Indent}
@begin{Description}
<ACATS_Test_File_Name>@\The name of an ACATS test source file. This can use
any file name acceptable to the implementation that compiled the tool (in
particular, full paths may be used on most implementations). The Summary
tool assumes that the simple file name of the test follows the naming
conventions as described in @RefSecNum{Naming Convention}; either the modern
or legacy naming conventions are acceptable.

@\The summary tool can be used on split tests and other Ada code not directly
part of the ACATS, so long as the ACATS naming conventions are followed (see
above), and the files contain no optional units (as the tool has no way to
discover optional units). This allows the Grading Tool to be used on tests
being developed for submission to the ACATS (see
@RefSecNum{Guidelines for Test Development}). We also expect that the tool
could be used on older ACATS versions, allowing the Grading Tool to be used
with those versions.

<Summary_of_Tests_File_Name>@\The name of the Test Summary File. If this file
exists, the new test summary records will be appended to it. Otherwise, the file
will be created.
@end{Description}
@end{Indent}

@leading@;The test summary tool needs to be run on each individual source file.
Typically, it makes sense to combine all of the tests of a single ACATS
directory into a single test summary file. This can easily be accomplished
on Microsoft Windows with a batch file containing the following:

@begin{Example}
   Rem Chapter 5 C-Tests
   Del C5-Sum.csv
   for %%i in (\My_ACATS\C5\*.A??) do Summary %%i C5-Sum.csv
   for %%i in (\My_ACATS\C5\*.D??) do Summary %%i C5-Sum.csv
@end{Example}

We first delete any existing summary file (so we don't accidentally double the
contents), then run the summary tool on all of the ACATS source code found in
the directory \My_ACATS\C5. (The second loop is needed in case there are any
legacy .DEP files; all of the other extensions start with 'A'.)

A similar technique can be used on other host operating systems.


@LabeledClause{Event Trace Files}

An event trace file includes each (interesting) event that occurs during the
compilation, binding/linking, and execution of one or more ACATS tests.
@Defn{event trace}@Defn2{Term=[file],Sec=[event trace]}

An event trace file provides a way to present the implementation-specific
format of events to the Grading Tool in a common format. In order for an
ACATS user to use the Grading Tool, they will need to provide a method to
get an event trace file from the implementation's processing of ACATS tests.
There are a number of ways to accomplish that; several are outlined in
following subclauses. No matter what method is selected, it should be possible
to use the same options/tools for future ACATS tests. (Developing a method to
create event trace files should be a one-time cost).

The events of an event trace file are intended to be abstract representations
of the processes of an implementation. It should be possible to map the
processes of any Ada implementation into an event trace file.

The event trace file was selected as the method of abstracting implementation
processing in order to avoid the ACATS Grading Tool from providing an
disincentive to innovation in error handling by Ada implementations. The files
are not intended to be useful (directly) to a human user, so their details
should have little effect on the human error handling interface for an
implementation. Moreover, while the event trace files provide values for
error messages and positioning, neither of these is required for formal
grading (they're provided to making easier for an ACATS user to figure out
why a test is reported as failing). As noted in
@RefSecNum{Expected Results for Class B Tests}, the actual text of an error
message is not used to determine pass or fail for grading purposing; only
the specified location of the error is used.


@LabeledSubClause{Event Trace File Reference}

An event trace file is a CSV (Comma Separated Value) file of event records.
See @RefSec{CSV File Reference} for the general rules for constructing
a CSV file.

@leading@;An event trace file record contains the following
comma-separated fields on a single line:

@begin{Indent}
@begin{Description}
Event@\One of UNKN (Unknown), CSTART (Compilation_Start),
CEND (Compilation_End), CERR (Compile_Error),
CWARN (Compile_Warning),
BSTART (Binder_Start), BEND (Binder_End),
BERR (Binder_Error), BWARN (Binder_Warning),
EXSTART (Execution_Start),
EXEND (Execution_End), EXFAIL (Execution_Failure),
EXNA (Execution_Not_Applicable), EXSACT (Execution_Special_Action),
EVENT (see below). These values are case-insensitive.
"EVENT" is treated as specifying a comment; it usually appears in column
headers.

Timestamp@\The timestamp, double quoted, in the format specified by
Ada.Calendar.Formatting.Image.

Name@\The double quoted name of the source file, main subprogram, or test.
For Compilation events, this is the simple name of the source file.
For Binder events, this is the name of the main subprogram.
For Execution events, this is the name of the test as passed to Report.Test.

Line@\For Compilation_Start, the first line of the current compilation unit.
(Usually 1, unless there are multiple compilation units in a single file.)
For Compile_Error or Compile_Warning, the line number where that error or
warning is reported. (This is critical to the correct operation of the
grading tool.) Otherwise, it is not used and can be omitted other than the
comma separator.

Position@\For Compile_Error or Compile_Warning, the position within the
line that on which the error is reported. An implementation does not have
to provide a meaningful Position for errors (use the -No_Position option
on the Grading Tool - see @RefSecNum{Grading Tool Reference} if this is true
for your implementation). Otherwise, it is not used and can be omitted other
than the comma separator.

Message@\The double quoted message (make sure to replace any double quotes,
as they are not allowed in double quoted strings). For Compile and Binder
Errors and Warnings, this is the message emitted by the appropriate tool.
For Execution events, this is the message passed to Report. For End events,
this is an implementation-defined result of the operation (OK, with Errors,
Passed, Failed, and so on). If there is no appropriate message, nothing need
be written for this field (as it is last, there is no trailing comma).
@end{Description}
@end{Indent}

The order of the event records in the event trace file is unimportant to the
Grading Tool; it will sort the records appropiately before grading. (The
order of the timestamps in the records does matter; running before compiling
and the like indicate a test processing problem.)

The record types used by the Ada implementation of this file can be found
in Trace.A.

There is an example of writing an event trace file in the file Report.A, in
procedure Put_Event_Trace. Most of the code involves limiting the length of, and
removing any double quotes from, the (quoted) message string. Note that
Put_Event_Trace writes column headers into a new file, so that headers exist
if the file is loaded into a spreadsheet or database. This is recommended for
any tool that creates an event trace.

The Grading Tool treats any record that starts with EVENT as a comment; this
skips any headers and allows event trace files to be concatenated together
for combined processing.

@thinline

For the purposes of an event trace, a "compile" is the part of an Ada
implementation that processes Ada source code and provides diagnostics to
diagnose Ada errors (specifically syntax errors, resolution errors, and
violations of Legality Rules). This does not need to be a single phase or
program; it could be several cooperating programs. Moreover, the "compile"
events only need to include parts of the implementation that are involved
in diagnosing errors. Code generation and optimization are part of a
conventional compiler that can be omitted from the "compile" as defined
for an event trace. (A failure in one of these phases not included in
"compile" would probably cause a test to be graded as crashed or with a
failed bind.)

Similarly, a "bind" is the part of an Ada implementation that creates an
Ada partition and enforces post-compilation rules not enforced by the
compile stage. (The compiler is allowed to enforce post-compilation rules
y the Ada Standard.)
This also does need not be a single program, and it only needs to include
phases that enforce Ada errors. For instance, a system linker need not be
included in the event trace for the "bind" operation.

@thinline

Not all of the information in an event trace is currently used by the Grading
Tool. We included additional information (like warnings) in part because future
versions of the ACATS tools might need them and changing the format in the
future could be very disruptive. In addition, it's possible that this
compiler-independent event format could be useful to other future ACATS
tools or even third-party tools having nothing to do with the ACATS. As such,
we included all of the information that seemed potentially useful.

Here is part of an event trace for Chapter 5 C-Tests:

@begin{Example}
Event,"Timestamp","Name","Line","Position","Message"
CSTART,"2016-05-16 23:16:41.05","C51004A.ADA", 1, 1,""
CEND,"2016-05-16 23:16:41.13","C51004A.ADA",,,"OK"
BSTART,"2016-05-16 23:16:41.14","C51004A",,,""
BEND,"2016-05-16 23:16:41.27","C51004A",,,"OK"
EXSTART,"2016-05-16 23:16:41.33","C51004A",,,"CHECK THAT LABELS, LOOP IDENTIFIERS, AND BLOCK"
EXEND,"2016-05-16 23:16:41.33","C51004A",,,"Passed"
CSTART,"2016-05-16 23:16:41.38","C52005A.ADA", 1, 1,""
CEND,"2016-05-16 23:16:41.44","C52005A.ADA",,,"OK"
BSTART,"2016-05-16 23:16:41.45","C52005A",,,""
BEND,"2016-05-16 23:16:41.56","C52005A",,,"OK"
EXSTART,"2016-05-16 23:16:41.64","C52005A",,,"CHECK THAT CONSTRAINT_ERROR EXCEPTION IS RAISED"
EXEND,"2016-05-16 23:16:41.64","C52005A",,,"Passed"
CSTART,"2016-05-16 23:16:41.70","C52005B.ADA", 1, 1,""
CEND,"2016-05-16 23:16:41.77","C52005B.ADA",,,"OK"
BSTART,"2016-05-16 23:16:41.78","C52005B",,,""
BEND,"2016-05-16 23:16:41.89","C52005B",,,"OK"
EXSTART,"2016-05-16 23:16:41.95","C52005B",,,"CHECK THAT CONSTRAINT_ERROR EXCEPTION IS RAISED"
EXEND,"2016-05-16 23:16:41.95","C52005B",,,"Passed"
CSTART,"2016-05-16 23:17:06.36","C55B07B.DEP", 1, 1,""
CERR,"2016-05-16 23:17:06.41","C55B07B.DEP", 45, 14,"Identifier is not defined"
CERR,"2016-05-16 23:17:06.41","C55B07B.DEP", 47, 39,"Identifier is not defined"
CERR,"2016-05-16 23:17:06.41","C55B07B.DEP", 51, 27,"Identifier is not defined"
CERR,"2016-05-16 23:17:06.41","C55B07B.DEP", 52, 27,"Identifier is not defined"
CERR,"2016-05-16 23:17:06.41","C55B07B.DEP", 57, 32,"Identifier is not defined"
CERR,"2016-05-16 23:17:06.41","C55B07B.DEP", 58, 32,"Identifier is not defined"
CERR,"2016-05-16 23:17:06.41","C55B07B.DEP", 58, 52,"Identifier is not defined"
CERR,"2016-05-16 23:17:06.41","C55B07B.DEP", 83, 21,"Identifier is not defined"
CERR,"2016-05-16 23:17:06.41","C55B07B.DEP", 83, 21,"Only discrete types may b
CERR,"2016-05-16 23:17:06.41","C55B07B.DEP", 99, 18,"Identifier is not defined"
CERR,"2016-05-16 23:17:06.41","C55B07B.DEP", 107, 18,"Identifier is not defined"
CERR,"2016-05-16 23:17:06.42","C55B07B.DEP", 109, 26,"Identifier is not defined"
CEND,"2016-05-16 23:17:06.42","C55B07B.DEP",,,"Aborted by semantic errors"
BSTART,"2016-05-16 23:17:06.44","C55B07B",,,""
BERR,"2016-05-16 23:17:06.44","C55B07B",,,"Main program file not found"
BEND,"2016-05-16 23:17:06.44","C55B07B",,,"Aborted by errors"

@end{Example}


@LabeledSubClause{Creating an Event Trace directly by the implementation}

Creating an event trace for a particular implementation is a problem for
the ACATS user and their implementer (the ACATS user need not be an
implementer, of course). We can only give general guidance on approaching
this problem.

One approach is the for implementer to directly add event trace creation
to their tools. In this approach, options would be added to the compiler
and binder that would write events to an event trace file. The ACATS user
would set the Generate_Event_Trace_File constant in Report.A to True.
The user would use this new compiler and binder option to write events
to ETrace.csv, as will the Report package. At strategic times, the ETrace.csv
file could be copied to a new name in order to keep its size manageable.

This approach is likely to be the easiest, as all of the information needed
to create an event is likely to be available in a compiler error handler.
Moreover, writing an event is an easy operation in Ada (see
Report.Put_Event_Trace) and likely most other languages. And by directly
including this capability in the compiler, it is available to all users
and likely needs little modification even if the compiler error handling is
changed significantly. The effort needed is small and one-time.

@leading@;However, this approach may not work in some cases:

@begin{Enumerate}
The ACATS user is not the implementer, so modifying the compiler is not
possible;

An important part of the compiler or binder is a third party tool, so
adding an option is not feasible;

The ACATS testing target does not have a file system, so having report write
an event trace is not possible.
@end{Enumerate}

In each of these cases, the technique outlined in the next section is
preferable.

Note that in case 2, not all parts of an Ada compilation system need to be
able to generate an event trace. Any part that can fail only in the case of
a bug elsewhere in the system (that is, does not diagnose errors in the Ada
code) can be ignored for the purposes of an event trace. For instance, the
use of a system-provided linker after the completion of the bind step need
not be reflected in an event trace, as it has no role in the enforcement of
Ada Post-Compilation Rules. For more on this topic,
see @RefSecNum{Event Trace File Reference}.


@LabeledSubClause{Creating an Event Trace from Listings}

If it is not possible or desirable to modify the implementation to create an
event trace, an event trace can be created from a listing of test processing.
Such a listing would capture all of the compile, bind, and run steps for
a set of tests. The Generate_Event_Trace_File constant in Report.A would
remain False.

A tool then would have to be constructed to read those listings and convert
them to an event trace. This would be specific to a particular implementation,
and the tool would potentially break due to future changes in an implementation.

The tool would probably have to gather information from several parts of a
listing. For instance, most programs don't display a time stamp when they run,
so it would be necessary to use a system tool or tiny Ada program to add those
to strategic points in a listing. (Using a single timestamp for an entire
compilation of a compilation unit, including any reported errors, is sufficient
to meet the requirements of the Grading Tool.) Note that the initial message
from Report includes a timestamp, so there is no need to add one for the
execution of tests.

The listing converter tool would have to be able to extract events from the
compiler, the binder, and running of ACATS tests. Each of these likely has
a different format, so the tool would need many rules to extract all of the
events.

While a listing converted tool is potentially more fragile (due to more
dependence on the exact layout of messages for an implementation), it would have
the possibility of detecting test failures after the reporting of Passed (due to
a finalization or tasking failure) - such "aberrant behavior" (as defined in
@RefSecNum{Expected Results for Executable Tests}) cannot be detected when
Report.A itself writes the event trace (as the failure happens outside of the
control of Report).


@LabeledClause{Test Summary Files}

A test summary file is a machine-readable distillation of one or more
ACATS tests. The summary includes the information about the test that is
needed by the grading tool, in a much more convenient format than the original
source files. Test summaries are created with the Summary
tool (see @RefSecNum{Workflow using the Grading Tool} and
@RefSecNum{Test Summary Tool Reference}).
@Defn{test summary}@Defn2{Term=[file],Sec=[test summary]}

@LabeledSubClause{Test Summary File Reference}

A test summary file is a CSV (Comma Separated Value) file of test summary
records. See @RefSec{CSV File Reference} for the general rules for constructing
a CSV file.

@leading@;A test summary file record contains the following
comma-separated fields on a single line:

@begin{Indent}
@begin{Description}
Kind@\KIND (see below), ERROR (Error), PERROR (Possible_Error),
OERROR (Optional_Error), NAERR (NA_Error), ACRQMT (Annex_C_Requirement),
OK (OK),
UPACKSPEC (Package_Specification), UFUNCSPEC (Function_Specification),
UPROCSPEC (Procedure_Specification), UGENPACK (Generic_Package),
UGENFUNC (Generic_Function), UGENPROC (Generic_Procedure),
UPACKBODY (Package_Body), UFUNCBODY (Function_Body),
UPROCBODY (Procedure_Body),
UPACKINST (Package_Instantiation), UFUNCINST (Function_Instantiation),
UPROCINST (Procedure_Instantiation), UPACKREN (Package_Renaming),
UFUNCREN (Function_Renaming), UPROCREN (Procedure_Renaming),
UGPACKREN (Generic_Package_Renaming),
UGFUNCREN (Generic_Function_Renaming),
UGPROCREN (Generic_Procedure_Renaming),
PACKSUB (Package_Subunit), PROCSUB (Procedure_Subunit),
FUNCSUB (Function_Subunit), TASKSUB (Task_Subunit),
PROTSUB (Protected_Subunit), PRAGMA (Configuration_Pragma)
These values are case-insensitive.
"KIND" is treated as specifying a comment; it usually appears in column
headers. All of the U kinds are varieties of compilation units; as are the
subunits and PRAGMA.

Source_Name@\The quoted simple name of the source file.

Start_Line@\For a compilation unit, the starting line of the compilation unit.
This may include leading whitespace and comments, but must be no later than
the first significant text of the unit, and it may not include part of some
preceding unit. For other kinds of records, this
represents the first line of the range that indicates a trigger for this
record. (For instance, for an ERROR record, this represents the first line
in the range in which an error must be reported.)

Start_Position@\For a compilation unit, the starting position within the
Start_Line for the compilation unit. This may include whitespace or comments
(always setting this to 1 would be allowed), but must be no later than the
first significant character of the line. (The ACATS never puts multiple
compilation units on the same line.)

@\For other kinds of records, the starting position within Start_Line for the
trigger for this record. Whether or not positions are used for grading is
selectable by an option to the Grading Tool - see
@RefSecNum{Grading Tool Reference}. (Use of positions is not required.)

End_Line@\For a compilation unit, the end line of the compilation unit.
This may include trailing whitespace and comments, but must be no earlier than
the last significant text of the unit, and it may not include part of some
succeeding unit. For other kinds of records, this
represents the last line of the range that indicates a trigger for this
record.

End_Position@\For a compilation unit, the end position within the
Start_Line for the compilation unit. This may include whitespace or comments,
but must be no earlier than the last significant character of the line.
For other kinds of records, this is the ending position within End_Line for
the trigger for this record.

Name_Label@\For a compilation unit, the quoted compilation unit name. For
a Possible_Error, the quoted set name. For other kinds of records, this
value can be omitted other than the separating comma.

Flag@\Only used for compilation units; for other kinds of records it can be
completely omitted (as it is last, there is no trailing comma). For
a compilation unit, it can be omitted (meaning a required, non-main subprogram
unit), MAIN (a required, main subprogram unit), OPT (an optional, non-main
subprogram unit), or OPTMAIN (an optional, main subprogram unit). An optional
unit is one that is not required to be processed (this is indicated by comments
within the test). As with kinds, this value is case-insensitive.

@end{Description}
@end{Indent}

These fields do not quite match the components used in the associated Ada
record; a few were combined to simplify the file format (at a cost of some
additional complexity in reading the records).

The order of records within a test summary file is unimportant to the
Grading Tool; it will sort the records appropiately before grading.

The compilation unit start and end information is primarily used to determine
to which compilation unit (and thus, which compilation) a particular ERROR or
other kind of record belongs. The compilation unit records themselves are
secondarily used to ensure that all required units have been processed.

The ranges for ERROR, OK, and other non-compilation unit record kinds are
used to determine if an appropriate error is (or is not) present for that
record. The limits can be determined by a range indicator (see
@RefSecNum{Range Indicators}) if present. Otherwise, the limits will
contain at least all of the significant text of the line that contains the tag.
(In some cases, these default limits may be expanded.)

The record types used by the Ada implementation of this file can be found
in Tst_Sum.A, along with a routine (Write_Summary_Record) used to write
a summary record to a test summary file. Note that Write_Summary_Record writes
column headers into a new file, so that headers exist if the file is loaded into
a spreadsheet or database. It is recommended to use Tst_Sum.A as part of any
tool that creates a test summary file (there being little reason to reinvent
this wheel).

Code to read a test summary file into an array can be found in Grd_Data.A.

Here is part of a test summary file for Chapter 5 C-Tests:

@begin{Example}
Kind,Source_Name,Start Line,Start Pos,End Line,End Pos,Name_Label,Flag
UPROCBODY,C51004A.ADA,38,1,261,12,C51004A,MAIN
UPROCBODY,C52005A.ADA,33,1,177,12,C52005A,MAIN
UPROCBODY,C52005B.ADA,33,1,115,12,C52005B,MAIN
UPROCBODY,C52005C.ADA,33,1,79,12,C52005C,MAIN
UPROCBODY,C52005D.ADA,32,1,182,12,C52005D,MAIN
UPROCBODY,C52005E.ADA,32,1,129,12,C52005E,MAIN
UPROCBODY,C52005F.ADA,32,1,86,12,C52005F,MAIN
UPACKSPEC,C540001.A,50,1,53,14,C540001_0,
UPACKSPEC,C540001.A,57,1,73,14,C540001_1,
UPACKBODY,C540001.A,77,1,105,14,C540001_1,
UGENPACK,C540001.A,109,1,121,14,C540001_2,
UPACKBODY,C540001.A,125,1,137,14,C540001_2,
UGENFUNC,C540001.A,141,1,149,69,C540001_3,
UFUNCBODY,C540001.A,153,1,157,14,C540001_3,
UGENPACK,C540001.A,161,1,172,14,C540001_4,
UPACKBODY,C540001.A,176,1,187,14,C540001_4,
UPACKSPEC,C540001.A,191,1,205,14,C540001_5,
UPACKBODY,C540001.A,209,1,222,14,C540001_5,
UPROCBODY,C540001.A,226,1,410,12,C540001,MAIN
UPROCBODY,C540002.A,66,1,235,12,C540002,MAIN
UPACKSPEC,C540003.A,62,1,81,14,C540003_0,
UPROCBODY,C540003.A,83,1,240,12,C540003,MAIN
NAERR,C55B07A.DEP,45,6,45,26,,
UPROCBODY,C55B07A.DEP,41,1,126,14,C55B07A,MAIN
NAERR,C55B07B.DEP,45,6,45,27,,
UPROCBODY,C55B07B.DEP,41,1,126,14,C55B07B,MAIN
@end{Example}


@LabeledSubClause{Range Indicators}

Optional range indicators (sometimes known as location indicators)
can appear after the various markers in an ACATS
test source file. These describe the exact range of the reported location
of an expected error. Range indicators are usually used to expand the range
of an error beyond the same line as the ERROR: or other marker.
@Defn{range indicator}@Defn{location indicator}

The format of a range indicator is:
@begin{Example}
   {[sl:]sp[;[el:]ep]}
@end{Example}

@Leading@;In the above, '{' and '}' are literal, while '[' and ']' indicate
optionality. Each of the four values is relative, so it is one or two digits,
with an optional minus sign for @Exam{el}. Omitted values are assumed to be
zero. Specifically:
@begin{Indent}
@begin{Description}
@Exam{sl}@\Start Line @en offset @i<before> the current line for the
start of the error range.

@Exam{sp}@\Start Position @en position offset in the line indicated by
@Exam{sl}, relative to the start of the line.

@Exam{el}@\End_Line @en offset @i<before> the current line for the end of
the error range. Can be negative if the end of the error
range follows the error tag. But almost always should be zero.

@Exam{ep}@\End Position @en position offset from the last significant
character in the line indication by End_Line. (The last
significant character is the last non-white-space character
not including any comment.)
@end{Description}
@end{Indent}

This compact representation was chosen because of the limited space given
the ACATS line length limit (see @RefSecNum{General Standards}) and a desire
to avoid unnecessarily cluttering tests with extraneous information.


@LabeledClause{Manual Grading Request Files}

The ACATS Grading Tool takes a file containing a list of tests that may
require manual grading.@Defn{manual grading request}

The file is just a list of ACATS test names (7 characters each), one per line.
Ada comments (anywhere on a line) and blank lines are also allowed.

If the Grading Tool encounters a failure for one of the tests in the manual
grading file for that grading run, it will report that the test needs manual
grading rather than that it failed. The manual grading list has no effect
on tests that are graded as Passed or Not Applicable.

An ACATS user can add tests to this file as needed. Some of the reasons that
one might want to do this are discussed in
@RefSecNum{Workflow using the Grading Tool}.

@Red{The ACAA would like to know which tests require manual grading for your
implementation. Please send manual grading files annotated with comments as
to why they require manual grading to the ACAA Technical Agent,
@urllink{URL=[mailto:agent@ada-auth.org],Text=[agent@ada-auth.org]}. The
ACAA will use this information to determine which,
if any tests, require repair to better provide error range information
(potentially including alternative error locations.)}

For formal testing, the ACAL should be aware of all tests included in any
manual grading file used.


@LabeledClause{CSV File Reference}

Several of the files used by the Grading Tool are .CSV (Comma Separated Value)
files. This format was chosen as it:

@begin{itemize}
is a pure text format that is easy to write in Ada. Records can be appended to
an existing file by opening the file with Text_IO in Append_Mode, Put_Line the
line of values for the record, then closing the file. No end markers need
to be moved.

is reasonably easy to read in Ada using Text_IO. Get_Line and some string
operations are sufficient.

can be read by most spreadsheet and database programs. Thus, we don't need
to provide tools to inspect, check, or edit the contents of these files.
@end{itemize}

A CSV consists of list of records, one record per line. Each line is a set
of values, separated by commas. Each line should have the same number of
values, so when loaded in a spreadsheet it becomes a series of columns
with each Ada component associated with a column.

There are a number of varieties of CSV files, so we have adopted one of the
simplest in order to have the widest applicability.

There are only a few restrictions on the values in our CSV files. Unquoted
values must not contain commas, spaces, tabs, or semicolons. A value can be
quoted with double quotes, in which case commas, spaces, tabs, and semicolons
are allowed, but double quotes are @b{not} allowed in quoted strings. Line
breaks are not allowed in or outside of values of a single record (they
delimit the records).

We require that values are limited to 150 characters (primarily to prevent
excessively long messages from making reports hard to read).

There is an example of writing a CSV file (an event trace - see
@RefSecNum{Event Trace Files}) in the file Report.A, in procedure
Put_Event_Trace. Most of the code involves limiting the length of, and removing
any double quotes from, the (quoted) message string.
