10946
Comment:
|
15188
Correction
|
Deletions are marked like this. | Additions are marked like this. |
Line 1: | Line 1: |
## page was renamed from AplAplTemplate | |
Line 22: | Line 21: |
1. All test functions inside that namespace are expected to start their names with the string `Test_` followed by digits. | 1. All test functions inside that namespace are expected to start their names with the string `Test_` followed by digits. '''Update''': since version 1.1 test functions can be grouped. For example, in order to group all functions testing a method `foo` the test functions are named `Test_foo_001` (or `Test_foo_01` or `Test_foo_00001`), `Test_foo_002`, `Test_foo_003` and so on. |
Line 29: | Line 28: |
1. `stopFlag` (0=use error trapping) | 1. `debugFlag` (0=use error trapping) |
Line 40: | Line 39: |
=== INI file s === | === INI files === |
Line 59: | Line 58: |
Use this function to create stuff that's needed by '''all''' test cases, or tell the user something important - if the batch flag is false. | Use this function to create stuff that's needed by '''all''' test cases, or tell the user something important - of course only if the batch flag is false. |
Line 75: | Line 74: |
== Best Pratices == * Try to keep your test cases simple and test just one thing at a time, if possible a method. * Create everything you need and tidy up afterwards. Or more precisely, tidy up (left overs!), create, test, tidy up again. |
== Helpers == Over time it emerged that certain helpers are very useful in the namespace hosting the test cases. Therefore with version 1.4.0 a method `EstablishHelpersIn` was introduced that takes a reference as the right argument. That ref should point to the namespace hosting the test cases. If you are executing `#.Testers.EstablishHelpersIn` from within that namespace you can specify an empty vector as the right argument: it then defaults to `⎕IO⊃⎕RSI`. One of the helpers is the method `ListHelpers` which returns a table with all names in the first column and the first line of the function (which is expected to be a comment telling what the function actually does) in the second column. Some of the Helpers are useful for flow control, most of the others are just shortcuts that call methods in `Testers`. === RunThese === A particularly helpful method while developing/enhancing stuff is `RunThese`. The function allows you to run just selected test functions rather than a whole test suite but at the same time process any INI files and execute `Initial` and `Cleanup` if they exist. `RunThese` offers the following options: {{{ RunThese 1 2 ⍝ Execute test cases 1 & 2 that do not belong to any group. RunThese ¯1 ¯2 ⍝ Same but stop just before a test case is actually executed. RunThese 'Group1' ⍝ Execute all test cases belonging to group 1. RunThese 'Group1' (2 3) ⍝ Execute test cases 2 &3 of group 1. RunThese 'Group1' 2 3 ⍝ Same as before. RunThese 'Test_Group1_001' ⍝ Executes just this test case. RunThese 0 ⍝ Run all test cases but stop just before the execution. }}} == Best Practices == * Try to keep your test cases simple and test just one thing at a time, for example just one method at a time. * Create everything you need and tidy up afterwards. Or more precisely, tidy up (left overs!), prepare, test, tidy up again. |
Line 85: | Line 107: |
Note that for convenience there are a couple of Helpers which are shortcuts for some of the methods listed here - see there. |
|
Line 106: | Line 130: |
Sets a global variable `TestCasesExecutedAt` in the namespace hosting the test cases. |
|
Line 117: | Line 143: |
Sets a global variable `TestCasesExecutedAt` in the namespace hosting the test cases. |
|
Line 124: | Line 152: |
This will work only if you use a particualar strategy when checking results in a test case | This will work only if you use a particular strategy when checking results in a test case. Sets a global variable `TestCasesExecutedAt` in the namespace hosting the test cases. |
Line 136: | Line 167: |
Sets a global variable `TestCasesExecutedAt` in the namespace hosting the test cases. |
|
Line 150: | Line 183: |
#.Tester.GetAllTestFns #.TestCases Test_001 Test_002 Test_003 Test_004 Test_005 ... ⊃#.Tester.ListTestCases #.TestCases |
)cs #.TestCases ⊣#.Tester.EstablishHelpersIn ⎕THIS Run Run all test cases RunDebug Run all test cases with DEBUG flag on RunThese Run just the specified tests RunBatchTests Run all batch tests E Get all functions starting their names with "Test_" into the editor. L Prints a list with all test cases and the first comment line to the session FailsIf Usage : →FailsIf x, where x is a boolean scalar PassesIf Usage : →PassesIf x, where x is a boolean scalar L |
Line 158: | Line 199: |
#.Tester.Run #.TestCases | Run |
Line 194: | Line 235: |
⎕←↑#.Tester.RunBatchTests #.TestCases | RunBatchTests |
Line 197: | Line 238: |
2 10 #.Tester.RunTheseIn #.TestCases ⍝ Just test cases 10 and 11 | RunThese 2 10 ⍝ Just test cases 2 and 10 |
Line 206: | Line 247: |
'MyGroup' (2 199 ) #.Tester.RunTheseIn #.TestCases ⍝ Test cases 2 and 199 of "MyGroup" | RunThese ¯2 ⍝ A negative number... --- Tests started at 2013-01-08 14:35:28 on #.TestCases -------------------------------------------------------- Run__[76] ⍝ ... stop just before the test cases gets executed →⎕LC ⍝ Test_002 (1 of 1) : Exercise "RmDir" ----------------------------------------------------------------------------------------------------------------- 1 test case executed 0 test cases failed 0 test cases broken ⍝ Use `RunThese 0` for executing all test cases with stops. RunThese 'MyGroup' (2 199 ) ⍝ Test cases 2 and 199 of "MyGroup" |
Line 215: | Line 269: |
('MyGroup' ⍬) #.Tester.RunTheseIn #.TestCases ⍝ All test cases of group "MyGroup" | ⍝ Since version 1.7.0 you can specify the group name with or without a prefix `Test_`: RunThese 'Test_MyGroup' (2 199 ) ⍝ Test cases 2 and 199 of "MyGroup" --- Tests started at 2011-10-18 07:15:25 on #.TestCases ------------------------------------- Test_MyGroup_002: Exercise MyGroup -2- Test_MyGroup_199: Exercise MyGroup -199- ---------------------------------------------------------------------------------------------- 2 test cases executed 0 test cases failed 0 test cases broken RunThese ('MyGroup' ⍬) ⍝ All test cases of group "MyGroup" |
Line 225: | Line 290: |
4 #.Tester.RunDebug #.TestCases ⍝ Stop at test case number 4 | RunDebug 1 ⍝ Stop just before any test case gets executed --- Tests started at 2012-12-21 09:05:59 on #.TestCases -------------------------------------- Run__[71] ⍝ Now you can trace into the test case if you want. RunDebug ¯4 ⍝ Stop at test case number 4 |
Line 227: | Line 298: |
Test_001: Exercise "MkDir" Test_002: Exercise "RmDir" Test_003: Exercise "Dir" Run__[61] ⍝ Now you can trace into the forth test case. |
Run__[71] ⍝ Now you can trace into test case number 4. |
Tester
Contents
Tester is part of the CategoryAplTree project.
Overview
The purpose of this class is to provide a framework for testing all the projects of the APLTree project. Only with such a framework is it possible to make changes to the test suite without too much hassle. You might find the framework flexible enough to suit your own needs when it comes to implementing tests.
Details
Terminology
Note that test cases causing a crash are considered "broken". Test cases that do not return the expected result are considered "failing".
Assumptions and preconditions
The #.Tester class assumes that all your tests are hosted by an ordinary namespace. All methods running test cases take a ref to that namespace as an argument.
All test functions inside that namespace are expected to start their names with the string Test_ followed by digits. Update: since version 1.1 test functions can be grouped. For example, in order to group all functions testing a method foo the test functions are named Test_foo_001 (or Test_foo_01 or Test_foo_00001), Test_foo_002, Test_foo_003 and so on.
- The first line after the header must carry a comment telling what is actually tested. Keep in mind that later you might be in need for finding out which test cases are testing a certain method!
- It is assumed that you have three different scenarios when you want to run test cases with a different behaviour:
Run with error trapping and return a log (vector of text vectors) reporting details. Run this before checking your code into a Code Management System like SubVersion. See Tester.Run for details.
Run without error trapping. Use this for investigating why test cases crash. See Tester.RunDebug for details.
Run "batch mode". It is assumed that those test cases that need a human being in front of the monitor are not executed when running in batch mode. See Tester.RunBatchTests for details. (Of course it is best not to rely on a human being but this cannot always be avoided)
- Every test function must accept a right argument which is a two-item vector of Booleans:
debugFlag (0=use error trapping)
batchFlag (0=does not run in batch mode)
- Every test function must return a scalar integer with one of:
0: the test case is okay.
1: the test case failed.
-1: means the test case did not execute itself because it is not designed to run in batch mode.
Work flow
No matter which of the three main methods (Run, RunBatchTests, RunDebug) you are going to call, the workflow is always the same:
INI files
First of all the Run* method checks whether there is a file testcases_{computername}.ini. If this is the case that INI file is processed. Use this to specify important computer-dependent variables.
It then checks whether there is a file testcases.ini. If this is the case that INI file is processed, too. Use this to specify general stuff that does not depend on a certain computer/environment.
Note that if one or both INI files exist there will be a flat namespace {yourTestNamespace}.INI, meaning that sections in the INI file are ignored. An example: if your test functions are hosted by a namespace "foo" and your INI file specifies a variable hello as holding "world" then:
'world'≡#.foo.INI.hello 1
Initialisation
Now the Run* method checks whether there is a function "Initial" in the hosting namespace. If that is the case then the function is executed.
Note that the function must be either niladic or monadic and must not return a result. If the function is monadic then a Boolean is passed via the right argument telling the function that the test suite is going to run in batch mode if true.
Use this function to create stuff that's needed by all test cases, or tell the user something important - of course only if the batch flag is false.
It might be a good idea to call a function tidying up in line 1, just in case a failing test case has left something behind; see below for details.
Finally: running the test cases
Now the test cases are executed one by one.
Tidying up
After the last test case was executed the Run* function checks whether there is a function "Cleanup" in the namespace hosting your test cases. If that's the case then this function is executed. Such a function must be a niladic, no-result returning function.
INI file again
Finally the namespace "INI" holding variables populated from your INI file(s) is deleted.
Helpers
Over time it emerged that certain helpers are very useful in the namespace hosting the test cases. Therefore with version 1.4.0 a method EstablishHelpersIn was introduced that takes a reference as the right argument. That ref should point to the namespace hosting the test cases. If you are executing #.Testers.EstablishHelpersIn from within that namespace you can specify an empty vector as the right argument: it then defaults to ⎕IO⊃⎕RSI.
One of the helpers is the method ListHelpers which returns a table with all names in the first column and the first line of the function (which is expected to be a comment telling what the function actually does) in the second column.
Some of the Helpers are useful for flow control, most of the others are just shortcuts that call methods in Testers.
RunThese
A particularly helpful method while developing/enhancing stuff is RunThese. The function allows you to run just selected test functions rather than a whole test suite but at the same time process any INI files and execute Initial and Cleanup if they exist.
RunThese offers the following options:
RunThese 1 2 ⍝ Execute test cases 1 & 2 that do not belong to any group. RunThese ¯1 ¯2 ⍝ Same but stop just before a test case is actually executed. RunThese 'Group1' ⍝ Execute all test cases belonging to group 1. RunThese 'Group1' (2 3) ⍝ Execute test cases 2 &3 of group 1. RunThese 'Group1' 2 3 ⍝ Same as before. RunThese 'Test_Group1_001' ⍝ Executes just this test case. RunThese 0 ⍝ Run all test cases but stop just before the execution.
Best Practices
- Try to keep your test cases simple and test just one thing at a time, for example just one method at a time.
- Create everything you need and tidy up afterwards. Or more precisely, tidy up (left overs!), prepare, test, tidy up again.
- ....
Methods
Show the methods
Examples
Show the examples
Project Page
For bug reports, future enhancements and a full version history see Tester/ProjectPage
Version Information
Original author: |
|
Responsible: |
|
Email: |