4193
Comment:
|
9144
Final version 1.0.0
|
Deletions are marked like this. | Additions are marked like this. |
Line 6: | Line 6: |
|| /!\ Work in progress || |
|
Line 19: | Line 17: |
Note that test case causing a crash are considered "broken" Test cases that do not get the expected result are considered "failing". | Note that test cases causing a crash are considered "broken" Test cases that do not get the expected result are considered "failing". |
Line 27: | Line 25: |
* Run with error trapping and return a log (vector of text vectors) reporting details. Run this before checking in. * Run without error trapping. Use this for investigating why test cases unexpectedly fail. * Run "batch mode". It is assumed that those test cases that need a human being in front of the monitor are '''not''' executed when running in batch mode. |
* Run with error trapping and return a log (vector of text vectors) reporting details. Run this before checking in. See `Tester.Run` for details. * Run without error trapping. Use this for investigating why test cases unexpectedly fail. See `Tester.RunDebug` for details. * Run "batch mode". It is assumed that those test cases that need a human being in front of the monitor are '''not''' executed when running in batch mode. See `Tester.RunBatchTests` for details. |
Line 38: | Line 36: |
== Work flow == No matter which of the three main methods (`Run`, `RunBatchTests`, `RunDebug`) you are going to call, the workflow is always the same: === INI file s === First of all the `Run*` method checks whether there is a file `testcases_{computername}.ini`. If this is the case that INI file is processed. Use this to specify important computer-dependent variables. It then checks whether there is a file `testcases.ini`. If this is the case that INI file is processed, too. Use this to specify general stuff that does not depend on a certain computer/environment. Note that if one or both INI files exist there will be a flat namespace `{yourTestNamespace}.INI`, meaning that sections in the INI file are ignored. An example: if your test functions are hosted by a namespace "foo" and your INI file specifies a variable hello as holding "world" then: {{{ 'world'≡#.foo.INI.hello 1 }}} === Initialisation === Now the `Run*` method checks whether there is a function "Initial" in the hosting namespace. If that is the case then the function is executed. Note that the function must be either niladic or monadic and must not return a result. If the function is monadic then a Boolean is passed via the right argument telling the function that the test suite is going to run in batch mode if true. Use this function to create stuff that's needed by '''all''' test cases, or tell the user something important - if the batch flag is false. It might be a good idea to call a function tidying up in line 1, just in case a failing test case has left something behind; see below for details. === Finally: running the test cases === Now the test cases are executed one by one. === Tidying up === After the last test case was executed the `Run*` function checks whether there is a function "Cleanup" in the namespace hosting your test cases. If that's the case then this function is executed. Such a function must be a niladic, no-result returning function. === INI file again === Finally the namespace "INI" holding variables populated from your INI file(s) is deleted. == Best Pratices == * Try to keep your test cases simple and test just one thing at a time, if possible a method. * Create everything you need and tidy up afterwards. Or more precisely, tidy up (left overs!), create, test, tidy up again. * .... |
|
Line 39: | Line 82: |
<<SeeSaw(section="methods", toshow="<<Show>> the methods", tohide="<<Hide>> the methods", bg="#FEE1A5", speed="Slow")>> {{{{#!wiki seesaw/methods/methods-bg/hide |
|
Line 78: | Line 123: |
Runs all test cases in "refToTestNamespace" <b>without</b> error trapping. If a test case encounters an invalid result it stops. Use this function to investigate the details after "Run" detected a problem. | Runs all test cases in "refToTestNamespace" '''without''' error trapping. If a test case encounters an invalid result it stops. Use this function to investigate the details after "Run" detected a problem. |
Line 83: | Line 128: |
{{{ {log}←testCaseNos RunTheseIn refToTestNamespace }}} Like `RunDebug` but it runs just "testCaseNos" in "refToTestNamespace". Use this in order to debug one or some particular test cases. |
|
Line 87: | Line 138: |
}}}} |
|
Line 88: | Line 141: |
<<SeeSaw(section="samples", toshow="<<Show>> the examples", tohide="<<Hide>> the examples", bg="#FEE1A5", speed="Slow")>> {{{{#!wiki seesaw/samples/samples-bg/hide {{{ )load APLTree\WinFile Loaded.... #.Tester.GetAllTestFns #.TestCases Test_001 Test_002 Test_003 Test_004 Test_005 ... ⊃#.Tester.ListTestCases #.TestCases Exercise "MkDir" Exercise "RmDir" Exercise "Dir" Exercise "DirX" |
|
Line 89: | Line 155: |
#.Tester.Run #.TestCases --- Tests started at 2011-10-18 07:15:25 on #.TestCases ------------------------------------- Test_001: Exercise "MkDir" Test_002: Exercise "RmDir" Test_003: Exercise "Dir" Test_004: Exercise "DirX" Test_005: Exercise "cd" Test_006: Exercise "Delete" Test_007: Exercise "DateOf" Test_008: Exercise "DoesExistDir" Test_009: Exercise "DoesExistFile" Test_010: Exercise "DoesExist" Test_011: Exercise "GetAllDrives" Test_012: Exercise "GetDriveAndType" Test_013: Exercise "IsDirEmpty" Test_014: Exercise "ListDirXIndices" Test_015: Exercise "ListFileAttributes" Test_016: Exercise "YoungerThan" Test_017: Exercise "GetTempFileName" Test_018: Exercise "GetTempPath" Test_019: Exercise "WriteAnsiFile" and "ReadAnsiFile" Test_020: Exercise "CopyTo" Test_021: Exercise "MoveTo" Test_022: Exercise "ListDirsOnly" Test_023: Exercise "CheckPath" Test_024: Exercise "PWD" Test_025: Exercise "IsFilenameOkay" Test_026: Exercise "IsFoldername" ---------------------------------------------------------------------------------------------- 26 test cases executed 0 test cases failed 0 test cases broken ⎕←↑#.Tester.RunBatchTests #.TestCases 0 4 #.Tester.RunDebug #.TestCases ⍝ Stop at test case number 4 --- Tests started at 2011-10-18 07:16:03 on #.TestCases ------------------------------------- Test_001: Exercise "MkDir" Test_002: Exercise "RmDir" Test_003: Exercise "Dir" Run__[61] ⍝ Now you can trace into the forth test case. }}} }}}} |
|
Line 99: | Line 212: |
== Download == |
Tester
Contents
Tester is part of the CategoryAplTree project.
Overview
The purpose of this class is to provide a framework for testing all the projects of the APLTree project. Only with such a framework is it possible to make changes to the test suite without too much hassle. You might find the framework flexible enough to suit your own needs when it comes to implementing tests.
Details
Terminology
Note that test cases causing a crash are considered "broken" Test cases that do not get the expected result are considered "failing".
Assumptions and preconditions
The #.Tester class assumes that all your tests are hosted by an ordinary namespace. All methods take a ref to that namespace as an argument.
All test functions inside that namespace are expected to start their names with the string Test_ followed by digits.
- The first line after the header must carry a comment telling what is actually tested. Keep in mind that later you might be in need for finding out which test cases are testing a certain method!
- It is assumed that you have three different scenarios when you want to run test cases with a different behaviour:
Run with error trapping and return a log (vector of text vectors) reporting details. Run this before checking in. See Tester.Run for details.
Run without error trapping. Use this for investigating why test cases unexpectedly fail. See Tester.RunDebug for details.
Run "batch mode". It is assumed that those test cases that need a human being in front of the monitor are not executed when running in batch mode. See Tester.RunBatchTests for details.
- Every test function must accept a right argument which is a two-item vector of Booleans:
stopFlag (0=use error trapping)
batchFlag (0=does not run in batch mode)
- Every test function must return a scalar integer with one of:
0: the test case is okay.
1: the test case failed.
-1: means the test case did not execute itself because it is not designed to run in batch mode.
Work flow
No matter which of the three main methods (Run, RunBatchTests, RunDebug) you are going to call, the workflow is always the same:
INI file s
First of all the Run* method checks whether there is a file testcases_{computername}.ini. If this is the case that INI file is processed. Use this to specify important computer-dependent variables.
It then checks whether there is a file testcases.ini. If this is the case that INI file is processed, too. Use this to specify general stuff that does not depend on a certain computer/environment.
Note that if one or both INI files exist there will be a flat namespace {yourTestNamespace}.INI, meaning that sections in the INI file are ignored. An example: if your test functions are hosted by a namespace "foo" and your INI file specifies a variable hello as holding "world" then:
'world'≡#.foo.INI.hello 1
Initialisation
Now the Run* method checks whether there is a function "Initial" in the hosting namespace. If that is the case then the function is executed.
Note that the function must be either niladic or monadic and must not return a result. If the function is monadic then a Boolean is passed via the right argument telling the function that the test suite is going to run in batch mode if true.
Use this function to create stuff that's needed by all test cases, or tell the user something important - if the batch flag is false.
It might be a good idea to call a function tidying up in line 1, just in case a failing test case has left something behind; see below for details.
Finally: running the test cases
Now the test cases are executed one by one.
Tidying up
After the last test case was executed the Run* function checks whether there is a function "Cleanup" in the namespace hosting your test cases. If that's the case then this function is executed. Such a function must be a niladic, no-result returning function.
INI file again
Finally the namespace "INI" holding variables populated from your INI file(s) is deleted.
Best Pratices
- Try to keep your test cases simple and test just one thing at a time, if possible a method.
- Create everything you need and tidy up afterwards. Or more precisely, tidy up (left overs!), create, test, tidy up again.
- ....
Methods
Show the methods
Examples
Show the examples
Project Page
For bug reports, future enhancements and a full version history see Tester/ProjectPage
Version Information
Original author: |
|
Responsible: |
|
Email: |