04.03 - Test suite definition


When you execute your test through the 'run' goal, you define the test suite to execute ( aka the list of tests to execute ) through the "ta.test.suite" parameter (More details on 'run' goal here). You could define your test suite by providing to this parameter filters (for filtered execution) or data structured in json ( for filtered or ordedered execution)


"ta.test.suite": filters (filtered execution)

  • Definition

The ta.test.suite parameter could be a list of filters separated by comma. A filter could be:
    • The test case path, absolute or relative to the automation project "tests" directory

The file path D:/myProject/src/squashTA/tests/foo/bar/baz.ta can be matched as:

  • D:/myProject/src/squashTA/tests/foo/bar/baz.ta
  • foo/bar/baz.ta
    • A path using character joker which select matching test script inside "tests" directory
      •  for any directory
      • * for 0,1 or many characters

The file path foo/bar/baz.ta can be matched as:

  • **/baz.ta
  • **/bar/baz.ta
  • foo/**/baz.ta
  • f*o/b*/baz.ta
  • **/b*z.ta
  • etc.
    • A regular expression, using the format regex'<myRegex>', which select matching test script inside "tests" directory
regex'<regular_expression>'

with regular_expression: The regular expression to use to select tests

  • Usage

In the sample below, ta.test.suite is compound of two filters:
    • "foo/bar/baz.ta" : will select file foo/bar/baz.ta in "tests" directory (if it exists) for execution
    • "sample/**/*.ta : will select for execution all file in "tests/sample" directory and its subdirectoy which name finish by ".ta"
mvn squash-ta:run -Dta.test.suite=foo/bar/baz.ta,sample/**/*.ta


"ta.test.suite": json data (@Since Squash-TA framework 1.8.0)

Through json data you could do a filtered execution or an ordered execution. 

  • Filtered execution

In the json data you could provide filters, as defined in the previous section, by using the syntax below:
{
    "filter" : "**/*.ta"
}

You could provide, in addition, some global parameters (More details on global parameters here TODO add link) :
{
    "filter" : "**/*.ta",
    "param" : {
        "property5" : "value13",
        "property6" : "value14"
    }
}

  • Ordered execution

The other possibility, in  json format, is to provide the list of tests to execute:
{
    "test" : [{
            "script" : "pathToMyscript1",              // Path to the test script to execute
            "id" : "TestCase1",                             // Test execution identifier
            "param" : {                                         // Script parameters
                "property1" : "value1",
                "property2" : "value2"
            }
        }, {
            "script" : "pathToMyscript2",
            "id" : "TestCase2",
            "param" : {
                "property3" : "value7",
                "property4" : "value8"
            }
        }, {
            "script" : "pathToMyscript1",
            "id" : "TestCase3",
            "param" : {
                "property1" : "value3",
                "property2" : "value4"
            }
        }
    ],
    "param" : {                                                 // Global parameters
        "property5" : "value13",
        "property6" : "value14"
    }
}
Where for each test:
    • "script" is the the path to the test to execute relative to the "tests" directory. This property is mandatory.
    • "id" is the test execution identifier. Only useful for Squash TM - Squash TA link. However, if "id" is define for one test then it should be define for all tests of the test suite.
    • "param" is a list of parameters (key/value) associate to the test. This property is optional.
As for json filtered execution, global parameters are optionals

When no "param" and "id" properties are defined for test, it's possible to use a simpler syntax:

{
    "test" :
    [
        "script1Path", "script2Path", "script3Path"
    ],
    "param" : {
        "property5" : "value13",
        "property6" : "value14"
    }
}
  • Usage
Json data could be provided through a string or a file.

Json provided through a String
    mvn squash-ta:run -Dta.test.suite={'test':[{'script':'pathToMyscript1','param':{'property1':'value1','property2':'value2'}},{'script':'pathToMyscript2'}]}

    Note that the double quote surrounding properties and value of the json data has been replaced. You could:

      • replaced them by  by simple quote (as it's done in the example)
      • escaping the double quote \"

    Json provided through a file
    mvn squash-ta:run -Dta.test.suite={file:pathToMyJsonFile}

    Where pathToMyJsonFile is the path to the json data file. This path could absolute or relative to the root directory of the automation project.


    Filtered execution vs Ordered execution

    • Filtered execution

    When you doing a filtered execution you provide filters. The list of test to execute is compound of all the tests in "tests" directory whose path match the filter. With this kind of execution:
      • A test could be execute only once during an execution
      • There is no execution order.
      • You can't provide specific parameters to  the script (However you can provide global parameters through json data) 

    • Ordered execution

    When you do a filtered execution you provide the list of tests to execute through json format. With this kind of execution:
      • A test could be execute as many time as needed
      • The tests are executed in the order that they were declared in the json data if the tests are in the same ecosystem. If the tests are not in the same ecosystem they are executed ecosystem by ecosystem. That means we execute all the tests of the first ecosystem used ( in the order they are declared) then the tests of the second ecosytem are executed, etc
    With the test tree below:
    tests
      |--foo
      |    |--test21.ta
      |    `--test22.ta
      |--test01.ta
      `--test02.ta


    When this json data is given as input:
    {
        "test" : [{
                "script" : "foo/test22.ta"
            }, {
                "script" : "test01.ta"
            }, {
                "script" : "test02.ta"
            }, {
                "script" : "foo/test21.ta"
            }
        ]
    }

    Then: 
    test22.ta and test21.ta of ecosystem foo will be executed then test01.ta and test02.ta will be executed.
      • Parameters could be specified for each test