mirror-webgpu-cts/docs/terms.md

Writing Tests

Each test suite is organized as a tree, both in the filesystem and further within each file.

• Suites, e.g. src/webgpu/.
  • READMEs, e.g. src/webgpu/README.txt.
  • Test Spec Files, e.g. src/webgpu/examples.spec.ts. Identified by their file path. Each test spec file provides a description and a Test Group. A Test Group defines a test fixture, and contains multiple:
    • Tests. Identified by a comma-separated list of parts (e.g. basic,async) which define a path through a filesystem-like tree (analogy: basic/async.txt). Defines a test function and contains multiple:
      • Test Cases. Identified by a list of Public Parameters (e.g. x = 1, y = 2). Each Test Case has the same test function but different Public Parameters.

Test Tree

A Test Tree is a tree whose leaves are individual Test Cases.

A Test Tree can be thought of as follows:

• Suite, which is the root of a tree with "leaves" which are:
  • Test Spec Files, each of which is a tree with "leaves" which are:
    • Tests, each of which is a tree with leaves which are:
      • Test Cases.

(In the implementation, this conceptual tree of trees is decomposed into one big tree whose leaves are Test Cases.)

Type: TestTree

Suite

A suite of tests. A single suite has a directory structure, and many test spec files (.spec.ts files containing tests) and READMEs. Each member of a suite is identified by its path within the suite.

Example: src/webgpu/

README

Example: src/webgpu/README.txt

Describes (in prose) the contents of a subdirectory in a suite.

READMEs are only processed at build time, when generating the Listing for a suite.

Type: TestSuiteListingEntryReadme

Queries

A Query is a structured object which specifies a subset of cases in exactly one Suite. A Query can be represented uniquely as a string. Queries are used to:

• Identify a subtree of a suite (by identifying the root node of that subtree).
• Identify individual cases.
• Represent the list of tests that a test runner (standalone, wpt, or cmdline) should run.
• Identify subtrees which should not be "collapsed" during WPT cts.html generation, so that that cts.html "variants" can have individual test expectations (i.e. marked as "expected to fail", "skip", etc.).

There are four types of TestQuery:

• TestQueryMultiFile represents any subtree of the file hierarchy:
  • suite:*
  • suite:path,to,*
  • suite:path,to,file,*
• TestQueryMultiTest represents any subtree of the test hierarchy:
  • suite:path,to,file:*
  • suite:path,to,file:path,to,*
  • suite:path,to,file:path,to,test,*
• TestQueryMultiCase represents any subtree of the case hierarchy:
  • suite:path,to,file:path,to,test:*
  • suite:path,to,file:path,to,test:my=0;*
  • suite:path,to,file:path,to,test:my=0;params="here";*
• TestQuerySingleCase represents as single case:
  • suite:path,to,file:path,to,test:my=0;params="here"

Test Queries are a weakly ordered set: any query is Unordered, Equal, StrictSuperset, or StrictSubset relative to any other. This property is used to construct the complete tree of test cases. In the examples above, every example query is a StrictSubset of the previous one (note: even :* is a subset of ,*).

In the WPT and standalone harnesses, the query is stored in the URL, e.g. index.html?q=q:u,e:r,y:*.

Queries are selectively URL-encoded for readability and compatibility with browsers (see encodeURIComponentSelectively).

Type: TestQuery

Listing

A listing of the test spec files in a suite.

This can be generated only in Node, which has filesystem access (see src/tools/crawl.ts). As part of the build step, a listing file is generated (see src/tools/gen.ts) so that the Test Spec Files can be discovered by the web runner (since it does not have filesystem access).

Type: TestSuiteListing

Listing File

Each Suite has one Listing File (suite/listing.[tj]s), containing a list of the files in the suite.

In src/suite/listing.ts, this is computed dynamically. In out/suite/listing.js, the listing has been pre-baked (by tools/gen_listings).

Type: Once imported, ListingFile

Example: out/webgpu/listing.js

Test Spec File

A Test Spec File has a description and a Test Group (under which tests and cases are defined).

Type: Once imported, SpecFile

Example: src/webgpu/**/*.spec.ts

Test Group

A subtree of tests. There is one Test Group per Test Spec File.

The Test Fixture used for tests is defined at TestGroup creation.

Type: TestGroup

Test

One test. It has a single test function.

It may represent multiple test cases, each of which runs the same Test Function with different Parameters.

A test is named using TestGroup.test(), which returns a TestBuilder. TestBuilder.params() can optionally be used to parameterize the test. Then, TestBuilder.fn() provides the Test Function.

Test Function

When a test case is run, the Test Function receives an instance of the Test Fixture provided to the Test Group, producing test results.

Type: TestFn

Test Case / Case

A single case of a test. It is identified by a TestCaseID: a test name, and its parameters.

Type: During test run time, a case is encapsulated as a RunCase.

Parameters / Params

Each Test Case has a (possibly empty) set of Parameters. The parameters are available to the Test Function f(t) via t.params.

A set of Public Parameters identifies a Test Case within a Test.

There are also Private Paremeters: any parameter name beginning with an underscore (_). These parameters are not part of the Test Case identification, but are still passed into the Test Function. They can be used to manually specify expected results.

Type: CaseParams

Test Fixture / Fixture

Test Fixtures provide helpers for tests to use. A new instance of the fixture is created for every run of every test case.

There is always one fixture class for a whole test group (though this may change).

The fixture is also how a test gets access to the case recorder, which allows it to produce test results.

They are also how tests produce results: .skip(), .fail(), etc.

Type: Fixture

UnitTest Fixture

Provides basic fixture utilities most useful in the unittests suite.

GPUTest Fixture

Provides utilities useful in WebGPU CTS tests.

Test Results

Logger

A logger logs the results of a whole test run.

It saves an empty LiveTestSpecResult into its results map, then creates a test spec recorder, which records the results for a group into the LiveTestSpecResult.

Type: Logger

Test Case Recorder

Refers to a LiveTestCaseResult created by the logger. Records the results of running a test case (its pass-status, run time, and logs) into it.

Types: TestCaseRecorder, LiveTestCaseResult

Test Case Status

The status of a LiveTestCaseResult can be one of:

• 'running' (only while still running)
• 'pass'
• 'skip'
• 'warn'
• 'fail'

The "worst" result from running a case is always reported (fail > warn > skip > pass). Note this means a test can still fail if it's "skipped", if it failed before .skip() was called.

Type: Status

Results Format

The results are returned in JSON format.

They are designed to be easily merged in JavaScript: the "results" can be passed into the constructor of Map and merged from there.

(TODO: Write a merge tool, if needed.)

{
  "version": "bf472c5698138cdf801006cd400f587e9b1910a5-dirty",
  "results": [
    [
      "unittests:async_mutex:basic:",
      { "status": "pass", "timems": 0.286, "logs": [] }
    ],
    [
      "unittests:async_mutex:serial:",
      { "status": "pass", "timems": 0.415, "logs": [] }
    ]
  ]
}