Cold-Air-Networks/mirror-webgpu-cts
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Make test cases hierarchical, and refactor/cleanup a lot in the process (#181)

Browse source 
* update tests, approximately

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* unittests:loading:* now passes

* passes 63 unittests, but not all are running...

* typechecks!!

* 58 unittests passing

* reenable getStackTrace tests

* wip

* finally all tests and checks pass!

* fixups

* wip

* replace all spaces with underscores in test names

* wip

* wip

* wip

* Dissolve TestLoader (now just use DefaultTestFileLoader)

* tighten ParamSpec type

* rename ParamSpec -> CaseParams

* make kParamSeparator ";", now standalone runner is working!

* add separate icon for leaf run buttons

* file renames

* remove user-select:all (not really needed anymore, triple click works fine now)

* cleanup some tests, add tests for test queries

* split interface for defining and running TestGroups

* more cleanup, fix URL encoding stuff, and standalone HTML

* fix readmes for parents of query

* clean up unused ReadmeFile interface

* dissolve id.ts

* test_suite_listing.ts

* split params_utils into param_utils + stringify_params

* rename encodeURLSelectively again

* add query.isMulti{File,Test,Case}

* fix gen_wpt_cts_html (at least for the simple mode; advanced mode probably still broken

* better run-one icon

* standalone: always make the url bar show the complete url

* revert tasks.json

* fix number out of bounds

* don't drop worker results, report input string on all query parse errors

* update docs/terms.md

* revisions

* more revisions

* make comparePublicParamsPaths order-agnostic (now allows queries against params out of order, but tree will not reorder itself accordingly)

* revise compareQueries, part 1

* revise compareQueries, part 2: compareOneLevel

* remove incorrect comment

* undo fast-glob dependency

* address Austin's comments

* address comments

* pass debug flag into test worker

* typo fixes
This commit is contained in:
Kai Ninomiya 2020-05-18 15:33:41 -07:00 committed by GitHub
parent ba023dd7e4
commit 9cbb800a96
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
91 changed files with 2318 additions and 2098 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
.eslintrc.json
View file

@ -14,7 +14,9 @@
}
],
"linebreak-style": ["error", "unix"],
"object-shorthand": "error",
"no-console": "error",
"no-useless-rename": "error",
"@typescript-eslint/no-inferrable-types": "off",
"import/order": [
"error",

2
Gruntfile.js
View file

@ -23,7 +23,7 @@ module.exports = function (grunt) {
},
test: {
cmd: 'node',
args: ['tools/run', 'unittests:'],
args: ['tools/run', 'unittests:*'],
},
'build-out': {
cmd: 'node',

282
docs/terms.md
View file

@ -1,11 +1,36 @@
# Writing Tests
- _Suites_ contain multiple:
- _READMEs_
- _Test Spec Files_. Contains one:
- _Test Group_. Defines a _test fixture_ and contains multiple:
- _Tests_. Defines a _test function_ and contains multiple:
- _Test Cases_, each with the same test function but different _parameters_.
Each test suite is organized as a tree, both in the filesystem and further within each file.
- _Suites_, e.g. `src/webgpu/`.
- _READMEs_, e.g. `src/webgpu/README.txt`.
- _Test Spec Files_, e.g. `src/webgpu/examples.spec.ts`.
Identified by their file path.
Each test spec file provides a description and a _Test Group_.
A _Test Group_ defines a test fixture, and contains multiple:
- _Tests_.
Identified by a comma-separated list of parts (e.g. `basic,async`)
which define a path through a filesystem-like tree (analogy: `basic/async.txt`).
Defines a _test function_ and contains multiple:
- _Test Cases_.
Identified by a list of _Public Parameters_ (e.g. `x` = `1`, `y` = `2`).
Each Test Case has the same test function but different Public Parameters.
## Test Tree
A _Test Tree_ is a tree whose leaves are individual Test Cases.
A Test Tree can be thought of as follows:
- Suite, which is the root of a tree with "leaves" which are:
- Test Spec Files, each of which is a tree with "leaves" which are:
- Tests, each of which is a tree with leaves which are:
- Test Cases.
(In the implementation, this conceptual tree of trees is decomposed into one big tree
whose leaves are Test Cases.)
**Type:** `TestTree`
## Suite
@ -22,30 +47,53 @@ Each member of a suite is identified by its path within the suite.
Describes (in prose) the contents of a subdirectory in a suite.
**Type:** After a README is loaded, it is stored as a `ReadmeFile`.
READMEs are only processed at build time, when generating the _Listing_ for a suite.
## IDs
**Type:** `TestSuiteListingEntryReadme`
### Test Spec ID
## Queries
Uniquely identifies a single test spec file.
Comprised of suite name (`suite`) and test spec file path relative to the suite root (`path`).
A _Query_ is a structured object which specifies a subset of cases in exactly one Suite.
A Query can be represented uniquely as a string.
Queries are used to:
**Type:** `TestSpecID`
- Identify a subtree of a suite (by identifying the root node of that subtree).
- Identify individual cases.
- Represent the list of tests that a test runner (standalone, wpt, or cmdline) should run.
- Identify subtrees which should not be "collapsed" during WPT `cts.html` generation,
so that that cts.html "variants" can have individual test expectations
(i.e. marked as "expected to fail", "skip", etc.).
**Example:** `{ suite: 'webgpu', path: 'command_buffer/compute/basic' }` corresponds to
`src/webgpu/command_buffer/compute/basic.spec.ts`.
There are four types of `TestQuery`:
### Test Case ID
- `TestQueryMultiFile` represents any subtree of the file hierarchy:
- `suite:*`
- `suite:path,to,*`
- `suite:path,to,file,*`
- `TestQueryMultiTest` represents any subtree of the test hierarchy:
- `suite:path,to,file:*`
- `suite:path,to,file:path,to,*`
- `suite:path,to,file:path,to,test,*`
- `TestQueryMultiCase` represents any subtree of the case hierarchy:
- `suite:path,to,file:path,to,test:*`
- `suite:path,to,file:path,to,test:my=0;*`
- `suite:path,to,file:path,to,test:my=0;params="here";*`
- `TestQuerySingleCase` represents as single case:
- `suite:path,to,file:path,to,test:my=0;params="here"`
Uniquely identifies a single test case within a test spec.
Test Queries are a **weakly ordered set**: any query is
_Unordered_, _Equal_, _StrictSuperset_, or _StrictSubset_ relative to any other.
This property is used to construct the complete tree of test cases.
In the examples above, every example query is a StrictSubset of the previous one
(note: even `:*` is a subset of `,*`).
Comprised of test name (`test`) and the parameters for a case (`params`).
(If `params` is null, there is only one case for the test.)
In the WPT and standalone harnesses, the query is stored in the URL, e.g.
`index.html?q=q:u,e:r,y:*`.
**Type:** `TestCaseID`
Queries are selectively URL-encoded for readability and compatibility with browsers
(see `encodeURIComponentSelectively`).
**Example:** `{ test: '', params: { 'value': 1 } }`
**Type:** `TestQuery`
## Listing
@ -53,39 +101,35 @@ A listing of the **test spec files** in a suite.
This can be generated only in Node, which has filesystem access (see `src/tools/crawl.ts`).
As part of the build step, a _listing file_ is generated (see `src/tools/gen.ts`) so that the
test files can be discovered by the web runner (since it does not have filesystem access).
Test Spec Files can be discovered by the web runner (since it does not have filesystem access).
**Type:** `TestSuiteListing`
### Listing File
Each Suite has one Listing File (`suite/listing.[tj]s`), containing a list of the files
in the suite.
In `src/suite/listing.ts`, this is computed dynamically.
In `out/suite/listing.js`, the listing has been pre-baked (by `tools/gen_listings`).
**Type:** Once `import`ed, `ListingFile`
**Example:** `out/webgpu/listing.js`
## Test Spec
## Test Spec File
May be either a _test spec file_ or a _filtered test spec_.
It is identified by a `TestSpecID`.
A Test Spec File has a `description` and a Test Group (under which tests and cases are defined).
Always contains one `RunCaseIterable`.
**Type:** `TestSpec`
### Test Spec File
A single `.spec.ts` file. It always contains one _test group_.
**Type:** Once `import`ed, `SpecFile`
**Example:** `src/webgpu/**/*.spec.ts`
### Filtered Test Spec
## Test Group
A subset of the tests in a single test spec file.
It is created at runtime, via a filter.
A subtree of tests. There is one Test Group per Test Spec File.
It contains one "virtual" test group, which has type `RunCaseIterable`.
## Test Group / Group
A collection of test cases. There is one per test spec file.
The Test Fixture used for tests is defined at TestGroup creation.
**Type:** `TestGroup`
@ -93,18 +137,17 @@ A collection of test cases. There is one per test spec file.
One test. It has a single _test function_.
It may represent multiple _test cases_, each of which runs the same test function with different **parameters**.
It may represent multiple _test cases_, each of which runs the same Test Function with different
Parameters.
It is created by `TestGroup.test()`.
At creation time, it can be parameterized via `Test.params()`.
**Type:** `Test`
A test is named using `TestGroup.test()`, which returns a `TestBuilder`.
`TestBuilder.params()` can optionally be used to parameterize the test.
Then, `TestBuilder.fn()` provides the Test Function.
### Test Function
A test function is defined inline inside of a `TestGroup.test()` call.
It receives an instance of the appropriate _test fixture_, through which it produce test results.
When a test case is run, the Test Function receives an instance of the
Test Fixture provided to the Test Group, producing test results.
**Type:** `TestFn`
@ -116,9 +159,20 @@ A single case of a test. It is identified by a `TestCaseID`: a test name, and it
## Parameters / Params
Each Test Case has a (possibly empty) set of Parameters.
The parameters are available to the Test Function `f(t)` via `t.params`.
A set of Public Parameters identifies a Test Case within a Test.
There are also Private Paremeters: any parameter name beginning with an underscore (`_`).
These parameters are not part of the Test Case identification, but are still passed into
the Test Function. They can be used to manually specify expected results.
**Type:** `CaseParams`
## Test Fixture / Fixture
Test fixtures provide helpers for tests to use.
_Test Fixtures_ provide helpers for tests to use.
A new instance of the fixture is created for every run of every test case.
There is always one fixture class for a whole test group (though this may change).
@ -126,7 +180,7 @@ There is always one fixture class for a whole test group (though this may change
The fixture is also how a test gets access to the _case recorder_,
which allows it to produce test results.
They are also how tests produce results: `.log()`, `.fail()`, etc.
They are also how tests produce results: `.skip()`, `.fail()`, etc.
**Type:** `Fixture`
@ -138,137 +192,61 @@ Provides basic fixture utilities most useful in the `unittests` suite.
Provides utilities useful in WebGPU CTS tests.
# Running Tests
- _Queries_ contain multiple:
- _Filters_ (positive or negative).
## Query
A query is a string which denotes a subset of a test suite to run.
It is comprised of a list of filters.
A case is included in the subset if it matches any of the filters.
In the WPT and standalone harnesses, the query is stored in the URL.
Queries are selectively URL-encoded for readability and compatibility with browsers.
**Example:** `?q=unittests:param_helpers:combine/=`
**Example:** `?q=unittests:param_helpers:combine/mixed=`
**Type:** None yet. TODO
### Filter
A filter matches a set of cases in a suite.
Each filter may match one of:
- `S:s` In one suite `S`, all specs whose paths start with `s` (which may be empty).
- `S:s:t` In one spec `S:s`, all tests whose names start with `t` (which may be empty).
- `S:s:t~c` In one test `S:s:t`, all cases whose params are a superset of `c`.
- `S:s:t=c` In one test `S:s:t`, the single case whose params equal `c` (empty string = `{}`).
**Type:** `TestFilter`
### Using filters to split expectations
A set of cases can be split using negative filters. For example, imagine you have one WPT test variant:
- `webgpu/cts.html?q=unittests:param_helpers:`
But one of the cases is failing. To be able to suppress the failing test without losing test coverage, the WPT test variant can be split into two variants:
- `webgpu/cts.html?q=unittests:param_helpers:&not=unittests:param_helpers:combine/mixed:`
- `webgpu/cts.html?q=unittests:param_helpers:combine/mixed:`
This runs the same set of cases, but in two separate page loads.
# Test Results
## Logger
A logger logs the results of a whole test run.
It saves an empty `LiveTestSpecResult` into its results array, then creates a
It saves an empty `LiveTestSpecResult` into its results map, then creates a
_test spec recorder_, which records the results for a group into the `LiveTestSpecResult`.
### Test Spec Recorder
Refers to a `LiveTestSpecResult` in the logger, and writes results into it.
It saves an empty `LiveTestCaseResult` into its `LiveTestSpecResult`, then creates a
_test case recorder_, which records the results for a case into the `LiveTestCaseResult`.
**Type:** `Logger`
### Test Case Recorder
Records the actual results of running a test case (its pass-status, run time, and logs)
into its `LiveTestCaseResult`.
Refers to a `LiveTestCaseResult` created by the logger.
Records the results of running a test case (its pass-status, run time, and logs) into it.
**Types:** `TestCaseRecorder`, `LiveTestCaseResult`
#### Test Case Status
The `status` of a `LiveTestCaseResult` can be one of:
- `'running'`
- `'fail'`
- `'warn'`
- `'running'` (only while still running)
- `'pass'`
- `'skip'`
- `'warn'`
- `'fail'`
The "worst" result from running a case is always reported.
The "worst" result from running a case is always reported (fail > warn > skip > pass).
Note this means a test can still fail if it's "skipped", if it failed before
`.skip()` was called.
**Type:** `Status`
## Results Format
The results are returned in JSON format.
They are designed to be easily merged in JavaScript.
(TODO: Write a merge tool.)
They are designed to be easily merged in JavaScript:
the `"results"` can be passed into the constructor of `Map` and merged from there.
(TODO: Update these docs if the format changes.)
(TODO: Write a merge tool, if needed.)
```js
{
"version": "e24c459b46c4815f93f0aed5261e28008d1f2882-dirty",
"version": "bf472c5698138cdf801006cd400f587e9b1910a5-dirty",
"results": [
// LiveTestSpecResult objects
{
"spec": "unittests:basic:",
"cases": [
// LiveTestCaseResult objects
{
"test": "test/sync",
"params": null,
"status": "pass",
"timems": 145,
"logs": []
},
{
"test": "test/async",
"params": null,
"status": "pass",
"timems": 26,
"logs": []
}
]
},
{
"spec": "unittests:test_group:",
"cases": [
{
"test": "invalid test name",
"params": { "char": "\"" },
"status": "pass",
"timems": 66,
"logs": ["OK: threw"]
},
{
"test": "invalid test name",
"params": { "char": "`" },
"status": "pass",
"timems": 15,
"logs": ["OK: threw"]
}
]
}
[
"unittests:async_mutex:basic:",
{ "status": "pass", "timems": 0.286, "logs": [] }
],
[
"unittests:async_mutex:serial:",
{ "status": "pass", "timems": 0.415, "logs": [] }
]
]
}
```

2
prettier.config.js
View file

@ -1,8 +1,8 @@
module.exports = {
printWidth: 100,
arrowParens: 'avoid',
bracketSpacing: true,
singleQuote: true,
trailingComma: 'es5',
arrowParens: 'avoid',
};

2
src/common/framework/allowed_characters.ts
View file

@ -1,2 +0,0 @@
// It may be OK to add more allowed characters here.
export const allowedTestNameCharacters = 'a-zA-Z0-9/_';

50
src/common/framework/file_loader.ts Normal file
View file

@ -0,0 +1,50 @@
import { parseQuery } from './query/parseQuery.js';
import { RunCaseIterable } from './test_group.js';
import { TestSuiteListing } from './test_suite_listing.js';
import { loadTreeForQuery, TestTree, TestTreeLeaf } from './tree.js';
// A listing file, e.g. either of:
// - `src/webgpu/listing.ts` (which is dynamically computed, has a Promise<TestSuiteListing>)
// - `out/webgpu/listing.js` (which is pre-baked, has a TestSuiteListing)
interface ListingFile {
listing: Promise<TestSuiteListing> | TestSuiteListing;
}
// A .spec.ts file, as imported.
export interface SpecFile {
readonly description: string;
readonly g: RunCaseIterable;
}
// Base class for DefaultTestFileLoader and FakeTestFileLoader.
export abstract class TestFileLoader {
abstract listing(suite: string): Promise<TestSuiteListing>;
protected abstract import(path: string): Promise<SpecFile>;
importSpecFile(suite: string, path: string[]): Promise<SpecFile> {
return this.import(`${suite}/${path.join('/')}.spec.js`);
}
async loadTree(query: string, subqueriesToExpand: string[] = []): Promise<TestTree> {
return loadTreeForQuery(
this,
parseQuery(query),
subqueriesToExpand.map(q => parseQuery(q))
);
}
async loadTests(query: string): Promise<IterableIterator<TestTreeLeaf>> {
const tree = await this.loadTree(query);
return tree.iterateLeaves();
}
}
export class DefaultTestFileLoader extends TestFileLoader {
async listing(suite: string): Promise<TestSuiteListing> {
return ((await import(`../../${suite}/listing.js`)) as ListingFile).listing;
}
import(path: string): Promise<SpecFile> {
return import(`../../${path}`);
}
}

14
src/common/framework/fixture.ts
View file

@ -1,5 +1,5 @@
import { TestCaseRecorder } from './logger.js';
import { ParamSpec } from './params_utils.js';
import { TestCaseRecorder } from './logging/test_case_recorder.js';
import { CaseParams } from './params_utils.js';
import { assert } from './util/util.js';
export class SkipTestCase extends Error {}
@ -13,7 +13,7 @@ export class Fixture {
private eventualExpectations: Array<Promise<unknown>> = [];
private numOutstandingAsyncExpectations = 0;
constructor(rec: TestCaseRecorder, params: ParamSpec) {
constructor(rec: TestCaseRecorder, params: CaseParams) {
this.rec = rec;
this.params = params;
}
@ -61,16 +61,16 @@ export class Fixture {
private expectErrorValue(expectedName: string, ex: unknown, niceStack: Error): void {
if (!(ex instanceof Error)) {
niceStack.message = 'THREW non-error value, of type ' + typeof ex + niceStack.message;
niceStack.message = `THREW non-error value, of type ${typeof ex}: ${ex}`;
this.rec.fail(niceStack);
return;
}
const actualName = ex.name;
if (actualName !== expectedName) {
niceStack.message = `THREW ${actualName}, instead of ${expectedName}` + niceStack.message;
niceStack.message = `THREW ${actualName}, instead of ${expectedName}: ${ex}`;
this.rec.fail(niceStack);
} else {
niceStack.message = 'OK: threw ' + actualName + niceStack.message;
niceStack.message = `OK: threw ${actualName}${ex.message}`;
this.rec.debug(niceStack);
}
}
@ -80,7 +80,7 @@ export class Fixture {
const m = msg ? ': ' + msg : '';
try {
await p;
niceStack.message = 'DID NOT THROW' + m;
niceStack.message = 'DID NOT REJECT' + m;
this.rec.fail(niceStack);
} catch (ex) {
niceStack.message = m;

112
src/common/framework/generate_minimal_query_list.ts
View file

@ -1,112 +0,0 @@
import { TestSpecOrTestOrCaseID } from './id.js';
import { Logger } from './logger.js';
import { makeFilter } from './test_filter/load_filter.js';
import { TestFilterResult } from './test_filter/test_filter_result.js';
import { FilterResultTreeNode, treeFromFilterResults } from './tree.js';
interface QuerySplitterTreeNode {
needsSplit: boolean;
children?: Map<string, QuerySplitterTreeNode>;
}
interface Expectation {
id: TestSpecOrTestOrCaseID;
line: string;
seen: boolean;
}
function makeQuerySplitterTree(
caselist: TestFilterResult[],
expectationStrings: string[]
): QuerySplitterTreeNode {
const expectations: Expectation[] = [];
for (const e of expectationStrings) {
const filter = makeFilter(e);
const id = filter.idIfSingle();
if (!id) {
throw new Error(
'Can only handle expectations which cover one file, one test, or one case. ' + e
);
}
expectations.push({ id, line: e, seen: false });
}
const convertToQuerySplitterTree = (
tree: FilterResultTreeNode,
name?: string
): QuerySplitterTreeNode => {
const children = tree.children;
let needsSplit = true;
if (name !== undefined) {
const filter = makeFilter(name);
const moreThanOneFile = !filter.definitelyOneFile();
const matchingExpectations = expectations.map(e => {
const matches = filter.matches(e.id);
if (matches) e.seen = true;
return matches;
});
needsSplit = matchingExpectations.some(m => m) || moreThanOneFile;
}
const queryNode: QuerySplitterTreeNode = { needsSplit };
if (children) {
queryNode.children = new Map();
for (const [k, v] of children) {
const subtree = convertToQuerySplitterTree(v, k);
queryNode.children.set(k, subtree);
}
}
return queryNode;
};
const log = new Logger();
const tree = treeFromFilterResults(log, caselist.values());
const queryTree = convertToQuerySplitterTree(tree)!;
for (const e of expectations) {
if (!e.seen) throw new Error('expectation had no effect: ' + e.line);
}
return queryTree;
}
// Takes a TestFilterResultIterator enumerating every test case in the suite, and a list of
// expectation queries from a browser's expectations file. Creates a minimal list of queries
// (i.e. wpt variant lines) such that:
//
// - There is at least one query per spec file.
// - Each of those those input queries is in the output, so that it can have its own expectation.
//
// It does this by creating a tree from the list of cases (same tree as the standalone runner uses),
// then marking every node which is a parent of a node that matches an expectation.
export async function generateMinimalQueryList(
caselist: TestFilterResult[],
expectationStrings: string[]
): Promise<string[]> {
const unsplitNodes: string[] = [];
const findUnsplitNodes = (name: string, node: QuerySplitterTreeNode | undefined) => {
if (node === undefined) {
return;
}
if (node.needsSplit && node.children) {
for (const [k, v] of node.children) {
findUnsplitNodes(k, v);
}
} else {
unsplitNodes.push(name);
}
};
const queryTree = makeQuerySplitterTree(caselist, expectationStrings);
findUnsplitNodes('', queryTree);
for (const exp of expectationStrings) {
if (!unsplitNodes.some(name => name === exp)) {
throw new Error(
'Something went wrong: all expectation strings should always appear exactly: ' + exp
);
}
}
return unsplitNodes;
}

2
src/common/framework/gpu/device_pool.ts
View file

@ -71,7 +71,7 @@ export class DevicePool {
// TODO: device.destroy()
// Mark the holder as free. (This only has an effect if the pool still has the holder.)
// This could be done at the top but is done here to guard againt async-races during release.
// This could be done at the top but is done here to guard against async-races during release.
holder.acquired = false;
}
}

25
src/common/framework/id.ts
View file

@ -1,25 +0,0 @@
import { ParamSpec } from './params_utils.js';
// Identifies a test spec file.
export interface TestSpecID {
// The spec's suite name, e.g. 'webgpu'.
readonly suite: string;
// The spec's path within the suite, e.g. 'command_buffer/compute/basic'.
readonly path: string;
}
export function testSpecEquals(x: TestSpecID, y: TestSpecID): boolean {
return x.suite === y.suite && x.path === y.path;
}
// Identifies a test case (a specific parameterization of a test), within its spec file.
export interface TestCaseID {
readonly test: string;
readonly params: ParamSpec | null;
}
export interface TestSpecOrTestOrCaseID {
readonly spec: TestSpecID;
readonly test?: string;
readonly params?: ParamSpec | null;
}

9
src/common/framework/listing.ts
View file

@ -1,9 +0,0 @@
// A listing of all specs within a single suite. This is the (awaited) type of
// `groups` in '{cts,unittests}/listing.ts' and the auto-generated
// 'out/{cts,unittests}/listing.js' files (see tools/gen_listings).
export type TestSuiteListing = Iterable<TestSuiteListingEntry>;
export interface TestSuiteListingEntry {
readonly path: string;
readonly description: string;
}

65
src/common/framework/loader.ts
View file

@ -1,65 +0,0 @@
import { TestSuiteListing } from './listing.js';
import { loadFilter } from './test_filter/load_filter.js';
import { TestFilterResult } from './test_filter/test_filter_result.js';
import { RunCaseIterable } from './test_group.js';
// One of the following:
// - An actual .spec.ts file, as imported.
// - A *filtered* list of cases from a single .spec.ts file.
export interface TestSpec {
readonly description: string;
readonly g: RunCaseIterable;
}
// A shell object describing a directory (from its README.txt).
export interface ReadmeFile {
readonly description: string;
}
export type TestSpecOrReadme = TestSpec | ReadmeFile;
type TestFilterResultIterator = IterableIterator<TestFilterResult>;
function* concat(lists: TestFilterResult[][]): TestFilterResultIterator {
for (const specs of lists) {
yield* specs;
}
}
export interface TestFileLoader {
listing(suite: string): Promise<TestSuiteListing>;
import(path: string): Promise<TestSpecOrReadme>;
}
class DefaultTestFileLoader implements TestFileLoader {
async listing(suite: string): Promise<TestSuiteListing> {
return (await import(`../../${suite}/listing.js`)).listing;
}
import(path: string): Promise<TestSpec> {
return import('../../' + path);
}
}
export class TestLoader {
private fileLoader: TestFileLoader;
constructor(fileLoader: TestFileLoader = new DefaultTestFileLoader()) {
this.fileLoader = fileLoader;
}
// TODO: Test
async loadTestsFromQuery(query: string): Promise<TestFilterResultIterator> {
return this.loadTests(new URLSearchParams(query).getAll('q'));
}
// TODO: Test
// TODO: Probably should actually not exist at all, just use queries on cmd line too.
async loadTestsFromCmdLine(filters: string[]): Promise<TestFilterResultIterator> {
return this.loadTests(filters);
}
async loadTests(filters: string[]): Promise<TestFilterResultIterator> {
const loads = filters.map(f => loadFilter(this.fileLoader, f));
return concat(await Promise.all(loads));
}
}

159
src/common/framework/logger.ts
View file

@ -1,159 +0,0 @@
import { SkipTestCase } from './fixture.js';
import { TestSpecID } from './id.js';
import { ParamSpec, extractPublicParams } from './params_utils.js';
import { makeQueryString } from './url_query.js';
import { getStackTrace } from './util/stack.js';
import { assert, now } from './util/util.js';
import { version } from './version.js';
type Status = 'running' | 'pass' | 'skip' | 'warn' | 'fail';
export interface LiveTestSpecResult {
readonly spec: string;
readonly cases: LiveTestCaseResult[];
}
interface TestCaseResult {
readonly test: string;
readonly params: ParamSpec | null;
status: Status;
timems: number;
}
export interface LiveTestCaseResult extends TestCaseResult {
logs?: LogMessageWithStack[];
}
export interface TransferredTestCaseResult extends TestCaseResult {
// When transferred from a worker, a LogMessageWithStack turns into a generic Error
// (its prototype gets lost and replaced with Error).
logs?: Error[];
}
export class LogMessageWithStack extends Error {
constructor(name: string, ex: Error, includeStack: boolean = true) {
super(ex.message);
this.name = name;
this.stack = includeStack ? ex.stack : undefined;
}
toJSON(): string {
let m = this.name;
if (this.message) {
m += ': ' + this.message;
}
if (this.stack) {
m += '\n' + getStackTrace(this);
}
return m;
}
}
export class Logger {
readonly results: LiveTestSpecResult[] = [];
constructor() {}
record(spec: TestSpecID): [TestSpecRecorder, LiveTestSpecResult] {
const result: LiveTestSpecResult = { spec: makeQueryString(spec), cases: [] };
this.results.push(result);
return [new TestSpecRecorder(result), result];
}
asJSON(space?: number): string {
return JSON.stringify({ version, results: this.results }, undefined, space);
}
}
export class TestSpecRecorder {
private result: LiveTestSpecResult;
constructor(result: LiveTestSpecResult) {
this.result = result;
}
record(test: string, params: ParamSpec | null): [TestCaseRecorder, LiveTestCaseResult] {
const result: LiveTestCaseResult = {
test,
params: params ? extractPublicParams(params) : null,
status: 'running',
timems: -1,
};
this.result.cases.push(result);
return [new TestCaseRecorder(result), result];
}
}
enum PassState {
pass = 0,
skip = 1,
warn = 2,
fail = 3,
}
export class TestCaseRecorder {
private result: LiveTestCaseResult;
private state = PassState.pass;
private startTime = -1;
private logs: LogMessageWithStack[] = [];
private debugging = false;
constructor(result: LiveTestCaseResult) {
this.result = result;
}
start(debug: boolean = false): void {
this.startTime = now();
this.logs = [];
this.state = PassState.pass;
this.debugging = debug;
}
finish(): void {
assert(this.startTime >= 0, 'finish() before start()');
const endTime = now();
// Round to next microsecond to avoid storing useless .xxxx00000000000002 in results.
this.result.timems = Math.ceil((endTime - this.startTime) * 1000) / 1000;
this.result.status = PassState[this.state] as Status;
this.result.logs = this.logs;
this.debugging = false;
}
debug(ex: Error): void {
if (!this.debugging) {
return;
}
this.logs.push(new LogMessageWithStack('DEBUG', ex, false));
}
warn(ex: Error): void {
this.setState(PassState.warn);
this.logs.push(new LogMessageWithStack('WARN', ex));
}
fail(ex: Error): void {
this.setState(PassState.fail);
this.logs.push(new LogMessageWithStack('FAIL', ex));
}
skipped(ex: SkipTestCase): void {
this.setState(PassState.skip);
this.logs.push(new LogMessageWithStack('SKIP', ex));
}
threw(ex: Error): void {
if (ex instanceof SkipTestCase) {
this.skipped(ex);
return;
}
this.setState(PassState.fail);
this.logs.push(new LogMessageWithStack('EXCEPTION', ex));
}
private setState(state: PassState): void {
this.state = Math.max(this.state, state);
}
}

21
src/common/framework/logging/log_message.ts Normal file
View file

@ -0,0 +1,21 @@
import { extractImportantStackTrace } from '../util/stack.js';
export class LogMessageWithStack extends Error {
constructor(name: string, ex: Error, includeStack: boolean = true) {
super(ex.message);
this.name = name;
this.stack = includeStack ? ex.stack : undefined;
}
toJSON(): string {
let m = this.name + ': ';
if (this.stack) {
// this.message is already included in this.stack
m += extractImportantStackTrace(this);
} else {
m += this.message;
}
return m;
}
}

25
src/common/framework/logging/logger.ts Normal file
View file

@ -0,0 +1,25 @@
import { version } from '../version.js';
import { LiveTestCaseResult } from './result.js';
import { TestCaseRecorder } from './test_case_recorder.js';
export type LogResults = Map<string, LiveTestCaseResult>;
export class Logger {
readonly debug: boolean;
readonly results: LogResults = new Map();
constructor(debug: boolean) {
this.debug = debug;
}
record(name: string): [TestCaseRecorder, LiveTestCaseResult] {
const result: LiveTestCaseResult = { status: 'running', timems: -1 };
this.results.set(name, result);
return [new TestCaseRecorder(result, this.debug), result];
}
asJSON(space?: number): string {
return JSON.stringify({ version, results: Array.from(this.results) }, undefined, space);
}
}

18
src/common/framework/logging/result.ts Normal file
View file

@ -0,0 +1,18 @@
import { LogMessageWithStack } from './log_message.js';
export type Status = 'running' | 'pass' | 'skip' | 'warn' | 'fail';
export interface TestCaseResult {
status: Status;
timems: number;
}
export interface LiveTestCaseResult extends TestCaseResult {
logs?: LogMessageWithStack[];
}
export interface TransferredTestCaseResult extends TestCaseResult {
// When transferred from a worker, a LogMessageWithStack turns into a generic Error
// (its prototype gets lost and replaced with Error).
logs?: Error[];
}

82
src/common/framework/logging/test_case_recorder.ts Normal file
View file

@ -0,0 +1,82 @@
import { SkipTestCase } from '../fixture.js';
import { now, assert } from '../util/util.js';
import { LogMessageWithStack } from './log_message.js';
import { LiveTestCaseResult, Status } from './result.js';
enum PassState {
pass = 0,
skip = 1,
warn = 2,
fail = 3,
}
// Holds onto a LiveTestCaseResult owned by the Logger, and writes the results into it.
export class TestCaseRecorder {
private result: LiveTestCaseResult;
private state = PassState.pass;
private startTime = -1;
private logs: LogMessageWithStack[] = [];
private debugging = false;
constructor(result: LiveTestCaseResult, debugging: boolean) {
this.result = result;
this.debugging = debugging;
}
start(): void {
assert(this.startTime < 0, 'TestCaseRecorder cannot be reused');
this.startTime = now();
}
finish(): void {
assert(this.startTime >= 0, 'finish() before start()');
const timeMilliseconds = now() - this.startTime;
// Round to next microsecond to avoid storing useless .xxxx00000000000002 in results.
this.result.timems = Math.ceil(timeMilliseconds * 1000) / 1000;
this.result.status = PassState[this.state] as Status; // Convert numeric enum back to string
this.result.logs = this.logs;
}
injectResult(injectedResult: LiveTestCaseResult): void {
Object.assign(this.result, injectedResult);
}
debug(ex: Error): void {
if (!this.debugging) {
return;
}
this.logs.push(new LogMessageWithStack('DEBUG', ex, false));
}
warn(ex: Error): void {
this.setState(PassState.warn);
this.logs.push(new LogMessageWithStack('WARN', ex));
}
fail(ex: Error): void {
this.setState(PassState.fail);
this.logs.push(new LogMessageWithStack('FAIL', ex));
}
skipped(ex: SkipTestCase): void {
this.setState(PassState.skip);
this.logs.push(new LogMessageWithStack('SKIP', ex));
}
threw(ex: Error): void {
if (ex instanceof SkipTestCase) {
this.skipped(ex);
return;
}
this.setState(PassState.fail);
this.logs.push(new LogMessageWithStack('EXCEPTION', ex));
}
private setState(state: PassState): void {
this.state = Math.max(this.state, state);
}
}

16
src/common/framework/params.ts → src/common/framework/params_builder.ts
View file

@ -1,4 +1,4 @@
import { ParamSpec, ParamSpecIterable, paramsEquals } from './params_utils.js';
import { CaseParams, CaseParamsIterable, publicParamsEquals } from './params_utils.js';
import { assert } from './util/util.js';
// https://stackoverflow.com/a/56375136
@ -33,11 +33,11 @@ export function params(): ParamsBuilder<{}> {
return new ParamsBuilder();
}
class ParamsBuilder<A extends {}> implements ParamSpecIterable {
private paramSpecs: ParamSpecIterable = [{}];
export class ParamsBuilder<A extends {}> implements CaseParamsIterable {
private paramSpecs: CaseParamsIterable = [{}];
[Symbol.iterator](): Iterator<A> {
const iter: Iterator<ParamSpec> = this.paramSpecs[Symbol.iterator]();
const iter: Iterator<CaseParams> = this.paramSpecs[Symbol.iterator]();
return iter as Iterator<A>;
}
@ -49,7 +49,7 @@ class ParamsBuilder<A extends {}> implements ParamSpecIterable {
yield mergeParams(a, b);
}
}
});
}) as CaseParamsIterable;
/* eslint-disable-next-line @typescript-eslint/no-explicit-any */
return this as any;
}
@ -62,7 +62,7 @@ class ParamsBuilder<A extends {}> implements ParamSpecIterable {
yield mergeParams(a, b);
}
}
});
}) as CaseParamsIterable;
/* eslint-disable-next-line @typescript-eslint/no-explicit-any */
return this as any;
}
@ -83,12 +83,12 @@ class ParamsBuilder<A extends {}> implements ParamSpecIterable {
return this.filter(x => !pred(x));
}
exclude(exclude: ParamSpecIterable): ParamsBuilder<A> {
exclude(exclude: CaseParamsIterable): ParamsBuilder<A> {
const excludeArray = Array.from(exclude);
const paramSpecs = this.paramSpecs;
this.paramSpecs = makeReusableIterable(function* () {
for (const p of paramSpecs) {
if (excludeArray.every(e => !paramsEquals(p, e))) {
if (excludeArray.every(e => !publicParamsEquals(p, e))) {
yield p;
}
}

77
src/common/framework/params_utils.ts
View file

@ -1,69 +1,32 @@
import { objectEquals } from './util/util.js';
import { comparePublicParamsPaths, Ordering } from './query/compare.js';
import { kWildcard, kParamSeparator } from './query/separators.js';
/* eslint-disable-next-line @typescript-eslint/no-explicit-any */
export type ParamArgument = any;
export interface ParamSpec {
// Consider adding more types here if needed
export type ParamArgument = void | undefined | number | string | boolean | number[];
export interface CaseParams {
readonly [k: string]: ParamArgument;
}
export interface CaseParamsRW {
[k: string]: ParamArgument;
}
export type ParamSpecIterable = Iterable<ParamSpec>;
export type ParamSpecIterator = IterableIterator<ParamSpec>;
export type CaseParamsIterable = Iterable<CaseParams>;
export function extractPublicParams(params: ParamSpec): ParamSpec {
const publicParams: ParamSpec = {};
export function paramKeyIsPublic(key: string): boolean {
return !key.startsWith('_');
}
export function extractPublicParams(params: CaseParams): CaseParams {
const publicParams: CaseParamsRW = {};
for (const k of Object.keys(params)) {
if (!k.startsWith('_')) {
if (paramKeyIsPublic(k)) {
publicParams[k] = params[k];
}
}
return publicParams;
}
export function stringifyPublicParams(p: ParamSpec | null): string {
if (p === null || paramsEquals(p, {})) {
return '';
}
return JSON.stringify(extractPublicParams(p));
}
export function paramsEquals(x: ParamSpec | null, y: ParamSpec | null): boolean {
if (x === y) {
return true;
}
if (x === null) {
x = {};
}
if (y === null) {
y = {};
}
for (const xk of Object.keys(x)) {
if (x[xk] !== undefined && !(xk in y)) {
return false;
}
if (!objectEquals(x[xk], y[xk])) {
return false;
}
}
for (const yk of Object.keys(y)) {
if (y[yk] !== undefined && !(yk in x)) {
return false;
}
}
return true;
}
export function paramsSupersets(sup: ParamSpec | null, sub: ParamSpec | null): boolean {
if (sub === null) {
return true;
}
if (sup === null) {
sup = {};
}
for (const k of Object.keys(sub)) {
if (!(k in sup) || sup[k] !== sub[k]) {
return false;
}
}
return true;
export const badParamValueChars = new RegExp('[=' + kParamSeparator + kWildcard + ']');
export function publicParamsEquals(x: CaseParams, y: CaseParams): boolean {
return comparePublicParamsPaths(x, y) === Ordering.Equal;
}

89
src/common/framework/query/compare.ts Normal file
View file

@ -0,0 +1,89 @@
import { CaseParams, extractPublicParams } from '../params_utils.js';
import { assert, objectEquals } from '../util/util.js';
import { TestQuery } from './query.js';
export const enum Ordering {
Unordered,
StrictSuperset,
Equal,
StrictSubset,
}
// Compares two queries for their ordering (which is used to build the tree).
// See src/unittests/query_compare.spec.ts for examples.
export function compareQueries(a: TestQuery, b: TestQuery): Ordering {
if (a.suite !== b.suite) {
return Ordering.Unordered;
}
const filePathOrdering = comparePaths(a.filePathParts, b.filePathParts);
if (filePathOrdering !== Ordering.Equal || a.isMultiFile || b.isMultiFile) {
return compareOneLevel(filePathOrdering, a.isMultiFile, b.isMultiFile);
}
assert('testPathParts' in a && 'testPathParts' in b);
const testPathOrdering = comparePaths(a.testPathParts, b.testPathParts);
if (testPathOrdering !== Ordering.Equal || a.isMultiTest || b.isMultiTest) {
return compareOneLevel(testPathOrdering, a.isMultiTest, b.isMultiTest);
}
assert('params' in a && 'params' in b);
const paramsPathOrdering = comparePublicParamsPaths(a.params, b.params);
if (paramsPathOrdering !== Ordering.Equal || a.isMultiCase || b.isMultiCase) {
return compareOneLevel(paramsPathOrdering, a.isMultiCase, b.isMultiCase);
}
return Ordering.Equal;
}
// Compares a single level of a query.
// "IsBig" means the query is big relative to the level, e.g. for test-level:
// anything >= suite:a,* is big
// anything <= suite:a:* is small
function compareOneLevel(ordering: Ordering, aIsBig: boolean, bIsBig: boolean): Ordering {
assert(ordering !== Ordering.Equal || aIsBig || bIsBig);
if (ordering === Ordering.Unordered) return Ordering.Unordered;
if (aIsBig && bIsBig) return ordering;
if (!aIsBig && !bIsBig) return Ordering.Unordered; // Equal case is already handled
// Exactly one of (a, b) is big.
if (aIsBig && ordering !== Ordering.StrictSubset) return Ordering.StrictSuperset;
if (bIsBig && ordering !== Ordering.StrictSuperset) return Ordering.StrictSubset;
return Ordering.Unordered;
}
function comparePaths(a: readonly string[], b: readonly string[]): Ordering {
const shorter = Math.min(a.length, b.length);
for (let i = 0; i < shorter; ++i) {
if (a[i] !== b[i]) {
return Ordering.Unordered;
}
}
if (a.length === b.length) {
return Ordering.Equal;
} else if (a.length < b.length) {
return Ordering.StrictSuperset;
} else {
return Ordering.StrictSubset;
}
}
export function comparePublicParamsPaths(a0: CaseParams, b0: CaseParams): Ordering {
const a = extractPublicParams(a0);
const b = extractPublicParams(b0);
const aKeys = Object.keys(a);
const commonKeys = new Set(aKeys.filter(k => k in b));
for (const k of commonKeys) {
if (!objectEquals(a[k], b[k])) {
return Ordering.Unordered;
}
}
const bKeys = Object.keys(b);
const aRemainingKeys = aKeys.length - commonKeys.size;
const bRemainingKeys = bKeys.length - commonKeys.size;
if (aRemainingKeys === 0 && bRemainingKeys === 0) return Ordering.Equal;
if (aRemainingKeys === 0) return Ordering.StrictSuperset;
if (bRemainingKeys === 0) return Ordering.StrictSubset;
return Ordering.Unordered;
}

16
src/common/framework/query/encode_selectively.ts Normal file
View file

@ -0,0 +1,16 @@
// Encodes a stringified TestQuery so that it can be placed in a `?q=` parameter in a URL.
// encodeURIComponent encodes in accordance with `application/x-www-form-urlencoded`, but URLs don't
// actually have to be as strict as HTML form encoding (we interpret this purely from JavaScript).
// So we encode the component, then selectively convert some %-encoded escape codes back to their
// original form for readability/copyability.
export function encodeURIComponentSelectively(s: string): string {
let ret = encodeURIComponent(s);
ret = ret.replace(/%22/g, '"'); // for JSON strings
ret = ret.replace(/%2C/g, ','); // for path separator, and JSON arrays
ret = ret.replace(/%3A/g, ':'); // for big separator
ret = ret.replace(/%3B/g, ';'); // for param separator
ret = ret.replace(/%3D/g, '='); // for params (k=v)
ret = ret.replace(/%5B/g, '['); // for JSON arrays
ret = ret.replace(/%5D/g, ']'); // for JSON arrays
return ret;
}

132
src/common/framework/query/parseQuery.ts Normal file
View file

@ -0,0 +1,132 @@
import {
CaseParamsRW,
ParamArgument,
badParamValueChars,
paramKeyIsPublic,
} from '../params_utils.js';
import { assert } from '../util/util.js';
import {
TestQuery,
TestQueryMultiFile,
TestQueryMultiTest,
TestQueryMultiCase,
TestQuerySingleCase,
} from './query.js';
import { kBigSeparator, kWildcard, kPathSeparator, kParamSeparator } from './separators.js';
import { validQueryPart } from './validQueryPart.js';
export function parseQuery(s: string): TestQuery {
try {
return parseQueryImpl(s);
} catch (ex) {
ex.message += '\n on: ' + s;
throw ex;
}
}
function parseQueryImpl(s: string): TestQuery {
// Undo encodeURIComponentSelectively
s = decodeURIComponent(s);
// bigParts are: suite, group, test, params (note kBigSeparator could appear in params)
const [suite, fileString, testString, paramsString] = s.split(kBigSeparator, 4);
assert(fileString !== undefined, `filter string must have at least one ${kBigSeparator}`);
const { parts: file, wildcard: filePathHasWildcard } = parseBigPart(fileString, kPathSeparator);
if (testString === undefined) {
// Query is file-level
assert(
filePathHasWildcard,
`File-level query without wildcard ${kWildcard}. Did you want a file-level query \
(append ${kPathSeparator}${kWildcard}) or test-level query (append ${kBigSeparator}${kWildcard})?`
);
return new TestQueryMultiFile(suite, file);
}
assert(!filePathHasWildcard, `Wildcard ${kWildcard} must be at the end of the query string`);
const { parts: test, wildcard: testPathHasWildcard } = parseBigPart(testString, kPathSeparator);
if (paramsString === undefined) {
// Query is test-level
assert(
testPathHasWildcard,
`Test-level query without wildcard ${kWildcard}; did you want a test-level query \
(append ${kPathSeparator}${kWildcard}) or case-level query (append ${kBigSeparator}${kWildcard})?`
);
assert(file.length > 0, 'File part of test-level query was empty (::)');
return new TestQueryMultiTest(suite, file, test);
}
// Query is case-level
assert(!testPathHasWildcard, `Wildcard ${kWildcard} must be at the end of the query string`);
const { parts: paramsParts, wildcard: paramsHasWildcard } = parseBigPart(
paramsString,
kParamSeparator
);
assert(test.length > 0, 'Test part of case-level query was empty (::)');
const params: CaseParamsRW = {};
for (const paramPart of paramsParts) {
const [k, v] = parseSingleParam(paramPart);
assert(validQueryPart.test(k), 'param key names must match ' + validQueryPart);
params[k] = v;
}
if (paramsHasWildcard) {
return new TestQueryMultiCase(suite, file, test, params);
} else {
return new TestQuerySingleCase(suite, file, test, params);
}
}
// webgpu:a,b,* or webgpu:a,b,c:*
const kExampleQueries = `\
webgpu${kBigSeparator}a${kPathSeparator}b${kPathSeparator}${kWildcard} or \
webgpu${kBigSeparator}a${kPathSeparator}b${kPathSeparator}c${kBigSeparator}${kWildcard}`;
function parseBigPart(
s: string,
separator: typeof kParamSeparator | typeof kPathSeparator
): { parts: string[]; wildcard: boolean } {
if (s === '') {
return { parts: [], wildcard: false };
}
const parts = s.split(separator);
let endsWithWildcard = false;
for (const [i, part] of parts.entries()) {
if (i === parts.length - 1) {
endsWithWildcard = part === kWildcard;
}
assert(
part.indexOf(kWildcard) === -1 || endsWithWildcard,
`Wildcard ${kWildcard} must be complete last part of a path (e.g. ${kExampleQueries})`
);
}
if (endsWithWildcard) {
// Remove the last element of the array (which is just the wildcard).
parts.length = parts.length - 1;
}
return { parts, wildcard: endsWithWildcard };
}
function parseSingleParam(paramSubstring: string): [string, ParamArgument] {
assert(paramSubstring !== '', 'Param in a query must not be blank (is there a trailing comma?)');
const i = paramSubstring.indexOf('=');
assert(i !== -1, 'Param in a query must be of form key=value');
const k = paramSubstring.substring(0, i);
assert(paramKeyIsPublic(k), 'Param in a query must not be private (start with _)');
const v = paramSubstring.substring(i + 1);
return [k, parseSingleParamValue(v)];
}
function parseSingleParamValue(s: string): ParamArgument {
assert(
!badParamValueChars.test(s),
`param value must not match ${badParamValueChars} - was ${s}`
);
return s === 'undefined' ? undefined : JSON.parse(s);
}

91
src/common/framework/query/query.ts Normal file
View file

@ -0,0 +1,91 @@
import { CaseParams } from '../params_utils.js';
import { assert } from '../util/util.js';
import { encodeURIComponentSelectively } from './encode_selectively.js';
import { kBigSeparator, kPathSeparator, kWildcard, kParamSeparator } from './separators.js';
import { stringifyPublicParams } from './stringify_params.js';
export type TestQuery =
| TestQuerySingleCase
| TestQueryMultiCase
| TestQueryMultiTest
| TestQueryMultiFile;
export class TestQueryMultiFile {
readonly isMultiFile: boolean = true;
readonly suite: string;
readonly filePathParts: readonly string[];
constructor(suite: string, file: readonly string[]) {
this.suite = suite;
this.filePathParts = [...file];
}
toString(): string {
return encodeURIComponentSelectively(this.toStringHelper().join(kBigSeparator));
}
toHTML(): string {
return this.toStringHelper().join(kBigSeparator + '<wbr>');
}
protected toStringHelper(): string[] {
return [this.suite, [...this.filePathParts, kWildcard].join(kPathSeparator)];
}
}
export class TestQueryMultiTest extends TestQueryMultiFile {
readonly isMultiFile: false = false;
readonly isMultiTest: boolean = true;
readonly testPathParts: readonly string[];
constructor(suite: string, file: readonly string[], test: readonly string[]) {
super(suite, file);
assert(file.length > 0, 'multi-test (or finer) query must have file-path');
this.testPathParts = [...test];
}
protected toStringHelper(): string[] {
return [
this.suite,
this.filePathParts.join(kPathSeparator),
[...this.testPathParts, kWildcard].join(kPathSeparator),
];
}
}
export class TestQueryMultiCase extends TestQueryMultiTest {
readonly isMultiTest: false = false;
readonly isMultiCase: boolean = true;
readonly params: CaseParams;
constructor(suite: string, file: readonly string[], test: readonly string[], params: CaseParams) {
super(suite, file, test);
assert(test.length > 0, 'multi-case (or finer) query must have test-path');
this.params = { ...params };
}
protected toStringHelper(): string[] {
const paramsParts = stringifyPublicParams(this.params);
return [
this.suite,
this.filePathParts.join(kPathSeparator),
this.testPathParts.join(kPathSeparator),
[...paramsParts, kWildcard].join(kParamSeparator),
];
}
}
export class TestQuerySingleCase extends TestQueryMultiCase {
readonly isMultiCase: false = false;
protected toStringHelper(): string[] {
const paramsParts = stringifyPublicParams(this.params);
return [
this.suite,
this.filePathParts.join(kPathSeparator),
this.testPathParts.join(kPathSeparator),
paramsParts.join(kParamSeparator),
];
}
}

5
src/common/framework/query/separators.ts Normal file
View file

@ -0,0 +1,5 @@
export const kBigSeparator = ':'; // Separator between big parts: suite:file:test:case
export const kPathSeparator = ','; // Separator between path,to,file or path,to,test
export const kParamSeparator = ';'; // Separator between k=v;k=v
export const kParamKVSeparator = '='; // Separator between key and value in k=v
export const kWildcard = '*'; // Final wildcard, if query is not single-case

27
src/common/framework/query/stringify_params.ts Normal file
View file

@ -0,0 +1,27 @@
import {
CaseParams,
extractPublicParams,
ParamArgument,
badParamValueChars,
} from '../params_utils.js';
import { assert } from '../util/util.js';
import { kParamKVSeparator } from './separators.js';
export function stringifyPublicParams(p: CaseParams): string[] {
const pub = extractPublicParams(p);
return Object.entries(pub).map(([k, v]) => stringifySingleParam(k, v));
}
export function stringifySingleParam(k: string, v: ParamArgument) {
return `${k}${kParamKVSeparator}${stringifySingleParamValue(v)}`;
}
function stringifySingleParamValue(v: ParamArgument): string {
const s = v === undefined ? 'undefined' : JSON.stringify(v);
assert(
!badParamValueChars.test(s),
`JSON.stringified param value must not match ${badParamValueChars} - was ${s}`
);
return s;
}

2
src/common/framework/query/validQueryPart.ts Normal file
View file

@ -0,0 +1,2 @@
// Applies to group parts, test parts, params keys.
export const validQueryPart = /^[a-zA-Z0-9_]+$/;

52
src/common/framework/test_filter/filter_by_group.ts
View file

@ -1,52 +0,0 @@
import { TestSpecOrTestOrCaseID } from '../id.js';
import { ReadmeFile, TestFileLoader, TestSpec } from '../loader.js';
import { TestFilter } from './internal.js';
import { TestFilterResult } from './test_filter_result.js';
export class FilterByGroup implements TestFilter {
private readonly suite: string;
private readonly specPathPrefix: string;
constructor(suite: string, groupPrefix: string) {
this.suite = suite;
this.specPathPrefix = groupPrefix;
}
matches(id: TestSpecOrTestOrCaseID): boolean {
return id.spec.suite === this.suite && this.pathMatches(id.spec.path);
}
async iterate(loader: TestFileLoader): Promise<TestFilterResult[]> {
const specs = await loader.listing(this.suite);
const entries: TestFilterResult[] = [];
const suite = this.suite;
for (const { path, description } of specs) {
if (this.pathMatches(path)) {
const isReadme = path === '' || path.endsWith('/');
const spec = isReadme
? ({ description } as ReadmeFile)
: ((await loader.import(`${suite}/${path}.spec.js`)) as TestSpec);
entries.push({ id: { suite, path }, spec });
}
}
return entries;
}
definitelyOneFile(): boolean {
// FilterByGroup could always possibly match multiple files, because it represents a prefix,
// e.g. "a:b" not "a:b:".
return false;
}
idIfSingle(): undefined {
// FilterByGroup could be one whole suite, but we only want whole files, tests, or cases.
return undefined;
}
private pathMatches(path: string): boolean {
return path.startsWith(this.specPathPrefix);
}
}

153
src/common/framework/test_filter/filter_one_file.ts
View file

@ -1,153 +0,0 @@
import { TestCaseID, TestSpecID, TestSpecOrTestOrCaseID, testSpecEquals } from '../id.js';
import { TestFileLoader, TestSpec } from '../loader.js';
import { TestSpecRecorder } from '../logger.js';
import { ParamSpec, paramsEquals, paramsSupersets } from '../params_utils.js';
import { RunCase, RunCaseIterable } from '../test_group.js';
import { TestFilter } from './internal.js';
import { TestFilterResult } from './test_filter_result.js';
abstract class FilterOneFile implements TestFilter {
protected readonly specId: TestSpecID;
constructor(specId: TestSpecID) {
this.specId = specId;
}
abstract getCases(spec: TestSpec): RunCaseIterable;
async iterate(loader: TestFileLoader): Promise<TestFilterResult[]> {
const spec = (await loader.import(
`${this.specId.suite}/${this.specId.path}.spec.js`
)) as TestSpec;
return [
{
id: this.specId,
spec: {
description: spec.description,
g: this.getCases(spec),
},
},
];
}
definitelyOneFile(): true {
return true;
}
abstract idIfSingle(): TestSpecOrTestOrCaseID | undefined;
abstract matches(id: TestSpecOrTestOrCaseID): boolean;
}
type TestGroupFilter = (testcase: TestCaseID) => boolean;
function filterTestGroup(group: RunCaseIterable, filter: TestGroupFilter): RunCaseIterable {
return {
*iterate(log: TestSpecRecorder): Iterable<RunCase> {
for (const rc of group.iterate(log)) {
if (filter(rc.id)) {
yield rc;
}
}
},
};
}
export class FilterByTestMatch extends FilterOneFile {
private readonly testPrefix: string;
constructor(specId: TestSpecID, testPrefix: string) {
super(specId);
this.testPrefix = testPrefix;
}
getCases(spec: TestSpec): RunCaseIterable {
return filterTestGroup(spec.g, testcase => this.testMatches(testcase.test));
}
idIfSingle(): TestSpecOrTestOrCaseID | undefined {
if (this.testPrefix.length !== 0) {
return undefined;
}
// This is one whole spec file.
return { spec: this.specId };
}
matches(id: TestSpecOrTestOrCaseID): boolean {
if (id.test === undefined) {
return false;
}
return testSpecEquals(id.spec, this.specId) && this.testMatches(id.test);
}
private testMatches(test: string): boolean {
return test.startsWith(this.testPrefix);
}
}
export class FilterByParamsMatch extends FilterOneFile {
private readonly test: string;
private readonly params: ParamSpec | null;
constructor(specId: TestSpecID, test: string, params: ParamSpec | null) {
super(specId);
this.test = test;
this.params = params;
}
getCases(spec: TestSpec): RunCaseIterable {
return filterTestGroup(spec.g, testcase => this.caseMatches(testcase.test, testcase.params));
}
idIfSingle(): TestSpecOrTestOrCaseID | undefined {
if (this.params !== null) {
return undefined;
}
// This is one whole test.
return { spec: this.specId, test: this.test };
}
matches(id: TestSpecOrTestOrCaseID): boolean {
if (id.test === undefined) {
return false;
}
return testSpecEquals(id.spec, this.specId) && this.caseMatches(id.test, id.params);
}
private caseMatches(test: string, params: ParamSpec | null | undefined): boolean {
if (params === undefined) {
return false;
}
return test === this.test && paramsSupersets(params, this.params);
}
}
export class FilterByParamsExact extends FilterOneFile {
private readonly test: string;
private readonly params: ParamSpec | null;
constructor(specId: TestSpecID, test: string, params: ParamSpec | null) {
super(specId);
this.test = test;
this.params = params;
}
getCases(spec: TestSpec): RunCaseIterable {
return filterTestGroup(spec.g, testcase => this.caseMatches(testcase.test, testcase.params));
}
idIfSingle(): TestSpecOrTestOrCaseID | undefined {
// This is one single test case.
return { spec: this.specId, test: this.test, params: this.params };
}
matches(id: TestSpecOrTestOrCaseID): boolean {
if (id.test === undefined || id.params === undefined) {
return false;
}
return testSpecEquals(id.spec, this.specId) && this.caseMatches(id.test, id.params);
}
private caseMatches(test: string, params: ParamSpec | null): boolean {
return test === this.test && paramsEquals(params, this.params);
}
}

17
src/common/framework/test_filter/internal.ts
View file

@ -1,17 +0,0 @@
import { TestSpecOrTestOrCaseID } from '../id.js';
import { TestFileLoader } from '../loader.js';
import { TestFilterResult } from './test_filter_result.js';
export interface TestFilter {
// Iterates over the test cases matched by the filter.
iterate(loader: TestFileLoader): Promise<TestFilterResult[]>;
// Iff the filter could not possibly match multiple files, returns true.
definitelyOneFile(): boolean;
// If the filter can accept one spec, one test, or one case, returns its ID.
idIfSingle(): TestSpecOrTestOrCaseID | undefined;
matches(id: TestSpecOrTestOrCaseID): boolean;
}

63
src/common/framework/test_filter/load_filter.ts
View file

@ -1,63 +0,0 @@
import { allowedTestNameCharacters } from '../allowed_characters.js';
import { TestFileLoader } from '../loader.js';
import { ParamSpec } from '../params_utils.js';
import { assert, unreachable } from '../util/util.js';
import { FilterByGroup } from './filter_by_group.js';
import { FilterByParamsExact, FilterByParamsMatch, FilterByTestMatch } from './filter_one_file.js';
import { TestFilter } from './internal.js';
import { TestFilterResult } from './test_filter_result.js';
// Each filter is of one of the forms below (urlencoded).
export function makeFilter(filter: string): TestFilter {
const i1 = filter.indexOf(':');
assert(i1 !== -1, 'Test queries must fully specify their suite name (e.g. "webgpu:")');
const suite = filter.substring(0, i1);
const i2 = filter.indexOf(':', i1 + 1);
if (i2 === -1) {
// - webgpu:
// - webgpu:buf
// - webgpu:buffers/
// - webgpu:buffers/map
const groupPrefix = filter.substring(i1 + 1);
return new FilterByGroup(suite, groupPrefix);
}
const path = filter.substring(i1 + 1, i2);
const endOfTestName = new RegExp('[^' + allowedTestNameCharacters + ']');
const i3sub = filter.substring(i2 + 1).search(endOfTestName);
if (i3sub === -1) {
// - webgpu:buffers/mapWriteAsync:
// - webgpu:buffers/mapWriteAsync:b
const testPrefix = filter.substring(i2 + 1);
return new FilterByTestMatch({ suite, path }, testPrefix);
}
const i3 = i2 + 1 + i3sub;
const test = filter.substring(i2 + 1, i3);
const token = filter.charAt(i3);
let params = null;
if (i3 + 1 < filter.length) {
params = JSON.parse(filter.substring(i3 + 1)) as ParamSpec;
}
if (token === '~') {
// - webgpu:buffers/mapWriteAsync:basic~
// - webgpu:buffers/mapWriteAsync:basic~{}
// - webgpu:buffers/mapWriteAsync:basic~{filter:"params"}
return new FilterByParamsMatch({ suite, path }, test, params);
} else if (token === '=') {
// - webgpu:buffers/mapWriteAsync:basic=
// - webgpu:buffers/mapWriteAsync:basic={}
// - webgpu:buffers/mapWriteAsync:basic={exact:"params"}
return new FilterByParamsExact({ suite, path }, test, params);
} else {
unreachable("invalid character after test name; must be '~' or '='");
}
}
export function loadFilter(loader: TestFileLoader, filter: string): Promise<TestFilterResult[]> {
return makeFilter(filter).iterate(loader);
}

8
src/common/framework/test_filter/test_filter_result.ts
View file

@ -1,8 +0,0 @@
import { TestSpecID } from '../id.js';
import { TestSpecOrReadme } from '../loader.js';
// Result of iterating a test filter. Contains a loaded spec (.spec.ts) file and its id.
export interface TestFilterResult {
readonly id: TestSpecID;
readonly spec: TestSpecOrReadme;
}

116
src/common/framework/test_group.ts
View file

@ -1,27 +1,50 @@
import { allowedTestNameCharacters } from './allowed_characters.js';
import { Fixture } from './fixture.js';
import { TestCaseID } from './id.js';
import { LiveTestCaseResult, TestCaseRecorder, TestSpecRecorder } from './logger.js';
import { ParamSpec, ParamSpecIterable, extractPublicParams, paramsEquals } from './params_utils.js';
import { checkPublicParamType } from './url_query.js';
import { TestCaseRecorder } from './logging/test_case_recorder.js';
import {
CaseParams,
CaseParamsIterable,
extractPublicParams,
publicParamsEquals,
} from './params_utils.js';
import { kPathSeparator } from './query/separators.js';
import { stringifySingleParam } from './query/stringify_params.js';
import { validQueryPart } from './query/validQueryPart.js';
import { assert } from './util/util.js';
export type RunFn = (rec: TestCaseRecorder) => Promise<void>;
export interface TestCaseID {
readonly test: readonly string[];
readonly params: CaseParams;
}
export interface RunCase {
readonly id: TestCaseID;
run(debug?: boolean): Promise<LiveTestCaseResult>;
injectResult(result: LiveTestCaseResult): void;
run: RunFn;
}
// Interface for defining tests
export interface TestGroupBuilder<F extends Fixture> {
test(name: string): TestBuilderWithName<F, never>;
}
export function makeTestGroup<F extends Fixture>(fixture: FixtureClass<F>): TestGroupBuilder<F> {
return new TestGroup(fixture);
}
// Interface for running tests
export interface RunCaseIterable {
iterate(rec: TestSpecRecorder): Iterable<RunCase>;
iterate(): Iterable<RunCase>;
}
export function makeTestGroupForUnitTesting<F extends Fixture>(
fixture: FixtureClass<F>
): TestGroup<F> {
return new TestGroup(fixture);
}
type FixtureClass<F extends Fixture> = new (log: TestCaseRecorder, params: ParamSpec) => F;
type FixtureClass<F extends Fixture> = new (log: TestCaseRecorder, params: CaseParams) => F;
type TestFn<F extends Fixture, P extends {}> = (t: F & { params: P }) => Promise<void> | void;
const validNames = new RegExp('^[' + allowedTestNameCharacters + ']+$');
export class TestGroup<F extends Fixture> implements RunCaseIterable {
class TestGroup<F extends Fixture> implements RunCaseIterable, TestGroupBuilder<F> {
private fixture: FixtureClass<F>;
private seen: Set<string> = new Set();
private tests: Array<TestBuilder<F, never>> = [];
@ -30,14 +53,13 @@ export class TestGroup<F extends Fixture> implements RunCaseIterable {
this.fixture = fixture;
}
*iterate(log: TestSpecRecorder): Iterable<RunCase> {
*iterate(): Iterable<RunCase> {
for (const test of this.tests) {
yield* test.iterate(log);
yield* test.iterate();
}
}
private checkName(name: string): void {
assert(validNames.test(name), `Invalid test name ${name}; must match [${validNames}]+`);
assert(
// Shouldn't happen due to the rule above. Just makes sure that treated
// unencoded strings as encoded strings is OK.
@ -51,13 +73,14 @@ export class TestGroup<F extends Fixture> implements RunCaseIterable {
// TODO: This could take a fixture, too, to override the one for the group.
test(name: string): TestBuilderWithName<F, never> {
// Replace spaces with underscores for readability.
assert(name.indexOf('_') === -1, 'Invalid test name ${name}: contains underscore (use space)');
name = name.replace(/ /g, '_');
this.checkName(name);
const test = new TestBuilder<F, never>(name, this.fixture);
const parts = name.split(kPathSeparator);
for (const p of parts) {
assert(validQueryPart.test(p), `Invalid test name part ${p}; must match ${validQueryPart}`);
}
const test = new TestBuilder<F, never>(parts, this.fixture);
this.tests.push(test);
return test;
}
@ -72,13 +95,13 @@ interface TestBuilderWithParams<F extends Fixture, P extends {}> {
}
class TestBuilder<F extends Fixture, P extends {}> {
private readonly name: string;
private readonly testPath: string[];
private readonly fixture: FixtureClass<F>;
private testFn: TestFn<F, P> | undefined;
private cases: ParamSpecIterable | null = null;
private cases?: CaseParamsIterable = undefined;
constructor(name: string, fixture: FixtureClass<F>) {
this.name = name;
constructor(testPath: string[], fixture: FixtureClass<F>) {
this.testPath = testPath;
this.fixture = fixture;
}
@ -86,22 +109,22 @@ class TestBuilder<F extends Fixture, P extends {}> {
this.testFn = fn;
}
params<NewP extends {}>(specs: Iterable<NewP>): TestBuilderWithParams<F, NewP> {
assert(this.cases === null, 'test case is already parameterized');
const cases = Array.from(specs);
const seen: ParamSpec[] = [];
params<NewP extends {}>(casesIterable: Iterable<NewP>): TestBuilderWithParams<F, NewP> {
assert(this.cases === undefined, 'test case is already parameterized');
const cases = Array.from(casesIterable);
const seen: CaseParams[] = [];
// This is n^2.
for (const spec of cases) {
const publicParams = extractPublicParams(spec);
// Check type of public params: can only be (currently):
// number, string, boolean, undefined, number[]
for (const v of Object.values(publicParams)) {
checkPublicParamType(v);
for (const [k, v] of Object.entries(publicParams)) {
stringifySingleParam(k, v); // To check for invalid params values
}
assert(
!seen.some(x => paramsEquals(x, publicParams)),
!seen.some(x => publicParamsEquals(x, publicParams)),
'Duplicate test case params: ' + JSON.stringify(publicParams)
);
seen.push(publicParams);
@ -111,39 +134,35 @@ class TestBuilder<F extends Fixture, P extends {}> {
return (this as unknown) as TestBuilderWithParams<F, NewP>;
}
*iterate(rec: TestSpecRecorder): IterableIterator<RunCase> {
*iterate(): IterableIterator<RunCase> {
assert(this.testFn !== undefined, 'internal error');
for (const params of this.cases || [null]) {
yield new RunCaseSpecific(rec, this.name, params, this.fixture, this.testFn);
for (const params of this.cases || [{}]) {
yield new RunCaseSpecific(this.testPath, params, this.fixture, this.testFn);
}
}
}
class RunCaseSpecific<F extends Fixture> implements RunCase {
readonly id: TestCaseID;
private readonly params: ParamSpec | null;
private readonly recorder: TestSpecRecorder;
private readonly params: CaseParams | null;
private readonly fixture: FixtureClass<F>;
private readonly fn: TestFn<F, never>;
constructor(
recorder: TestSpecRecorder,
test: string,
params: ParamSpec | null,
testPath: string[],
params: CaseParams,
fixture: FixtureClass<F>,
fn: TestFn<F, never>
) {
this.id = { test, params: params ? extractPublicParams(params) : null };
this.id = { test: testPath, params: extractPublicParams(params) };
this.params = params;
this.recorder = recorder;
this.fixture = fixture;
this.fn = fn;
}
async run(debug: boolean): Promise<LiveTestCaseResult> {
const [rec, res] = this.recorder.record(this.id.test, this.id.params);
rec.start(debug);
async run(rec: TestCaseRecorder): Promise<void> {
rec.start();
try {
const inst = new this.fixture(rec, this.params || {});
@ -162,13 +181,6 @@ class RunCaseSpecific<F extends Fixture> implements RunCase {
// or unexpected validation/OOM error from the GPUDevice.
rec.threw(ex);
}
rec.finish();
return res;
}
injectResult(result: LiveTestCaseResult): void {
const [, res] = this.recorder.record(this.id.test, this.id.params);
Object.assign(res, result);
}
}

16
src/common/framework/test_suite_listing.ts Normal file
View file

@ -0,0 +1,16 @@
// A listing of all specs within a single suite. This is the (awaited) type of
// `groups` in '{cts,unittests}/listing.ts' and `listing` in the auto-generated
// 'out/{cts,unittests}/listing.js' files (see tools/gen_listings).
export type TestSuiteListing = Iterable<TestSuiteListingEntry>;
export type TestSuiteListingEntry = TestSuiteListingEntrySpec | TestSuiteListingEntryReadme;
interface TestSuiteListingEntrySpec {
readonly file: string[];
readonly description: string;
}
interface TestSuiteListingEntryReadme {
readonly file: string[];
readonly readme: string;
}

410
src/common/framework/tree.ts
View file

@ -1,82 +1,362 @@
import { Logger } from './logger.js';
import { stringifyPublicParams } from './params_utils.js';
import { TestFilterResult } from './test_filter/test_filter_result.js';
import { RunCase } from './test_group.js';
import { TestFileLoader } from './file_loader.js';
import { TestCaseRecorder } from './logging/test_case_recorder.js';
import { CaseParamsRW } from './params_utils.js';
import { compareQueries, Ordering } from './query/compare.js';
import {
TestQuery,
TestQueryMultiCase,
TestQuerySingleCase,
TestQueryMultiFile,
TestQueryMultiTest,
} from './query/query.js';
import { stringifySingleParam } from './query/stringify_params.js';
import { RunCase, RunFn } from './test_group.js';
import { assert } from './util/util.js';
export interface FilterResultTreeNode {
// `loadTreeForQuery()` loads a TestTree for a given queryToLoad.
// The resulting tree is a linked-list all the way from `suite:*` to queryToLoad,
// and under queryToLoad is a tree containing every case matched by queryToLoad.
//
// `subqueriesToExpand` influences the `collapsible` flag on nodes in the resulting tree.
// A node is considered "collapsible" if none of the subqueriesToExpand is a StrictSubset
// of that node.
//
// In WebKit/Blink-style web_tests, an expectation file marks individual cts.html "variants" as
// "Failure", "Crash", etc.
// By passing in the list of expectations as the subqueriesToExpand, we can programmatically
// subdivide the cts.html "variants" list to be able to implement arbitrarily-fine suppressions
// (instead of having to suppress entire test files, which would lose a lot of coverage).
//
// `iterateCollapsedQueries()` produces the list of queries for the variants list.
//
// Though somewhat complicated, this system has important benefits:
// - Avoids having to suppress entire test files, which would cause large test coverage loss.
// - Minimizes the number of page loads needed for fine-grained suppressions.
// (In the naive case, we could do one page load per test case - but the test suite would
// take impossibly long to run.)
// - Enables developers to put any number of tests in one file as appropriate, without worrying
// about expectation granularity.
export interface TestSubtree<T extends TestQuery = TestQuery> {
readonly query: T;
readonly children: Map<string, TestTreeNode>;
readonly collapsible: boolean;
description?: string;
runCase?: RunCase;
children?: Map<string, FilterResultTreeNode>;
}
// e.g. iteratePath('a/b/c/d', ':') yields ['a/', 'a/b/', 'a/b/c/', 'a/b/c/d:']
function* iteratePath(path: string, terminator: string): IterableIterator<string> {
const parts = path.split('/');
if (parts.length > 1) {
let partial = parts[0] + '/';
yield partial;
for (let i = 1; i < parts.length - 1; ++i) {
partial += parts[i] + '/';
yield partial;
}
// Path ends in '/' (so is a README).
if (parts[parts.length - 1] === '') {
return;
}
}
yield path + terminator;
export interface TestTreeLeaf {
readonly query: TestQuerySingleCase;
readonly run: RunFn;
}
export function treeFromFilterResults(
log: Logger,
listing: IterableIterator<TestFilterResult>
): FilterResultTreeNode {
function getOrInsert(n: FilterResultTreeNode, k: string): FilterResultTreeNode {
const children = n.children!;
if (children.has(k)) {
return children.get(k)!;
}
const v = { children: new Map() };
children.set(k, v);
return v;
export type TestTreeNode = TestSubtree | TestTreeLeaf;
export class TestTree {
readonly root: TestSubtree;
constructor(root: TestSubtree) {
this.root = root;
}
const tree = { children: new Map() };
for (const f of listing) {
const files = getOrInsert(tree, f.id.suite + ':');
if (f.id.path === '') {
// This is a suite README.
files.description = f.spec.description.trim();
continue;
}
iterateCollapsedQueries(): IterableIterator<TestQuery> {
return TestTree.iterateSubtreeCollapsedQueries(this.root);
}
let tests = files;
for (const path of iteratePath(f.id.path, ':')) {
tests = getOrInsert(tests, f.id.suite + ':' + path);
}
if (f.spec.description) {
// This is a directory README or spec file.
tests.description = f.spec.description.trim();
}
iterateLeaves(): IterableIterator<TestTreeLeaf> {
return TestTree.iterateSubtreeLeaves(this.root);
}
if (!('g' in f.spec)) {
// This is a directory README.
continue;
}
toString(): string {
return TestTree.subtreeToString('(root)', this.root, '');
}
const [tRec] = log.record(f.id);
const fId = f.id.suite + ':' + f.id.path;
for (const t of f.spec.g.iterate(tRec)) {
let cases = tests;
for (const path of iteratePath(t.id.test, '~')) {
cases = getOrInsert(cases, fId + ':' + path);
static *iterateSubtreeCollapsedQueries(subtree: TestSubtree): IterableIterator<TestQuery> {
for (const [, child] of subtree.children) {
if ('children' in child && !child.collapsible) {
yield* TestTree.iterateSubtreeCollapsedQueries(child);
} else {
yield child.query;
}
}
}
static *iterateSubtreeLeaves(subtree: TestSubtree): IterableIterator<TestTreeLeaf> {
for (const [, child] of subtree.children) {
if ('children' in child) {
yield* TestTree.iterateSubtreeLeaves(child);
} else {
yield child;
}
}
}
static subtreeToString(name: string, tree: TestTreeNode, indent: string): string {
const collapsible = 'run' in tree ? '>' : tree.collapsible ? '+' : '-';
let s =
indent +
`${collapsible} ${JSON.stringify(name)} => ` +
`${tree.query} ${JSON.stringify(tree.query)}`;
if ('children' in tree) {
if (tree.description !== undefined) {
s += indent + `\n | ${JSON.stringify(tree.description)}`;
}
const p = stringifyPublicParams(t.id.params);
cases.children!.set(fId + ':' + t.id.test + '=' + p, {
runCase: t,
});
for (const [name, child] of tree.children) {
s += '\n' + TestTree.subtreeToString(name, child, indent + ' ');
}
}
return s;
}
}
// TODO: Consider having subqueriesToExpand actually impact the depth-order of params in the tree.
export async function loadTreeForQuery(
loader: TestFileLoader,
queryToLoad: TestQuery,
subqueriesToExpand: TestQuery[]
): Promise<TestTree> {
const suite = queryToLoad.suite;
const specs = await loader.listing(suite);
const subqueriesToExpandEntries = Array.from(subqueriesToExpand.entries());
const seenSubqueriesToExpand: boolean[] = new Array(subqueriesToExpand.length);
seenSubqueriesToExpand.fill(false);
const isCollapsible = (subquery: TestQuery) =>
subqueriesToExpandEntries.every(([i, toExpand]) => {
const ordering = compareQueries(toExpand, subquery);
// If toExpand == subquery, no expansion is needed (but it's still "seen").
if (ordering === Ordering.Equal) seenSubqueriesToExpand[i] = true;
return ordering !== Ordering.StrictSubset;
});
// L0 = suite-level, e.g. suite:*
// L1 = file-level, e.g. suite:a,b:*
// L2 = test-level, e.g. suite:a,b:c,d:*
// L3 = case-level, e.g. suite:a,b:c,d:
let foundCase = false;
// L0 is suite:*
const subtreeL0 = makeTreeForSuite(suite);
isCollapsible(subtreeL0.query); // mark seenSubqueriesToExpand
for (const entry of specs) {
if (entry.file.length === 0 && 'readme' in entry) {
// Suite-level readme.
assert(subtreeL0.description === undefined);
subtreeL0.description = entry.readme.trim();
continue;
}
{
const queryL1 = new TestQueryMultiFile(suite, entry.file);
const orderingL1 = compareQueries(queryL1, queryToLoad);
if (orderingL1 === Ordering.Unordered) {
// File path is not matched by this query.
continue;
}
}
if ('readme' in entry) {
// Entry is a README that is an ancestor or descendant of the query.
// (It's included for display in the standalone runner.)
// readmeSubtree is suite:a,b,*
// (This is always going to dedup with a file path, if there are any test spec files under
// the directory that has the README).
const readmeSubtree: TestSubtree<TestQueryMultiFile> = addSubtreeForDirPath(
subtreeL0,
entry.file
);
assert(readmeSubtree.description === undefined);
readmeSubtree.description = entry.readme.trim();
continue;
}
// Entry is a spec file.
const spec = await loader.importSpecFile(queryToLoad.suite, entry.file);
const description = spec.description.trim();
// subtreeL1 is suite:a,b:*
const subtreeL1: TestSubtree<TestQueryMultiTest> = addSubtreeForFilePath(
subtreeL0,
entry.file,
description,
isCollapsible
);
// TODO: If tree generation gets too slow, avoid actually iterating the cases in a file
// if there's no need to (based on the subqueriesToExpand).
for (const t of spec.g.iterate()) {
{
const queryL3 = new TestQuerySingleCase(suite, entry.file, t.id.test, t.id.params);
const orderingL3 = compareQueries(queryL3, queryToLoad);
if (orderingL3 === Ordering.Unordered || orderingL3 === Ordering.StrictSuperset) {
// Case is not matched by this query.
continue;
}
}
// subtreeL2 is suite:a,b:c,d:*
const subtreeL2: TestSubtree<TestQueryMultiCase> = addSubtreeForTestPath(
subtreeL1,
t.id.test,
isCollapsible
);
// Leaf for case is suite:a,b:c,d:x=1;y=2
addLeafForCase(subtreeL2, t, isCollapsible);
foundCase = true;
}
}
const tree = new TestTree(subtreeL0);
for (const [i, sq] of subqueriesToExpandEntries) {
const seen = seenSubqueriesToExpand[i];
assert(
seen,
`subqueriesToExpand entry did not match anything \
(can happen due to overlap with another subquery): ${sq.toString()}`
);
}
assert(foundCase, 'Query does not match any cases');
// TODO: Contains lots of single-child subtrees. Consider cleaning those up (as postprocess?).
return tree;
}
function makeTreeForSuite(suite: string): TestSubtree<TestQueryMultiFile> {
return {
query: new TestQueryMultiFile(suite, []),
children: new Map(),
collapsible: false,
};
}
function addSubtreeForDirPath(
tree: TestSubtree<TestQueryMultiFile>,
file: string[]
): TestSubtree<TestQueryMultiFile> {
const subqueryFile: string[] = [];
// To start, tree is suite:*
// This loop goes from that -> suite:a,* -> suite:a,b,*
for (const part of file) {
subqueryFile.push(part);
tree = getOrInsertSubtree(part, tree, () => {
const query = new TestQueryMultiFile(tree.query.suite, subqueryFile);
return { query, collapsible: false };
});
}
return tree;
}
function addSubtreeForFilePath(
tree: TestSubtree<TestQueryMultiFile>,
file: string[],
description: string,
checkCollapsible: (sq: TestQuery) => boolean
): TestSubtree<TestQueryMultiTest> {
// To start, tree is suite:*
// This goes from that -> suite:a,* -> suite:a,b,*
tree = addSubtreeForDirPath(tree, file);
// This goes from that -> suite:a,b:*
const subtree = getOrInsertSubtree('', tree, () => {
const query = new TestQueryMultiTest(tree.query.suite, tree.query.filePathParts, []);
return { query, description, collapsible: checkCollapsible(query) };
});
return subtree;
}
function addSubtreeForTestPath(
tree: TestSubtree<TestQueryMultiTest>,
test: readonly string[],
isCollapsible: (sq: TestQuery) => boolean
): TestSubtree<TestQueryMultiCase> {
const subqueryTest: string[] = [];
// To start, tree is suite:a,b:*
// This loop goes from that -> suite:a,b:c,* -> suite:a,b:c,d,*
for (const part of test) {
subqueryTest.push(part);
tree = getOrInsertSubtree(part, tree, () => {
const query = new TestQueryMultiTest(
tree.query.suite,
tree.query.filePathParts,
subqueryTest
);
return { query, collapsible: isCollapsible(query) };
});
}
// This goes from that -> suite:a,b:c,d:*
return getOrInsertSubtree('', tree, () => {
const query = new TestQueryMultiCase(
tree.query.suite,
tree.query.filePathParts,
subqueryTest,
{}
);
return { query, collapsible: isCollapsible(query) };
});
}
function addLeafForCase(
tree: TestSubtree<TestQueryMultiTest>,
t: RunCase,
checkCollapsible: (sq: TestQuery) => boolean
): void {
const query = tree.query;
let name: string = '';
const subqueryParams: CaseParamsRW = {};
// To start, tree is suite:a,b:c,d:*
// This loop goes from that -> suite:a,b:c,d:x=1;* -> suite:a,b:c,d:x=1;y=2;*
for (const [k, v] of Object.entries(t.id.params)) {
name = stringifySingleParam(k, v);
subqueryParams[k] = v;
tree = getOrInsertSubtree(name, tree, () => {
const subquery = new TestQueryMultiCase(
query.suite,
query.filePathParts,
query.testPathParts,
subqueryParams
);
return { query: subquery, collapsible: checkCollapsible(subquery) };
});
}
// This goes from that -> suite:a,b:c,d:x=1;y=2
const subquery = new TestQuerySingleCase(
query.suite,
query.filePathParts,
query.testPathParts,
subqueryParams
);
checkCollapsible(subquery); // mark seenSubqueriesToExpand
insertLeaf(tree, subquery, t);
}
function getOrInsertSubtree<T extends TestQuery>(
key: string,
parent: TestSubtree,
createSubtree: () => Omit<TestSubtree<T>, 'children'>
): TestSubtree<T> {
let v: TestSubtree<T>;
const child = parent.children.get(key);
if (child !== undefined) {
assert('children' in child); // Make sure cached subtree is not actually a leaf
v = child as TestSubtree<T>;
} else {
v = { ...createSubtree(), children: new Map() };
parent.children.set(key, v);
}
return v;
}
function insertLeaf(parent: TestSubtree, query: TestQuerySingleCase, t: RunCase) {
const key = '';
const leaf: TestTreeLeaf = {
query,
run: (rec: TestCaseRecorder) => t.run(rec),
};
assert(!parent.children.has(key));
parent.children.set(key, leaf);
}

42
src/common/framework/url_query.ts
View file

@ -1,42 +0,0 @@
import { TestCaseID, TestSpecID } from './id.js';
import { ParamArgument, stringifyPublicParams } from './params_utils.js';
import { unreachable } from './util/util.js';
export function encodeSelectively(s: string): string {
let ret = encodeURIComponent(s);
ret = ret.replace(/%22/g, '"');
ret = ret.replace(/%2C/g, ',');
ret = ret.replace(/%2F/g, '/');
ret = ret.replace(/%3A/g, ':');
ret = ret.replace(/%3D/g, '=');
ret = ret.replace(/%5B/g, '[');
ret = ret.replace(/%5D/g, ']');
ret = ret.replace(/%7B/g, '{');
ret = ret.replace(/%7D/g, '}');
return ret;
}
export function checkPublicParamType(v: ParamArgument): void {
if (typeof v === 'number' || typeof v === 'string' || typeof v === 'boolean' || v === undefined) {
return;
}
if (v instanceof Array) {
for (const x of v) {
if (typeof x !== 'number') {
break;
}
}
return;
}
unreachable('Invalid type for test case params ' + v);
}
export function makeQueryString(spec: TestSpecID, testcase?: TestCaseID): string {
let s = spec.suite + ':';
s += spec.path + ':';
if (testcase !== undefined) {
s += testcase.test + '=';
s += stringifyPublicParams(testcase.params);
}
return encodeSelectively(s);
}

2
src/common/framework/collect_garbage.ts → src/common/framework/util/collect_garbage.ts
View file

@ -1,4 +1,4 @@
import { resolveOnTimeout } from './util/util.js';
import { resolveOnTimeout } from './util.js';
/* eslint-disable-next-line @typescript-eslint/no-explicit-any */
declare const Components: any;

34
src/common/framework/util/stack.ts
View file

@ -1,33 +1,17 @@
// Takes a stack trace, and extracts only the first continuous range of lines
// containing '/(webgpu|unittests)/', which should provide only the useful part
// of the stack to the caller (for logging).
export function getStackTrace(e: Error): string {
// Returns the stack trace of an Error, but without the extra boilerplate at the bottom
// (e.g. RunCaseSpecific, processTicksAndRejections, etc.), for logging.
export function extractImportantStackTrace(e: Error): string {
if (!e.stack) {
return '';
}
const parts = e.stack.split('\n');
const stack = [];
const moreStack = [];
let found = false;
const commonRegex = /[/\\](webgpu|unittests)[/\\]/;
for (let i = 0; i < parts.length; ++i) {
const part = parts[i].trim();
const isSuites = commonRegex.test(part); // approximate
if (found && !isSuites) {
moreStack.push(part);
}
if (isSuites) {
if (moreStack.length) {
stack.push(...moreStack);
moreStack.length = 0;
}
stack.push(part);
found = true;
const lines = e.stack.split('\n');
for (let i = lines.length - 1; i >= 0; --i) {
const line = lines[i];
if (line.indexOf('.spec.') !== -1) {
return lines.slice(0, i + 1).join('\n');
}
}
return stack.join('\n');
return e.stack;
}
// *** Examples ***

77
src/common/runtime/cmdline.ts
View file

@ -4,10 +4,9 @@
import * as fs from 'fs';
import * as process from 'process';
import { TestSpecID } from '../framework/id.js';
import { TestLoader } from '../framework/loader.js';
import { LiveTestCaseResult, Logger } from '../framework/logger.js';
import { makeQueryString } from '../framework/url_query.js';
import { DefaultTestFileLoader } from '../framework/file_loader.js';
import { Logger } from '../framework/logging/logger.js';
import { LiveTestCaseResult } from '../framework/logging/result.js';
import { assert, unreachable } from '../framework/util/util.js';
function usage(rc: number): never {
@ -29,7 +28,7 @@ if (!fs.existsSync('src/common/runtime/cmdline.ts')) {
let verbose = false;
let debug = false;
let printJSON = false;
const filterArgs = [];
const filterArgs: string[] = [];
for (const a of process.argv.slice(2)) {
if (a.startsWith('-')) {
if (a === '--verbose') {
@ -52,44 +51,42 @@ if (filterArgs.length === 0) {
(async () => {
try {
const loader = new TestLoader();
const files = await loader.loadTestsFromCmdLine(filterArgs);
const loader = new DefaultTestFileLoader();
assert(filterArgs.length === 1, 'currently, there must be exactly one query on the cmd line');
const testcases = await loader.loadTests(filterArgs[0]);
const log = new Logger();
const log = new Logger(debug);
const failed: Array<[TestSpecID, LiveTestCaseResult]> = [];
const warned: Array<[TestSpecID, LiveTestCaseResult]> = [];
const skipped: Array<[TestSpecID, LiveTestCaseResult]> = [];
const failed: Array<[string, LiveTestCaseResult]> = [];
const warned: Array<[string, LiveTestCaseResult]> = [];
const skipped: Array<[string, LiveTestCaseResult]> = [];
let total = 0;
for (const f of files) {
if (!('g' in f.spec)) {
continue;
for (const testcase of testcases) {
const name = testcase.query.toString();
const [rec, res] = log.record(name);
await testcase.run(rec);
if (verbose) {
printResults([[name, res]]);
}
const [rec] = log.record(f.id);
for (const t of f.spec.g.iterate(rec)) {
const res = await t.run(debug);
if (verbose) {
printResults([[f.id, res]]);
}
total++;
switch (res.status) {
case 'pass':
break;
case 'fail':
failed.push([f.id, res]);
break;
case 'warn':
warned.push([f.id, res]);
break;
case 'skip':
skipped.push([f.id, res]);
break;
default:
unreachable('unrecognized status');
}
total++;
switch (res.status) {
case 'pass':
break;
case 'fail':
failed.push([name, res]);
break;
case 'warn':
warned.push([name, res]);
break;
case 'skip':
skipped.push([name, res]);
break;
default:
unreachable('unrecognized status');
}
}
@ -138,9 +135,9 @@ Failed = ${rpt(failed.length)}`);
}
})();
function printResults(results: Array<[TestSpecID, LiveTestCaseResult]>): void {
for (const [id, r] of results) {
console.log(`[${r.status}] ${makeQueryString(id, r)} (${r.timems}ms). Log:`);
function printResults(results: Array<[string, LiveTestCaseResult]>): void {
for (const [name, r] of results) {
console.log(`[${r.status}] ${name} (${r.timems}ms). Log:`);
if (r.logs) {
for (const l of r.logs) {
console.log(' - ' + l.toJSON().replace(/\n/g, '\n '));

26
src/common/runtime/helper/test_worker-worker.ts
View file

@ -1,28 +1,24 @@
import { TestLoader } from '../../framework/loader.js';
import { Logger } from '../../framework/logger.js';
import { DefaultTestFileLoader } from '../../framework/file_loader.js';
import { Logger } from '../../framework/logging/logger.js';
import { assert } from '../../framework/util/util.js';
/* eslint-disable-next-line @typescript-eslint/no-explicit-any */
declare const self: any; // should be DedicatedWorkerGlobalScope
const log = new Logger();
const loader = new TestLoader();
const loader = new DefaultTestFileLoader();
self.onmessage = async (ev: MessageEvent) => {
const { query, debug } = ev.data;
const query: string = ev.data.query;
const debug: boolean = ev.data.debug;
const files = Array.from(await loader.loadTests([query]));
assert(files.length === 1, 'worker query resulted in != 1 files');
const log = new Logger(debug);
const f = files[0];
const [rec] = log.record(f.id);
assert('g' in f.spec, 'worker query resulted in README');
const testcases = Array.from(await loader.loadTests(query));
assert(testcases.length === 1, 'worker query resulted in != 1 cases');
const cases = Array.from(f.spec.g.iterate(rec));
assert(cases.length === 1, 'worker query resulted in != 1 cases');
const c = cases[0];
const result = await c.run(debug);
const testcase = testcases[0];
const [rec, result] = log.record(testcase.query.toString());
await testcase.run(rec);
self.postMessage({ query, result });
};

24
src/common/runtime/helper/test_worker.ts
View file

@ -1,14 +1,15 @@
import {
LiveTestCaseResult,
LogMessageWithStack,
TransferredTestCaseResult,
} from '../../framework/logger.js';
import { LogMessageWithStack } from '../../framework/logging/log_message.js';
import { TransferredTestCaseResult, LiveTestCaseResult } from '../../framework/logging/result.js';
import { TestCaseRecorder } from '../../framework/logging/test_case_recorder.js';
export class TestWorker {
private worker: Worker;
private resolvers = new Map<string, (result: LiveTestCaseResult) => void>();
private readonly debug: boolean;
private readonly worker: Worker;
private readonly resolvers = new Map<string, (result: LiveTestCaseResult) => void>();
constructor(debug: boolean) {
this.debug = debug;
constructor() {
const selfPath = import.meta.url;
const selfPathDir = selfPath.substring(0, selfPath.lastIndexOf('/'));
const workerPath = selfPathDir + '/test_worker-worker.js';
@ -28,10 +29,11 @@ export class TestWorker {
};
}
run(query: string, debug: boolean = false): Promise<LiveTestCaseResult> {
this.worker.postMessage({ query, debug });
return new Promise(resolve => {
async run(rec: TestCaseRecorder, query: string): Promise<void> {
this.worker.postMessage({ query, debug: this.debug });
const workerResult = await new Promise<LiveTestCaseResult>(resolve => {
this.resolvers.set(query, resolve);
});
rec.injectResult(workerResult);
}
}

104
src/common/runtime/standalone.ts
View file

@ -1,10 +1,10 @@
// Implements the standalone test runner (see also: /standalone/index.html).
import { TestLoader } from '../framework/loader.js';
import { LiveTestCaseResult, Logger } from '../framework/logger.js';
import { RunCase } from '../framework/test_group.js';
import { FilterResultTreeNode, treeFromFilterResults } from '../framework/tree.js';
import { encodeSelectively } from '../framework/url_query.js';
import { DefaultTestFileLoader } from '../framework/file_loader.js';
import { Logger } from '../framework/logging/logger.js';
import { TestQuery } from '../framework/query/query.js';
import { TestTreeNode, TestSubtree, TestTreeLeaf } from '../framework/tree.js';
import { assert } from '../framework/util/util.js';
import { optionEnabled } from './helper/options.js';
import { TestWorker } from './helper/test_worker.js';
@ -15,12 +15,13 @@ window.onbeforeunload = () => {
};
let haveSomeResults = false;
const log = new Logger();
const runnow = optionEnabled('runnow');
const debug = optionEnabled('debug');
const worker = optionEnabled('worker') ? new TestWorker() : undefined;
const logger = new Logger(debug);
const worker = optionEnabled('worker') ? new TestWorker(debug) : undefined;
const resultsVis = document.getElementById('resultsVis')!;
const resultsJSON = document.getElementById('resultsJSON')!;
@ -29,25 +30,25 @@ type RunSubtree = () => Promise<void>;
// DOM generation
function makeTreeNodeHTML(name: string, tree: FilterResultTreeNode): [HTMLElement, RunSubtree] {
if (tree.children) {
return makeSubtreeHTML(name, tree);
function makeTreeNodeHTML(tree: TestTreeNode): [HTMLElement, RunSubtree] {
if ('children' in tree) {
return makeSubtreeHTML(tree);
} else {
return makeCaseHTML(name, tree.runCase!);
return makeCaseHTML(tree);
}
}
function makeCaseHTML(name: string, t: RunCase): [HTMLElement, RunSubtree] {
function makeCaseHTML(t: TestTreeLeaf): [HTMLElement, RunSubtree] {
const div = $('<div>').addClass('testcase');
const name = t.query.toString();
const runSubtree = async () => {
haveSomeResults = true;
let res: LiveTestCaseResult;
const [rec, res] = logger.record(name);
if (worker) {
res = await worker.run(name, debug);
t.injectResult(res);
await worker.run(rec, name);
} else {
res = await t.run(debug);
await t.run(rec);
}
casetime.text(res.timems.toFixed(4) + ' ms');
@ -72,7 +73,7 @@ function makeCaseHTML(name: string, t: RunCase): [HTMLElement, RunSubtree] {
}
};
const casehead = makeTreeNodeHeaderHTML(name, undefined, runSubtree);
const casehead = makeTreeNodeHeaderHTML(t.query, undefined, runSubtree, true);
div.append(casehead);
const casetime = $('<div>').addClass('testcasetime').html('ms').appendTo(casehead);
const caselogs = $('<div>').addClass('testcaselogs').appendTo(div);
@ -80,27 +81,22 @@ function makeCaseHTML(name: string, t: RunCase): [HTMLElement, RunSubtree] {
return [div[0], runSubtree];
}
function makeSubtreeHTML(name: string, subtree: FilterResultTreeNode): [HTMLElement, RunSubtree] {
function makeSubtreeHTML(t: TestSubtree): [HTMLElement, RunSubtree] {
const div = $('<div>').addClass('subtree');
const header = makeTreeNodeHeaderHTML(name, subtree.description, () => {
return runSubtree();
});
const header = makeTreeNodeHeaderHTML(t.query, t.description, () => runSubtree(), false);
div.append(header);
const subtreeHTML = $('<div>').addClass('subtreechildren').appendTo(div);
const runSubtree = makeSubtreeChildrenHTML(subtreeHTML[0], subtree.children!);
const runSubtree = makeSubtreeChildrenHTML(subtreeHTML[0], t.children.values());
return [div[0], runSubtree];
}
function makeSubtreeChildrenHTML(
div: HTMLElement,
children: Map<string, FilterResultTreeNode>
): RunSubtree {
function makeSubtreeChildrenHTML(div: HTMLElement, children: Iterable<TestTreeNode>): RunSubtree {
const runSubtreeFns: RunSubtree[] = [];
for (const [name, subtree] of children) {
const [subtreeHTML, runSubtree] = makeTreeNodeHTML(name, subtree);
for (const subtree of children) {
const [subtreeHTML, runSubtree] = makeTreeNodeHTML(subtree);
div.append(subtreeHTML);
runSubtreeFns.push(runSubtree);
}
@ -113,24 +109,16 @@ function makeSubtreeChildrenHTML(
}
function makeTreeNodeHeaderHTML(
name: string,
query: TestQuery,
description: string | undefined,
runSubtree: RunSubtree
runSubtree: RunSubtree,
isLeaf: boolean
): HTMLElement {
const div = $('<div>').addClass('nodeheader');
const nameEncoded = encodeSelectively(name);
let nameHTML;
{
const i = nameEncoded.indexOf('{');
const n1 = i === -1 ? nameEncoded : nameEncoded.slice(0, i + 1);
const n2 = i === -1 ? '' : nameEncoded.slice(i + 1);
nameHTML = n1.replace(/:/g, ':<wbr>') + '<wbr>' + n2.replace(/,/g, ',<wbr>');
}
const href = `?${worker ? 'worker&' : ''}${debug ? 'debug&' : ''}q=${nameEncoded}`;
const href = `?${worker ? 'worker&' : ''}${debug ? 'debug&' : ''}q=${query.toString()}`;
$('<button>')
.addClass('noderun')
.addClass(isLeaf ? 'leafrun' : 'subtreerun')
.attr('alt', 'Run subtree')
.attr('title', 'Run subtree')
.on('click', async () => {
@ -145,7 +133,7 @@ function makeTreeNodeHeaderHTML(
.attr('title', 'Open')
.appendTo(div);
const nodetitle = $('<div>').addClass('nodetitle').appendTo(div);
$('<span>').addClass('nodename').html(nameHTML).appendTo(nodetitle);
$('<span>').addClass('nodename').html(query.toHTML()).appendTo(nodetitle);
if (description) {
$('<div>')
.addClass('nodedescription')
@ -156,18 +144,38 @@ function makeTreeNodeHeaderHTML(
}
function updateJSON(): void {
resultsJSON.textContent = log.asJSON(2);
resultsJSON.textContent = logger.asJSON(2);
}
(async () => {
const loader = new TestLoader();
const loader = new DefaultTestFileLoader();
// TODO: everything after this point is very similar across the three runtimes.
// TODO: start populating page before waiting for everything to load?
const files = await loader.loadTestsFromQuery(window.location.search);
const tree = treeFromFilterResults(log, files);
const qs = new URLSearchParams(window.location.search).getAll('q');
if (qs.length === 0) {
qs.push('webgpu:*');
}
const runSubtree = makeSubtreeChildrenHTML(resultsVis, tree.children!);
// Update the URL bar to match the exact current options.
{
let url = window.location.protocol + '//' + window.location.host + window.location.pathname;
url +=
'?' +
new URLSearchParams([
['runnow', runnow ? '1' : '0'],
['worker', worker ? '1' : '0'],
['debug', debug ? '1' : '0'],
]).toString() +
'&' +
qs.map(q => 'q=' + q).join('&');
window.history.replaceState(null, '', url);
}
assert(qs.length === 1, 'currently, there must be exactly one ?q=');
const tree = await loader.loadTree(qs[0]);
const [el, runSubtree] = makeSubtreeHTML(tree.root);
resultsVis.append(el);
if (runnow) {
runSubtree();

93
src/common/runtime/wpt.ts
View file

@ -1,7 +1,8 @@
import { TestLoader } from '../framework/loader.js';
import { LiveTestCaseResult, Logger } from '../framework/logger.js';
import { makeQueryString } from '../framework/url_query.js';
import { DefaultTestFileLoader } from '../framework/file_loader.js';
import { Logger } from '../framework/logging/logger.js';
import { TestTreeLeaf } from '../framework/tree.js';
import { AsyncMutex } from '../framework/util/async_mutex.js';
import { assert } from '../framework/util/util.js';
import { optionEnabled } from './helper/options.js';
import { TestWorker } from './helper/test_worker.js';
@ -15,51 +16,51 @@ declare interface WptTestObject {
declare function async_test(f: (this: WptTestObject) => Promise<void>, name: string): void;
(async () => {
const loader = new TestLoader();
const files = await loader.loadTestsFromQuery(window.location.search);
const loader = new DefaultTestFileLoader();
const qs = new URLSearchParams(window.location.search).getAll('q');
assert(qs.length === 1, 'currently, there must be exactly one ?q=');
const testcases = await loader.loadTests(qs[0]);
const worker = optionEnabled('worker') ? new TestWorker() : undefined;
const log = await addWPTTests(testcases);
const log = new Logger();
const mutex = new AsyncMutex();
const running: Array<Promise<void>> = [];
for (const f of files) {
if (!('g' in f.spec)) {
continue;
}
const [rec] = log.record(f.id);
for (const t of f.spec.g.iterate(rec)) {
const name = makeQueryString(f.id, t.id);
// Note: apparently, async_tests must ALL be added within the same task.
async_test(function (this: WptTestObject): Promise<void> {
const p = mutex.with(async () => {
let r: LiveTestCaseResult;
if (worker) {
r = await worker.run(name);
t.injectResult(r);
} else {
r = await t.run();
}
this.step(() => {
// Unfortunately, it seems not possible to surface any logs for warn/skip.
if (r.status === 'fail') {
throw (r.logs || []).map(s => s.toJSON()).join('\n\n');
}
});
this.done();
});
running.push(p);
return p;
}, name);
}
}
await Promise.all(running);
const resultsElem = document.getElementById('results') as HTMLElement;
resultsElem.textContent = log.asJSON(2);
})();
// Note: async_tests must ALL be added within the same task. This function *must not* be async.
function addWPTTests(testcases: IterableIterator<TestTreeLeaf>): Promise<Logger> {
const worker = optionEnabled('worker') ? new TestWorker(false) : undefined;
const log = new Logger(false);
const mutex = new AsyncMutex();
const running: Array<Promise<void>> = [];
for (const testcase of testcases) {
const name = testcase.query.toString();
const wpt_fn = function (this: WptTestObject): Promise<void> {
const p = mutex.with(async () => {
const [rec, res] = log.record(name);
if (worker) {
await worker.run(rec, name);
} else {
await testcase.run(rec);
}
this.step(() => {
// Unfortunately, it seems not possible to surface any logs for warn/skip.
if (res.status === 'fail') {
throw (res.logs || []).map(s => s.toJSON()).join('\n\n');
}
});
this.done();
});
running.push(p);
return p;
};
async_test(wpt_fn, name);
}
return Promise.all(running).then(() => log);
}

59
src/common/tools/crawl.ts
View file

@ -5,54 +5,53 @@
import * as fs from 'fs';
import * as path from 'path';
import { TestSuiteListingEntry } from '../framework/listing.js';
import { TestSpec } from '../framework/loader.js';
import { assert } from '../framework/util/util.js';
import { SpecFile } from '../framework/file_loader.js';
import { TestSuiteListingEntry, TestSuiteListing } from '../framework/test_suite_listing.js';
import { assert, unreachable } from '../framework/util/util.js';
const fg = require('fast-glob');
const specSuffix = '.spec.ts';
const specFileSuffix = '.spec.ts';
export async function crawl(suite: string): Promise<TestSuiteListingEntry[]> {
const specDir = 'src/' + suite;
if (!fs.existsSync(specDir)) {
console.error(`Could not find ${specDir}`);
const suiteDir = 'src/' + suite;
if (!fs.existsSync(suiteDir)) {
console.error(`Could not find ${suiteDir}`);
process.exit(1);
}
const specFiles = await fg(`${specDir}/**/{README.txt,*${specSuffix}}`, { onlyFiles: true });
specFiles.sort();
const glob = `${suiteDir}/**/{README.txt,*${specFileSuffix}}`;
const filesToEnumerate: string[] = await fg(glob, { onlyFiles: true });
filesToEnumerate.sort();
const groups: TestSuiteListingEntry[] = [];
for (const file of specFiles) {
const f = file.substring((specDir + '/').length);
if (f.endsWith(specSuffix)) {
const testPath = f.substring(0, f.length - specSuffix.length);
const filename = `../../../${specDir}/${testPath}.spec.js`;
const mod = (await import(filename)) as TestSpec;
const entries: TestSuiteListingEntry[] = [];
for (const file of filesToEnumerate) {
const f = file.substring((suiteDir + '/').length); // Suite-relative file path
if (f.endsWith(specFileSuffix)) {
const filepathWithoutExtension = f.substring(0, f.length - specFileSuffix.length);
const filename = `../../../${suiteDir}/${filepathWithoutExtension}.spec.js`;
const mod = (await import(filename)) as SpecFile;
assert(mod.description !== undefined, 'Test spec file missing description: ' + filename);
assert(mod.g !== undefined, 'Test spec file missing TestGroup definition: ' + filename);
groups.push({
path: testPath,
description: mod.description.trim(),
});
const path = filepathWithoutExtension.split('/');
entries.push({ file: path, description: mod.description.trim() });
} else if (path.basename(file) === 'README.txt') {
const group = f.substring(0, f.length - 'README.txt'.length);
const description = fs.readFileSync(file, 'utf8').trim();
groups.push({
path: group,
description,
});
const filepathWithoutExtension = f.substring(0, f.length - '/README.txt'.length);
const readme = fs.readFileSync(file, 'utf8').trim();
const path = filepathWithoutExtension ? filepathWithoutExtension.split('/') : [];
entries.push({ file: path, readme });
} else {
console.error('Unrecognized file: ' + file);
process.exit(1);
unreachable(`glob ${glob} matched an unrecognized filename ${file}`);
}
}
return groups;
return entries;
}
export function makeListing(filename: string): Promise<TestSuiteListingEntry[]> {
export function makeListing(filename: string): Promise<TestSuiteListing> {
const suite = path.basename(path.dirname(filename));
return crawl(suite);
}

28
src/common/tools/gen_wpt_cts_html.ts
View file

@ -1,9 +1,10 @@
import { promises as fs } from 'fs';
import { listing } from '../../webgpu/listing.js';
import { generateMinimalQueryList } from '../framework/generate_minimal_query_list.js';
import { TestSuiteListingEntry } from '../framework/listing.js';
import { TestLoader } from '../framework/loader.js';
import { DefaultTestFileLoader } from '../framework/file_loader.js';
import { TestQueryMultiTest } from '../framework/query/query.js';
import { kBigSeparator, kWildcard } from '../framework/query/separators.js';
import { TestSuiteListingEntry } from '../framework/test_suite_listing.js';
function printUsageAndExit(rc: number): void {
console.error(`\
@ -49,8 +50,8 @@ const [
const entries = (await listing) as TestSuiteListingEntry[];
const lines = entries
// Exclude READMEs.
.filter(l => l.path.length !== 0 && !l.path.endsWith('/'))
.map(l => '?q=webgpu:' + l.path);
.filter(l => !('readme' in l))
.map(l => '?q=' + new TestQueryMultiTest('webgpu', l.file, []).toString());
await generateFile(lines);
} else {
// Prefixes sorted from longest to shortest
@ -79,14 +80,17 @@ const [
throw new Error('All input lines must start with one of the prefixes. ' + exp);
}
const loader = new TestLoader();
const files = Array.from(await loader.loadTestsFromCmdLine([suite + ':']));
const lines = [];
const loader = new DefaultTestFileLoader();
const lines: Array<string | undefined> = [];
for (const prefix of argsPrefixes) {
lines.push(undefined);
for (const q of await generateMinimalQueryList(files, expectations.get(prefix)!)) {
lines.push(prefix + q);
const tree = await loader.loadTree(
suite + kBigSeparator + kWildcard,
expectations.get(prefix)!
);
lines.push(undefined); // output blank line between prefixes
for (const q of tree.iterateCollapsedQueries()) {
lines.push(prefix + q.toString());
}
}
await generateFile(lines);

8
src/unittests/async_mutex.spec.ts
View file

@ -2,13 +2,13 @@ export const description = `
Tests for AsyncMutex.
`;
import { TestGroup } from '../common/framework/test_group.js';
import { makeTestGroup } from '../common/framework/test_group.js';
import { AsyncMutex } from '../common/framework/util/async_mutex.js';
import { objectEquals } from '../common/framework/util/util.js';
import { UnitTest } from './unit_test.js';
export const g = new TestGroup(UnitTest);
export const g = makeTestGroup(UnitTest);
/* eslint-disable-next-line @typescript-eslint/no-unused-vars */
g.test('basic').fn(async t => {
@ -34,7 +34,7 @@ g.test('parallel').fn(async t => {
]);
});
g.test('parallel/many').fn(async t => {
g.test('parallel,many').fn(async t => {
const mutex = new AsyncMutex();
const actual: number[] = [];
const expected = [];
@ -54,7 +54,7 @@ g.test('return').fn(async t => {
t.expect(ret === 123);
});
g.test('return/parallel').fn(async t => {
g.test('return,parallel').fn(async t => {
const mutex = new AsyncMutex();
const ret = await Promise.all([
mutex.with(async () => 1),

14
src/unittests/basic.spec.ts
View file

@ -2,31 +2,31 @@ export const description = `
Basic unit tests for test framework.
`;
import { TestGroup } from '../common/framework/test_group.js';
import { makeTestGroup } from '../common/framework/test_group.js';
import { UnitTest } from './unit_test.js';
export const g = new TestGroup(UnitTest);
export const g = makeTestGroup(UnitTest);
/* eslint-disable-next-line @typescript-eslint/no-unused-vars */
g.test('test/sync').fn(t => {});
g.test('test,sync').fn(t => {});
/* eslint-disable-next-line @typescript-eslint/no-unused-vars */
g.test('test/async').fn(async t => {});
g.test('test,async').fn(async t => {});
g.test('testp/sync')
g.test('test_with_params,sync')
.params([{}])
.fn(t => {
t.debug(JSON.stringify(t.params));
});
g.test('testp/async')
g.test('test_with_params,async')
.params([{}])
.fn(async t => {
t.debug(JSON.stringify(t.params));
});
g.test('testp/private')
g.test('test_with_params,private_params')
.params([
{ a: 1, b: 2, _result: 3 }, //
{ a: 4, b: -3, _result: 1 },

36
src/unittests/getStackTrace.spec.ts
View file

@ -2,18 +2,18 @@ export const description = `
Tests for getStackTrace.
`;
import { TestGroup } from '../common/framework/test_group.js';
import { getStackTrace } from '../common/framework/util/stack.js';
import { makeTestGroup } from '../common/framework/test_group.js';
import { extractImportantStackTrace } from '../common/framework/util/stack.js';
import { UnitTest } from './unit_test.js';
export const g = new TestGroup(UnitTest);
export const g = makeTestGroup(UnitTest);
g.test('stacks')
.params([
{
case: 'node_fail',
_expectedLines: 1,
_expectedLines: 3,
_stack: `Error:
at CaseRecorder.fail (/Users/kainino/src/cts/src/common/framework/logger.ts:99:30)
at RunCaseSpecific.exports.g.test.t [as fn] (/Users/kainino/src/cts/src/unittests/logger.spec.ts:80:7)
@ -23,7 +23,7 @@ g.test('stacks')
{
// TODO: make sure this test case actually matches what happens on windows
case: 'node_fail_backslash',
_expectedLines: 1,
_expectedLines: 3,
_stack: `Error:
at CaseRecorder.fail (C:\\Users\\kainino\\src\\cts\\src\\common\\framework\\logger.ts:99:30)
at RunCaseSpecific.exports.g.test.t [as fn] (C:\\Users\\kainino\\src\\cts\\src\\unittests\\logger.spec.ts:80:7)
@ -32,7 +32,7 @@ g.test('stacks')
},
{
case: 'node_fail_processTicksAndRejections',
_expectedLines: 3,
_expectedLines: 5,
_stack: `Error: expectation had no effect: suite1:foo:
at Object.generateMinimalQueryList (/Users/kainino/src/cts/src/common/framework/generate_minimal_query_list.ts:72:24)
at testGenerateMinimalQueryList (/Users/kainino/src/cts/src/unittests/loading.spec.ts:289:25)
@ -45,7 +45,7 @@ g.test('stacks')
},
{
case: 'node_throw',
_expectedLines: 1,
_expectedLines: 2,
_stack: `Error: hello
at RunCaseSpecific.g.test.t [as fn] (/Users/kainino/src/cts/src/unittests/test_group.spec.ts:51:11)
at RunCaseSpecific.run (/Users/kainino/src/cts/src/common/framework/test_group.ts:121:18)
@ -53,7 +53,7 @@ g.test('stacks')
},
{
case: 'firefox_fail',
_expectedLines: 1,
_expectedLines: 3,
_stack: `fail@http://localhost:8080/out/common/framework/logger.js:104:30
expect@http://localhost:8080/out/common/framework/default_fixture.js:59:16
@http://localhost:8080/out/unittests/util.spec.js:35:5
@ -67,7 +67,7 @@ run@http://localhost:8080/out/common/framework/test_group.js:119:18`,
},
{
case: 'safari_fail',
_expectedLines: 1,
_expectedLines: 3,
_stack: `fail@http://localhost:8080/out/common/framework/logger.js:104:39
expect@http://localhost:8080/out/common/framework/default_fixture.js:59:20
http://localhost:8080/out/unittests/util.spec.js:35:11
@ -87,7 +87,7 @@ promiseReactionJob@[native code]`,
},
{
case: 'chrome_fail',
_expectedLines: 1,
_expectedLines: 4,
_stack: `Error
at CaseRecorder.fail (http://localhost:8080/out/common/framework/logger.js:104:30)
at DefaultFixture.expect (http://localhost:8080/out/common/framework/default_fixture.js:59:16)
@ -98,7 +98,7 @@ promiseReactionJob@[native code]`,
},
{
case: 'chrome_throw',
_expectedLines: 5,
_expectedLines: 6,
_stack: `Error: hello
at RunCaseSpecific.fn (http://localhost:8080/out/unittests/test_group.spec.js:48:11)
at RunCaseSpecific.run (http://localhost:8080/out/common/framework/test_group.js:119:18)"
@ -111,7 +111,7 @@ promiseReactionJob@[native code]`,
},
{
case: 'multiple_lines',
_expectedLines: 7,
_expectedLines: 8,
_stack: `Error: hello
at RunCaseSpecific.fn (http://localhost:8080/out/unittests/test_group.spec.js:48:11)
at RunCaseSpecific.fn (http://localhost:8080/out/unittests/test_group.spec.js:48:11)
@ -126,17 +126,13 @@ promiseReactionJob@[native code]`,
},
])
.fn(t => {
const lines: number = t.params._expectedLines;
const ex = new Error();
ex.stack = t.params._stack;
t.expect(ex.stack === t.params._stack);
const stringified = getStackTrace(ex);
const stringified = extractImportantStackTrace(ex);
const parts = stringified.split('\n');
t.expect(parts.length === lines);
const fst = parts[0];
const lst = parts[parts.length - 1];
t.expect(fst.indexOf('/unittests/') !== -1 || fst.indexOf('\\unittests\\') !== -1);
t.expect(lst.indexOf('/unittests/') !== -1 || lst.indexOf('\\unittests\\') !== -1);
t.expect(parts.length === t.params._expectedLines);
const last = parts[parts.length - 1];
t.expect(last.indexOf('/unittests/') !== -1 || last.indexOf('\\unittests\\') !== -1);
});

2
src/unittests/listing.ts
View file

@ -1,4 +1,4 @@
import { TestSuiteListing } from '../common/framework/listing.js';
import { TestSuiteListing } from '../common/framework/test_suite_listing.js';
import { makeListing } from '../common/tools/crawl.js';
export const listing: Promise<TestSuiteListing> = makeListing(__filename);

356
src/unittests/loaders_and_trees.spec.ts Normal file
View file

@ -0,0 +1,356 @@
export const description = `
Tests for queries/filtering, loading, and running.
`;
import { TestFileLoader, SpecFile } from '../common/framework/file_loader.js';
import { Logger } from '../common/framework/logging/logger.js';
import { Status } from '../common/framework/logging/result.js';
import {
TestQuery,
TestQuerySingleCase,
TestQueryMultiCase,
TestQueryMultiTest,
TestQueryMultiFile,
} from '../common/framework/query/query.js';
import { makeTestGroup, makeTestGroupForUnitTesting } from '../common/framework/test_group.js';
import { TestSuiteListing, TestSuiteListingEntry } from '../common/framework/test_suite_listing.js';
import { TestTreeLeaf } from '../common/framework/tree.js';
import { assert, objectEquals } from '../common/framework/util/util.js';
import { UnitTest } from './unit_test.js';
const listingData: { [k: string]: TestSuiteListingEntry[] } = {
suite1: [
{ file: [], readme: 'desc 1a' },
{ file: ['foo'], description: 'desc 1b' },
{ file: ['bar'], readme: 'desc 1c' },
{ file: ['bar', 'buzz', 'buzz'], description: 'desc 1d' },
{ file: ['baz'], description: 'desc 1e' },
],
suite2: [
{ file: [], readme: 'desc 2a' },
{ file: ['foof'], description: 'desc 2b' },
],
};
const specsData: { [k: string]: SpecFile } = {
'suite1/foo.spec.js': {
description: 'desc 1b',
g: (() => {
const g = makeTestGroupForUnitTesting(UnitTest);
g.test('hello').fn(() => {});
g.test('bonjour').fn(() => {});
g.test('hola').fn(() => {});
return g;
})(),
},
'suite1/bar/biz.spec.js': {
description: 'desc 1f',
g: makeTestGroupForUnitTesting(UnitTest),
},
'suite1/bar/bez.spec.js': {
description: 'desc 1g',
g: makeTestGroupForUnitTesting(UnitTest),
},
'suite1/bar/buzz/buzz.spec.js': {
description: 'desc 1d',
g: (() => {
const g = makeTestGroupForUnitTesting(UnitTest);
g.test('zap').fn(() => {});
return g;
})(),
},
'suite1/baz.spec.js': {
description: 'desc 1e',
g: (() => {
const g = makeTestGroupForUnitTesting(UnitTest);
g.test('wye')
.params([{}, { x: 1 }])
.fn(() => {});
g.test('zed')
.params([
{ a: 1, b: 2, _c: 0 },
{ b: 3, a: 1, _c: 0 },
])
.fn(() => {});
return g;
})(),
},
'suite2/foof.spec.js': {
description: 'desc 2b',
g: (() => {
const g = makeTestGroupForUnitTesting(UnitTest);
g.test('blah').fn(t => {
t.debug('OK');
});
g.test('bleh')
.params([{ a: 1 }])
.fn(t => {
t.debug('OK');
t.debug('OK');
});
g.test('bluh,a').fn(t => {
t.fail('bye');
});
return g;
})(),
},
};
class FakeTestFileLoader extends TestFileLoader {
async listing(suite: string): Promise<TestSuiteListing> {
return listingData[suite];
}
async import(path: string): Promise<SpecFile> {
assert(path in specsData, '[test] mock file ' + path + ' does not exist');
return specsData[path];
}
}
class LoadingTest extends UnitTest {
static readonly loader = new FakeTestFileLoader();
async load(filter: string): Promise<TestTreeLeaf[]> {
return Array.from(await LoadingTest.loader.loadTests(filter));
}
async loadNames(filter: string): Promise<string[]> {
return (await this.load(filter)).map(c => c.query.toString());
}
}
export const g = makeTestGroup(LoadingTest);
g.test('suite').fn(async t => {
t.shouldReject('Error', t.load('suite1'));
t.shouldReject('Error', t.load('suite1:'));
});
g.test('group').fn(async t => {
t.expect((await t.load('suite1:*')).length === 8);
t.expect((await t.load('suite1:foo,*')).length === 3); // x:foo,* matches x:foo:
t.expect((await t.load('suite1:bar,*')).length === 1);
t.expect((await t.load('suite1:bar,buzz,buzz,*')).length === 1);
t.shouldReject('Error', t.load('suite1:f*'));
{
const s = new TestQueryMultiFile('suite1', ['bar', 'buzz']).toString();
t.expect((await t.load(s)).length === 1);
}
});
g.test('test').fn(async t => {
t.shouldReject('Error', t.load('suite1::'));
t.shouldReject('Error', t.load('suite1:bar:'));
t.shouldReject('Error', t.load('suite1:bar,:'));
t.shouldReject('Error', t.load('suite1::*'));
t.shouldReject('Error', t.load('suite1:bar,:*'));
t.shouldReject('Error', t.load('suite1:bar:*'));
t.expect((await t.load('suite1:foo:*')).length === 3);
t.expect((await t.load('suite1:bar,buzz,buzz:*')).length === 1);
t.expect((await t.load('suite1:baz:*')).length === 4);
t.expect((await t.load('suite2:foof:bluh,*')).length === 1);
t.expect((await t.load('suite2:foof:bluh,a,*')).length === 1);
{
const s = new TestQueryMultiTest('suite2', ['foof'], ['bluh']).toString();
t.expect((await t.load(s)).length === 1);
}
});
g.test('case').fn(async t => {
t.shouldReject('Error', t.load('suite1:foo::'));
t.shouldReject('Error', t.load('suite1:bar:zed,:'));
t.shouldReject('Error', t.load('suite1:foo:h*'));
t.shouldReject('Error', t.load('suite1:foo::*'));
t.shouldReject('Error', t.load('suite1:baz::*'));
t.shouldReject('Error', t.load('suite1:baz:zed,:*'));
t.shouldReject('Error', t.load('suite1:baz:zed:'));
t.shouldReject('Error', t.load('suite1:baz:zed:a=1;b=2*'));
t.shouldReject('Error', t.load('suite1:baz:zed:a=1;b=2;'));
t.shouldReject('SyntaxError', t.load('suite1:baz:zed:a=1;b=2,')); // tries to parse '2,' as JSON
t.shouldReject('Error', t.load('suite1:baz:zed:a=1,b=2')); // '=' not allowed in value '1,b=2'
t.shouldReject('Error', t.load('suite1:baz:zed:b=2*'));
t.shouldReject('Error', t.load('suite1:baz:zed:b=2;a=1;_c=0'));
t.shouldReject('Error', t.load('suite1:baz:zed:a=1,*'));
t.expect((await t.load('suite1:baz:zed:*')).length === 2);
t.expect((await t.load('suite1:baz:zed:a=1;*')).length === 2);
t.expect((await t.load('suite1:baz:zed:a=1;b=2')).length === 1);
t.expect((await t.load('suite1:baz:zed:a=1;b=2;*')).length === 1);
t.expect((await t.load('suite1:baz:zed:b=2;*')).length === 1);
t.expect((await t.load('suite1:baz:zed:b=2;a=1')).length === 1);
t.expect((await t.load('suite1:baz:zed:b=2;a=1;*')).length === 1);
t.expect((await t.load('suite1:baz:zed:b=3;a=1')).length === 1);
t.expect((await t.load('suite1:baz:zed:a=1;b=3')).length === 1);
t.expect((await t.load('suite1:foo:hello:')).length === 1);
{
const s = new TestQueryMultiCase('suite1', ['baz'], ['zed'], { a: 1, b: 2 }).toString();
t.expect((await t.load(s)).length === 1);
}
{
const s = new TestQuerySingleCase('suite1', ['baz'], ['zed'], { a: 1, b: 2 }).toString();
t.expect((await t.load(s)).length === 1);
}
});
g.test('end2end').fn(async t => {
const l = await t.load('suite2:foof:*');
assert(l.length === 3, 'listing length');
const log = new Logger(true);
const exp = async (
i: number,
query: TestQuery,
status: Status,
logs: (s: string[]) => boolean
) => {
t.expect(objectEquals(l[i].query, query));
const name = l[i].query.toString();
const [rec, res] = log.record(name);
await l[i].run(rec);
t.expect(log.results.get(name) === res);
t.expect(res.status === status);
t.expect(res.timems > 0);
assert(res.logs !== undefined); // only undefined while pending
t.expect(logs(res.logs.map(l => JSON.stringify(l))));
};
await exp(0, new TestQuerySingleCase('suite2', ['foof'], ['blah'], {}), 'pass', logs =>
objectEquals(logs, ['"DEBUG: OK"'])
);
await exp(1, new TestQuerySingleCase('suite2', ['foof'], ['bleh'], { a: 1 }), 'pass', logs =>
objectEquals(logs, ['"DEBUG: OK"', '"DEBUG: OK"'])
);
await exp(
2,
new TestQuerySingleCase('suite2', ['foof'], ['bluh', 'a'], {}),
'fail',
logs =>
logs.length === 1 &&
logs[0].startsWith('"FAIL: Error: bye\\n') &&
logs[0].indexOf('loaders_and_trees.spec.') !== -1
);
});
async function testIterateCollapsed(
t: LoadingTest,
expectations: string[],
expectedResult: 'throws' | string[]
) {
const treePromise = LoadingTest.loader.loadTree('suite1:*', expectations);
if (expectedResult === 'throws') {
t.shouldReject('Error', treePromise, 'loadTree should have thrown Error');
return;
}
const tree = await treePromise;
const actual = Array.from(tree.iterateCollapsedQueries(), q => q.toString());
if (!objectEquals(actual, expectedResult)) {
t.fail(
`iterateCollapsed failed:
got [${actual.join(', ')}]
exp [${expectedResult.join(', ')}]
${tree.toString()}`
);
}
}
g.test('print').fn(async () => {
const tree = await LoadingTest.loader.loadTree('suite1:*');
tree.toString();
});
g.test('iterateCollapsed').fn(async t => {
// No effect
await testIterateCollapsed(t, [], ['suite1:foo:*', 'suite1:bar,buzz,buzz:*', 'suite1:baz:*']);
await testIterateCollapsed(
t,
['suite1:*'],
['suite1:foo:*', 'suite1:bar,buzz,buzz:*', 'suite1:baz:*']
);
await testIterateCollapsed(
t,
['suite1:foo:*'],
['suite1:foo:*', 'suite1:bar,buzz,buzz:*', 'suite1:baz:*']
);
await testIterateCollapsed(
t,
['suite1:bar,buzz,buzz:*'],
['suite1:foo:*', 'suite1:bar,buzz,buzz:*', 'suite1:baz:*']
);
// Some effect
await testIterateCollapsed(
t,
['suite1:baz:wye:*'],
['suite1:foo:*', 'suite1:bar,buzz,buzz:*', 'suite1:baz:wye:*', 'suite1:baz:zed,*']
);
await testIterateCollapsed(
t,
['suite1:baz:zed:*'],
['suite1:foo:*', 'suite1:bar,buzz,buzz:*', 'suite1:baz:wye,*', 'suite1:baz:zed:*']
);
await testIterateCollapsed(
t,
['suite1:baz:wye:*', 'suite1:baz:zed:*'],
['suite1:foo:*', 'suite1:bar,buzz,buzz:*', 'suite1:baz:wye:*', 'suite1:baz:zed:*']
);
await testIterateCollapsed(
t,
['suite1:baz:wye:'],
[
'suite1:foo:*',
'suite1:bar,buzz,buzz:*',
'suite1:baz:wye:',
'suite1:baz:wye:x=1;*',
'suite1:baz:zed,*',
]
);
await testIterateCollapsed(
t,
['suite1:baz:wye:x=1'],
[
'suite1:foo:*',
'suite1:bar,buzz,buzz:*',
'suite1:baz:wye:',
'suite1:baz:wye:x=1',
'suite1:baz:zed,*',
]
);
await testIterateCollapsed(
t,
['suite1:foo:*', 'suite1:baz:wye:'],
[
'suite1:foo:*',
'suite1:bar,buzz,buzz:*',
'suite1:baz:wye:',
'suite1:baz:wye:x=1;*',
'suite1:baz:zed,*',
]
);
// Invalid expectation queries
await testIterateCollapsed(t, ['garbage'], 'throws');
await testIterateCollapsed(t, ['garbage*'], 'throws');
await testIterateCollapsed(t, ['suite1*'], 'throws');
await testIterateCollapsed(t, ['suite1:foo*'], 'throws');
await testIterateCollapsed(t, ['suite1:foo:ba*'], 'throws');
// Valid expectation queries but they don't match anything
await testIterateCollapsed(t, ['garbage:*'], 'throws');
await testIterateCollapsed(t, ['suite1:doesntexist:*'], 'throws');
await testIterateCollapsed(t, ['suite2:foo:*'], 'throws');
// Doesn't match anything because we collapse this unnecessary node into just 'suite1:foo:*'
await testIterateCollapsed(t, ['suite1:foo,*'], 'throws');
});

354
src/unittests/loading.spec.ts
View file

@ -1,354 +0,0 @@
export const description = `
Tests for queries/filtering, loading, and running.
`;
import { generateMinimalQueryList } from '../common/framework/generate_minimal_query_list.js';
import { TestSuiteListing, TestSuiteListingEntry } from '../common/framework/listing.js';
import { TestFileLoader, TestLoader, TestSpecOrReadme } from '../common/framework/loader.js';
import { Logger } from '../common/framework/logger.js';
import { paramsEquals } from '../common/framework/params_utils.js';
import { TestFilterResult } from '../common/framework/test_filter/test_filter_result.js';
import { RunCase, TestGroup } from '../common/framework/test_group.js';
import { makeQueryString } from '../common/framework/url_query.js';
import { assert, objectEquals } from '../common/framework/util/util.js';
import { UnitTest } from './unit_test.js';
const listingData: { [k: string]: TestSuiteListingEntry[] } = {
suite1: [
{ path: '', description: 'desc 1a' },
{ path: 'foo', description: 'desc 1b' },
{ path: 'bar/', description: 'desc 1c' },
{ path: 'bar/buzz', description: 'desc 1d' },
{ path: 'baz', description: 'desc 1e' },
],
suite2: [
{ path: '', description: 'desc 2a' },
{ path: 'foof', description: 'desc 2b' },
],
};
const specsData: { [k: string]: TestSpecOrReadme } = {
'suite1/README.txt': { description: 'desc 1a' },
'suite1/foo.spec.js': {
description: 'desc 1b',
g: (() => {
const g = new TestGroup(UnitTest);
g.test('hello').fn(() => {});
g.test('bonjour').fn(() => {});
g.test('hola').fn(() => {});
return g;
})(),
},
'suite1/bar/README.txt': { description: 'desc 1c' },
'suite1/bar/biz.spec.js': {
description: 'desc 1f',
g: new TestGroup(UnitTest),
},
'suite1/bar/bez.spec.js': {
description: 'desc 1g',
g: new TestGroup(UnitTest),
},
'suite1/bar/buzz.spec.js': {
description: 'desc 1d',
g: (() => {
const g = new TestGroup(UnitTest);
g.test('zap').fn(() => {});
return g;
})(),
},
'suite1/baz.spec.js': {
description: 'desc 1e',
g: (() => {
const g = new TestGroup(UnitTest);
g.test('wye')
.params([
{}, //
{ x: 1 },
])
.fn(() => {});
g.test('zed')
.params([
{ a: 1, b: 2, _c: 0 }, //
{ a: 1, b: 3, _c: 0 },
])
.fn(() => {});
return g;
})(),
},
'suite2/foof.spec.js': {
description: 'desc 2b',
g: (() => {
const g = new TestGroup(UnitTest);
g.test('blah').fn(t => {
t.debug('OK');
});
g.test('bleh')
.params([{}])
.fn(t => {
t.debug('OK');
t.debug('OK');
});
g.test('bluh').fn(t => {
t.fail('bye');
});
return g;
})(),
},
};
class FakeTestFileLoader implements TestFileLoader {
async listing(suite: string): Promise<TestSuiteListing> {
return listingData[suite];
}
async import(path: string): Promise<TestSpecOrReadme> {
assert(path in specsData, '[test] mock file ' + path + ' does not exist');
return specsData[path];
}
}
class LoadingTest extends UnitTest {
static readonly loader: TestLoader = new TestLoader(new FakeTestFileLoader());
async load(filters: string[]): Promise<TestFilterResult[]> {
return Array.from(await LoadingTest.loader.loadTestsFromCmdLine(filters));
}
async singleGroup(query: string): Promise<RunCase[]> {
const [rec] = new Logger().record({ suite: '', path: '' });
const a = await this.load([query]);
assert(a.length === 1, 'more than one group');
const spec = a[0].spec;
assert('g' in spec, 'group undefined');
return Array.from(spec.g.iterate(rec));
}
}
export const g = new TestGroup(LoadingTest);
g.test('whole suite').fn(async t => {
t.shouldReject('Error', t.load(['suite1']));
t.expect((await t.load(['suite1:'])).length === 5);
});
g.test('partial suite').fn(async t => {
t.expect((await t.load(['suite1:f'])).length === 1);
t.expect((await t.load(['suite1:fo'])).length === 1);
t.expect((await t.load(['suite1:foo'])).length === 1);
t.expect((await t.load(['suite1:foof'])).length === 0);
t.expect((await t.load(['suite1:ba'])).length === 3);
t.expect((await t.load(['suite1:bar'])).length === 2);
t.expect((await t.load(['suite1:bar/'])).length === 2);
t.expect((await t.load(['suite1:bar/b'])).length === 1);
});
g.test('whole group').fn(async t => {
t.shouldReject('Error', t.load(['suite1::']));
t.shouldReject('Error', t.load(['suite1:bar:']));
t.shouldReject('Error', t.load(['suite1:bar/:']));
t.expect((await t.singleGroup('suite1:bar/buzz:')).length === 1);
t.expect((await t.singleGroup('suite1:baz:')).length === 4);
{
const foo = (await t.load(['suite1:foo:']))[0];
t.expect(foo.id.suite === 'suite1');
t.expect(foo.id.path === 'foo');
assert('g' in foo.spec, 'foo group');
const [rec] = new Logger().record({ suite: '', path: '' });
t.expect(Array.from(foo.spec.g.iterate(rec)).length === 3);
}
});
g.test('partial group').fn(async t => {
t.expect((await t.singleGroup('suite1:foo:h')).length === 2);
t.expect((await t.singleGroup('suite1:foo:he')).length === 1);
t.expect((await t.singleGroup('suite1:foo:hello')).length === 1);
t.expect((await t.singleGroup('suite1:baz:zed')).length === 2);
});
g.test('partial test/exact').fn(async t => {
t.expect((await t.singleGroup('suite1:foo:hello=')).length === 1);
t.expect((await t.singleGroup('suite1:baz:zed=')).length === 0);
t.expect((await t.singleGroup('suite1:baz:zed=')).length === 0);
t.expect((await t.singleGroup('suite1:baz:zed={}')).length === 0);
t.expect((await t.singleGroup('suite1:baz:zed={"a":1,"b":2}')).length === 1);
t.expect((await t.singleGroup('suite1:baz:zed={"b":2,"a":1}')).length === 1);
t.expect((await t.singleGroup('suite1:baz:zed={"a":1,"b":2,"_c":0}')).length === 0);
});
g.test('partial test/makeQueryString').fn(async t => {
const s = makeQueryString(
{ suite: 'suite1', path: 'baz' },
{ test: 'zed', params: { a: 1, b: 2 } }
);
t.expect((await t.singleGroup(s)).length === 1);
});
g.test('partial test/match').fn(async t => {
t.expect((await t.singleGroup('suite1:baz:zed~')).length === 2);
t.expect((await t.singleGroup('suite1:baz:zed~{}')).length === 2);
t.expect((await t.singleGroup('suite1:baz:zed~{"a":1}')).length === 2);
t.expect((await t.singleGroup('suite1:baz:zed~{"a":1,"b":2}')).length === 1);
t.expect((await t.singleGroup('suite1:baz:zed~{"b":2,"a":1}')).length === 1);
t.expect((await t.singleGroup('suite1:baz:zed~{"b":2}')).length === 1);
t.expect((await t.singleGroup('suite1:baz:zed~{"a":2}')).length === 0);
t.expect((await t.singleGroup('suite1:baz:zed~{"c":0}')).length === 0);
t.expect((await t.singleGroup('suite1:baz:zed~{"_c":0}')).length === 0);
});
g.test('end2end').fn(async t => {
const l = await t.load(['suite2:foof']);
assert(l.length === 1, 'listing length');
t.expect(l[0].id.suite === 'suite2');
t.expect(l[0].id.path === 'foof');
t.expect(l[0].spec.description === 'desc 2b');
assert('g' in l[0].spec);
t.expect(l[0].spec.g.iterate instanceof Function);
const log = new Logger();
const [rec, res] = log.record(l[0].id);
const rcs = Array.from(l[0].spec.g.iterate(rec));
assert(rcs.length === 3, 'iterate length');
t.expect(rcs[0].id.test === 'blah');
t.expect(rcs[0].id.params === null);
t.expect(rcs[1].id.test === 'bleh');
t.expect(paramsEquals(rcs[1].id.params, {}));
t.expect(log.results[0] === res);
t.expect(res.spec === 'suite2:foof:');
t.expect(res.cases.length === 0);
{
const cases = res.cases;
const res0 = await rcs[0].run(true);
assert(cases.length === 1, 'results cases length');
t.expect(res.cases[0] === res0);
t.expect(res0.test === 'blah');
t.expect(res0.params === null);
t.expect(res0.status === 'pass');
t.expect(res0.timems >= 0);
assert(res0.logs !== undefined, 'results case logs');
t.expect(objectEquals(JSON.stringify(res0.logs), '["DEBUG: OK"]'));
}
{
// Store cases off to a separate variable due to a typescript bug
// where it can't detect that res.cases.length might change between
// above and here.
const cases = res.cases;
const res1 = await rcs[1].run(true);
assert(cases.length === 2, 'results cases length');
t.expect(res.cases[1] === res1);
t.expect(res1.test === 'bleh');
t.expect(paramsEquals(res1.params, {}));
t.expect(res1.status === 'pass');
t.expect(res1.timems >= 0);
assert(res1.logs !== undefined, 'results case logs');
t.expect(objectEquals(JSON.stringify(res1.logs), '["DEBUG: OK","DEBUG: OK"]'));
}
{
// Store cases off to a separate variable due to a typescript bug
// where it can't detect that res.cases.length might change between
// above and here.
const cases = res.cases;
const res2 = await rcs[2].run(true);
assert(cases.length === 3, 'results cases length');
t.expect(res.cases[2] === res2);
t.expect(res2.test === 'bluh');
t.expect(res2.params === null);
t.expect(res2.status === 'fail');
t.expect(res2.timems >= 0);
assert(res2.logs !== undefined, 'results case logs');
t.expect(res2.logs.length === 1);
const l = res2.logs[0].toJSON();
t.expect(l.startsWith('FAIL: bye\n'));
t.expect(l.indexOf('loading.spec.') !== -1);
}
});
const testGenerateMinimalQueryList = async (
t: LoadingTest,
expectations: string[],
result: string[]
) => {
const l = await t.load(['suite1:']);
const queries = await generateMinimalQueryList(l, expectations);
t.expect(objectEquals(queries, result));
};
g.test('generateMinimalQueryList/errors').fn(async t => {
t.shouldReject('Error', testGenerateMinimalQueryList(t, ['garbage'], []));
t.shouldReject('Error', testGenerateMinimalQueryList(t, ['suite1:'], []));
t.shouldReject('Error', testGenerateMinimalQueryList(t, ['suite1:foo'], []));
t.shouldReject('Error', testGenerateMinimalQueryList(t, ['suite1:foo:ba'], []));
t.shouldReject('Error', testGenerateMinimalQueryList(t, ['suite2:foo:'], []));
});
g.test('generateMinimalQueryList').fn(async t => {
await testGenerateMinimalQueryList(
t, //
[],
['suite1:foo:', 'suite1:bar/buzz:', 'suite1:baz:']
);
await testGenerateMinimalQueryList(
t, //
['suite1:foo:'],
['suite1:foo:', 'suite1:bar/buzz:', 'suite1:baz:']
);
await testGenerateMinimalQueryList(
t, //
['suite1:bar/buzz:'],
['suite1:foo:', 'suite1:bar/buzz:', 'suite1:baz:']
);
await testGenerateMinimalQueryList(
t, //
['suite1:baz:wye~'],
['suite1:foo:', 'suite1:bar/buzz:', 'suite1:baz:wye~', 'suite1:baz:zed~']
);
await testGenerateMinimalQueryList(
t, //
['suite1:baz:zed~'],
['suite1:foo:', 'suite1:bar/buzz:', 'suite1:baz:wye~', 'suite1:baz:zed~']
);
await testGenerateMinimalQueryList(
t, //
['suite1:baz:wye~', 'suite1:baz:zed~'],
['suite1:foo:', 'suite1:bar/buzz:', 'suite1:baz:wye~', 'suite1:baz:zed~']
);
await testGenerateMinimalQueryList(
t, //
['suite1:baz:wye='],
[
'suite1:foo:',
'suite1:bar/buzz:',
'suite1:baz:wye=',
'suite1:baz:wye={"x":1}',
'suite1:baz:zed~',
]
);
await testGenerateMinimalQueryList(
t, //
['suite1:baz:wye={"x":1}'],
[
'suite1:foo:',
'suite1:bar/buzz:',
'suite1:baz:wye=',
'suite1:baz:wye={"x":1}',
'suite1:baz:zed~',
]
);
await testGenerateMinimalQueryList(
t, //
['suite1:foo:', 'suite1:baz:wye='],
[
'suite1:foo:',
'suite1:bar/buzz:',
'suite1:baz:wye=',
'suite1:baz:wye={"x":1}',
'suite1:baz:zed~',
]
);
});

96
src/unittests/logger.spec.ts
View file

@ -5,82 +5,61 @@ Also serves as a larger test of async test functions, and of the logging system.
`;
import { SkipTestCase } from '../common/framework/fixture.js';
import { Logger } from '../common/framework/logger.js';
import { paramsEquals } from '../common/framework/params_utils.js';
import { TestGroup } from '../common/framework/test_group.js';
import { Logger } from '../common/framework/logging/logger.js';
import { makeTestGroup } from '../common/framework/test_group.js';
import { assert } from '../common/framework/util/util.js';
import { UnitTest } from './unit_test.js';
export const g = new TestGroup(UnitTest);
export const g = makeTestGroup(UnitTest);
g.test('construct').fn(t => {
const mylog = new Logger();
const [testrec, testres] = mylog.record({ suite: 'a', path: 'foo/bar' });
const [, res1] = testrec.record('baz', null);
const params2 = {};
const [, res2] = testrec.record('qux', params2);
const mylog = new Logger(true);
const [, res1] = mylog.record('one');
const [, res2] = mylog.record('two');
t.expect(testres.spec === 'a:foo/bar:');
t.expect(testres.cases.length === 2);
t.expect(testres.cases[0] === res1);
t.expect(testres.cases[1] === res2);
t.expect(res1.test === 'baz');
t.expect(res1.params === null);
t.expect(mylog.results.get('one') === res1);
t.expect(mylog.results.get('two') === res2);
t.expect(res1.logs === undefined);
t.expect(res1.status === 'running');
t.expect(res1.timems < 0);
t.expect(res2.test === 'qux');
t.expect(paramsEquals(res2.params, params2));
t.expect(res2.logs === undefined);
t.expect(res2.status === 'running');
t.expect(res2.timems < 0);
});
g.test('private params').fn(t => {
const mylog = new Logger();
const [testrec] = mylog.record({ suite: 'a', path: 'foo/bar' });
const [, res] = testrec.record('baz', { a: 1, _b: 2 });
t.expect(paramsEquals(res.params, { a: 1 }));
});
g.test('empty').fn(t => {
const mylog = new Logger();
const [testrec] = mylog.record({ suite: '', path: '' });
const [rec, res] = testrec.record('baz', null);
const mylog = new Logger(true);
const [rec, res] = mylog.record('one');
rec.start();
t.expect(res.status === 'running');
rec.finish();
t.expect(res.status === 'pass');
t.expect(res.timems >= 0);
});
g.test('pass').fn(t => {
const mylog = new Logger();
const [testrec] = mylog.record({ suite: '', path: '' });
const [rec, res] = testrec.record('baz', null);
const mylog = new Logger(true);
const [rec, res] = mylog.record('one');
rec.start();
rec.debug(new Error('hello'));
t.expect(res.status === 'running');
rec.finish();
t.expect(res.status === 'pass');
t.expect(res.timems >= 0);
});
g.test('skip').fn(t => {
const mylog = new Logger();
const [testrec] = mylog.record({ suite: '', path: '' });
const [rec, res] = testrec.record('baz', null);
const mylog = new Logger(true);
const [rec, res] = mylog.record('one');
rec.start();
t.expect(res.status === 'running');
rec.skipped(new SkipTestCase());
rec.debug(new Error('hello'));
t.expect(res.status === 'running');
rec.finish();
t.expect(res.status === 'skip');
@ -88,17 +67,12 @@ g.test('skip').fn(t => {
});
g.test('warn').fn(t => {
const mylog = new Logger();
const [testrec] = mylog.record({ suite: '', path: '' });
const [rec, res] = testrec.record('baz', null);
const mylog = new Logger(true);
const [rec, res] = mylog.record('one');
rec.start();
t.expect(res.status === 'running');
rec.warn(new Error());
rec.warn(new Error('hello'));
rec.skipped(new SkipTestCase());
t.expect(res.status === 'running');
rec.finish();
t.expect(res.status === 'warn');
@ -106,18 +80,13 @@ g.test('warn').fn(t => {
});
g.test('fail').fn(t => {
const mylog = new Logger();
const [testrec] = mylog.record({ suite: '', path: '' });
const [rec, res] = testrec.record('baz', null);
const mylog = new Logger(true);
const [rec, res] = mylog.record('one');
rec.start();
t.expect(res.status === 'running');
rec.fail(new Error('bye'));
rec.warn(new Error());
rec.skipped(new SkipTestCase());
t.expect(res.status === 'running');
rec.finish();
t.expect(res.status === 'fail');
@ -126,20 +95,25 @@ g.test('fail').fn(t => {
g.test('debug')
.params([
{ debug: true, _logsCount: 1 }, //
{ debug: false, _logsCount: 0 },
{ debug: true, _logsCount: 5 }, //
{ debug: false, _logsCount: 3 },
])
.fn(t => {
const { debug, _logsCount } = t.params;
const mylog = new Logger();
const [testrec] = mylog.record({ suite: '', path: '' });
const [rec, res] = testrec.record('baz', null);
const mylog = new Logger(debug);
const [rec, res] = mylog.record('one');
rec.start(debug);
rec.start();
rec.debug(new Error('hello'));
rec.fail(new Error('bye'));
rec.warn(new Error());
rec.skipped(new SkipTestCase());
rec.debug(new Error('foo'));
rec.finish();
t.expect(res.status === 'pass');
t.expect(res.status === 'fail');
t.expect(res.timems >= 0);
t.expect(res.logs!.length === _logsCount);
assert(res.logs !== undefined);
t.expect(res.logs.length === _logsCount);
});

11
src/unittests/make_filter.spec.ts
View file

@ -1,11 +0,0 @@
export const description = `
Unit tests for makeFilter.
`;
import { TestGroup } from '../common/framework/test_group.js';
import { UnitTest } from './unit_test.js';
export const g = new TestGroup(UnitTest);
// TODO: tests for makeFilter, in particular idIfSingle()

31
src/unittests/param_helpers.spec.ts → src/unittests/params_builder_and_utils.spec.ts
View file

@ -2,20 +2,24 @@ export const description = `
Unit tests for parameterization helpers.
`;
import { poptions, params } from '../common/framework/params.js';
import { ParamSpec, ParamSpecIterable, paramsEquals } from '../common/framework/params_utils.js';
import { TestGroup } from '../common/framework/test_group.js';
import { poptions, params } from '../common/framework/params_builder.js';
import {
CaseParams,
CaseParamsIterable,
publicParamsEquals,
} from '../common/framework/params_utils.js';
import { makeTestGroup } from '../common/framework/test_group.js';
import { UnitTest } from './unit_test.js';
class ParamsTest extends UnitTest {
expectSpecEqual(act: ParamSpecIterable, exp: ParamSpec[]): void {
expectSpecEqual(act: CaseParamsIterable, exp: CaseParams[]): void {
const a = Array.from(act);
this.expect(a.length === exp.length && a.every((x, i) => paramsEquals(x, exp[i])));
this.expect(a.length === exp.length && a.every((x, i) => publicParamsEquals(x, exp[i])));
}
}
export const g = new TestGroup(ParamsTest);
export const g = makeTestGroup(ParamsTest);
g.test('options').fn(t => {
t.expectSpecEqual(poptions('hello', [1, 2, 3]), [{ hello: 1 }, { hello: 2 }, { hello: 3 }]);
@ -25,14 +29,14 @@ g.test('params').fn(t => {
t.expectSpecEqual(params(), [{}]);
});
g.test('combine/zeroes and ones').fn(t => {
g.test('combine,zeroes_and_ones').fn(t => {
t.expectSpecEqual(params().combine([]).combine([]), []);
t.expectSpecEqual(params().combine([]).combine([{}]), []);
t.expectSpecEqual(params().combine([{}]).combine([]), []);
t.expectSpecEqual(params().combine([{}]).combine([{}]), [{}]);
});
g.test('combine/mixed').fn(t => {
g.test('combine,mixed').fn(t => {
t.expectSpecEqual(
params()
.combine(poptions('x', [1, 2]))
@ -110,7 +114,7 @@ g.test('expand').fn(t => {
);
});
g.test('expand/invalid').fn(t => {
g.test('expand,invalid').fn(t => {
const p = params()
.combine([{ x: 1 }])
.expand(function* (p) {
@ -122,8 +126,13 @@ g.test('expand/invalid').fn(t => {
});
g.test('undefined').fn(t => {
t.expectSpecEqual([{ a: undefined }], [{}]);
t.expectSpecEqual([{}], [{ a: undefined }]);
t.expect(!publicParamsEquals({ a: undefined }, {}));
t.expect(!publicParamsEquals({}, { a: undefined }));
});
g.test('private').fn(t => {
t.expect(publicParamsEquals({ _a: 0 }, {}));
t.expect(publicParamsEquals({}, { _a: 0 }));
});
g.test('arrays').fn(t => {

28
src/unittests/params.spec.ts → src/unittests/params_builder_toplevel.spec.ts
View file

@ -2,14 +2,14 @@ export const description = `
Unit tests for parameterization.
`;
import { params } from '../common/framework/params.js';
import { ParamSpec } from '../common/framework/params_utils.js';
import { TestGroup } from '../common/framework/test_group.js';
import { params } from '../common/framework/params_builder.js';
import { CaseParams } from '../common/framework/params_utils.js';
import { makeTestGroup, makeTestGroupForUnitTesting } from '../common/framework/test_group.js';
import { TestGroupTest } from './test_group_test.js';
import { UnitTest } from './unit_test.js';
export const g = new TestGroup(TestGroupTest);
export const g = makeTestGroup(TestGroupTest);
g.test('none')
.params([])
@ -17,7 +17,7 @@ g.test('none')
t.fail("this test shouldn't run");
});
g.test('combine none')
g.test('combine_none')
.params(params().combine([]))
.fn(t => {
t.fail("this test shouldn't run");
@ -66,13 +66,13 @@ g.test('exclude')
});
g.test('generator').fn(t0 => {
const g = new TestGroup(UnitTest);
const g = makeTestGroupForUnitTesting(UnitTest);
const ran: ParamSpec[] = [];
const ran: CaseParams[] = [];
g.test('generator')
.params(
(function* (): IterableIterator<ParamSpec> {
(function* (): IterableIterator<CaseParams> {
for (let x = 0; x < 3; ++x) {
for (let y = 0; y < 2; ++y) {
yield { x, y };
@ -85,11 +85,11 @@ g.test('generator').fn(t0 => {
});
t0.expectCases(g, [
{ test: 'generator', params: { x: 0, y: 0 } },
{ test: 'generator', params: { x: 0, y: 1 } },
{ test: 'generator', params: { x: 1, y: 0 } },
{ test: 'generator', params: { x: 1, y: 1 } },
{ test: 'generator', params: { x: 2, y: 0 } },
{ test: 'generator', params: { x: 2, y: 1 } },
{ test: ['generator'], params: { x: 0, y: 0 } },
{ test: ['generator'], params: { x: 0, y: 1 } },
{ test: ['generator'], params: { x: 1, y: 0 } },
{ test: ['generator'], params: { x: 1, y: 1 } },
{ test: ['generator'], params: { x: 2, y: 0 } },
{ test: ['generator'], params: { x: 2, y: 1 } },
]);
});

133
src/unittests/query_compare.spec.ts Normal file
View file

@ -0,0 +1,133 @@
export const description = `
Tests for TestQuery comparison
`;
import { compareQueries, Ordering } from '../common/framework/query/compare.js';
import {
TestQuery,
TestQuerySingleCase,
TestQueryMultiFile,
TestQueryMultiTest,
TestQueryMultiCase,
} from '../common/framework/query/query.js';
import { makeTestGroup } from '../common/framework/test_group.js';
import { UnitTest } from './unit_test.js';
class F extends UnitTest {
expectQ(a: TestQuery, exp: '<' | '=' | '>' | '!', b: TestQuery) {
const [expOrdering, expInvOrdering] =
exp === '<'
? [Ordering.StrictSubset, Ordering.StrictSuperset]
: exp === '='
? [Ordering.Equal, Ordering.Equal]
: exp === '>'
? [Ordering.StrictSuperset, Ordering.StrictSubset]
: [Ordering.Unordered, Ordering.Unordered];
{
const act = compareQueries(a, b);
this.expect(act === expOrdering, `${a} ${b} got ${act}, exp ${expOrdering}`);
}
{
const act = compareQueries(a, b);
this.expect(act === expOrdering, `${b} ${a} got ${act}, exp ${expInvOrdering}`);
}
}
expectWellOrdered(...qs: TestQuery[]) {
for (let i = 0; i < qs.length; ++i) {
this.expectQ(qs[i], '=', qs[i]);
for (let j = i + 1; j < qs.length; ++j) {
this.expectQ(qs[i], '>', qs[j]);
}
}
}
expectUnordered(...qs: TestQuery[]) {
for (let i = 0; i < qs.length; ++i) {
this.expectQ(qs[i], '=', qs[i]);
for (let j = i + 1; j < qs.length; ++j) {
this.expectQ(qs[i], '!', qs[j]);
}
}
}
}
export const g = makeTestGroup(F);
// suite:* > suite:a,* > suite:a,b,* > suite:a,b:*
// suite:a,b:* > suite:a,b:c,* > suite:a,b:c,d,* > suite:a,b:c,d:*
// suite:a,b:c,d:* > suite:a,b:c,d:x=1;* > suite:a,b:c,d:x=1;y=2;* > suite:a,b:c,d:x=1;y=2
// suite:a;* (unordered) suite:b;*
g.test('well_ordered').fn(t => {
t.expectWellOrdered(
new TestQueryMultiFile('suite', []),
new TestQueryMultiFile('suite', ['a']),
new TestQueryMultiFile('suite', ['a', 'b']),
new TestQueryMultiTest('suite', ['a', 'b'], []),
new TestQueryMultiTest('suite', ['a', 'b'], ['c']),
new TestQueryMultiTest('suite', ['a', 'b'], ['c', 'd']),
new TestQueryMultiCase('suite', ['a', 'b'], ['c', 'd'], {}),
new TestQueryMultiCase('suite', ['a', 'b'], ['c', 'd'], { x: 1 }),
new TestQueryMultiCase('suite', ['a', 'b'], ['c', 'd'], { x: 1, y: 2 }),
new TestQuerySingleCase('suite', ['a', 'b'], ['c', 'd'], { x: 1, y: 2 })
);
t.expectWellOrdered(
new TestQueryMultiFile('suite', []),
new TestQueryMultiFile('suite', ['a']),
new TestQueryMultiFile('suite', ['a', 'b']),
new TestQueryMultiTest('suite', ['a', 'b'], []),
new TestQueryMultiTest('suite', ['a', 'b'], ['c']),
new TestQueryMultiTest('suite', ['a', 'b'], ['c', 'd']),
new TestQueryMultiCase('suite', ['a', 'b'], ['c', 'd'], {}),
new TestQuerySingleCase('suite', ['a', 'b'], ['c', 'd'], {})
);
});
g.test('unordered').fn(t => {
t.expectUnordered(
new TestQueryMultiFile('suite', ['a']), //
new TestQueryMultiFile('suite', ['x'])
);
t.expectUnordered(
new TestQueryMultiFile('suite', ['a', 'b']),
new TestQueryMultiFile('suite', ['a', 'x'])
);
t.expectUnordered(
new TestQueryMultiTest('suite', ['a', 'b'], ['c']),
new TestQueryMultiTest('suite', ['a', 'b'], ['x']),
new TestQueryMultiTest('suite', ['a'], []),
new TestQueryMultiTest('suite', ['a', 'x'], [])
);
t.expectUnordered(
new TestQueryMultiTest('suite', ['a', 'b'], ['c', 'd']),
new TestQueryMultiTest('suite', ['a', 'b'], ['c', 'x']),
new TestQueryMultiTest('suite', ['a'], []),
new TestQueryMultiTest('suite', ['a', 'x'], [])
);
t.expectUnordered(
new TestQueryMultiTest('suite', ['a', 'b'], ['c', 'd']),
new TestQueryMultiTest('suite', ['a', 'b'], ['c', 'x']),
new TestQueryMultiTest('suite', ['a'], []),
new TestQueryMultiTest('suite', ['a', 'x'], ['c'])
);
t.expectUnordered(
new TestQueryMultiCase('suite', ['a', 'b'], ['c', 'd'], { x: 1 }),
new TestQueryMultiCase('suite', ['a', 'b'], ['c', 'd'], { x: 9 }),
new TestQueryMultiCase('suite', ['a', 'b'], ['c'], { x: 9 })
);
t.expectUnordered(
new TestQueryMultiCase('suite', ['a', 'b'], ['c', 'd'], { x: 1, y: 2 }),
new TestQueryMultiCase('suite', ['a', 'b'], ['c', 'd'], { x: 1, y: 8 }),
new TestQueryMultiCase('suite', ['a', 'b'], ['c'], { x: 1, y: 8 })
);
t.expectUnordered(
new TestQuerySingleCase('suite', ['a', 'b'], ['c', 'd'], { x: 1, y: 2 }),
new TestQuerySingleCase('suite', ['a', 'b'], ['c', 'd'], { x: 1, y: 8 }),
new TestQuerySingleCase('suite', ['a', 'b'], ['c'], { x: 1, y: 8 })
);
t.expectUnordered(
new TestQuerySingleCase('suite1', ['bar', 'buzz', 'buzz'], ['zap'], {}),
new TestQueryMultiTest('suite1', ['bar'], [])
);
});

82
src/unittests/test_filter.spec.ts
View file

@ -1,82 +0,0 @@
export const description = `
Unit tests for test_filter .matches() methods.
`;
import { FilterByGroup } from '../common/framework/test_filter/filter_by_group.js';
import {
FilterByParamsExact,
FilterByParamsMatch,
FilterByTestMatch,
} from '../common/framework/test_filter/filter_one_file.js';
import { TestGroup } from '../common/framework/test_group.js';
import { UnitTest } from './unit_test.js';
export const g = new TestGroup(UnitTest);
g.test('FilterByGroup/matches').fn(t => {
{
const f = new FilterByGroup('ab', 'de/f');
t.expect(f.matches({ spec: { suite: 'ab', path: 'de/fg' } }));
t.expect(f.matches({ spec: { suite: 'ab', path: 'de/fg' }, test: 'x' }));
t.expect(f.matches({ spec: { suite: 'ab', path: 'de/fg' }, test: 'x', params: null }));
t.expect(f.matches({ spec: { suite: 'ab', path: 'de/fg' }, test: 'x', params: {} }));
t.expect(f.matches({ spec: { suite: 'ab', path: 'de/fg/' }, test: 'x', params: null }));
t.expect(f.matches({ spec: { suite: 'ab', path: 'de/f/g' }, test: 'x', params: null }));
t.expect(f.matches({ spec: { suite: 'ab', path: 'de/f/g/' }, test: 'x', params: null }));
}
{
const f = new FilterByGroup('ab', 'de/f/');
t.expect(!f.matches({ spec: { suite: 'ab', path: 'de/fg' } }));
t.expect(!f.matches({ spec: { suite: 'ab', path: 'de/fg' }, test: 'x' }));
t.expect(!f.matches({ spec: { suite: 'ab', path: 'de/fg' }, test: 'x', params: null }));
t.expect(!f.matches({ spec: { suite: 'ab', path: 'de/fg' }, test: 'x', params: {} }));
t.expect(!f.matches({ spec: { suite: 'ab', path: 'de/fg/' }, test: 'x', params: null }));
t.expect(f.matches({ spec: { suite: 'ab', path: 'de/f/g' }, test: 'x', params: null }));
t.expect(f.matches({ spec: { suite: 'ab', path: 'de/f/g/' }, test: 'x', params: null }));
}
});
g.test('FilterByTestMatch/matches').fn(t => {
const f = new FilterByTestMatch({ suite: 'ab', path: 'de/f' }, 'xy');
t.expect(f.matches({ spec: { suite: 'ab', path: 'de/f' }, test: 'xy' }));
t.expect(f.matches({ spec: { suite: 'ab', path: 'de/f' }, test: 'xy', params: null }));
t.expect(f.matches({ spec: { suite: 'ab', path: 'de/f' }, test: 'xyz', params: null }));
t.expect(f.matches({ spec: { suite: 'ab', path: 'de/f' }, test: 'xyz', params: {} }));
t.expect(f.matches({ spec: { suite: 'ab', path: 'de/f' }, test: 'xyz', params: { z: 1 } }));
t.expect(!f.matches({ spec: { suite: 'ab', path: 'de/f' } }));
t.expect(!f.matches({ spec: { suite: 'ab', path: 'de/f/' }, test: 'xyz', params: null }));
t.expect(!f.matches({ spec: { suite: 'ab', path: 'de/fg' }, test: 'xyz', params: null }));
});
g.test('FilterByParamsMatch/matches').fn(t => {
{
const f = new FilterByParamsMatch({ suite: 'ab', path: 'de/f' }, 'xy', null);
t.expect(f.matches({ spec: { suite: 'ab', path: 'de/f' }, test: 'xy', params: null }));
t.expect(f.matches({ spec: { suite: 'ab', path: 'de/f' }, test: 'xy', params: {} }));
t.expect(f.matches({ spec: { suite: 'ab', path: 'de/f' }, test: 'xy', params: { z: 1 } }));
t.expect(!f.matches({ spec: { suite: 'ab', path: 'de/f' } }));
t.expect(!f.matches({ spec: { suite: 'ab', path: 'de/f' }, test: 'xy' }));
t.expect(!f.matches({ spec: { suite: 'ab', path: 'de/f' }, test: 'xy', params: undefined }));
t.expect(!f.matches({ spec: { suite: 'ab', path: 'de/f' }, test: 'xyz', params: {} }));
t.expect(!f.matches({ spec: { suite: 'ab', path: 'de/f/' }, test: 'xy', params: {} }));
}
{
const f = new FilterByParamsMatch({ suite: 'ab', path: 'de/f' }, 'xy', {});
t.expect(f.matches({ spec: { suite: 'ab', path: 'de/f' }, test: 'xy', params: null }));
t.expect(f.matches({ spec: { suite: 'ab', path: 'de/f' }, test: 'xy', params: {} }));
t.expect(!f.matches({ spec: { suite: 'ab', path: 'de/f' } }));
t.expect(!f.matches({ spec: { suite: 'ab', path: 'de/f' }, test: 'xy' }));
}
});
g.test('FilterByParamsExact').fn(t => {
const f = new FilterByParamsExact({ suite: 'ab', path: 'de/f' }, 'xy', {});
t.expect(f.matches({ spec: { suite: 'ab', path: 'de/f' }, test: 'xy', params: null }));
t.expect(f.matches({ spec: { suite: 'ab', path: 'de/f' }, test: 'xy', params: {} }));
t.expect(!f.matches({ spec: { suite: 'ab', path: 'de/f' }, test: 'xy', params: { z: 1 } }));
t.expect(!f.matches({ spec: { suite: 'ab', path: 'de/f' }, test: 'xyz', params: {} }));
t.expect(!f.matches({ spec: { suite: 'ab', path: 'de/f/' }, test: 'xy', params: {} }));
t.expect(!f.matches({ spec: { suite: 'ab', path: 'de/f' } }));
t.expect(!f.matches({ spec: { suite: 'ab', path: 'de/f' }, test: 'xy' }));
});

86
src/unittests/test_group.spec.ts
View file

@ -3,23 +3,22 @@ Unit tests for TestGroup.
`;
import { Fixture } from '../common/framework/fixture.js';
import { poptions } from '../common/framework/params.js';
import { TestGroup } from '../common/framework/test_group.js';
import { makeTestGroup, makeTestGroupForUnitTesting } from '../common/framework/test_group.js';
import { assert } from '../common/framework/util/util.js';
import { TestGroupTest } from './test_group_test.js';
import { UnitTest } from './unit_test.js';
export const g = new TestGroup(TestGroupTest);
export const g = makeTestGroup(TestGroupTest);
g.test('UnitTest fixture').fn(async t0 => {
g.test('UnitTest_fixture').fn(async t0 => {
let seen = 0;
/* eslint-disable-next-line @typescript-eslint/no-unused-vars */
function count(t: Fixture): void {
seen++;
}
const g = new TestGroup(UnitTest);
const g = makeTestGroupForUnitTesting(UnitTest);
g.test('test').fn(count);
g.test('testp')
@ -30,7 +29,7 @@ g.test('UnitTest fixture').fn(async t0 => {
t0.expect(seen === 2);
});
g.test('custom fixture').fn(async t0 => {
g.test('custom_fixture').fn(async t0 => {
let seen = 0;
class Counter extends UnitTest {
count(): void {
@ -38,7 +37,7 @@ g.test('custom fixture').fn(async t0 => {
}
}
const g = new TestGroup(Counter);
const g = makeTestGroupForUnitTesting(Counter);
g.test('test').fn(t => {
t.count();
@ -54,7 +53,7 @@ g.test('custom fixture').fn(async t0 => {
});
g.test('stack').fn(async t0 => {
const g = new TestGroup(UnitTest);
const g = makeTestGroupForUnitTesting(UnitTest);
const doNestedThrow1 = () => {
throw new Error('goodbye');
@ -70,24 +69,23 @@ g.test('stack').fn(async t0 => {
throw new Error('hello');
});
/* eslint-disable-next-line @typescript-eslint/no-unused-vars */
g.test('throw nested').fn(t => {
g.test('throw_nested').fn(t => {
doNestedThrow2();
});
const res = await t0.run(g);
const search = /unittests[/\\]test_group\.spec\.[tj]s|suites[/\\]unittests[/\\]unit_test\.[tj]s/;
for (const { logs } of res.cases) {
const search = /unittests[/\\]test_group\.spec\.[tj]s/;
t0.expect(res.size > 0);
for (const { logs } of res.values()) {
assert(logs !== undefined, 'expected logs');
const l = logs[0].toJSON();
t0.expect(search.test(l));
const st = l.split('\n');
t0.expect(search.test(st[st.length - 1]));
t0.expect(logs.some(l => search.test(l.toJSON())));
t0.expect(search.test(logs[logs.length - 1].toJSON()));
}
});
g.test('duplicate test name').fn(t => {
const g = new TestGroup(UnitTest);
g.test('duplicate_test_name').fn(t => {
const g = makeTestGroupForUnitTesting(UnitTest);
g.test('abc').fn(() => {});
t.shouldThrow('Error', () => {
@ -95,8 +93,8 @@ g.test('duplicate test name').fn(t => {
});
});
g.test('duplicate test params').fn(t => {
const g = new TestGroup(UnitTest);
g.test('duplicate_test_params').fn(t => {
const g = makeTestGroupForUnitTesting(UnitTest);
t.shouldThrow('Error', () => {
g.test('abc')
@ -110,8 +108,8 @@ g.test('duplicate test params').fn(t => {
});
});
g.test('duplicate test params/with different private params').fn(t => {
const g = new TestGroup(UnitTest);
g.test('duplicate_test_params,with_different_private_params').fn(t => {
const g = makeTestGroupForUnitTesting(UnitTest);
t.shouldThrow('Error', () => {
g.test('abc')
@ -125,19 +123,29 @@ g.test('duplicate test params/with different private params').fn(t => {
});
});
const badChars = Array.from('"`~@#$+=\\|!^&*[]<>{}-\'.,');
g.test('invalid test name')
.params(poptions('char', badChars))
.fn(t => {
const g = new TestGroup(UnitTest);
g.test('invalid_test_name').fn(t => {
const g = makeTestGroupForUnitTesting(UnitTest);
t.shouldThrow('Error', () => {
g.test('a' + t.params.char + 'b').fn(() => {});
});
});
const badChars = Array.from('"`~@#$+=\\|!^&*[]<>{}-\'. ');
for (const char of badChars) {
const name = 'a' + char + 'b';
t.shouldThrow(
'Error',
() => {
g.test(name).fn(() => {});
},
name
);
}
});
g.test('valid_param_value').fn(() => {
const g = makeTestGroup(UnitTest);
g.test('a').params([{ x: JSON.stringify({ a: 1, b: 2 }) }]);
});
g.test('throws').fn(async t0 => {
const g = new TestGroup(UnitTest);
const g = makeTestGroupForUnitTesting(UnitTest);
/* eslint-disable-next-line @typescript-eslint/no-unused-vars */
g.test('a').fn(t => {
@ -145,7 +153,9 @@ g.test('throws').fn(async t0 => {
});
const result = await t0.run(g);
t0.expect(result.cases[0].status === 'fail');
const values = Array.from(result.values());
t0.expect(values.length === 1);
t0.expect(values[0].status === 'fail');
});
g.test('shouldThrow').fn(async t0 => {
@ -153,7 +163,7 @@ g.test('shouldThrow').fn(async t0 => {
throw new TypeError();
});
const g = new TestGroup(UnitTest);
const g = makeTestGroupForUnitTesting(UnitTest);
g.test('a').fn(t => {
t.shouldThrow('Error', () => {
@ -162,7 +172,9 @@ g.test('shouldThrow').fn(async t0 => {
});
const result = await t0.run(g);
t0.expect(result.cases[0].status === 'fail');
const values = Array.from(result.values());
t0.expect(values.length === 1);
t0.expect(values[0].status === 'fail');
});
g.test('shouldReject').fn(async t0 => {
@ -173,7 +185,7 @@ g.test('shouldReject').fn(async t0 => {
})()
);
const g = new TestGroup(UnitTest);
const g = makeTestGroupForUnitTesting(UnitTest);
g.test('a').fn(async t => {
t.shouldReject(
@ -186,5 +198,7 @@ g.test('shouldReject').fn(async t0 => {
const result = await t0.run(g);
// Fails even though shouldReject doesn't fail until after the test function ends
t0.expect(result.cases[0].status === 'fail');
const values = Array.from(result.values());
t0.expect(values.length === 1);
t0.expect(values[0].status === 'fail');
});

42
src/unittests/test_group_test.ts
View file

@ -1,35 +1,23 @@
import { Fixture } from '../common/framework/fixture.js';
import { TestCaseID } from '../common/framework/id.js';
import { LiveTestSpecResult, Logger } from '../common/framework/logger.js';
import { paramsEquals } from '../common/framework/params_utils.js';
import { TestGroup } from '../common/framework/test_group.js';
import { Logger, LogResults } from '../common/framework/logging/logger.js';
import { TestQuerySingleCase } from '../common/framework/query/query.js';
import { RunCaseIterable, TestCaseID } from '../common/framework/test_group.js';
import { objectEquals } from '../common/framework/util/util.js';
import { UnitTest } from './unit_test.js';
export class TestGroupTest extends UnitTest {
async run<F extends Fixture>(g: TestGroup<F>): Promise<LiveTestSpecResult> {
const [rec, res] = new Logger().record({ suite: '', path: '' });
await Promise.all(Array.from(g.iterate(rec)).map(test => test.run()));
return res;
async run(g: RunCaseIterable): Promise<LogResults> {
const logger = new Logger(true);
for (const rc of await Promise.all(g.iterate())) {
const query = new TestQuerySingleCase('xx', ['yy'], rc.id.test, rc.id.params);
const [rec] = logger.record(query.toString());
await rc.run(rec);
}
return logger.results;
}
enumerate<F extends Fixture>(g: TestGroup<F>): TestCaseID[] {
const cases = [];
const [rec] = new Logger().record({ suite: '', path: '' });
for (const test of g.iterate(rec)) {
cases.push(test.id);
}
return cases;
}
expectCases<F extends Fixture>(g: TestGroup<F>, cases: TestCaseID[]): void {
const gcases = this.enumerate(g);
if (this.expect(gcases.length === cases.length)) {
for (let i = 0; i < cases.length; ++i) {
this.expect(gcases[i].test === cases[i].test);
this.expect(paramsEquals(gcases[i].params, cases[i].params));
}
}
expectCases(g: RunCaseIterable, cases: TestCaseID[]): void {
const gcases = Array.from(g.iterate()).map(rc => rc.id);
this.expect(objectEquals(gcases, cases));
}
}

56
src/unittests/test_query.spec.ts Normal file
View file

@ -0,0 +1,56 @@
export const description = `
Tests for TestQuery
`;
import {
TestQueryMultiFile,
TestQueryMultiTest,
TestQueryMultiCase,
TestQuerySingleCase,
TestQuery,
} from '../common/framework/query/query.js';
import { makeTestGroup } from '../common/framework/test_group.js';
import { UnitTest } from './unit_test.js';
class F extends UnitTest {
expectToString(q: TestQuery, exp: string) {
this.expect(q.toString() === exp);
}
}
export const g = makeTestGroup(F);
g.test('constructor').fn(t => {
t.shouldThrow('Error', () => new TestQueryMultiTest('suite', [], []));
t.shouldThrow('Error', () => new TestQueryMultiCase('suite', ['a'], [], {}));
t.shouldThrow('Error', () => new TestQueryMultiCase('suite', [], ['c'], {}));
t.shouldThrow('Error', () => new TestQueryMultiCase('suite', [], [], {}));
t.shouldThrow('Error', () => new TestQuerySingleCase('suite', ['a'], [], {}));
t.shouldThrow('Error', () => new TestQuerySingleCase('suite', [], ['c'], {}));
t.shouldThrow('Error', () => new TestQuerySingleCase('suite', [], [], {}));
});
g.test('toString').fn(t => {
t.expectToString(new TestQueryMultiFile('s', []), 's:*');
t.expectToString(new TestQueryMultiFile('s', ['a']), 's:a,*');
t.expectToString(new TestQueryMultiFile('s', ['a', 'b']), 's:a,b,*');
t.expectToString(new TestQueryMultiTest('s', ['a', 'b'], []), 's:a,b:*');
t.expectToString(new TestQueryMultiTest('s', ['a', 'b'], ['c']), 's:a,b:c,*');
t.expectToString(new TestQueryMultiTest('s', ['a', 'b'], ['c', 'd']), 's:a,b:c,d,*');
t.expectToString(new TestQueryMultiCase('s', ['a', 'b'], ['c', 'd'], {}), 's:a,b:c,d:*');
t.expectToString(
new TestQueryMultiCase('s', ['a', 'b'], ['c', 'd'], { x: 1 }),
's:a,b:c,d:x=1;*'
);
t.expectToString(
new TestQueryMultiCase('s', ['a', 'b'], ['c', 'd'], { x: 1, y: 2 }),
's:a,b:c,d:x=1;y=2;*'
);
t.expectToString(
new TestQuerySingleCase('s', ['a', 'b'], ['c', 'd'], { x: 1, y: 2 }),
's:a,b:c,d:x=1;y=2'
);
t.expectToString(new TestQuerySingleCase('s', ['a', 'b'], ['c', 'd'], {}), 's:a,b:c,d:');
});

82
src/unittests/url_query.spec.ts
View file

@ -2,27 +2,75 @@ export const description = `
Unit tests for URL queries.
`;
import { TestGroup } from '../common/framework/test_group.js';
import { makeQueryString } from '../common/framework/url_query.js';
import {
TestQuery,
TestQuerySingleCase,
TestQueryMultiCase,
TestQueryMultiTest,
TestQueryMultiFile,
} from '../common/framework/query/query.js';
import { makeTestGroup } from '../common/framework/test_group.js';
import { UnitTest } from './unit_test.js';
export const g = new TestGroup(UnitTest);
class T extends UnitTest {
expectQueryString(q: TestQuery, exp: string): void {
const s = q.toString();
this.expect(s === exp, `got ${s} expected ${exp}`);
}
}
g.test('makeQueryString').fn(t => {
const s = makeQueryString({ suite: 'a', path: 'b/c' }, { test: 'd/e', params: { f: 'g', h: 3 } });
t.expect(s === 'a:b/c:d/e={"f":"g","h":3}');
});
export const g = makeTestGroup(T);
g.test('makeQueryString/private').fn(t => {
const s = makeQueryString({ suite: 'a', path: 'b' }, { test: 'c', params: { pub: 2, _pri: 3 } });
t.expect(s === 'a:b:c={"pub":2}');
});
g.test('makeQueryString/undefined').fn(t => {
const s = makeQueryString(
{ suite: 'a', path: 'b' },
{ test: 'c', params: { f: undefined, h: 3 } }
g.test('stringifyQuery,single_case').fn(t => {
t.expectQueryString(
new TestQuerySingleCase('a', ['b_1', '2_c'], ['d_3', '4_e'], {
f: 'g',
_pri1: 0,
a: 3,
_pri2: 1,
}),
'a:b_1,2_c:d_3,4_e:f="g";a=3'
);
});
g.test('stringifyQuery,multi_case').fn(t => {
t.expectQueryString(
new TestQueryMultiCase('a', ['b_1', '2_c'], ['d_3', '4_e'], {
f: 'g',
_pri1: 0,
a: 3,
_pri2: 1,
}),
'a:b_1,2_c:d_3,4_e:f="g";a=3;*'
);
t.expectQueryString(
new TestQueryMultiCase('a', ['b_1', '2_c'], ['d_3', '4_e'], {}),
'a:b_1,2_c:d_3,4_e:*'
);
});
g.test('stringifyQuery,multi_test').fn(t => {
t.expectQueryString(
new TestQueryMultiTest('a', ['b_1', '2_c'], ['d_3', '4_e']),
'a:b_1,2_c:d_3,4_e,*'
);
t.expectQueryString(
new TestQueryMultiTest('a', ['b_1', '2_c'], []), //
'a:b_1,2_c:*'
);
});
g.test('stringifyQuery,multi_file').fn(t => {
t.expectQueryString(
new TestQueryMultiFile('a', ['b_1', '2_c']), //
'a:b_1,2_c,*'
);
t.expectQueryString(
new TestQueryMultiFile('a', []), //
'a:*'
);
t.expect(s === 'a:b:c={"h":3}');
});

6
src/webgpu/api/operation/buffers/create_mapped.spec.ts
View file

@ -1,11 +1,11 @@
export const description = '';
import { params, pbool, poptions } from '../../../../common/framework/params.js';
import { TestGroup } from '../../../../common/framework/test_group.js';
import { params, pbool, poptions } from '../../../../common/framework/params_builder.js';
import { makeTestGroup } from '../../../../common/framework/test_group.js';
import { MappingTest } from './mapping_test.js';
export const g = new TestGroup(MappingTest);
export const g = makeTestGroup(MappingTest);
g.test('createBufferMapped')
.params(

6
src/webgpu/api/operation/buffers/map.spec.ts
View file

@ -1,11 +1,11 @@
export const description = '';
import { pbool, poptions, params } from '../../../../common/framework/params.js';
import { TestGroup } from '../../../../common/framework/test_group.js';
import { pbool, poptions, params } from '../../../../common/framework/params_builder.js';
import { makeTestGroup } from '../../../../common/framework/test_group.js';
import { MappingTest } from './mapping_test.js';
export const g = new TestGroup(MappingTest);
export const g = makeTestGroup(MappingTest);
g.test('mapWriteAsync')
.params(poptions('size', [12, 512 * 1024]))

6
src/webgpu/api/operation/buffers/map_detach.spec.ts
View file

@ -1,6 +1,6 @@
export const description = '';
import { TestGroup } from '../../../../common/framework/test_group.js';
import { makeTestGroup } from '../../../../common/framework/test_group.js';
import { GPUTest } from '../../../gpu_test.js';
class F extends GPUTest {
@ -17,7 +17,7 @@ class F extends GPUTest {
}
}
export const g = new TestGroup(F);
export const g = makeTestGroup(F);
g.test('mapWriteAsync')
.params([
@ -43,7 +43,7 @@ g.test('mapReadAsync')
t.checkDetach(buffer, arrayBuffer, t.params.unmap, t.params.destroy);
});
g.test('create mapped')
g.test('create_mapped')
.params([
{ unmap: true, destroy: false },
{ unmap: false, destroy: true },

4
src/webgpu/api/operation/buffers/map_oom.spec.ts
View file

@ -1,6 +1,6 @@
export const description = '';
import { TestGroup } from '../../../../common/framework/test_group.js';
import { makeTestGroup } from '../../../../common/framework/test_group.js';
import { GPUTest } from '../../../gpu_test.js';
function getBufferDesc(): GPUBufferDescriptor {
@ -10,7 +10,7 @@ function getBufferDesc(): GPUBufferDescriptor {
};
}
export const g = new TestGroup(GPUTest);
export const g = makeTestGroup(GPUTest);
g.test('mapWriteAsync').fn(async t => {
const buffer = t.device.createBuffer(getBufferDesc());

4
src/webgpu/api/operation/command_buffer/basic.spec.ts
View file

@ -2,10 +2,10 @@ export const description = `
Basic tests.
`;
import { TestGroup } from '../../../../common/framework/test_group.js';
import { makeTestGroup } from '../../../../common/framework/test_group.js';
import { GPUTest } from '../../../gpu_test.js';
export const g = new TestGroup(GPUTest);
export const g = makeTestGroup(GPUTest);
g.test('empty').fn(async t => {
const encoder = t.device.createCommandEncoder();

4
src/webgpu/api/operation/command_buffer/copies.spec.ts
View file

@ -2,10 +2,10 @@ export const description = `
copy{Buffer,Texture}To{Buffer,Texture} tests.
`;
import { TestGroup } from '../../../../common/framework/test_group.js';
import { makeTestGroup } from '../../../../common/framework/test_group.js';
import { GPUTest } from '../../../gpu_test.js';
export const g = new TestGroup(GPUTest);
export const g = makeTestGroup(GPUTest);
g.test('b2b').fn(async t => {
const data = new Uint32Array([0x01020304]);

4
src/webgpu/api/operation/command_buffer/render/basic.spec.ts
View file

@ -2,10 +2,10 @@ export const description = `
Basic command buffer rendering tests.
`;
import { TestGroup } from '../../../../../common/framework/test_group.js';
import { makeTestGroup } from '../../../../../common/framework/test_group.js';
import { GPUTest } from '../../../../gpu_test.js';
export const g = new TestGroup(GPUTest);
export const g = makeTestGroup(GPUTest);
g.test('clear').fn(async t => {
const dst = t.device.createBuffer({

32
src/webgpu/api/operation/fences.spec.ts
View file

@ -1,29 +1,29 @@
export const description = '';
import { attemptGarbageCollection } from '../../../common/framework/collect_garbage.js';
import { TestGroup } from '../../../common/framework/test_group.js';
import { makeTestGroup } from '../../../common/framework/test_group.js';
import { attemptGarbageCollection } from '../../../common/framework/util/collect_garbage.js';
import { raceWithRejectOnTimeout } from '../../../common/framework/util/util.js';
import { GPUTest } from '../../gpu_test.js';
export const g = new TestGroup(GPUTest);
export const g = makeTestGroup(GPUTest);
g.test('initial/no descriptor').fn(t => {
g.test('initial,no_descriptor').fn(t => {
const fence = t.queue.createFence();
t.expect(fence.getCompletedValue() === 0);
});
g.test('initial/empty descriptor').fn(t => {
g.test('initial,empty_descriptor').fn(t => {
const fence = t.queue.createFence({});
t.expect(fence.getCompletedValue() === 0);
});
g.test('initial/descriptor with initialValue').fn(t => {
g.test('initial,descriptor_with_initialValue').fn(t => {
const fence = t.queue.createFence({ initialValue: 2 });
t.expect(fence.getCompletedValue() === 2);
});
// Promise resolves when onCompletion value is less than signal value.
g.test('wait/less than signaled').fn(async t => {
g.test('wait,less_than_signaled').fn(async t => {
const fence = t.queue.createFence();
t.queue.signal(fence, 2);
await fence.onCompletion(1);
@ -31,7 +31,7 @@ g.test('wait/less than signaled').fn(async t => {
});
// Promise resolves when onCompletion value is equal to signal value.
g.test('wait/equal to signaled').fn(async t => {
g.test('wait,equal_to_signaled').fn(async t => {
const fence = t.queue.createFence();
t.queue.signal(fence, 2);
await fence.onCompletion(2);
@ -39,7 +39,7 @@ g.test('wait/equal to signaled').fn(async t => {
});
// All promises resolve when signal is called once.
g.test('wait/signaled once').fn(async t => {
g.test('wait,signaled_once').fn(async t => {
const fence = t.queue.createFence();
t.queue.signal(fence, 20);
const promises = [];
@ -54,7 +54,7 @@ g.test('wait/signaled once').fn(async t => {
});
// Promise resolves when signal is called multiple times.
g.test('wait/signaled multiple times').fn(async t => {
g.test('wait,signaled_multiple_times').fn(async t => {
const fence = t.queue.createFence();
t.queue.signal(fence, 1);
t.queue.signal(fence, 2);
@ -63,7 +63,7 @@ g.test('wait/signaled multiple times').fn(async t => {
});
// Promise resolves if fence has already completed.
g.test('wait/already completed').fn(async t => {
g.test('wait,already_completed').fn(async t => {
const fence = t.queue.createFence();
t.queue.signal(fence, 2);
@ -81,7 +81,7 @@ g.test('wait/already completed').fn(async t => {
});
// Test many calls to signal and wait on fence values one at a time.
g.test('wait/many/serially').fn(async t => {
g.test('wait,many,serially').fn(async t => {
const fence = t.queue.createFence();
for (let i = 1; i <= 20; ++i) {
t.queue.signal(fence, i);
@ -91,7 +91,7 @@ g.test('wait/many/serially').fn(async t => {
});
// Test many calls to signal and wait on all fence values.
g.test('wait/many/parallel').fn(async t => {
g.test('wait,many,parallel').fn(async t => {
const fence = t.queue.createFence();
const promises = [];
for (let i = 1; i <= 20; ++i) {
@ -107,7 +107,7 @@ g.test('wait/many/parallel').fn(async t => {
});
// Test onCompletion promise resolves within a time limit.
g.test('wait/resolves within timeout').fn(t => {
g.test('wait,resolves_within_timeout').fn(t => {
const fence = t.queue.createFence();
t.queue.signal(fence, 2);
@ -122,7 +122,7 @@ g.test('wait/resolves within timeout').fn(t => {
});
// Test dropping references to the fence and onCompletion promise does not crash.
g.test('drop/fence and promise').fn(async t => {
g.test('drop,fence_and_promise').fn(async t => {
{
const fence = t.queue.createFence();
t.queue.signal(fence, 2);
@ -132,7 +132,7 @@ g.test('drop/fence and promise').fn(async t => {
});
// Test dropping references to the fence and holding the promise does not crash.
g.test('drop/promise').fn(async t => {
g.test('drop,promise').fn(async t => {
let promise;
{
const fence = t.queue.createFence();

6
src/webgpu/api/operation/resource_init/copied_texture_clear.spec.ts
View file

@ -1,7 +1,7 @@
export const description = 'Test uninitialized textures are initialized to zero when copied.';
import * as C from '../../../../common/constants.js';
import { TestGroup } from '../../../../common/framework/test_group.js';
import { makeTestGroup } from '../../../../common/framework/test_group.js';
import { assert, unreachable } from '../../../../common/framework/util/util.js';
import { SubresourceRange } from '../../../util/texture/subresource.js';
@ -78,9 +78,9 @@ class CopiedTextureClearTest extends TextureZeroInitTest {
}
}
export const g = new TestGroup(CopiedTextureClearTest);
export const g = makeTestGroup(CopiedTextureClearTest);
g.test('uninitialized texture is zero')
g.test('uninitialized_texture_is_zero')
.params(TextureZeroInitTest.generateParams([ReadMethod.CopyToBuffer, ReadMethod.CopyToTexture]))
.fn(t => {
t.run();

8
src/webgpu/api/operation/resource_init/texture_zero_init_test.ts
View file

@ -1,7 +1,7 @@
import * as C from '../../../../common/constants.js';
import { TestCaseRecorder } from '../../../../common/framework/logger.js';
import { params, poptions, pbool } from '../../../../common/framework/params.js';
import { ParamSpec } from '../../../../common/framework/params_utils.js';
import { TestCaseRecorder } from '../../../../common/framework/logging/test_case_recorder.js';
import { params, poptions, pbool } from '../../../../common/framework/params_builder.js';
import { CaseParams } from '../../../../common/framework/params_utils.js';
import { assert, unreachable } from '../../../../common/framework/util/util.js';
import { kTextureAspects, kTextureFormatInfo, kTextureFormats } from '../../../capability_info.js';
import { GPUTest } from '../../../gpu_test.js';
@ -214,7 +214,7 @@ function getRequiredTextureUsage(
export abstract class TextureZeroInitTest extends GPUTest {
protected stateToTexelComponents: { [k in InitializedState]: PerTexelComponent<number> };
constructor(rec: TestCaseRecorder, params: ParamSpec) {
constructor(rec: TestCaseRecorder, params: CaseParams) {
super(rec, params);
const stateToTexelComponents = (state: InitializedState) => {

20
src/webgpu/api/validation/createBindGroup.spec.ts
View file

@ -3,8 +3,8 @@ createBindGroup validation tests.
`;
import * as C from '../../../common/constants.js';
import { poptions, params } from '../../../common/framework/params.js';
import { TestGroup } from '../../../common/framework/test_group.js';
import { poptions, params } from '../../../common/framework/params_builder.js';
import { makeTestGroup } from '../../../common/framework/test_group.js';
import { unreachable } from '../../../common/framework/util/util.js';
import {
kBindingTypes,
@ -21,9 +21,9 @@ function clone<T extends GPUTextureDescriptor>(descriptor: T): T {
return JSON.parse(JSON.stringify(descriptor));
}
export const g = new TestGroup(ValidationTest);
export const g = makeTestGroup(ValidationTest);
g.test('binding count mismatch').fn(async t => {
g.test('binding_count_mismatch').fn(async t => {
const bindGroupLayout = t.device.createBindGroupLayout({
entries: [{ binding: 0, visibility: GPUShaderStage.COMPUTE, type: 'storage-buffer' }],
});
@ -51,7 +51,7 @@ g.test('binding count mismatch').fn(async t => {
});
});
g.test('binding must be present in layout').fn(async t => {
g.test('binding_must_be_present_in_layout').fn(async t => {
const bindGroupLayout = t.device.createBindGroupLayout({
entries: [{ binding: 0, visibility: GPUShaderStage.COMPUTE, type: 'storage-buffer' }],
});
@ -75,7 +75,7 @@ g.test('binding must be present in layout').fn(async t => {
});
});
g.test('buffer binding must contain exactly one buffer of its type')
g.test('buffer_binding_must_contain_exactly_one_buffer_of_its_type')
.params(
params()
.combine(poptions('bindingType', kBindingTypes))
@ -100,7 +100,7 @@ g.test('buffer binding must contain exactly one buffer of its type')
}, !resourceBindingMatches);
});
g.test('texture binding must have correct usage')
g.test('texture_binding_must_have_correct_usage')
.params(
params()
.combine(poptions('type', kTextureBindingTypes))
@ -130,7 +130,7 @@ g.test('texture binding must have correct usage')
}, shouldError);
});
g.test('texture must have correct component type')
g.test('texture_must_have_correct_component_type')
.params(
poptions('textureComponentType', [
C.TextureComponentType.Float,
@ -208,7 +208,7 @@ g.test('texture must have correct component type')
});
// TODO: Write test for all dimensions.
g.test('texture must have correct dimension').fn(async t => {
g.test('texture_must_have_correct_dimension').fn(async t => {
const bindGroupLayout = t.device.createBindGroupLayout({
entries: [
{
@ -244,7 +244,7 @@ g.test('texture must have correct dimension').fn(async t => {
});
});
g.test('buffer offset and size for bind groups match')
g.test('buffer_offset_and_size_for_bind_groups_match')
.params([
{ offset: 0, size: 512, _success: true }, // offset 0 is valid
{ offset: 256, size: 256, _success: true }, // offset 256 (aligned) is valid

18
src/webgpu/api/validation/createBindGroupLayout.spec.ts
View file

@ -3,8 +3,8 @@ createBindGroupLayout validation tests.
`;
import * as C from '../../../common/constants.js';
import { poptions, params } from '../../../common/framework/params.js';
import { TestGroup } from '../../../common/framework/test_group.js';
import { poptions, params } from '../../../common/framework/params_builder.js';
import { makeTestGroup } from '../../../common/framework/test_group.js';
import {
kBindingTypeInfo,
kBindingTypes,
@ -18,9 +18,9 @@ function clone<T extends GPUBindGroupLayoutDescriptor>(descriptor: T): T {
return JSON.parse(JSON.stringify(descriptor));
}
export const g = new TestGroup(ValidationTest);
export const g = makeTestGroup(ValidationTest);
g.test('some binding index was specified more than once').fn(async t => {
g.test('some_binding_index_was_specified_more_than_once').fn(async t => {
const goodDescriptor = {
entries: [
{ binding: 0, visibility: GPUShaderStage.COMPUTE, type: C.BindingType.StorageBuffer },
@ -40,13 +40,13 @@ g.test('some binding index was specified more than once').fn(async t => {
});
});
g.test('Visibility of bindings can be 0').fn(async t => {
g.test('visibility_of_bindings_can_be_0').fn(async t => {
t.device.createBindGroupLayout({
entries: [{ binding: 0, visibility: 0, type: 'storage-buffer' }],
});
});
g.test('number of dynamic buffers exceeds the maximum value')
g.test('number_of_dynamic_buffers_exceeds_the_maximum_value')
.params([
{ type: C.BindingType.StorageBuffer, maxDynamicBufferCount: 4 },
{ type: C.BindingType.UniformBuffer, maxDynamicBufferCount: 8 },
@ -88,7 +88,7 @@ g.test('number of dynamic buffers exceeds the maximum value')
});
});
g.test('dynamic set to true is allowed only for buffers')
g.test('dynamic_set_to_true_is_allowed_only_for_buffers')
.params(poptions('type', kBindingTypes))
.fn(async t => {
const { type } = t.params;
@ -143,7 +143,7 @@ const kCasesForMaxResourcesPerStageTests = params()
// Should never fail unless kMaxBindingsPerBindGroup is exceeded, because the validation for
// resources-of-type-per-stage is in pipeline layout creation.
g.test('max resources per stage/in bind group layout')
g.test('max_resources_per_stage,in_bind_group_layout')
.params(kCasesForMaxResourcesPerStageTests)
.fn(async t => {
const { maxedType, extraType, maxedVisibility, extraVisibility } = t.params;
@ -184,7 +184,7 @@ g.test('max resources per stage/in bind group layout')
// One pipeline layout can have a maximum number of each type of binding *per stage* (which is
// different for each type). Test that the max works, then add one more binding of same-or-different
// type and same-or-different visibility.
g.test('max resources per stage/in pipeline layout')
g.test('max_resources_per_stage,in_pipeline_layout')
.params(kCasesForMaxResourcesPerStageTests)
.fn(async t => {
const { maxedType, extraType, maxedVisibility, extraVisibility } = t.params;

12
src/webgpu/api/validation/createPipelineLayout.spec.ts
View file

@ -3,8 +3,8 @@ createPipelineLayout validation tests.
`;
import * as C from '../../../common/constants.js';
import { pbool, poptions, params } from '../../../common/framework/params.js';
import { TestGroup } from '../../../common/framework/test_group.js';
import { pbool, poptions, params } from '../../../common/framework/params_builder.js';
import { makeTestGroup } from '../../../common/framework/test_group.js';
import {
kBindingTypeInfo,
kBindingTypes,
@ -17,9 +17,9 @@ function clone<T extends GPUBindGroupLayoutDescriptor>(descriptor: T): T {
return JSON.parse(JSON.stringify(descriptor));
}
export const g = new TestGroup(ValidationTest);
export const g = makeTestGroup(ValidationTest);
g.test('number of dynamic buffers exceeds the maximum value')
g.test('number_of_dynamic_buffers_exceeds_the_maximum_value')
.params(
params()
.combine(poptions('visibility', [0, 2, 4, 6]))
@ -74,7 +74,7 @@ g.test('number of dynamic buffers exceeds the maximum value')
});
});
g.test('visibility and dynamic offsets')
g.test('visibility_and_dynamic_offsets')
.params(
params()
.combine(poptions('type', kBindingTypes))
@ -101,7 +101,7 @@ g.test('visibility and dynamic offsets')
}, !success);
});
g.test('number of bind group layouts exceeds the maximum value').fn(async t => {
g.test('number_of_bind_group_layouts_exceeds_the_maximum_value').fn(async t => {
const bindGroupLayoutDescriptor: GPUBindGroupLayoutDescriptor = {
entries: [],
};

18
src/webgpu/api/validation/createTexture.spec.ts
View file

@ -2,8 +2,8 @@ export const description = `
createTexture validation tests.
`;
import { poptions } from '../../../common/framework/params.js';
import { TestGroup } from '../../../common/framework/test_group.js';
import { poptions } from '../../../common/framework/params_builder.js';
import { makeTestGroup } from '../../../common/framework/test_group.js';
import { kTextureFormatInfo, kTextureFormats } from '../../capability_info.js';
import { ValidationTest } from './validation_test.js';
@ -38,9 +38,9 @@ class F extends ValidationTest {
}
}
export const g = new TestGroup(F);
export const g = makeTestGroup(F);
g.test('validation of sampleCount')
g.test('validation_of_sampleCount')
.params([
// TODO: Consider making a list of "valid"+"invalid" texture descriptors in capability_info.
{ sampleCount: 0, _success: false }, // sampleCount of 0 is not allowed
@ -63,7 +63,7 @@ g.test('validation of sampleCount')
}, !_success);
});
g.test('validation of mipLevelCount')
g.test('validation_of_mipLevelCount')
.params([
{ width: 32, height: 32, mipLevelCount: 1, _success: true }, // mipLevelCount of 1 is allowed
{ width: 32, height: 32, mipLevelCount: 0, _success: false }, // mipLevelCount of 0 is not allowed
@ -85,20 +85,20 @@ g.test('validation of mipLevelCount')
}, !_success);
});
g.test('it is valid to destroy a texture').fn(t => {
g.test('it_is_valid_to_destroy_a_texture').fn(t => {
const descriptor = t.getDescriptor();
const texture = t.device.createTexture(descriptor);
texture.destroy();
});
g.test('it is valid to destroy a destroyed texture').fn(t => {
g.test('it_is_valid_to_destroy_a_destroyed_texture').fn(t => {
const descriptor = t.getDescriptor();
const texture = t.device.createTexture(descriptor);
texture.destroy();
texture.destroy();
});
g.test('it is invalid to submit a destroyed texture before and after encode')
g.test('it_is_invalid_to_submit_a_destroyed_texture_before_and_after_encode')
.params([
{ destroyBeforeEncode: false, destroyAfterEncode: false, _success: true },
{ destroyBeforeEncode: true, destroyAfterEncode: false, _success: false },
@ -136,7 +136,7 @@ g.test('it is invalid to submit a destroyed texture before and after encode')
}, !_success);
});
g.test('it is invalid to have an output attachment texture with non renderable format')
g.test('it_is_invalid_to_have_an_output_attachment_texture_with_non_renderable_format')
.params(poptions('format', kTextureFormats))
.fn(async t => {
const format: GPUTextureFormat = t.params.format;

20
src/webgpu/api/validation/createView.spec.ts
View file

@ -3,7 +3,7 @@ createView validation tests.
`;
import * as C from '../../../common/constants.js';
import { TestGroup } from '../../../common/framework/test_group.js';
import { makeTestGroup } from '../../../common/framework/test_group.js';
import { ValidationTest } from './validation_test.js';
@ -68,9 +68,9 @@ class F extends ValidationTest {
}
}
export const g = new TestGroup(F);
export const g = makeTestGroup(F);
g.test('creating texture view on a 2D non array texture')
g.test('creating_texture_view_on_a_2D_non_array_texture')
.params([
{ _success: true }, // default view works
{ arrayLayerCount: 1, _success: true }, // it is OK to create a 2D texture view on a 2D texture
@ -107,7 +107,7 @@ g.test('creating texture view on a 2D non array texture')
}, !_success);
});
g.test('creating texture view on a 2D array texture')
g.test('creating_texture_view_on_a_2D_array_texture')
.params([
{ _success: true }, // default view works
{ dimension: C.TextureViewDimension.E2d, arrayLayerCount: 1, _success: true }, // it is OK to create a 2D texture view on a 2D array texture
@ -139,7 +139,7 @@ g.test('creating texture view on a 2D array texture')
}, !_success);
});
g.test('Using defaults validates the same as setting values for more than 1 array layer')
g.test('Using_defaults_validates_the_same_as_setting_values_for_more_than_1_array_layer')
.params([
{ _success: true },
{ format: C.TextureFormat.RGBA8Unorm, _success: true },
@ -171,7 +171,7 @@ g.test('Using defaults validates the same as setting values for more than 1 arra
}, !_success);
});
g.test('Using defaults validates the same as setting values for only 1 array layer')
g.test('Using_defaults_validates_the_same_as_setting_values_for_only_1_array_layer')
.params([
{ _success: true },
{ format: C.TextureFormat.RGBA8Unorm, _success: true },
@ -196,7 +196,7 @@ g.test('Using defaults validates the same as setting values for only 1 array lay
}, !_success);
});
g.test('creating cube map texture view')
g.test('creating_cube_map_texture_view')
.params([
{ dimension: C.TextureViewDimension.Cube, arrayLayerCount: 6, _success: true }, // it is OK to create a cube map texture view with arrayLayerCount == 6
// it is an error to create a cube map texture view with arrayLayerCount != 6
@ -224,7 +224,7 @@ g.test('creating cube map texture view')
}, !_success);
});
g.test('creating cube map texture view with a non square texture')
g.test('creating_cube_map_texture_view_with_a_non_square_texture')
.params([
{ dimension: C.TextureViewDimension.Cube, arrayLayerCount: 6 }, // it is an error to create a cube map texture view with width != height.
{ dimension: C.TextureViewDimension.CubeArray, arrayLayerCount: 12 }, // it is an error to create a cube map array texture view with width != height.
@ -250,7 +250,7 @@ g.test('creating cube map texture view with a non square texture')
});
// TODO: add more tests when rules are fully implemented.
g.test('test the format compatibility rules when creating a texture view').fn(async t => {
g.test('test_the_format_compatibility_rules_when_creating_a_texture_view').fn(async t => {
const texture = t.createTexture({ arrayLayerCount: 1 });
const descriptor = t.getDescriptor({
@ -263,7 +263,7 @@ g.test('test the format compatibility rules when creating a texture view').fn(as
});
});
g.test('it is invalid to use a texture view created from a destroyed texture').fn(async t => {
g.test('it_is_invalid_to_use_a_texture_view_created_from_a_destroyed_texture').fn(async t => {
const texture = t.createTexture({ arrayLayerCount: 1 });
const commandEncoder = t.device.createCommandEncoder();

16
src/webgpu/api/validation/error_scope.spec.ts
View file

@ -4,7 +4,7 @@ error scope validation tests.
import { Fixture } from '../../../common/framework/fixture.js';
import { getGPU } from '../../../common/framework/gpu/implementation.js';
import { TestGroup } from '../../../common/framework/test_group.js';
import { makeTestGroup } from '../../../common/framework/test_group.js';
import { assert, raceWithRejectOnTimeout } from '../../../common/framework/util/util.js';
class F extends Fixture {
@ -58,9 +58,9 @@ class F extends Fixture {
}
}
export const g = new TestGroup(F);
export const g = makeTestGroup(F);
g.test('simple case where the error scope catches an error').fn(async t => {
g.test('simple_case_where_the_error_scope_catches_an_error').fn(async t => {
t.device.pushErrorScope('validation');
t.createErrorBuffer();
@ -69,7 +69,7 @@ g.test('simple case where the error scope catches an error').fn(async t => {
t.expect(error instanceof GPUValidationError);
});
g.test('errors bubble to the parent scope if not handled by the current scope').fn(async t => {
g.test('errors_bubble_to_the_parent_scope_if_not_handled_by_the_current_scope').fn(async t => {
t.device.pushErrorScope('validation');
t.device.pushErrorScope('out-of-memory');
@ -85,7 +85,7 @@ g.test('errors bubble to the parent scope if not handled by the current scope').
}
});
g.test('if an error scope matches an error it does not bubble to the parent scope').fn(async t => {
g.test('if_an_error_scope_matches_an_error_it_does_not_bubble_to_the_parent_scope').fn(async t => {
t.device.pushErrorScope('validation');
t.device.pushErrorScope('validation');
@ -101,7 +101,7 @@ g.test('if an error scope matches an error it does not bubble to the parent scop
}
});
g.test('if no error scope handles an error it fires an uncapturederror event').fn(async t => {
g.test('if_no_error_scope_handles_an_error_it_fires_an_uncapturederror_event').fn(async t => {
t.device.pushErrorScope('out-of-memory');
const uncapturedErrorEvent = await t.expectUncapturedError(() => {
@ -113,7 +113,7 @@ g.test('if no error scope handles an error it fires an uncapturederror event').f
t.expect(error === null);
});
g.test('push/popping sibling error scopes must be balanced').fn(async t => {
g.test('push,popping_sibling_error_scopes_must_be_balanced').fn(async t => {
{
const promise = t.device.popErrorScope();
t.shouldReject('OperationError', promise);
@ -133,7 +133,7 @@ g.test('push/popping sibling error scopes must be balanced').fn(async t => {
}
});
g.test('push/popping nested error scopes must be balanced').fn(async t => {
g.test('push,popping_nested_error_scopes_must_be_balanced').fn(async t => {
{
const promise = t.device.popErrorScope();
t.shouldReject('OperationError', promise);

18
src/webgpu/api/validation/fences.spec.ts
View file

@ -2,14 +2,14 @@ export const description = `
fences validation tests.
`;
import { TestGroup } from '../../../common/framework/test_group.js';
import { makeTestGroup } from '../../../common/framework/test_group.js';
import { ValidationTest } from './validation_test.js';
export const g = new TestGroup(ValidationTest);
export const g = makeTestGroup(ValidationTest);
// TODO: Remove if https://github.com/gpuweb/gpuweb/issues/377 is decided
g.test('wait on a fence without signaling the value is invalid').fn(async t => {
g.test('wait_on_a_fence_without_signaling_the_value_is_invalid').fn(async t => {
const fence = t.queue.createFence();
t.expectValidationError(() => {
@ -19,7 +19,7 @@ g.test('wait on a fence without signaling the value is invalid').fn(async t => {
});
// TODO: Remove if https://github.com/gpuweb/gpuweb/issues/377 is decided
g.test('wait on a fence with a value greater than signaled value is invalid').fn(async t => {
g.test('wait_on_a_fence_with_a_value_greater_than_signaled_value_is_invalid').fn(async t => {
const fence = t.queue.createFence();
t.queue.signal(fence, 2);
@ -29,7 +29,7 @@ g.test('wait on a fence with a value greater than signaled value is invalid').fn
});
});
g.test('signal a value lower than signaled value is invalid').fn(async t => {
g.test('signal_a_value_lower_than_signaled_value_is_invalid').fn(async t => {
const fence = t.queue.createFence({ initialValue: 1 });
t.expectValidationError(() => {
@ -37,7 +37,7 @@ g.test('signal a value lower than signaled value is invalid').fn(async t => {
});
});
g.test('signal a value equal to signaled value is invalid').fn(async t => {
g.test('signal_a_value_equal_to_signaled_value_is_invalid').fn(async t => {
const fence = t.queue.createFence({ initialValue: 1 });
t.expectValidationError(() => {
@ -45,7 +45,7 @@ g.test('signal a value equal to signaled value is invalid').fn(async t => {
});
});
g.test('increasing fence value by more than 1 succeeds').fn(async t => {
g.test('increasing_fence_value_by_more_than_1_succeeds').fn(async t => {
const fence = t.queue.createFence();
t.queue.signal(fence, 2);
@ -55,7 +55,7 @@ g.test('increasing fence value by more than 1 succeeds').fn(async t => {
await fence.onCompletion(6);
});
g.test('signal a fence on a different device than it was created on is invalid').fn(async t => {
g.test('signal_a_fence_on_a_different_device_than_it_was_created_on_is_invalid').fn(async t => {
const fence = t.queue.createFence();
const anotherDevice = await t.device.adapter.requestDevice();
@ -66,7 +66,7 @@ g.test('signal a fence on a different device than it was created on is invalid')
});
});
g.test('signal a fence on a different device does not update fence signaled value').fn(async t => {
g.test('signal_a_fence_on_a_different_device_does_not_update_fence_signaled_value').fn(async t => {
const fence = t.queue.createFence({ initialValue: 1 });
const anotherDevice = await t.device.adapter.requestDevice();

6
src/webgpu/api/validation/queue_submit.spec.ts
View file

@ -2,13 +2,13 @@ export const description = `
queue submit validation tests.
`;
import { TestGroup } from '../../../common/framework/test_group.js';
import { makeTestGroup } from '../../../common/framework/test_group.js';
import { ValidationTest } from './validation_test.js';
export const g = new TestGroup(ValidationTest);
export const g = makeTestGroup(ValidationTest);
g.test('submitting with a mapped buffer is disallowed').fn(async t => {
g.test('submitting_with_a_mapped_buffer_is_disallowed').fn(async t => {
const buffer = t.device.createBuffer({
size: 4,
usage: GPUBufferUsage.MAP_WRITE | GPUBufferUsage.COPY_SRC,

40
src/webgpu/api/validation/render_pass_descriptor.spec.ts
View file

@ -2,7 +2,7 @@ export const description = `
render pass descriptor validation tests.
`;
import { TestGroup } from '../../../common/framework/test_group.js';
import { makeTestGroup } from '../../../common/framework/test_group.js';
import { ValidationTest } from './validation_test.js';
@ -75,9 +75,9 @@ class F extends ValidationTest {
}
}
export const g = new TestGroup(F);
export const g = makeTestGroup(F);
g.test('a render pass with only one color is ok').fn(t => {
g.test('a_render_pass_with_only_one_color_is_ok').fn(t => {
const colorTexture = t.createTexture({ format: 'rgba8unorm' });
const descriptor = {
colorAttachments: [t.getColorAttachment(colorTexture)],
@ -86,7 +86,7 @@ g.test('a render pass with only one color is ok').fn(t => {
t.tryRenderPass(true, descriptor);
});
g.test('a render pass with only one depth attachment is ok').fn(t => {
g.test('a_render_pass_with_only_one_depth_attachment_is_ok').fn(t => {
const depthStencilTexture = t.createTexture({ format: 'depth24plus-stencil8' });
const descriptor = {
colorAttachments: [],
@ -96,7 +96,7 @@ g.test('a render pass with only one depth attachment is ok').fn(t => {
t.tryRenderPass(true, descriptor);
});
g.test('OOB color attachment indices are handled')
g.test('OOB_color_attachment_indices_are_handled')
.params([
{ colorAttachmentsCount: 4, _success: true }, // Control case
{ colorAttachmentsCount: 5, _success: false }, // Out of bounds
@ -113,7 +113,7 @@ g.test('OOB color attachment indices are handled')
await t.tryRenderPass(_success, { colorAttachments });
});
g.test('attachments must have the same size').fn(async t => {
g.test('attachments_must_have_the_same_size').fn(async t => {
const colorTexture1x1A = t.createTexture({ width: 1, height: 1, format: 'rgba8unorm' });
const colorTexture1x1B = t.createTexture({ width: 1, height: 1, format: 'rgba8unorm' });
const colorTexture2x2 = t.createTexture({ width: 2, height: 2, format: 'rgba8unorm' });
@ -165,7 +165,7 @@ g.test('attachments must have the same size').fn(async t => {
}
});
g.test('attachments must match whether they are used for color or depth stencil').fn(async t => {
g.test('attachments_must_match_whether_they_are_used_for_color_or_depth_stencil').fn(async t => {
const colorTexture = t.createTexture({ format: 'rgba8unorm' });
const depthStencilTexture = t.createTexture({ format: 'depth24plus-stencil8' });
@ -188,7 +188,7 @@ g.test('attachments must match whether they are used for color or depth stencil'
}
});
g.test('check layer count for color or depth stencil')
g.test('check_layer_count_for_color_or_depth_stencil')
.params([
{ arrayLayerCount: 5, baseArrayLayer: 0, _success: false }, // using 2D array texture view with arrayLayerCount > 1 is not allowed
{ arrayLayerCount: 1, baseArrayLayer: 0, _success: true }, // using 2D array texture view that covers the first layer of the texture is OK
@ -257,7 +257,7 @@ g.test('check layer count for color or depth stencil')
}
});
g.test('check mip level count for color or depth stencil')
g.test('check_mip_level_count_for_color_or_depth_stencil')
.params([
{ mipLevelCount: 2, baseMipLevel: 0, _success: false }, // using 2D texture view with mipLevelCount > 1 is not allowed
{ mipLevelCount: 1, baseMipLevel: 0, _success: true }, // using 2D texture view that covers the first level of the texture is OK
@ -326,7 +326,7 @@ g.test('check mip level count for color or depth stencil')
}
});
g.test('it is invalid to set resolve target if color attachment is non multisampled').fn(
g.test('it_is_invalid_to_set_resolve_target_if_color_attachment_is_non_multisampled').fn(
async t => {
const colorTexture = t.createTexture({ sampleCount: 1 });
const resolveTargetTexture = t.createTexture({ sampleCount: 1 });
@ -345,7 +345,7 @@ g.test('it is invalid to set resolve target if color attachment is non multisamp
}
);
g.test('check the use of multisampled textures as color attachments').fn(async t => {
g.test('check_the_use_of_multisampled_textures_as_color_attachments').fn(async t => {
const colorTexture = t.createTexture({ sampleCount: 1 });
const multisampledColorTexture = t.createTexture({ sampleCount: 4 });
@ -369,7 +369,7 @@ g.test('check the use of multisampled textures as color attachments').fn(async t
}
});
g.test('it is invalid to use a multisampled resolve target').fn(async t => {
g.test('it_is_invalid_to_use_a_multisampled_resolve_target').fn(async t => {
const multisampledColorTexture = t.createTexture({ sampleCount: 4 });
const multisampledResolveTargetTexture = t.createTexture({ sampleCount: 4 });
@ -383,7 +383,7 @@ g.test('it is invalid to use a multisampled resolve target').fn(async t => {
await t.tryRenderPass(false, descriptor);
});
g.test('it is invalid to use a resolve target with array layer count greater than 1').fn(
g.test('it_is_invalid_to_use_a_resolve_target_with_array_layer_count_greater_than_1').fn(
async t => {
const multisampledColorTexture = t.createTexture({ sampleCount: 4 });
const resolveTargetTexture = t.createTexture({ arrayLayerCount: 2 });
@ -399,7 +399,7 @@ g.test('it is invalid to use a resolve target with array layer count greater tha
}
);
g.test('it is invalid to use a resolve target with mipmap level count greater than 1').fn(
g.test('it_is_invalid_to_use_a_resolve_target_with_mipmap_level_count_greater_than_1').fn(
async t => {
const multisampledColorTexture = t.createTexture({ sampleCount: 4 });
const resolveTargetTexture = t.createTexture({ mipLevelCount: 2 });
@ -415,7 +415,7 @@ g.test('it is invalid to use a resolve target with mipmap level count greater th
}
);
g.test('it is invalid to use a resolve target whose usage is not output attachment').fn(async t => {
g.test('it_is_invalid_to_use_a_resolve_target_whose_usage_is_not_output_attachment').fn(async t => {
const multisampledColorTexture = t.createTexture({ sampleCount: 4 });
const resolveTargetTexture = t.createTexture({
usage: GPUTextureUsage.COPY_SRC | GPUTextureUsage.COPY_DST,
@ -431,7 +431,7 @@ g.test('it is invalid to use a resolve target whose usage is not output attachme
await t.tryRenderPass(false, descriptor);
});
g.test('it is invalid to use a resolve target in error state').fn(async t => {
g.test('it_is_invalid_to_use_a_resolve_target_in_error_state').fn(async t => {
const ARRAY_LAYER_COUNT = 1;
const multisampledColorTexture = t.createTexture({ sampleCount: 4 });
@ -453,7 +453,7 @@ g.test('it is invalid to use a resolve target in error state').fn(async t => {
await t.tryRenderPass(false, descriptor);
});
g.test('use of multisampled attachment and non multisampled resolve target is allowed').fn(
g.test('use_of_multisampled_attachment_and_non_multisampled_resolve_target_is_allowed').fn(
async t => {
const multisampledColorTexture = t.createTexture({ sampleCount: 4 });
const resolveTargetTexture = t.createTexture({ sampleCount: 1 });
@ -469,7 +469,7 @@ g.test('use of multisampled attachment and non multisampled resolve target is al
}
);
g.test('use a resolve target in a format different than the attachment is not allowed').fn(
g.test('use_a_resolve_target_in_a_format_different_than_the_attachment_is_not_allowed').fn(
async t => {
const multisampledColorTexture = t.createTexture({ sampleCount: 4 });
const resolveTargetTexture = t.createTexture({ format: 'bgra8unorm' });
@ -485,7 +485,7 @@ g.test('use a resolve target in a format different than the attachment is not al
}
);
g.test('size of the resolve target must be the same as the color attachment').fn(async t => {
g.test('size_of_the_resolve_target_must_be_the_same_as_the_color_attachment').fn(async t => {
const size = 16;
const multisampledColorTexture = t.createTexture({ width: size, height: size, sampleCount: 4 });
const resolveTargetTexture = t.createTexture({
@ -523,7 +523,7 @@ g.test('size of the resolve target must be the same as the color attachment').fn
}
});
g.test('check depth stencil attachment sample counts mismatch').fn(async t => {
g.test('check_depth_stencil_attachment_sample_counts_mismatch').fn(async t => {
const multisampledDepthStencilTexture = t.createTexture({
sampleCount: 4,
format: 'depth24plus-stencil8',

14
src/webgpu/api/validation/setBindGroup.spec.ts
View file

@ -2,8 +2,8 @@ export const description = `
setBindGroup validation tests.
`;
import { poptions, params } from '../../../common/framework/params.js';
import { TestGroup } from '../../../common/framework/test_group.js';
import { poptions, params } from '../../../common/framework/params_builder.js';
import { makeTestGroup } from '../../../common/framework/test_group.js';
import { ValidationTest } from './validation_test.js';
@ -48,9 +48,9 @@ class F extends ValidationTest {
}
}
export const g = new TestGroup(F);
export const g = makeTestGroup(F);
g.test('dynamic offsets passed but not expected/compute pass')
g.test('dynamic_offsets_passed_but_not_expected,compute_pass')
.params(poptions('type', ['compute', 'renderpass', 'renderbundle']))
.fn(async t => {
const bindGroupLayout = t.device.createBindGroupLayout({ entries: [] });
@ -91,7 +91,7 @@ g.test('dynamic offsets passed but not expected/compute pass')
});
});
g.test('dynamic offsets match expectations in pass encoder')
g.test('dynamic_offsets_match_expectations_in_pass_encoder')
.params(
params()
.combine(poptions('type', ['compute', 'renderpass', 'renderbundle']))
@ -107,12 +107,12 @@ g.test('dynamic offsets match expectations in pass encoder')
// Dynamic uniform buffer out of bounds because of binding size
{ dynamicOffsets: [512, 0], _success: false },
{ dynamicOffsets: [1024, 0], _success: false },
{ dynamicOffsets: [Number.MAX_SAFE_INTEGER, 0], _success: false },
{ dynamicOffsets: [0xffffffff, 0], _success: false },
// Dynamic storage buffer out of bounds because of binding size
{ dynamicOffsets: [0, 512], _success: false },
{ dynamicOffsets: [0, 1024], _success: false },
{ dynamicOffsets: [0, Number.MAX_SAFE_INTEGER], _success: false },
{ dynamicOffsets: [0, 0xffffffff], _success: false },
])
)
.fn(async t => {

8
src/webgpu/api/validation/setBlendColor.spec.ts
View file

@ -2,7 +2,7 @@ export const description = `
setBlendColor validation tests.
`;
import { TestGroup } from '../../../common/framework/test_group.js';
import { makeTestGroup } from '../../../common/framework/test_group.js';
import { ValidationTest } from './validation_test.js';
@ -26,9 +26,9 @@ class F extends ValidationTest {
}
}
export const g = new TestGroup(F);
export const g = makeTestGroup(F);
g.test('basic use of setBlendColor').fn(t => {
g.test('basic_use_of_setBlendColor').fn(t => {
const commandEncoder = t.device.createCommandEncoder();
const renderPass = t.beginRenderPass(commandEncoder);
renderPass.setBlendColor({ r: 0, g: 0, b: 0, a: 0 });
@ -36,7 +36,7 @@ g.test('basic use of setBlendColor').fn(t => {
commandEncoder.finish();
});
g.test('setBlendColor allows any number value').fn(t => {
g.test('setBlendColor_allows_any_number_value').fn(t => {
const values = [Number.MIN_SAFE_INTEGER, Number.MAX_SAFE_INTEGER];
for (const value of values) {
const commandEncoder = t.device.createCommandEncoder();

6
src/webgpu/api/validation/setScissorRect.spec.ts
View file

@ -2,7 +2,7 @@ export const description = `
setScissorRect validation tests.
`;
import { TestGroup } from '../../../common/framework/test_group.js';
import { makeTestGroup } from '../../../common/framework/test_group.js';
import { ValidationTest } from './validation_test.js';
@ -29,9 +29,9 @@ class F extends ValidationTest {
}
}
export const g = new TestGroup(F);
export const g = makeTestGroup(F);
g.test('use of setScissorRect')
g.test('use_of_setScissorRect')
.params([
{ x: 0, y: 0, width: 1, height: 1, _success: true }, // Basic use
{ x: 0, y: 0, width: 0, height: 1, _success: false }, // Width of zero is not allowed

8
src/webgpu/api/validation/setStencilReference.spec.ts
View file

@ -2,8 +2,8 @@ export const description = `
setStencilReference validation tests.
`;
import { poptions } from '../../../common/framework/params.js';
import { TestGroup } from '../../../common/framework/test_group.js';
import { poptions } from '../../../common/framework/params_builder.js';
import { makeTestGroup } from '../../../common/framework/test_group.js';
import { ValidationTest } from './validation_test.js';
@ -27,9 +27,9 @@ class F extends ValidationTest {
}
}
export const g = new TestGroup(F);
export const g = makeTestGroup(F);
g.test('use of setStencilReference')
g.test('use_of_setStencilReference')
.params(poptions('reference', [0, 0xffffffff]))
.fn(t => {
const { reference } = t.params;

6
src/webgpu/api/validation/setViewport.spec.ts
View file

@ -2,7 +2,7 @@ export const description = `
setViewport validation tests.
`;
import { TestGroup } from '../../../common/framework/test_group.js';
import { makeTestGroup } from '../../../common/framework/test_group.js';
import { ValidationTest } from './validation_test.js';
@ -29,9 +29,9 @@ class F extends ValidationTest {
}
}
export const g = new TestGroup(F);
export const g = makeTestGroup(F);
g.test('use of setViewport')
g.test('use_of_setViewport')
.params([
{ x: 0, y: 0, width: 1, height: 1, minDepth: 0, maxDepth: 1, _success: true }, // Basic use
{ x: 0, y: 0, width: 0, height: 1, minDepth: 0, maxDepth: 1, _success: false }, // Width of zero is not allowed

14
src/webgpu/examples.spec.ts
View file

@ -4,7 +4,7 @@ Examples of writing CTS tests with various features.
Start here when looking for examples of basic framework usage.
`;
import { TestGroup } from '../common/framework/test_group.js';
import { makeTestGroup } from '../common/framework/test_group.js';
import { GPUTest } from './gpu_test.js';
@ -18,11 +18,11 @@ import { GPUTest } from './gpu_test.js';
// - ?q=webgpu:examples:basic/
// - ?q=webgpu:examples:
export const g = new TestGroup(GPUTest);
export const g = makeTestGroup(GPUTest);
// Note: spaces in test names are replaced with underscores: webgpu:examples:test_name=
/* eslint-disable-next-line @typescript-eslint/no-unused-vars */
g.test('test name').fn(t => {});
g.test('test_name').fn(t => {});
g.test('basic').fn(t => {
t.expect(true);
@ -40,7 +40,7 @@ g.test('basic').fn(t => {
);
});
g.test('basic/async').fn(async t => {
g.test('basic,async').fn(async t => {
// shouldReject must be awaited to ensure it can wait for the promise before the test ends.
t.shouldReject(
// The expected '.name' of the thrown error.
@ -69,7 +69,7 @@ g.test('basic/async').fn(async t => {
//
// - webgpu:examples:basic/params={"x":2,"y":4} runs with t.params = {x: 2, y: 5, _result: 6}.
// - webgpu:examples:basic/params={"x":-10,"y":18} runs with t.params = {x: -10, y: 18, _result: 8}.
g.test('basic/params')
g.test('basic,params')
.params([
{ x: 2, y: 4, _result: 6 }, //
{ x: -10, y: 18, _result: 8 },
@ -79,14 +79,14 @@ g.test('basic/params')
});
// (note the blank comment above to enforce newlines on autoformat)
g.test('gpu/async').fn(async t => {
g.test('gpu,async').fn(async t => {
const fence = t.queue.createFence();
t.queue.signal(fence, 2);
await fence.onCompletion(1);
t.expect(fence.getCompletedValue() === 2);
});
g.test('gpu/buffers').fn(async t => {
g.test('gpu,buffers').fn(async t => {
const data = new Uint32Array([0, 1234, 0]);
const [src, map] = t.device.createBufferMapped({
size: 12,

2
src/webgpu/gpu_test.ts
View file

@ -1,6 +1,6 @@
import { attemptGarbageCollection } from '../common/framework/collect_garbage.js';
import { Fixture } from '../common/framework/fixture.js';
import { DevicePool, TestOOMedShouldAttemptGC } from '../common/framework/gpu/device_pool.js';
import { attemptGarbageCollection } from '../common/framework/util/collect_garbage.js';
import { assert } from '../common/framework/util/util.js';
import {

2
src/webgpu/listing.ts
View file

@ -1,4 +1,4 @@
import { TestSuiteListing } from '../common/framework/listing.js';
import { TestSuiteListing } from '../common/framework/test_suite_listing.js';
import { makeListing } from '../common/tools/crawl.js';
export const listing: Promise<TestSuiteListing> = makeListing(__filename);

2
src/webgpu/util/texture/texelData.ts
View file

@ -183,7 +183,7 @@ export interface TexelDataRepresentation {
}
class TexelDataRepresentationImpl implements TexelDataRepresentation {
// TODO: Determine endianess of the GPU data?
// TODO: Determine endianness of the GPU data?
private isGPULittleEndian = true;
constructor(

6
src/webgpu/web-platform/canvas/context_creation.spec.ts
View file

@ -1,11 +1,11 @@
export const description = '';
import { Fixture } from '../../../common/framework/fixture.js';
import { TestGroup } from '../../../common/framework/test_group.js';
import { makeTestGroup } from '../../../common/framework/test_group.js';
export const g = new TestGroup(Fixture);
export const g = makeTestGroup(Fixture);
g.test('canvas element getContext returns GPUCanvasContext').fn(async t => {
g.test('canvas_element_getContext_returns_GPUCanvasContext').fn(async t => {
if (typeof document === 'undefined') {
// Skip if there is no document (Workers, Node)
t.skip('DOM is not available to create canvas element');

10
src/webgpu/web-platform/copyImageBitmapToTexture.spec.ts
View file

@ -2,8 +2,8 @@ export const description = `
copy imageBitmap To texture tests.
`;
import { poptions, params } from '../../common/framework/params.js';
import { TestGroup } from '../../common/framework/test_group.js';
import { poptions, params } from '../../common/framework/params_builder.js';
import { makeTestGroup } from '../../common/framework/test_group.js';
import { GPUTest } from '../gpu_test.js';
function calculateRowPitch(width: number, bytesPerPixel: number): number {
@ -114,9 +114,9 @@ class F extends GPUTest {
}
}
export const g = new TestGroup(F);
export const g = makeTestGroup(F);
g.test('from ImageData')
g.test('from_ImageData')
.params(
params()
.combine(poptions('width', [1, 2, 4, 15, 255, 256]))
@ -195,7 +195,7 @@ g.test('from ImageData')
);
});
g.test('from canvas')
g.test('from_canvas')
.params(
params()
.combine(poptions('width', [1, 2, 4, 15, 255, 256]))

22
standalone/index.html
View file

@ -1,6 +1,6 @@
<html>
<head>
<title></title>
<title>WebGPU CTS</title>
<link
id="favicon"
rel="shortcut icon"
@ -41,7 +41,8 @@
width: 100%;
padding: 1px 2px 1px 1px;
}
.noderun,
.subtreerun,
.leafrun,
.nodelink,
.testcaselogbtn {
display: inline-block;
@ -54,7 +55,8 @@
border: 1px solid #888;
}
@media (pointer: fine) {
.noderun,
.subtreerun,
.leafrun,
.nodelink,
.testcaselogbtn {
flex-basis: 24px;
@ -64,7 +66,8 @@
}
}
@media (pointer: coarse) {
.noderun,
.subtreerun,
.leafrun,
.nodelink,
.testcaselogbtn {
flex-basis: 36px;
@ -73,9 +76,12 @@
height: 36px;
}
}
.noderun {
.subtreerun {
background-image: url(data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAkAAAAJCAYAAADgkQYQAAAANklEQVQoU2NkYGD4zwABjFAagwJJwBTBJDEUY1OEoRifIrhiYhSBHYvuJnSHM5LtJry+wxlOAGPTCQmAB/WwAAAAAElFTkSuQmCC);
}
.leafrun {
background-image: url(data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAkAAAAJAQMAAADaX5RTAAAABlBMVEUAAAAAAAClZ7nPAAAAAXRSTlMAQObYZgAAAB5JREFUCNdjOMDAsIGBoYeBoZmBoaEBRPaARQ4wAABTfwX/l/WQvgAAAABJRU5ErkJggg==);
}
.nodelink {
background-image: url(data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAwAAAAMAQMAAABsu86kAAAABlBMVEUAAAAAAAClZ7nPAAAAAXRSTlMAQObYZgAAACRJREFUCNdjYGBg+P+BoUGAYesFhj4BhvsFDPYNDHwMCMTAAACqJwbp3VgbrAAAAABJRU5ErkJggg==);
}
@ -83,7 +89,6 @@
flex: 10 0 4em;
}
.nodename {
user-select: all;
word-break: break-word;
}
@media only screen and (max-width: 600px) {
@ -174,11 +179,6 @@
}
}
</style>
<script>
if (!window.location.search) {
window.location += '?runnow=0&worker=0&debug=0&q=webgpu:';
}
</script>
</head>
<body>
<h1>WebGPU Conformance Test Suite</h1>