Make test cases hierarchical, and refactor/cleanup a lot in the process (#181)

* update tests, approximately

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* unittests:loading:* now passes

* passes 63 unittests, but not all are running...

* typechecks!!

* 58 unittests passing

* reenable getStackTrace tests

* wip

* finally all tests and checks pass!

* fixups

* wip

* replace all spaces with underscores in test names

* wip

* wip

* wip

* Dissolve TestLoader (now just use DefaultTestFileLoader)

* tighten ParamSpec type

* rename ParamSpec -> CaseParams

* make kParamSeparator ";", now standalone runner is working!

* add separate icon for leaf run buttons

* file renames

* remove user-select:all (not really needed anymore, triple click works fine now)

* cleanup some tests, add tests for test queries

* split interface for defining and running TestGroups

* more cleanup, fix URL encoding stuff, and standalone HTML

* fix readmes for parents of query

* clean up unused ReadmeFile interface

* dissolve id.ts

* test_suite_listing.ts

* split params_utils into param_utils + stringify_params

* rename encodeURLSelectively again

* add query.isMulti{File,Test,Case}

* fix gen_wpt_cts_html (at least for the simple mode; advanced mode probably still broken

* better run-one icon

* standalone: always make the url bar show the complete url

* revert tasks.json

* fix number out of bounds

* don't drop worker results, report input string on all query parse errors

* update docs/terms.md

* revisions

* more revisions

* make comparePublicParamsPaths order-agnostic (now allows queries against params out of order, but tree will not reorder itself accordingly)

* revise compareQueries, part 1

* revise compareQueries, part 2: compareOneLevel

* remove incorrect comment

* undo fast-glob dependency

* address Austin's comments

* address comments

* pass debug flag into test worker

* typo fixes

.eslintrc.json

@@ -14,7 +14,9 @@
       }
     ],
     "linebreak-style": ["error", "unix"],
+    "object-shorthand": "error",
     "no-console": "error",
+    "no-useless-rename": "error",
     "@typescript-eslint/no-inferrable-types": "off",
     "import/order": [
       "error",

Gruntfile.js

@@ -23,7 +23,7 @@ module.exports = function (grunt) {
       },
       test: {
         cmd: 'node',
-        args: ['tools/run', 'unittests:'],
+        args: ['tools/run', 'unittests:*'],
       },
       'build-out': {
         cmd: 'node',

docs/terms.md

@@ -1,11 +1,36 @@
 # Writing Tests
 
-- _Suites_ contain multiple:
+Each test suite is organized as a tree, both in the filesystem and further within each file.
-  - _READMEs_
+
-  - _Test Spec Files_. Contains one:
+- _Suites_, e.g. `src/webgpu/`.
-    - _Test Group_. Defines a _test fixture_ and contains multiple:
+  - _READMEs_, e.g. `src/webgpu/README.txt`.
-      - _Tests_. Defines a _test function_ and contains multiple:
+  - _Test Spec Files_, e.g. `src/webgpu/examples.spec.ts`.
-        - _Test Cases_, each with the same test function but different _parameters_.
+    Identified by their file path.
+    Each test spec file provides a description and a _Test Group_.
+    A _Test Group_ defines a test fixture, and contains multiple:
+    - _Tests_.
+      Identified by a comma-separated list of parts (e.g. `basic,async`)
+      which define a path through a filesystem-like tree (analogy: `basic/async.txt`).
+      Defines a _test function_ and contains multiple:
+      - _Test Cases_.
+        Identified by a list of _Public Parameters_ (e.g. `x` = `1`, `y` = `2`).
+        Each Test Case has the same test function but different Public Parameters.
+
+## Test Tree
+
+A _Test Tree_ is a tree whose leaves are individual Test Cases.
+
+A Test Tree can be thought of as follows:
+
+- Suite, which is the root of a tree with "leaves" which are:
+  - Test Spec Files, each of which is a tree with "leaves" which are:
+    - Tests, each of which is a tree with leaves which are:
+      - Test Cases.
+
+(In the implementation, this conceptual tree of trees is decomposed into one big tree
+whose leaves are Test Cases.)
+
+**Type:** `TestTree`
 
 ## Suite
 
@@ -22,30 +47,53 @@ Each member of a suite is identified by its path within the suite.
 
 Describes (in prose) the contents of a subdirectory in a suite.
 
-**Type:** After a README is loaded, it is stored as a `ReadmeFile`.
+READMEs are only processed at build time, when generating the _Listing_ for a suite.
 
-## IDs
+**Type:** `TestSuiteListingEntryReadme`
 
-### Test Spec ID
+## Queries
 
-Uniquely identifies a single test spec file.
+A _Query_ is a structured object which specifies a subset of cases in exactly one Suite.
-Comprised of suite name (`suite`) and test spec file path relative to the suite root (`path`).
+A Query can be represented uniquely as a string.
+Queries are used to:
 
-**Type:** `TestSpecID`
+- Identify a subtree of a suite (by identifying the root node of that subtree).
+- Identify individual cases.
+- Represent the list of tests that a test runner (standalone, wpt, or cmdline) should run.
+- Identify subtrees which should not be "collapsed" during WPT `cts.html` generation,
+  so that that cts.html "variants" can have individual test expectations
+  (i.e. marked as "expected to fail", "skip", etc.).
 
-**Example:** `{ suite: 'webgpu', path: 'command_buffer/compute/basic' }` corresponds to
+There are four types of `TestQuery`:
-`src/webgpu/command_buffer/compute/basic.spec.ts`.
 
-### Test Case ID
+- `TestQueryMultiFile` represents any subtree of the file hierarchy:
+  - `suite:*`
+  - `suite:path,to,*`
+  - `suite:path,to,file,*`
+- `TestQueryMultiTest` represents any subtree of the test hierarchy:
+  - `suite:path,to,file:*`
+  - `suite:path,to,file:path,to,*`
+  - `suite:path,to,file:path,to,test,*`
+- `TestQueryMultiCase` represents any subtree of the case hierarchy:
+  - `suite:path,to,file:path,to,test:*`
+  - `suite:path,to,file:path,to,test:my=0;*`
+  - `suite:path,to,file:path,to,test:my=0;params="here";*`
+- `TestQuerySingleCase` represents as single case:
+  - `suite:path,to,file:path,to,test:my=0;params="here"`
 
-Uniquely identifies a single test case within a test spec.
+Test Queries are a **weakly ordered set**: any query is
+_Unordered_, _Equal_, _StrictSuperset_, or _StrictSubset_ relative to any other.
+This property is used to construct the complete tree of test cases.
+In the examples above, every example query is a StrictSubset of the previous one
+(note: even `:*` is a subset of `,*`).
 
-Comprised of test name (`test`) and the parameters for a case (`params`).
+In the WPT and standalone harnesses, the query is stored in the URL, e.g.
-(If `params` is null, there is only one case for the test.)
+`index.html?q=q:u,e:r,y:*`.
 
-**Type:** `TestCaseID`
+Queries are selectively URL-encoded for readability and compatibility with browsers
+(see `encodeURIComponentSelectively`).
 
-**Example:** `{ test: '', params: { 'value': 1 } }`
+**Type:** `TestQuery`
 
 ## Listing
 
@@ -53,39 +101,35 @@ A listing of the **test spec files** in a suite.
 
 This can be generated only in Node, which has filesystem access (see `src/tools/crawl.ts`).
 As part of the build step, a _listing file_ is generated (see `src/tools/gen.ts`) so that the
-test files can be discovered by the web runner (since it does not have filesystem access).
+Test Spec Files can be discovered by the web runner (since it does not have filesystem access).
 
 **Type:** `TestSuiteListing`
 
 ### Listing File
 
+Each Suite has one Listing File (`suite/listing.[tj]s`), containing a list of the files
+in the suite.
+
+In `src/suite/listing.ts`, this is computed dynamically.
+In `out/suite/listing.js`, the listing has been pre-baked (by `tools/gen_listings`).
+
+**Type:** Once `import`ed, `ListingFile`
+
 **Example:** `out/webgpu/listing.js`
 
-## Test Spec
+## Test Spec File
 
-May be either a _test spec file_ or a _filtered test spec_.
+A Test Spec File has a `description` and a Test Group (under which tests and cases are defined).
-It is identified by a `TestSpecID`.
 
-Always contains one `RunCaseIterable`.
+**Type:** Once `import`ed, `SpecFile`
-
-**Type:** `TestSpec`
-
-### Test Spec File
-
-A single `.spec.ts` file. It always contains one _test group_.
 
 **Example:** `src/webgpu/**/*.spec.ts`
 
-### Filtered Test Spec
+## Test Group
 
-A subset of the tests in a single test spec file.
+A subtree of tests. There is one Test Group per Test Spec File.
-It is created at runtime, via a filter.
 
-It contains one "virtual" test group, which has type `RunCaseIterable`.
+The Test Fixture used for tests is defined at TestGroup creation.
-
-## Test Group / Group
-
-A collection of test cases. There is one per test spec file.
 
 **Type:** `TestGroup`
 
@@ -93,18 +137,17 @@ A collection of test cases. There is one per test spec file.
 
 One test. It has a single _test function_.
 
-It may represent multiple _test cases_, each of which runs the same test function with different **parameters**.
+It may represent multiple _test cases_, each of which runs the same Test Function with different
+Parameters.
 
-It is created by `TestGroup.test()`.
+A test is named using `TestGroup.test()`, which returns a `TestBuilder`.
-At creation time, it can be parameterized via `Test.params()`.
+`TestBuilder.params()` can optionally be used to parameterize the test.
-
+Then, `TestBuilder.fn()` provides the Test Function.
-**Type:** `Test`
 
 ### Test Function
 
-A test function is defined inline inside of a `TestGroup.test()` call.
+When a test case is run, the Test Function receives an instance of the
-
+Test Fixture provided to the Test Group, producing test results.
-It receives an instance of the appropriate _test fixture_, through which it produce test results.
 
 **Type:** `TestFn`
 
@@ -116,9 +159,20 @@ A single case of a test. It is identified by a `TestCaseID`: a test name, and it
 
 ## Parameters / Params
 
+Each Test Case has a (possibly empty) set of Parameters.
+The parameters are available to the Test Function `f(t)` via `t.params`.
+
+A set of Public Parameters identifies a Test Case within a Test.
+
+There are also Private Paremeters: any parameter name beginning with an underscore (`_`).
+These parameters are not part of the Test Case identification, but are still passed into
+the Test Function. They can be used to manually specify expected results.
+
+**Type:** `CaseParams`
+
 ## Test Fixture / Fixture
 
-Test fixtures provide helpers for tests to use.
+_Test Fixtures_ provide helpers for tests to use.
 A new instance of the fixture is created for every run of every test case.
 
 There is always one fixture class for a whole test group (though this may change).
@@ -126,7 +180,7 @@ There is always one fixture class for a whole test group (though this may change
 The fixture is also how a test gets access to the _case recorder_,
 which allows it to produce test results.
 
-They are also how tests produce results: `.log()`, `.fail()`, etc.
+They are also how tests produce results: `.skip()`, `.fail()`, etc.
 
 **Type:** `Fixture`
 
@@ -138,137 +192,61 @@ Provides basic fixture utilities most useful in the `unittests` suite.
 
 Provides utilities useful in WebGPU CTS tests.
 
-# Running Tests
-
-- _Queries_ contain multiple:
-  - _Filters_ (positive or negative).
-
-## Query
-
-A query is a string which denotes a subset of a test suite to run.
-It is comprised of a list of filters.
-A case is included in the subset if it matches any of the filters.
-
-In the WPT and standalone harnesses, the query is stored in the URL.
-
-Queries are selectively URL-encoded for readability and compatibility with browsers.
-
-**Example:** `?q=unittests:param_helpers:combine/=`
-**Example:** `?q=unittests:param_helpers:combine/mixed=`
-
-**Type:** None yet. TODO
-
-### Filter
-
-A filter matches a set of cases in a suite.
-
-Each filter may match one of:
-
-- `S:s` In one suite `S`, all specs whose paths start with `s` (which may be empty).
-- `S:s:t` In one spec `S:s`, all tests whose names start with `t` (which may be empty).
-- `S:s:t~c` In one test `S:s:t`, all cases whose params are a superset of `c`.
-- `S:s:t=c` In one test `S:s:t`, the single case whose params equal `c` (empty string = `{}`).
-
-**Type:** `TestFilter`
-
-### Using filters to split expectations
-
-A set of cases can be split using negative filters. For example, imagine you have one WPT test variant:
-
-- `webgpu/cts.html?q=unittests:param_helpers:`
-
-But one of the cases is failing. To be able to suppress the failing test without losing test coverage, the WPT test variant can be split into two variants:
-
-- `webgpu/cts.html?q=unittests:param_helpers:&not=unittests:param_helpers:combine/mixed:`
-- `webgpu/cts.html?q=unittests:param_helpers:combine/mixed:`
-
-This runs the same set of cases, but in two separate page loads.
-
 # Test Results
 
 ## Logger
 
 A logger logs the results of a whole test run.
 
-It saves an empty `LiveTestSpecResult` into its results array, then creates a
+It saves an empty `LiveTestSpecResult` into its results map, then creates a
 _test spec recorder_, which records the results for a group into the `LiveTestSpecResult`.
 
-### Test Spec Recorder
+**Type:** `Logger`
-
-Refers to a `LiveTestSpecResult` in the logger, and writes results into it.
-
-It saves an empty `LiveTestCaseResult` into its `LiveTestSpecResult`, then creates a
-_test case recorder_, which records the results for a case into the `LiveTestCaseResult`.
 
 ### Test Case Recorder
 
-Records the actual results of running a test case (its pass-status, run time, and logs)
+Refers to a `LiveTestCaseResult` created by the logger.
-into its `LiveTestCaseResult`.
+Records the results of running a test case (its pass-status, run time, and logs) into it.
+
+**Types:** `TestCaseRecorder`, `LiveTestCaseResult`
 
 #### Test Case Status
 
 The `status` of a `LiveTestCaseResult` can be one of:
 
-- `'running'`
+- `'running'` (only while still running)
-- `'fail'`
-- `'warn'`
 - `'pass'`
+- `'skip'`
+- `'warn'`
+- `'fail'`
 
-The "worst" result from running a case is always reported.
+The "worst" result from running a case is always reported (fail > warn > skip > pass).
+Note this means a test can still fail if it's "skipped", if it failed before
+`.skip()` was called.
+
+**Type:** `Status`
 
 ## Results Format
 
 The results are returned in JSON format.
 
-They are designed to be easily merged in JavaScript.
+They are designed to be easily merged in JavaScript:
-(TODO: Write a merge tool.)
+the `"results"` can be passed into the constructor of `Map` and merged from there.
 
-(TODO: Update these docs if the format changes.)
+(TODO: Write a merge tool, if needed.)
 
 ```js
 {
-  "version": "e24c459b46c4815f93f0aed5261e28008d1f2882-dirty",
+  "version": "bf472c5698138cdf801006cd400f587e9b1910a5-dirty",
   "results": [
-    // LiveTestSpecResult objects
+    [
-    {
+      "unittests:async_mutex:basic:",
-      "spec": "unittests:basic:",
+      { "status": "pass", "timems": 0.286, "logs": [] }
-      "cases": [
+    ],
-        // LiveTestCaseResult objects
+    [
-        {
+      "unittests:async_mutex:serial:",
-          "test": "test/sync",
+      { "status": "pass", "timems": 0.415, "logs": [] }
-          "params": null,
+    ]
-          "status": "pass",
-          "timems": 145,
-          "logs": []
-        },
-        {
-          "test": "test/async",
-          "params": null,
-          "status": "pass",
-          "timems": 26,
-          "logs": []
-        }
-      ]
-    },
-    {
-      "spec": "unittests:test_group:",
-      "cases": [
-        {
-          "test": "invalid test name",
-          "params": { "char": "\"" },
-          "status": "pass",
-          "timems": 66,
-          "logs": ["OK: threw"]
-        },
-        {
-          "test": "invalid test name",
-          "params": { "char": "`" },
-          "status": "pass",
-          "timems": 15,
-          "logs": ["OK: threw"]
-        }
-      ]
-    }
   ]
 }
 ```

prettier.config.js

@@ -1,8 +1,8 @@
 module.exports = {
   printWidth: 100,
+
   arrowParens: 'avoid',
   bracketSpacing: true,
   singleQuote: true,
   trailingComma: 'es5',
-  arrowParens: 'avoid',
 };

src/common/framework/allowed_characters.ts

@@ -1,2 +0,0 @@
-// It may be OK to add more allowed characters here.
-export const allowedTestNameCharacters = 'a-zA-Z0-9/_';

src/common/framework/file_loader.ts

@@ -0,0 +1,50 @@
+import { parseQuery } from './query/parseQuery.js';
+import { RunCaseIterable } from './test_group.js';
+import { TestSuiteListing } from './test_suite_listing.js';
+import { loadTreeForQuery, TestTree, TestTreeLeaf } from './tree.js';
+
+// A listing file, e.g. either of:
+// - `src/webgpu/listing.ts` (which is dynamically computed, has a Promise<TestSuiteListing>)
+// - `out/webgpu/listing.js` (which is pre-baked, has a TestSuiteListing)
+interface ListingFile {
+  listing: Promise<TestSuiteListing> | TestSuiteListing;
+}
+
+// A .spec.ts file, as imported.
+export interface SpecFile {
+  readonly description: string;
+  readonly g: RunCaseIterable;
+}
+
+// Base class for DefaultTestFileLoader and FakeTestFileLoader.
+export abstract class TestFileLoader {
+  abstract listing(suite: string): Promise<TestSuiteListing>;
+  protected abstract import(path: string): Promise<SpecFile>;
+
+  importSpecFile(suite: string, path: string[]): Promise<SpecFile> {
+    return this.import(`${suite}/${path.join('/')}.spec.js`);
+  }
+
+  async loadTree(query: string, subqueriesToExpand: string[] = []): Promise<TestTree> {
+    return loadTreeForQuery(
+      this,
+      parseQuery(query),
+      subqueriesToExpand.map(q => parseQuery(q))
+    );
+  }
+
+  async loadTests(query: string): Promise<IterableIterator<TestTreeLeaf>> {
+    const tree = await this.loadTree(query);
+    return tree.iterateLeaves();
+  }
+}
+
+export class DefaultTestFileLoader extends TestFileLoader {
+  async listing(suite: string): Promise<TestSuiteListing> {
+    return ((await import(`../../${suite}/listing.js`)) as ListingFile).listing;
+  }
+
+  import(path: string): Promise<SpecFile> {
+    return import(`../../${path}`);
+  }
+}

src/common/framework/fixture.ts

@@ -1,5 +1,5 @@
-import { TestCaseRecorder } from './logger.js';
+import { TestCaseRecorder } from './logging/test_case_recorder.js';
-import { ParamSpec } from './params_utils.js';
+import { CaseParams } from './params_utils.js';
 import { assert } from './util/util.js';
 
 export class SkipTestCase extends Error {}
@@ -13,7 +13,7 @@ export class Fixture {
   private eventualExpectations: Array<Promise<unknown>> = [];
   private numOutstandingAsyncExpectations = 0;
 
-  constructor(rec: TestCaseRecorder, params: ParamSpec) {
+  constructor(rec: TestCaseRecorder, params: CaseParams) {
     this.rec = rec;
     this.params = params;
   }
@@ -61,16 +61,16 @@ export class Fixture {
 
   private expectErrorValue(expectedName: string, ex: unknown, niceStack: Error): void {
     if (!(ex instanceof Error)) {
-      niceStack.message = 'THREW non-error value, of type ' + typeof ex + niceStack.message;
+      niceStack.message = `THREW non-error value, of type ${typeof ex}: ${ex}`;
       this.rec.fail(niceStack);
       return;
     }
     const actualName = ex.name;
     if (actualName !== expectedName) {
-      niceStack.message = `THREW ${actualName}, instead of ${expectedName}` + niceStack.message;
+      niceStack.message = `THREW ${actualName}, instead of ${expectedName}: ${ex}`;
       this.rec.fail(niceStack);
     } else {
-      niceStack.message = 'OK: threw ' + actualName + niceStack.message;
+      niceStack.message = `OK: threw ${actualName}${ex.message}`;
       this.rec.debug(niceStack);
     }
   }
@@ -80,7 +80,7 @@ export class Fixture {
       const m = msg ? ': ' + msg : '';
       try {
         await p;
-        niceStack.message = 'DID NOT THROW' + m;
+        niceStack.message = 'DID NOT REJECT' + m;
         this.rec.fail(niceStack);
       } catch (ex) {
         niceStack.message = m;

src/common/framework/generate_minimal_query_list.ts

@@ -1,112 +0,0 @@
-import { TestSpecOrTestOrCaseID } from './id.js';
-import { Logger } from './logger.js';
-import { makeFilter } from './test_filter/load_filter.js';
-import { TestFilterResult } from './test_filter/test_filter_result.js';
-import { FilterResultTreeNode, treeFromFilterResults } from './tree.js';
-
-interface QuerySplitterTreeNode {
-  needsSplit: boolean;
-  children?: Map<string, QuerySplitterTreeNode>;
-}
-
-interface Expectation {
-  id: TestSpecOrTestOrCaseID;
-  line: string;
-  seen: boolean;
-}
-
-function makeQuerySplitterTree(
-  caselist: TestFilterResult[],
-  expectationStrings: string[]
-): QuerySplitterTreeNode {
-  const expectations: Expectation[] = [];
-  for (const e of expectationStrings) {
-    const filter = makeFilter(e);
-    const id = filter.idIfSingle();
-    if (!id) {
-      throw new Error(
-        'Can only handle expectations which cover one file, one test, or one case. ' + e
-      );
-    }
-    expectations.push({ id, line: e, seen: false });
-  }
-
-  const convertToQuerySplitterTree = (
-    tree: FilterResultTreeNode,
-    name?: string
-  ): QuerySplitterTreeNode => {
-    const children = tree.children;
-    let needsSplit = true;
-
-    if (name !== undefined) {
-      const filter = makeFilter(name);
-      const moreThanOneFile = !filter.definitelyOneFile();
-      const matchingExpectations = expectations.map(e => {
-        const matches = filter.matches(e.id);
-        if (matches) e.seen = true;
-        return matches;
-      });
-      needsSplit = matchingExpectations.some(m => m) || moreThanOneFile;
-    }
-
-    const queryNode: QuerySplitterTreeNode = { needsSplit };
-    if (children) {
-      queryNode.children = new Map();
-      for (const [k, v] of children) {
-        const subtree = convertToQuerySplitterTree(v, k);
-        queryNode.children.set(k, subtree);
-      }
-    }
-    return queryNode;
-  };
-
-  const log = new Logger();
-  const tree = treeFromFilterResults(log, caselist.values());
-  const queryTree = convertToQuerySplitterTree(tree)!;
-
-  for (const e of expectations) {
-    if (!e.seen) throw new Error('expectation had no effect: ' + e.line);
-  }
-
-  return queryTree;
-}
-
-// Takes a TestFilterResultIterator enumerating every test case in the suite, and a list of
-// expectation queries from a browser's expectations file. Creates a minimal list of queries
-// (i.e. wpt variant lines) such that:
-//
-// - There is at least one query per spec file.
-// - Each of those those input queries is in the output, so that it can have its own expectation.
-//
-// It does this by creating a tree from the list of cases (same tree as the standalone runner uses),
-// then marking every node which is a parent of a node that matches an expectation.
-export async function generateMinimalQueryList(
-  caselist: TestFilterResult[],
-  expectationStrings: string[]
-): Promise<string[]> {
-  const unsplitNodes: string[] = [];
-  const findUnsplitNodes = (name: string, node: QuerySplitterTreeNode | undefined) => {
-    if (node === undefined) {
-      return;
-    }
-    if (node.needsSplit && node.children) {
-      for (const [k, v] of node.children) {
-        findUnsplitNodes(k, v);
-      }
-    } else {
-      unsplitNodes.push(name);
-    }
-  };
-
-  const queryTree = makeQuerySplitterTree(caselist, expectationStrings);
-  findUnsplitNodes('', queryTree);
-
-  for (const exp of expectationStrings) {
-    if (!unsplitNodes.some(name => name === exp)) {
-      throw new Error(
-        'Something went wrong: all expectation strings should always appear exactly: ' + exp
-      );
-    }
-  }
-  return unsplitNodes;
-}

src/common/framework/gpu/device_pool.ts

@@ -71,7 +71,7 @@ export class DevicePool {
       // TODO: device.destroy()
 
       // Mark the holder as free. (This only has an effect if the pool still has the holder.)
-      // This could be done at the top but is done here to guard againt async-races during release.
+      // This could be done at the top but is done here to guard against async-races during release.
       holder.acquired = false;
     }
   }

src/common/framework/id.ts

@@ -1,25 +0,0 @@
-import { ParamSpec } from './params_utils.js';
-
-// Identifies a test spec file.
-export interface TestSpecID {
-  // The spec's suite name, e.g. 'webgpu'.
-  readonly suite: string;
-  // The spec's path within the suite, e.g. 'command_buffer/compute/basic'.
-  readonly path: string;
-}
-
-export function testSpecEquals(x: TestSpecID, y: TestSpecID): boolean {
-  return x.suite === y.suite && x.path === y.path;
-}
-
-// Identifies a test case (a specific parameterization of a test), within its spec file.
-export interface TestCaseID {
-  readonly test: string;
-  readonly params: ParamSpec | null;
-}
-
-export interface TestSpecOrTestOrCaseID {
-  readonly spec: TestSpecID;
-  readonly test?: string;
-  readonly params?: ParamSpec | null;
-}

src/common/framework/listing.ts

@@ -1,9 +0,0 @@
-// A listing of all specs within a single suite. This is the (awaited) type of
-// `groups` in '{cts,unittests}/listing.ts' and the auto-generated
-// 'out/{cts,unittests}/listing.js' files (see tools/gen_listings).
-export type TestSuiteListing = Iterable<TestSuiteListingEntry>;
-
-export interface TestSuiteListingEntry {
-  readonly path: string;
-  readonly description: string;
-}

src/common/framework/loader.ts

@@ -1,65 +0,0 @@
-import { TestSuiteListing } from './listing.js';
-import { loadFilter } from './test_filter/load_filter.js';
-import { TestFilterResult } from './test_filter/test_filter_result.js';
-import { RunCaseIterable } from './test_group.js';
-
-// One of the following:
-// - An actual .spec.ts file, as imported.
-// - A *filtered* list of cases from a single .spec.ts file.
-export interface TestSpec {
-  readonly description: string;
-  readonly g: RunCaseIterable;
-}
-
-// A shell object describing a directory (from its README.txt).
-export interface ReadmeFile {
-  readonly description: string;
-}
-
-export type TestSpecOrReadme = TestSpec | ReadmeFile;
-
-type TestFilterResultIterator = IterableIterator<TestFilterResult>;
-function* concat(lists: TestFilterResult[][]): TestFilterResultIterator {
-  for (const specs of lists) {
-    yield* specs;
-  }
-}
-
-export interface TestFileLoader {
-  listing(suite: string): Promise<TestSuiteListing>;
-  import(path: string): Promise<TestSpecOrReadme>;
-}
-
-class DefaultTestFileLoader implements TestFileLoader {
-  async listing(suite: string): Promise<TestSuiteListing> {
-    return (await import(`../../${suite}/listing.js`)).listing;
-  }
-
-  import(path: string): Promise<TestSpec> {
-    return import('../../' + path);
-  }
-}
-
-export class TestLoader {
-  private fileLoader: TestFileLoader;
-
-  constructor(fileLoader: TestFileLoader = new DefaultTestFileLoader()) {
-    this.fileLoader = fileLoader;
-  }
-
-  // TODO: Test
-  async loadTestsFromQuery(query: string): Promise<TestFilterResultIterator> {
-    return this.loadTests(new URLSearchParams(query).getAll('q'));
-  }
-
-  // TODO: Test
-  // TODO: Probably should actually not exist at all, just use queries on cmd line too.
-  async loadTestsFromCmdLine(filters: string[]): Promise<TestFilterResultIterator> {
-    return this.loadTests(filters);
-  }
-
-  async loadTests(filters: string[]): Promise<TestFilterResultIterator> {
-    const loads = filters.map(f => loadFilter(this.fileLoader, f));
-    return concat(await Promise.all(loads));
-  }
-}

src/common/framework/logger.ts

@@ -1,159 +0,0 @@
-import { SkipTestCase } from './fixture.js';
-import { TestSpecID } from './id.js';
-import { ParamSpec, extractPublicParams } from './params_utils.js';
-import { makeQueryString } from './url_query.js';
-import { getStackTrace } from './util/stack.js';
-import { assert, now } from './util/util.js';
-import { version } from './version.js';
-
-type Status = 'running' | 'pass' | 'skip' | 'warn' | 'fail';
-export interface LiveTestSpecResult {
-  readonly spec: string;
-  readonly cases: LiveTestCaseResult[];
-}
-
-interface TestCaseResult {
-  readonly test: string;
-  readonly params: ParamSpec | null;
-  status: Status;
-  timems: number;
-}
-
-export interface LiveTestCaseResult extends TestCaseResult {
-  logs?: LogMessageWithStack[];
-}
-
-export interface TransferredTestCaseResult extends TestCaseResult {
-  // When transferred from a worker, a LogMessageWithStack turns into a generic Error
-  // (its prototype gets lost and replaced with Error).
-  logs?: Error[];
-}
-
-export class LogMessageWithStack extends Error {
-  constructor(name: string, ex: Error, includeStack: boolean = true) {
-    super(ex.message);
-
-    this.name = name;
-    this.stack = includeStack ? ex.stack : undefined;
-  }
-
-  toJSON(): string {
-    let m = this.name;
-    if (this.message) {
-      m += ': ' + this.message;
-    }
-    if (this.stack) {
-      m += '\n' + getStackTrace(this);
-    }
-    return m;
-  }
-}
-
-export class Logger {
-  readonly results: LiveTestSpecResult[] = [];
-
-  constructor() {}
-
-  record(spec: TestSpecID): [TestSpecRecorder, LiveTestSpecResult] {
-    const result: LiveTestSpecResult = { spec: makeQueryString(spec), cases: [] };
-    this.results.push(result);
-    return [new TestSpecRecorder(result), result];
-  }
-
-  asJSON(space?: number): string {
-    return JSON.stringify({ version, results: this.results }, undefined, space);
-  }
-}
-
-export class TestSpecRecorder {
-  private result: LiveTestSpecResult;
-
-  constructor(result: LiveTestSpecResult) {
-    this.result = result;
-  }
-
-  record(test: string, params: ParamSpec | null): [TestCaseRecorder, LiveTestCaseResult] {
-    const result: LiveTestCaseResult = {
-      test,
-      params: params ? extractPublicParams(params) : null,
-      status: 'running',
-      timems: -1,
-    };
-    this.result.cases.push(result);
-    return [new TestCaseRecorder(result), result];
-  }
-}
-
-enum PassState {
-  pass = 0,
-  skip = 1,
-  warn = 2,
-  fail = 3,
-}
-
-export class TestCaseRecorder {
-  private result: LiveTestCaseResult;
-  private state = PassState.pass;
-  private startTime = -1;
-  private logs: LogMessageWithStack[] = [];
-  private debugging = false;
-
-  constructor(result: LiveTestCaseResult) {
-    this.result = result;
-  }
-
-  start(debug: boolean = false): void {
-    this.startTime = now();
-    this.logs = [];
-    this.state = PassState.pass;
-    this.debugging = debug;
-  }
-
-  finish(): void {
-    assert(this.startTime >= 0, 'finish() before start()');
-
-    const endTime = now();
-    // Round to next microsecond to avoid storing useless .xxxx00000000000002 in results.
-    this.result.timems = Math.ceil((endTime - this.startTime) * 1000) / 1000;
-    this.result.status = PassState[this.state] as Status;
-
-    this.result.logs = this.logs;
-    this.debugging = false;
-  }
-
-  debug(ex: Error): void {
-    if (!this.debugging) {
-      return;
-    }
-    this.logs.push(new LogMessageWithStack('DEBUG', ex, false));
-  }
-
-  warn(ex: Error): void {
-    this.setState(PassState.warn);
-    this.logs.push(new LogMessageWithStack('WARN', ex));
-  }
-
-  fail(ex: Error): void {
-    this.setState(PassState.fail);
-    this.logs.push(new LogMessageWithStack('FAIL', ex));
-  }
-
-  skipped(ex: SkipTestCase): void {
-    this.setState(PassState.skip);
-    this.logs.push(new LogMessageWithStack('SKIP', ex));
-  }
-
-  threw(ex: Error): void {
-    if (ex instanceof SkipTestCase) {
-      this.skipped(ex);
-      return;
-    }
-
-    this.setState(PassState.fail);
-    this.logs.push(new LogMessageWithStack('EXCEPTION', ex));
-  }
-
-  private setState(state: PassState): void {
-    this.state = Math.max(this.state, state);
-  }
-}

src/common/framework/logging/log_message.ts

@@ -0,0 +1,21 @@
+import { extractImportantStackTrace } from '../util/stack.js';
+
+export class LogMessageWithStack extends Error {
+  constructor(name: string, ex: Error, includeStack: boolean = true) {
+    super(ex.message);
+
+    this.name = name;
+    this.stack = includeStack ? ex.stack : undefined;
+  }
+
+  toJSON(): string {
+    let m = this.name + ': ';
+    if (this.stack) {
+      // this.message is already included in this.stack
+      m += extractImportantStackTrace(this);
+    } else {
+      m += this.message;
+    }
+    return m;
+  }
+}

src/common/framework/logging/logger.ts

@@ -0,0 +1,25 @@
+import { version } from '../version.js';
+
+import { LiveTestCaseResult } from './result.js';
+import { TestCaseRecorder } from './test_case_recorder.js';
+
+export type LogResults = Map<string, LiveTestCaseResult>;
+
+export class Logger {
+  readonly debug: boolean;
+  readonly results: LogResults = new Map();
+
+  constructor(debug: boolean) {
+    this.debug = debug;
+  }
+
+  record(name: string): [TestCaseRecorder, LiveTestCaseResult] {
+    const result: LiveTestCaseResult = { status: 'running', timems: -1 };
+    this.results.set(name, result);
+    return [new TestCaseRecorder(result, this.debug), result];
+  }
+
+  asJSON(space?: number): string {
+    return JSON.stringify({ version, results: Array.from(this.results) }, undefined, space);
+  }
+}

src/common/framework/logging/result.ts

@@ -0,0 +1,18 @@
+import { LogMessageWithStack } from './log_message.js';
+
+export type Status = 'running' | 'pass' | 'skip' | 'warn' | 'fail';
+
+export interface TestCaseResult {
+  status: Status;
+  timems: number;
+}
+
+export interface LiveTestCaseResult extends TestCaseResult {
+  logs?: LogMessageWithStack[];
+}
+
+export interface TransferredTestCaseResult extends TestCaseResult {
+  // When transferred from a worker, a LogMessageWithStack turns into a generic Error
+  // (its prototype gets lost and replaced with Error).
+  logs?: Error[];
+}

src/common/framework/logging/test_case_recorder.ts

@@ -0,0 +1,82 @@
+import { SkipTestCase } from '../fixture.js';
+import { now, assert } from '../util/util.js';
+
+import { LogMessageWithStack } from './log_message.js';
+import { LiveTestCaseResult, Status } from './result.js';
+
+enum PassState {
+  pass = 0,
+  skip = 1,
+  warn = 2,
+  fail = 3,
+}
+
+// Holds onto a LiveTestCaseResult owned by the Logger, and writes the results into it.
+export class TestCaseRecorder {
+  private result: LiveTestCaseResult;
+  private state = PassState.pass;
+  private startTime = -1;
+  private logs: LogMessageWithStack[] = [];
+  private debugging = false;
+
+  constructor(result: LiveTestCaseResult, debugging: boolean) {
+    this.result = result;
+    this.debugging = debugging;
+  }
+
+  start(): void {
+    assert(this.startTime < 0, 'TestCaseRecorder cannot be reused');
+    this.startTime = now();
+  }
+
+  finish(): void {
+    assert(this.startTime >= 0, 'finish() before start()');
+
+    const timeMilliseconds = now() - this.startTime;
+    // Round to next microsecond to avoid storing useless .xxxx00000000000002 in results.
+    this.result.timems = Math.ceil(timeMilliseconds * 1000) / 1000;
+    this.result.status = PassState[this.state] as Status; // Convert numeric enum back to string
+
+    this.result.logs = this.logs;
+  }
+
+  injectResult(injectedResult: LiveTestCaseResult): void {
+    Object.assign(this.result, injectedResult);
+  }
+
+  debug(ex: Error): void {
+    if (!this.debugging) {
+      return;
+    }
+    this.logs.push(new LogMessageWithStack('DEBUG', ex, false));
+  }
+
+  warn(ex: Error): void {
+    this.setState(PassState.warn);
+    this.logs.push(new LogMessageWithStack('WARN', ex));
+  }
+
+  fail(ex: Error): void {
+    this.setState(PassState.fail);
+    this.logs.push(new LogMessageWithStack('FAIL', ex));
+  }
+
+  skipped(ex: SkipTestCase): void {
+    this.setState(PassState.skip);
+    this.logs.push(new LogMessageWithStack('SKIP', ex));
+  }
+
+  threw(ex: Error): void {
+    if (ex instanceof SkipTestCase) {
+      this.skipped(ex);
+      return;
+    }
+
+    this.setState(PassState.fail);
+    this.logs.push(new LogMessageWithStack('EXCEPTION', ex));
+  }
+
+  private setState(state: PassState): void {
+    this.state = Math.max(this.state, state);
+  }
+}

src/common/framework/params_builder.ts

@@ -1,4 +1,4 @@
-import { ParamSpec, ParamSpecIterable, paramsEquals } from './params_utils.js';
+import { CaseParams, CaseParamsIterable, publicParamsEquals } from './params_utils.js';
 import { assert } from './util/util.js';
 
 // https://stackoverflow.com/a/56375136
@@ -33,11 +33,11 @@ export function params(): ParamsBuilder<{}> {
   return new ParamsBuilder();
 }
 
-class ParamsBuilder<A extends {}> implements ParamSpecIterable {
+export class ParamsBuilder<A extends {}> implements CaseParamsIterable {
-  private paramSpecs: ParamSpecIterable = [{}];
+  private paramSpecs: CaseParamsIterable = [{}];
 
   [Symbol.iterator](): Iterator<A> {
-    const iter: Iterator<ParamSpec> = this.paramSpecs[Symbol.iterator]();
+    const iter: Iterator<CaseParams> = this.paramSpecs[Symbol.iterator]();
     return iter as Iterator<A>;
   }
 
@@ -49,7 +49,7 @@ class ParamsBuilder<A extends {}> implements ParamSpecIterable {
           yield mergeParams(a, b);
         }
       }
-    });
+    }) as CaseParamsIterable;
     /* eslint-disable-next-line @typescript-eslint/no-explicit-any */
     return this as any;
   }
@@ -62,7 +62,7 @@ class ParamsBuilder<A extends {}> implements ParamSpecIterable {
           yield mergeParams(a, b);
         }
       }
-    });
+    }) as CaseParamsIterable;
     /* eslint-disable-next-line @typescript-eslint/no-explicit-any */
     return this as any;
   }
@@ -83,12 +83,12 @@ class ParamsBuilder<A extends {}> implements ParamSpecIterable {
     return this.filter(x => !pred(x));
   }
 
-  exclude(exclude: ParamSpecIterable): ParamsBuilder<A> {
+  exclude(exclude: CaseParamsIterable): ParamsBuilder<A> {
     const excludeArray = Array.from(exclude);
     const paramSpecs = this.paramSpecs;
     this.paramSpecs = makeReusableIterable(function* () {
       for (const p of paramSpecs) {
-        if (excludeArray.every(e => !paramsEquals(p, e))) {
+        if (excludeArray.every(e => !publicParamsEquals(p, e))) {
           yield p;
         }
       }

src/common/framework/params_utils.ts

@@ -1,69 +1,32 @@
-import { objectEquals } from './util/util.js';
+import { comparePublicParamsPaths, Ordering } from './query/compare.js';
+import { kWildcard, kParamSeparator } from './query/separators.js';
 
-/* eslint-disable-next-line @typescript-eslint/no-explicit-any */
+// Consider adding more types here if needed
-export type ParamArgument = any;
+export type ParamArgument = void | undefined | number | string | boolean | number[];
-export interface ParamSpec {
+export interface CaseParams {
+  readonly [k: string]: ParamArgument;
+}
+export interface CaseParamsRW {
   [k: string]: ParamArgument;
 }
-export type ParamSpecIterable = Iterable<ParamSpec>;
+export type CaseParamsIterable = Iterable<CaseParams>;
-export type ParamSpecIterator = IterableIterator<ParamSpec>;
 
-export function extractPublicParams(params: ParamSpec): ParamSpec {
+export function paramKeyIsPublic(key: string): boolean {
-  const publicParams: ParamSpec = {};
+  return !key.startsWith('_');
+}
+
+export function extractPublicParams(params: CaseParams): CaseParams {
+  const publicParams: CaseParamsRW = {};
   for (const k of Object.keys(params)) {
-    if (!k.startsWith('_')) {
+    if (paramKeyIsPublic(k)) {
       publicParams[k] = params[k];
     }
   }
   return publicParams;
 }
 
-export function stringifyPublicParams(p: ParamSpec | null): string {
+export const badParamValueChars = new RegExp('[=' + kParamSeparator + kWildcard + ']');
-  if (p === null || paramsEquals(p, {})) {
+
-    return '';
+export function publicParamsEquals(x: CaseParams, y: CaseParams): boolean {
-  }
+  return comparePublicParamsPaths(x, y) === Ordering.Equal;
-  return JSON.stringify(extractPublicParams(p));
-}
-
-export function paramsEquals(x: ParamSpec | null, y: ParamSpec | null): boolean {
-  if (x === y) {
-    return true;
-  }
-  if (x === null) {
-    x = {};
-  }
-  if (y === null) {
-    y = {};
-  }
-
-  for (const xk of Object.keys(x)) {
-    if (x[xk] !== undefined && !(xk in y)) {
-      return false;
-    }
-    if (!objectEquals(x[xk], y[xk])) {
-      return false;
-    }
-  }
-
-  for (const yk of Object.keys(y)) {
-    if (y[yk] !== undefined && !(yk in x)) {
-      return false;
-    }
-  }
-  return true;
-}
-
-export function paramsSupersets(sup: ParamSpec | null, sub: ParamSpec | null): boolean {
-  if (sub === null) {
-    return true;
-  }
-  if (sup === null) {
-    sup = {};
-  }
-  for (const k of Object.keys(sub)) {
-    if (!(k in sup) || sup[k] !== sub[k]) {
-      return false;
-    }
-  }
-  return true;
 }

src/common/framework/query/compare.ts

@@ -0,0 +1,89 @@
+import { CaseParams, extractPublicParams } from '../params_utils.js';
+import { assert, objectEquals } from '../util/util.js';
+
+import { TestQuery } from './query.js';
+
+export const enum Ordering {
+  Unordered,
+  StrictSuperset,
+  Equal,
+  StrictSubset,
+}
+
+// Compares two queries for their ordering (which is used to build the tree).
+// See src/unittests/query_compare.spec.ts for examples.
+export function compareQueries(a: TestQuery, b: TestQuery): Ordering {
+  if (a.suite !== b.suite) {
+    return Ordering.Unordered;
+  }
+
+  const filePathOrdering = comparePaths(a.filePathParts, b.filePathParts);
+  if (filePathOrdering !== Ordering.Equal || a.isMultiFile || b.isMultiFile) {
+    return compareOneLevel(filePathOrdering, a.isMultiFile, b.isMultiFile);
+  }
+  assert('testPathParts' in a && 'testPathParts' in b);
+
+  const testPathOrdering = comparePaths(a.testPathParts, b.testPathParts);
+  if (testPathOrdering !== Ordering.Equal || a.isMultiTest || b.isMultiTest) {
+    return compareOneLevel(testPathOrdering, a.isMultiTest, b.isMultiTest);
+  }
+  assert('params' in a && 'params' in b);
+
+  const paramsPathOrdering = comparePublicParamsPaths(a.params, b.params);
+  if (paramsPathOrdering !== Ordering.Equal || a.isMultiCase || b.isMultiCase) {
+    return compareOneLevel(paramsPathOrdering, a.isMultiCase, b.isMultiCase);
+  }
+  return Ordering.Equal;
+}
+
+// Compares a single level of a query.
+// "IsBig" means the query is big relative to the level, e.g. for test-level:
+//   anything >= suite:a,* is big
+//   anything <= suite:a:* is small
+function compareOneLevel(ordering: Ordering, aIsBig: boolean, bIsBig: boolean): Ordering {
+  assert(ordering !== Ordering.Equal || aIsBig || bIsBig);
+  if (ordering === Ordering.Unordered) return Ordering.Unordered;
+  if (aIsBig && bIsBig) return ordering;
+  if (!aIsBig && !bIsBig) return Ordering.Unordered; // Equal case is already handled
+  // Exactly one of (a, b) is big.
+  if (aIsBig && ordering !== Ordering.StrictSubset) return Ordering.StrictSuperset;
+  if (bIsBig && ordering !== Ordering.StrictSuperset) return Ordering.StrictSubset;
+  return Ordering.Unordered;
+}
+
+function comparePaths(a: readonly string[], b: readonly string[]): Ordering {
+  const shorter = Math.min(a.length, b.length);
+
+  for (let i = 0; i < shorter; ++i) {
+    if (a[i] !== b[i]) {
+      return Ordering.Unordered;
+    }
+  }
+  if (a.length === b.length) {
+    return Ordering.Equal;
+  } else if (a.length < b.length) {
+    return Ordering.StrictSuperset;
+  } else {
+    return Ordering.StrictSubset;
+  }
+}
+
+export function comparePublicParamsPaths(a0: CaseParams, b0: CaseParams): Ordering {
+  const a = extractPublicParams(a0);
+  const b = extractPublicParams(b0);
+  const aKeys = Object.keys(a);
+  const commonKeys = new Set(aKeys.filter(k => k in b));
+
+  for (const k of commonKeys) {
+    if (!objectEquals(a[k], b[k])) {
+      return Ordering.Unordered;
+    }
+  }
+  const bKeys = Object.keys(b);
+  const aRemainingKeys = aKeys.length - commonKeys.size;
+  const bRemainingKeys = bKeys.length - commonKeys.size;
+  if (aRemainingKeys === 0 && bRemainingKeys === 0) return Ordering.Equal;
+  if (aRemainingKeys === 0) return Ordering.StrictSuperset;
+  if (bRemainingKeys === 0) return Ordering.StrictSubset;
+  return Ordering.Unordered;
+}

src/common/framework/query/encode_selectively.ts

@@ -0,0 +1,16 @@
+// Encodes a stringified TestQuery so that it can be placed in a `?q=` parameter in a URL.
+// encodeURIComponent encodes in accordance with `application/x-www-form-urlencoded`, but URLs don't
+// actually have to be as strict as HTML form encoding (we interpret this purely from JavaScript).
+// So we encode the component, then selectively convert some %-encoded escape codes back to their
+// original form for readability/copyability.
+export function encodeURIComponentSelectively(s: string): string {
+  let ret = encodeURIComponent(s);
+  ret = ret.replace(/%22/g, '"'); // for JSON strings
+  ret = ret.replace(/%2C/g, ','); // for path separator, and JSON arrays
+  ret = ret.replace(/%3A/g, ':'); // for big separator
+  ret = ret.replace(/%3B/g, ';'); // for param separator
+  ret = ret.replace(/%3D/g, '='); // for params (k=v)
+  ret = ret.replace(/%5B/g, '['); // for JSON arrays
+  ret = ret.replace(/%5D/g, ']'); // for JSON arrays
+  return ret;
+}

src/common/framework/query/parseQuery.ts

@@ -0,0 +1,132 @@
+import {
+  CaseParamsRW,
+  ParamArgument,
+  badParamValueChars,
+  paramKeyIsPublic,
+} from '../params_utils.js';
+import { assert } from '../util/util.js';
+
+import {
+  TestQuery,
+  TestQueryMultiFile,
+  TestQueryMultiTest,
+  TestQueryMultiCase,
+  TestQuerySingleCase,
+} from './query.js';
+import { kBigSeparator, kWildcard, kPathSeparator, kParamSeparator } from './separators.js';
+import { validQueryPart } from './validQueryPart.js';
+
+export function parseQuery(s: string): TestQuery {
+  try {
+    return parseQueryImpl(s);
+  } catch (ex) {
+    ex.message += '\n  on: ' + s;
+    throw ex;
+  }
+}
+
+function parseQueryImpl(s: string): TestQuery {
+  // Undo encodeURIComponentSelectively
+  s = decodeURIComponent(s);
+
+  // bigParts are: suite, group, test, params (note kBigSeparator could appear in params)
+  const [suite, fileString, testString, paramsString] = s.split(kBigSeparator, 4);
+  assert(fileString !== undefined, `filter string must have at least one ${kBigSeparator}`);
+
+  const { parts: file, wildcard: filePathHasWildcard } = parseBigPart(fileString, kPathSeparator);
+
+  if (testString === undefined) {
+    // Query is file-level
+    assert(
+      filePathHasWildcard,
+      `File-level query without wildcard ${kWildcard}. Did you want a file-level query \
+(append ${kPathSeparator}${kWildcard}) or test-level query (append ${kBigSeparator}${kWildcard})?`
+    );
+    return new TestQueryMultiFile(suite, file);
+  }
+  assert(!filePathHasWildcard, `Wildcard ${kWildcard} must be at the end of the query string`);
+
+  const { parts: test, wildcard: testPathHasWildcard } = parseBigPart(testString, kPathSeparator);
+
+  if (paramsString === undefined) {
+    // Query is test-level
+    assert(
+      testPathHasWildcard,
+      `Test-level query without wildcard ${kWildcard}; did you want a test-level query \
+(append ${kPathSeparator}${kWildcard}) or case-level query (append ${kBigSeparator}${kWildcard})?`
+    );
+    assert(file.length > 0, 'File part of test-level query was empty (::)');
+    return new TestQueryMultiTest(suite, file, test);
+  }
+
+  // Query is case-level
+  assert(!testPathHasWildcard, `Wildcard ${kWildcard} must be at the end of the query string`);
+
+  const { parts: paramsParts, wildcard: paramsHasWildcard } = parseBigPart(
+    paramsString,
+    kParamSeparator
+  );
+
+  assert(test.length > 0, 'Test part of case-level query was empty (::)');
+
+  const params: CaseParamsRW = {};
+  for (const paramPart of paramsParts) {
+    const [k, v] = parseSingleParam(paramPart);
+    assert(validQueryPart.test(k), 'param key names must match ' + validQueryPart);
+    params[k] = v;
+  }
+  if (paramsHasWildcard) {
+    return new TestQueryMultiCase(suite, file, test, params);
+  } else {
+    return new TestQuerySingleCase(suite, file, test, params);
+  }
+}
+
+// webgpu:a,b,* or webgpu:a,b,c:*
+const kExampleQueries = `\
+webgpu${kBigSeparator}a${kPathSeparator}b${kPathSeparator}${kWildcard} or \
+webgpu${kBigSeparator}a${kPathSeparator}b${kPathSeparator}c${kBigSeparator}${kWildcard}`;
+
+function parseBigPart(
+  s: string,
+  separator: typeof kParamSeparator | typeof kPathSeparator
+): { parts: string[]; wildcard: boolean } {
+  if (s === '') {
+    return { parts: [], wildcard: false };
+  }
+  const parts = s.split(separator);
+
+  let endsWithWildcard = false;
+  for (const [i, part] of parts.entries()) {
+    if (i === parts.length - 1) {
+      endsWithWildcard = part === kWildcard;
+    }
+    assert(
+      part.indexOf(kWildcard) === -1 || endsWithWildcard,
+      `Wildcard ${kWildcard} must be complete last part of a path (e.g. ${kExampleQueries})`
+    );
+  }
+  if (endsWithWildcard) {
+    // Remove the last element of the array (which is just the wildcard).
+    parts.length = parts.length - 1;
+  }
+  return { parts, wildcard: endsWithWildcard };
+}
+
+function parseSingleParam(paramSubstring: string): [string, ParamArgument] {
+  assert(paramSubstring !== '', 'Param in a query must not be blank (is there a trailing comma?)');
+  const i = paramSubstring.indexOf('=');
+  assert(i !== -1, 'Param in a query must be of form key=value');
+  const k = paramSubstring.substring(0, i);
+  assert(paramKeyIsPublic(k), 'Param in a query must not be private (start with _)');
+  const v = paramSubstring.substring(i + 1);
+  return [k, parseSingleParamValue(v)];
+}
+
+function parseSingleParamValue(s: string): ParamArgument {
+  assert(
+    !badParamValueChars.test(s),
+    `param value must not match ${badParamValueChars} - was ${s}`
+  );
+  return s === 'undefined' ? undefined : JSON.parse(s);
+}

src/common/framework/query/query.ts

@@ -0,0 +1,91 @@
+import { CaseParams } from '../params_utils.js';
+import { assert } from '../util/util.js';
+
+import { encodeURIComponentSelectively } from './encode_selectively.js';
+import { kBigSeparator, kPathSeparator, kWildcard, kParamSeparator } from './separators.js';
+import { stringifyPublicParams } from './stringify_params.js';
+
+export type TestQuery =
+  | TestQuerySingleCase
+  | TestQueryMultiCase
+  | TestQueryMultiTest
+  | TestQueryMultiFile;
+
+export class TestQueryMultiFile {
+  readonly isMultiFile: boolean = true;
+  readonly suite: string;
+  readonly filePathParts: readonly string[];
+
+  constructor(suite: string, file: readonly string[]) {
+    this.suite = suite;
+    this.filePathParts = [...file];
+  }
+
+  toString(): string {
+    return encodeURIComponentSelectively(this.toStringHelper().join(kBigSeparator));
+  }
+
+  toHTML(): string {
+    return this.toStringHelper().join(kBigSeparator + '<wbr>');
+  }
+
+  protected toStringHelper(): string[] {
+    return [this.suite, [...this.filePathParts, kWildcard].join(kPathSeparator)];
+  }
+}
+
+export class TestQueryMultiTest extends TestQueryMultiFile {
+  readonly isMultiFile: false = false;
+  readonly isMultiTest: boolean = true;
+  readonly testPathParts: readonly string[];
+
+  constructor(suite: string, file: readonly string[], test: readonly string[]) {
+    super(suite, file);
+    assert(file.length > 0, 'multi-test (or finer) query must have file-path');
+    this.testPathParts = [...test];
+  }
+
+  protected toStringHelper(): string[] {
+    return [
+      this.suite,
+      this.filePathParts.join(kPathSeparator),
+      [...this.testPathParts, kWildcard].join(kPathSeparator),
+    ];
+  }
+}
+
+export class TestQueryMultiCase extends TestQueryMultiTest {
+  readonly isMultiTest: false = false;
+  readonly isMultiCase: boolean = true;
+  readonly params: CaseParams;
+
+  constructor(suite: string, file: readonly string[], test: readonly string[], params: CaseParams) {
+    super(suite, file, test);
+    assert(test.length > 0, 'multi-case (or finer) query must have test-path');
+    this.params = { ...params };
+  }
+
+  protected toStringHelper(): string[] {
+    const paramsParts = stringifyPublicParams(this.params);
+    return [
+      this.suite,
+      this.filePathParts.join(kPathSeparator),
+      this.testPathParts.join(kPathSeparator),
+      [...paramsParts, kWildcard].join(kParamSeparator),
+    ];
+  }
+}
+
+export class TestQuerySingleCase extends TestQueryMultiCase {
+  readonly isMultiCase: false = false;
+
+  protected toStringHelper(): string[] {
+    const paramsParts = stringifyPublicParams(this.params);
+    return [
+      this.suite,
+      this.filePathParts.join(kPathSeparator),
+      this.testPathParts.join(kPathSeparator),
+      paramsParts.join(kParamSeparator),
+    ];
+  }
+}

src/common/framework/query/separators.ts

@@ -0,0 +1,5 @@
+export const kBigSeparator = ':'; // Separator between big parts: suite:file:test:case
+export const kPathSeparator = ','; // Separator between path,to,file or path,to,test
+export const kParamSeparator = ';'; // Separator between k=v;k=v
+export const kParamKVSeparator = '='; // Separator between key and value in k=v
+export const kWildcard = '*'; // Final wildcard, if query is not single-case

src/common/framework/query/stringify_params.ts

@@ -0,0 +1,27 @@
+import {
+  CaseParams,
+  extractPublicParams,
+  ParamArgument,
+  badParamValueChars,
+} from '../params_utils.js';
+import { assert } from '../util/util.js';
+
+import { kParamKVSeparator } from './separators.js';
+
+export function stringifyPublicParams(p: CaseParams): string[] {
+  const pub = extractPublicParams(p);
+  return Object.entries(pub).map(([k, v]) => stringifySingleParam(k, v));
+}
+
+export function stringifySingleParam(k: string, v: ParamArgument) {
+  return `${k}${kParamKVSeparator}${stringifySingleParamValue(v)}`;
+}
+
+function stringifySingleParamValue(v: ParamArgument): string {
+  const s = v === undefined ? 'undefined' : JSON.stringify(v);
+  assert(
+    !badParamValueChars.test(s),
+    `JSON.stringified param value must not match ${badParamValueChars} - was ${s}`
+  );
+  return s;
+}

src/common/framework/query/validQueryPart.ts

@@ -0,0 +1,2 @@
+// Applies to group parts, test parts, params keys.
+export const validQueryPart = /^[a-zA-Z0-9_]+$/;

src/common/framework/test_filter/filter_by_group.ts

@@ -1,52 +0,0 @@
-import { TestSpecOrTestOrCaseID } from '../id.js';
-import { ReadmeFile, TestFileLoader, TestSpec } from '../loader.js';
-
-import { TestFilter } from './internal.js';
-import { TestFilterResult } from './test_filter_result.js';
-
-export class FilterByGroup implements TestFilter {
-  private readonly suite: string;
-  private readonly specPathPrefix: string;
-
-  constructor(suite: string, groupPrefix: string) {
-    this.suite = suite;
-    this.specPathPrefix = groupPrefix;
-  }
-
-  matches(id: TestSpecOrTestOrCaseID): boolean {
-    return id.spec.suite === this.suite && this.pathMatches(id.spec.path);
-  }
-
-  async iterate(loader: TestFileLoader): Promise<TestFilterResult[]> {
-    const specs = await loader.listing(this.suite);
-    const entries: TestFilterResult[] = [];
-
-    const suite = this.suite;
-    for (const { path, description } of specs) {
-      if (this.pathMatches(path)) {
-        const isReadme = path === '' || path.endsWith('/');
-        const spec = isReadme
-          ? ({ description } as ReadmeFile)
-          : ((await loader.import(`${suite}/${path}.spec.js`)) as TestSpec);
-        entries.push({ id: { suite, path }, spec });
-      }
-    }
-
-    return entries;
-  }
-
-  definitelyOneFile(): boolean {
-    // FilterByGroup could always possibly match multiple files, because it represents a prefix,
-    // e.g. "a:b" not "a:b:".
-    return false;
-  }
-
-  idIfSingle(): undefined {
-    // FilterByGroup could be one whole suite, but we only want whole files, tests, or cases.
-    return undefined;
-  }
-
-  private pathMatches(path: string): boolean {
-    return path.startsWith(this.specPathPrefix);
-  }
-}

src/common/framework/test_filter/filter_one_file.ts

@@ -1,153 +0,0 @@
-import { TestCaseID, TestSpecID, TestSpecOrTestOrCaseID, testSpecEquals } from '../id.js';
-import { TestFileLoader, TestSpec } from '../loader.js';
-import { TestSpecRecorder } from '../logger.js';
-import { ParamSpec, paramsEquals, paramsSupersets } from '../params_utils.js';
-import { RunCase, RunCaseIterable } from '../test_group.js';
-
-import { TestFilter } from './internal.js';
-import { TestFilterResult } from './test_filter_result.js';
-
-abstract class FilterOneFile implements TestFilter {
-  protected readonly specId: TestSpecID;
-
-  constructor(specId: TestSpecID) {
-    this.specId = specId;
-  }
-
-  abstract getCases(spec: TestSpec): RunCaseIterable;
-
-  async iterate(loader: TestFileLoader): Promise<TestFilterResult[]> {
-    const spec = (await loader.import(
-      `${this.specId.suite}/${this.specId.path}.spec.js`
-    )) as TestSpec;
-    return [
-      {
-        id: this.specId,
-        spec: {
-          description: spec.description,
-          g: this.getCases(spec),
-        },
-      },
-    ];
-  }
-
-  definitelyOneFile(): true {
-    return true;
-  }
-
-  abstract idIfSingle(): TestSpecOrTestOrCaseID | undefined;
-  abstract matches(id: TestSpecOrTestOrCaseID): boolean;
-}
-
-type TestGroupFilter = (testcase: TestCaseID) => boolean;
-function filterTestGroup(group: RunCaseIterable, filter: TestGroupFilter): RunCaseIterable {
-  return {
-    *iterate(log: TestSpecRecorder): Iterable<RunCase> {
-      for (const rc of group.iterate(log)) {
-        if (filter(rc.id)) {
-          yield rc;
-        }
-      }
-    },
-  };
-}
-
-export class FilterByTestMatch extends FilterOneFile {
-  private readonly testPrefix: string;
-
-  constructor(specId: TestSpecID, testPrefix: string) {
-    super(specId);
-    this.testPrefix = testPrefix;
-  }
-
-  getCases(spec: TestSpec): RunCaseIterable {
-    return filterTestGroup(spec.g, testcase => this.testMatches(testcase.test));
-  }
-
-  idIfSingle(): TestSpecOrTestOrCaseID | undefined {
-    if (this.testPrefix.length !== 0) {
-      return undefined;
-    }
-    // This is one whole spec file.
-    return { spec: this.specId };
-  }
-
-  matches(id: TestSpecOrTestOrCaseID): boolean {
-    if (id.test === undefined) {
-      return false;
-    }
-    return testSpecEquals(id.spec, this.specId) && this.testMatches(id.test);
-  }
-
-  private testMatches(test: string): boolean {
-    return test.startsWith(this.testPrefix);
-  }
-}
-
-export class FilterByParamsMatch extends FilterOneFile {
-  private readonly test: string;
-  private readonly params: ParamSpec | null;
-
-  constructor(specId: TestSpecID, test: string, params: ParamSpec | null) {
-    super(specId);
-    this.test = test;
-    this.params = params;
-  }
-
-  getCases(spec: TestSpec): RunCaseIterable {
-    return filterTestGroup(spec.g, testcase => this.caseMatches(testcase.test, testcase.params));
-  }
-
-  idIfSingle(): TestSpecOrTestOrCaseID | undefined {
-    if (this.params !== null) {
-      return undefined;
-    }
-    // This is one whole test.
-    return { spec: this.specId, test: this.test };
-  }
-
-  matches(id: TestSpecOrTestOrCaseID): boolean {
-    if (id.test === undefined) {
-      return false;
-    }
-    return testSpecEquals(id.spec, this.specId) && this.caseMatches(id.test, id.params);
-  }
-
-  private caseMatches(test: string, params: ParamSpec | null | undefined): boolean {
-    if (params === undefined) {
-      return false;
-    }
-    return test === this.test && paramsSupersets(params, this.params);
-  }
-}
-
-export class FilterByParamsExact extends FilterOneFile {
-  private readonly test: string;
-  private readonly params: ParamSpec | null;
-
-  constructor(specId: TestSpecID, test: string, params: ParamSpec | null) {
-    super(specId);
-    this.test = test;
-    this.params = params;
-  }
-
-  getCases(spec: TestSpec): RunCaseIterable {
-    return filterTestGroup(spec.g, testcase => this.caseMatches(testcase.test, testcase.params));
-  }
-
-  idIfSingle(): TestSpecOrTestOrCaseID | undefined {
-    // This is one single test case.
-    return { spec: this.specId, test: this.test, params: this.params };
-  }
-
-  matches(id: TestSpecOrTestOrCaseID): boolean {
-    if (id.test === undefined || id.params === undefined) {
-      return false;
-    }
-    return testSpecEquals(id.spec, this.specId) && this.caseMatches(id.test, id.params);
-  }
-
-  private caseMatches(test: string, params: ParamSpec | null): boolean {
-    return test === this.test && paramsEquals(params, this.params);
-  }
-}

src/common/framework/test_filter/internal.ts

@@ -1,17 +0,0 @@
-import { TestSpecOrTestOrCaseID } from '../id.js';
-import { TestFileLoader } from '../loader.js';
-
-import { TestFilterResult } from './test_filter_result.js';
-
-export interface TestFilter {
-  // Iterates over the test cases matched by the filter.
-  iterate(loader: TestFileLoader): Promise<TestFilterResult[]>;
-
-  // Iff the filter could not possibly match multiple files, returns true.
-  definitelyOneFile(): boolean;
-
-  // If the filter can accept one spec, one test, or one case, returns its ID.
-  idIfSingle(): TestSpecOrTestOrCaseID | undefined;
-
-  matches(id: TestSpecOrTestOrCaseID): boolean;
-}

src/common/framework/test_filter/load_filter.ts

@@ -1,63 +0,0 @@
-import { allowedTestNameCharacters } from '../allowed_characters.js';
-import { TestFileLoader } from '../loader.js';
-import { ParamSpec } from '../params_utils.js';
-import { assert, unreachable } from '../util/util.js';
-
-import { FilterByGroup } from './filter_by_group.js';
-import { FilterByParamsExact, FilterByParamsMatch, FilterByTestMatch } from './filter_one_file.js';
-import { TestFilter } from './internal.js';
-import { TestFilterResult } from './test_filter_result.js';
-
-// Each filter is of one of the forms below (urlencoded).
-export function makeFilter(filter: string): TestFilter {
-  const i1 = filter.indexOf(':');
-  assert(i1 !== -1, 'Test queries must fully specify their suite name (e.g. "webgpu:")');
-
-  const suite = filter.substring(0, i1);
-  const i2 = filter.indexOf(':', i1 + 1);
-  if (i2 === -1) {
-    // - webgpu:
-    // - webgpu:buf
-    // - webgpu:buffers/
-    // - webgpu:buffers/map
-    const groupPrefix = filter.substring(i1 + 1);
-    return new FilterByGroup(suite, groupPrefix);
-  }
-
-  const path = filter.substring(i1 + 1, i2);
-  const endOfTestName = new RegExp('[^' + allowedTestNameCharacters + ']');
-  const i3sub = filter.substring(i2 + 1).search(endOfTestName);
-  if (i3sub === -1) {
-    // - webgpu:buffers/mapWriteAsync:
-    // - webgpu:buffers/mapWriteAsync:b
-    const testPrefix = filter.substring(i2 + 1);
-    return new FilterByTestMatch({ suite, path }, testPrefix);
-  }
-
-  const i3 = i2 + 1 + i3sub;
-  const test = filter.substring(i2 + 1, i3);
-  const token = filter.charAt(i3);
-
-  let params = null;
-  if (i3 + 1 < filter.length) {
-    params = JSON.parse(filter.substring(i3 + 1)) as ParamSpec;
-  }
-
-  if (token === '~') {
-    // - webgpu:buffers/mapWriteAsync:basic~
-    // - webgpu:buffers/mapWriteAsync:basic~{}
-    // - webgpu:buffers/mapWriteAsync:basic~{filter:"params"}
-    return new FilterByParamsMatch({ suite, path }, test, params);
-  } else if (token === '=') {
-    // - webgpu:buffers/mapWriteAsync:basic=
-    // - webgpu:buffers/mapWriteAsync:basic={}
-    // - webgpu:buffers/mapWriteAsync:basic={exact:"params"}
-    return new FilterByParamsExact({ suite, path }, test, params);
-  } else {
-    unreachable("invalid character after test name; must be '~' or '='");
-  }
-}
-
-export function loadFilter(loader: TestFileLoader, filter: string): Promise<TestFilterResult[]> {
-  return makeFilter(filter).iterate(loader);
-}

src/common/framework/test_filter/test_filter_result.ts

@@ -1,8 +0,0 @@
-import { TestSpecID } from '../id.js';
-import { TestSpecOrReadme } from '../loader.js';
-
-// Result of iterating a test filter. Contains a loaded spec (.spec.ts) file and its id.
-export interface TestFilterResult {
-  readonly id: TestSpecID;
-  readonly spec: TestSpecOrReadme;
-}

src/common/framework/test_group.ts

@@ -1,27 +1,50 @@
-import { allowedTestNameCharacters } from './allowed_characters.js';
 import { Fixture } from './fixture.js';
-import { TestCaseID } from './id.js';
+import { TestCaseRecorder } from './logging/test_case_recorder.js';
-import { LiveTestCaseResult, TestCaseRecorder, TestSpecRecorder } from './logger.js';
+import {
-import { ParamSpec, ParamSpecIterable, extractPublicParams, paramsEquals } from './params_utils.js';
+  CaseParams,
-import { checkPublicParamType } from './url_query.js';
+  CaseParamsIterable,
+  extractPublicParams,
+  publicParamsEquals,
+} from './params_utils.js';
+import { kPathSeparator } from './query/separators.js';
+import { stringifySingleParam } from './query/stringify_params.js';
+import { validQueryPart } from './query/validQueryPart.js';
 import { assert } from './util/util.js';
 
+export type RunFn = (rec: TestCaseRecorder) => Promise<void>;
+
+export interface TestCaseID {
+  readonly test: readonly string[];
+  readonly params: CaseParams;
+}
+
 export interface RunCase {
   readonly id: TestCaseID;
-  run(debug?: boolean): Promise<LiveTestCaseResult>;
+  run: RunFn;
-  injectResult(result: LiveTestCaseResult): void;
 }
 
+// Interface for defining tests
+export interface TestGroupBuilder<F extends Fixture> {
+  test(name: string): TestBuilderWithName<F, never>;
+}
+export function makeTestGroup<F extends Fixture>(fixture: FixtureClass<F>): TestGroupBuilder<F> {
+  return new TestGroup(fixture);
+}
+
+// Interface for running tests
 export interface RunCaseIterable {
-  iterate(rec: TestSpecRecorder): Iterable<RunCase>;
+  iterate(): Iterable<RunCase>;
+}
+export function makeTestGroupForUnitTesting<F extends Fixture>(
+  fixture: FixtureClass<F>
+): TestGroup<F> {
+  return new TestGroup(fixture);
 }
 
-type FixtureClass<F extends Fixture> = new (log: TestCaseRecorder, params: ParamSpec) => F;
+type FixtureClass<F extends Fixture> = new (log: TestCaseRecorder, params: CaseParams) => F;
 type TestFn<F extends Fixture, P extends {}> = (t: F & { params: P }) => Promise<void> | void;
 
-const validNames = new RegExp('^[' + allowedTestNameCharacters + ']+$');
+class TestGroup<F extends Fixture> implements RunCaseIterable, TestGroupBuilder<F> {
-
-export class TestGroup<F extends Fixture> implements RunCaseIterable {
   private fixture: FixtureClass<F>;
   private seen: Set<string> = new Set();
   private tests: Array<TestBuilder<F, never>> = [];
@@ -30,14 +53,13 @@ export class TestGroup<F extends Fixture> implements RunCaseIterable {
     this.fixture = fixture;
   }
 
-  *iterate(log: TestSpecRecorder): Iterable<RunCase> {
+  *iterate(): Iterable<RunCase> {
     for (const test of this.tests) {
-      yield* test.iterate(log);
+      yield* test.iterate();
     }
   }
 
   private checkName(name: string): void {
-    assert(validNames.test(name), `Invalid test name ${name}; must match [${validNames}]+`);
     assert(
       // Shouldn't happen due to the rule above. Just makes sure that treated
       // unencoded strings as encoded strings is OK.
@@ -51,13 +73,14 @@ export class TestGroup<F extends Fixture> implements RunCaseIterable {
 
   // TODO: This could take a fixture, too, to override the one for the group.
   test(name: string): TestBuilderWithName<F, never> {
-    // Replace spaces with underscores for readability.
-    assert(name.indexOf('_') === -1, 'Invalid test name ${name}: contains underscore (use space)');
-    name = name.replace(/ /g, '_');
-
     this.checkName(name);
 
-    const test = new TestBuilder<F, never>(name, this.fixture);
+    const parts = name.split(kPathSeparator);
+    for (const p of parts) {
+      assert(validQueryPart.test(p), `Invalid test name part ${p}; must match ${validQueryPart}`);
+    }
+
+    const test = new TestBuilder<F, never>(parts, this.fixture);
     this.tests.push(test);
     return test;
   }
@@ -72,13 +95,13 @@ interface TestBuilderWithParams<F extends Fixture, P extends {}> {
 }
 
 class TestBuilder<F extends Fixture, P extends {}> {
-  private readonly name: string;
+  private readonly testPath: string[];
   private readonly fixture: FixtureClass<F>;
   private testFn: TestFn<F, P> | undefined;
-  private cases: ParamSpecIterable | null = null;
+  private cases?: CaseParamsIterable = undefined;
 
-  constructor(name: string, fixture: FixtureClass<F>) {
+  constructor(testPath: string[], fixture: FixtureClass<F>) {
-    this.name = name;
+    this.testPath = testPath;
     this.fixture = fixture;
   }
 
@@ -86,22 +109,22 @@ class TestBuilder<F extends Fixture, P extends {}> {
     this.testFn = fn;
   }
 
-  params<NewP extends {}>(specs: Iterable<NewP>): TestBuilderWithParams<F, NewP> {
+  params<NewP extends {}>(casesIterable: Iterable<NewP>): TestBuilderWithParams<F, NewP> {
-    assert(this.cases === null, 'test case is already parameterized');
+    assert(this.cases === undefined, 'test case is already parameterized');
-    const cases = Array.from(specs);
+    const cases = Array.from(casesIterable);
-    const seen: ParamSpec[] = [];
+    const seen: CaseParams[] = [];
     // This is n^2.
     for (const spec of cases) {
       const publicParams = extractPublicParams(spec);
 
       // Check type of public params: can only be (currently):
       // number, string, boolean, undefined, number[]
-      for (const v of Object.values(publicParams)) {
+      for (const [k, v] of Object.entries(publicParams)) {
-        checkPublicParamType(v);
+        stringifySingleParam(k, v); // To check for invalid params values
       }
 
       assert(
-        !seen.some(x => paramsEquals(x, publicParams)),
+        !seen.some(x => publicParamsEquals(x, publicParams)),
         'Duplicate test case params: ' + JSON.stringify(publicParams)
       );
       seen.push(publicParams);
@@ -111,39 +134,35 @@ class TestBuilder<F extends Fixture, P extends {}> {
     return (this as unknown) as TestBuilderWithParams<F, NewP>;
   }
 
-  *iterate(rec: TestSpecRecorder): IterableIterator<RunCase> {
+  *iterate(): IterableIterator<RunCase> {
     assert(this.testFn !== undefined, 'internal error');
-    for (const params of this.cases || [null]) {
+    for (const params of this.cases || [{}]) {
-      yield new RunCaseSpecific(rec, this.name, params, this.fixture, this.testFn);
+      yield new RunCaseSpecific(this.testPath, params, this.fixture, this.testFn);
     }
   }
 }
 
 class RunCaseSpecific<F extends Fixture> implements RunCase {
   readonly id: TestCaseID;
-  private readonly params: ParamSpec | null;
+
-  private readonly recorder: TestSpecRecorder;
+  private readonly params: CaseParams | null;
   private readonly fixture: FixtureClass<F>;
   private readonly fn: TestFn<F, never>;
 
   constructor(
-    recorder: TestSpecRecorder,
+    testPath: string[],
-    test: string,
+    params: CaseParams,
-    params: ParamSpec | null,
     fixture: FixtureClass<F>,
     fn: TestFn<F, never>
   ) {
-    this.id = { test, params: params ? extractPublicParams(params) : null };
+    this.id = { test: testPath, params: extractPublicParams(params) };
     this.params = params;
-    this.recorder = recorder;
     this.fixture = fixture;
     this.fn = fn;
   }
 
-  async run(debug: boolean): Promise<LiveTestCaseResult> {
+  async run(rec: TestCaseRecorder): Promise<void> {
-    const [rec, res] = this.recorder.record(this.id.test, this.id.params);
+    rec.start();
-    rec.start(debug);
-
     try {
       const inst = new this.fixture(rec, this.params || {});
 
@@ -162,13 +181,6 @@ class RunCaseSpecific<F extends Fixture> implements RunCase {
       // or unexpected validation/OOM error from the GPUDevice.
       rec.threw(ex);
     }
-
     rec.finish();
-    return res;
-  }
-
-  injectResult(result: LiveTestCaseResult): void {
-    const [, res] = this.recorder.record(this.id.test, this.id.params);
-    Object.assign(res, result);
   }
 }

src/common/framework/test_suite_listing.ts

@@ -0,0 +1,16 @@
+// A listing of all specs within a single suite. This is the (awaited) type of
+// `groups` in '{cts,unittests}/listing.ts' and `listing` in the auto-generated
+// 'out/{cts,unittests}/listing.js' files (see tools/gen_listings).
+export type TestSuiteListing = Iterable<TestSuiteListingEntry>;
+
+export type TestSuiteListingEntry = TestSuiteListingEntrySpec | TestSuiteListingEntryReadme;
+
+interface TestSuiteListingEntrySpec {
+  readonly file: string[];
+  readonly description: string;
+}
+
+interface TestSuiteListingEntryReadme {
+  readonly file: string[];
+  readonly readme: string;
+}

src/common/framework/tree.ts

@@ -1,82 +1,362 @@
-import { Logger } from './logger.js';
+import { TestFileLoader } from './file_loader.js';
-import { stringifyPublicParams } from './params_utils.js';
+import { TestCaseRecorder } from './logging/test_case_recorder.js';
-import { TestFilterResult } from './test_filter/test_filter_result.js';
+import { CaseParamsRW } from './params_utils.js';
-import { RunCase } from './test_group.js';
+import { compareQueries, Ordering } from './query/compare.js';
+import {
+  TestQuery,
+  TestQueryMultiCase,
+  TestQuerySingleCase,
+  TestQueryMultiFile,
+  TestQueryMultiTest,
+} from './query/query.js';
+import { stringifySingleParam } from './query/stringify_params.js';
+import { RunCase, RunFn } from './test_group.js';
+import { assert } from './util/util.js';
 
-export interface FilterResultTreeNode {
+// `loadTreeForQuery()` loads a TestTree for a given queryToLoad.
+// The resulting tree is a linked-list all the way from `suite:*` to queryToLoad,
+// and under queryToLoad is a tree containing every case matched by queryToLoad.
+//
+// `subqueriesToExpand` influences the `collapsible` flag on nodes in the resulting tree.
+// A node is considered "collapsible" if none of the subqueriesToExpand is a StrictSubset
+// of that node.
+//
+// In WebKit/Blink-style web_tests, an expectation file marks individual cts.html "variants" as
+// "Failure", "Crash", etc.
+// By passing in the list of expectations as the subqueriesToExpand, we can programmatically
+// subdivide the cts.html "variants" list to be able to implement arbitrarily-fine suppressions
+// (instead of having to suppress entire test files, which would lose a lot of coverage).
+//
+// `iterateCollapsedQueries()` produces the list of queries for the variants list.
+//
+// Though somewhat complicated, this system has important benefits:
+//   - Avoids having to suppress entire test files, which would cause large test coverage loss.
+//   - Minimizes the number of page loads needed for fine-grained suppressions.
+//     (In the naive case, we could do one page load per test case - but the test suite would
+//     take impossibly long to run.)
+//   - Enables developers to put any number of tests in one file as appropriate, without worrying
+//     about expectation granularity.
+
+export interface TestSubtree<T extends TestQuery = TestQuery> {
+  readonly query: T;
+  readonly children: Map<string, TestTreeNode>;
+  readonly collapsible: boolean;
   description?: string;
-  runCase?: RunCase;
-  children?: Map<string, FilterResultTreeNode>;
 }
 
-// e.g. iteratePath('a/b/c/d', ':') yields ['a/', 'a/b/', 'a/b/c/', 'a/b/c/d:']
+export interface TestTreeLeaf {
-function* iteratePath(path: string, terminator: string): IterableIterator<string> {
+  readonly query: TestQuerySingleCase;
-  const parts = path.split('/');
+  readonly run: RunFn;
-  if (parts.length > 1) {
-    let partial = parts[0] + '/';
-    yield partial;
-    for (let i = 1; i < parts.length - 1; ++i) {
-      partial += parts[i] + '/';
-      yield partial;
-    }
-    // Path ends in '/' (so is a README).
-    if (parts[parts.length - 1] === '') {
-      return;
-    }
-  }
-  yield path + terminator;
 }
 
-export function treeFromFilterResults(
+export type TestTreeNode = TestSubtree | TestTreeLeaf;
-  log: Logger,
+
-  listing: IterableIterator<TestFilterResult>
+export class TestTree {
-): FilterResultTreeNode {
+  readonly root: TestSubtree;
-  function getOrInsert(n: FilterResultTreeNode, k: string): FilterResultTreeNode {
+
-    const children = n.children!;
+  constructor(root: TestSubtree) {
-    if (children.has(k)) {
+    this.root = root;
-      return children.get(k)!;
-    }
-    const v = { children: new Map() };
-    children.set(k, v);
-    return v;
   }
 
-  const tree = { children: new Map() };
+  iterateCollapsedQueries(): IterableIterator<TestQuery> {
-  for (const f of listing) {
+    return TestTree.iterateSubtreeCollapsedQueries(this.root);
-    const files = getOrInsert(tree, f.id.suite + ':');
+  }
-    if (f.id.path === '') {
-      // This is a suite README.
-      files.description = f.spec.description.trim();
-      continue;
-    }
 
-    let tests = files;
+  iterateLeaves(): IterableIterator<TestTreeLeaf> {
-    for (const path of iteratePath(f.id.path, ':')) {
+    return TestTree.iterateSubtreeLeaves(this.root);
-      tests = getOrInsert(tests, f.id.suite + ':' + path);
+  }
-    }
-    if (f.spec.description) {
-      // This is a directory README or spec file.
-      tests.description = f.spec.description.trim();
-    }
 
-    if (!('g' in f.spec)) {
+  toString(): string {
-      // This is a directory README.
+    return TestTree.subtreeToString('(root)', this.root, '');
-      continue;
+  }
-    }
 
-    const [tRec] = log.record(f.id);
+  static *iterateSubtreeCollapsedQueries(subtree: TestSubtree): IterableIterator<TestQuery> {
-    const fId = f.id.suite + ':' + f.id.path;
+    for (const [, child] of subtree.children) {
-    for (const t of f.spec.g.iterate(tRec)) {
+      if ('children' in child && !child.collapsible) {
-      let cases = tests;
+        yield* TestTree.iterateSubtreeCollapsedQueries(child);
-      for (const path of iteratePath(t.id.test, '~')) {
+      } else {
-        cases = getOrInsert(cases, fId + ':' + path);
+        yield child.query;
+      }
+    }
+  }
+
+  static *iterateSubtreeLeaves(subtree: TestSubtree): IterableIterator<TestTreeLeaf> {
+    for (const [, child] of subtree.children) {
+      if ('children' in child) {
+        yield* TestTree.iterateSubtreeLeaves(child);
+      } else {
+        yield child;
+      }
+    }
+  }
+
+  static subtreeToString(name: string, tree: TestTreeNode, indent: string): string {
+    const collapsible = 'run' in tree ? '>' : tree.collapsible ? '+' : '-';
+    let s =
+      indent +
+      `${collapsible} ${JSON.stringify(name)} => ` +
+      `${tree.query}        ${JSON.stringify(tree.query)}`;
+    if ('children' in tree) {
+      if (tree.description !== undefined) {
+        s += indent + `\n    | ${JSON.stringify(tree.description)}`;
       }
 
-      const p = stringifyPublicParams(t.id.params);
+      for (const [name, child] of tree.children) {
-      cases.children!.set(fId + ':' + t.id.test + '=' + p, {
+        s += '\n' + TestTree.subtreeToString(name, child, indent + '  ');
-        runCase: t,
+      }
-      });
     }
+    return s;
+  }
+}
+
+// TODO: Consider having subqueriesToExpand actually impact the depth-order of params in the tree.
+export async function loadTreeForQuery(
+  loader: TestFileLoader,
+  queryToLoad: TestQuery,
+  subqueriesToExpand: TestQuery[]
+): Promise<TestTree> {
+  const suite = queryToLoad.suite;
+  const specs = await loader.listing(suite);
+
+  const subqueriesToExpandEntries = Array.from(subqueriesToExpand.entries());
+  const seenSubqueriesToExpand: boolean[] = new Array(subqueriesToExpand.length);
+  seenSubqueriesToExpand.fill(false);
+
+  const isCollapsible = (subquery: TestQuery) =>
+    subqueriesToExpandEntries.every(([i, toExpand]) => {
+      const ordering = compareQueries(toExpand, subquery);
+
+      // If toExpand == subquery, no expansion is needed (but it's still "seen").
+      if (ordering === Ordering.Equal) seenSubqueriesToExpand[i] = true;
+      return ordering !== Ordering.StrictSubset;
+    });
+
+  // L0 = suite-level, e.g. suite:*
+  // L1 =  file-level, e.g. suite:a,b:*
+  // L2 =  test-level, e.g. suite:a,b:c,d:*
+  // L3 =  case-level, e.g. suite:a,b:c,d:
+  let foundCase = false;
+  // L0 is suite:*
+  const subtreeL0 = makeTreeForSuite(suite);
+  isCollapsible(subtreeL0.query); // mark seenSubqueriesToExpand
+  for (const entry of specs) {
+    if (entry.file.length === 0 && 'readme' in entry) {
+      // Suite-level readme.
+      assert(subtreeL0.description === undefined);
+      subtreeL0.description = entry.readme.trim();
+      continue;
+    }
+
+    {
+      const queryL1 = new TestQueryMultiFile(suite, entry.file);
+      const orderingL1 = compareQueries(queryL1, queryToLoad);
+      if (orderingL1 === Ordering.Unordered) {
+        // File path is not matched by this query.
+        continue;
+      }
+    }
+
+    if ('readme' in entry) {
+      // Entry is a README that is an ancestor or descendant of the query.
+      // (It's included for display in the standalone runner.)
+
+      // readmeSubtree is suite:a,b,*
+      // (This is always going to dedup with a file path, if there are any test spec files under
+      // the directory that has the README).
+      const readmeSubtree: TestSubtree<TestQueryMultiFile> = addSubtreeForDirPath(
+        subtreeL0,
+        entry.file
+      );
+      assert(readmeSubtree.description === undefined);
+      readmeSubtree.description = entry.readme.trim();
+      continue;
+    }
+    // Entry is a spec file.
+
+    const spec = await loader.importSpecFile(queryToLoad.suite, entry.file);
+    const description = spec.description.trim();
+    // subtreeL1 is suite:a,b:*
+    const subtreeL1: TestSubtree<TestQueryMultiTest> = addSubtreeForFilePath(
+      subtreeL0,
+      entry.file,
+      description,
+      isCollapsible
+    );
+
+    // TODO: If tree generation gets too slow, avoid actually iterating the cases in a file
+    // if there's no need to (based on the subqueriesToExpand).
+    for (const t of spec.g.iterate()) {
+      {
+        const queryL3 = new TestQuerySingleCase(suite, entry.file, t.id.test, t.id.params);
+        const orderingL3 = compareQueries(queryL3, queryToLoad);
+        if (orderingL3 === Ordering.Unordered || orderingL3 === Ordering.StrictSuperset) {
+          // Case is not matched by this query.
+          continue;
+        }
+      }
+
+      // subtreeL2 is suite:a,b:c,d:*
+      const subtreeL2: TestSubtree<TestQueryMultiCase> = addSubtreeForTestPath(
+        subtreeL1,
+        t.id.test,
+        isCollapsible
+      );
+
+      // Leaf for case is suite:a,b:c,d:x=1;y=2
+      addLeafForCase(subtreeL2, t, isCollapsible);
+
+      foundCase = true;
+    }
+  }
+  const tree = new TestTree(subtreeL0);
+
+  for (const [i, sq] of subqueriesToExpandEntries) {
+    const seen = seenSubqueriesToExpand[i];
+    assert(
+      seen,
+      `subqueriesToExpand entry did not match anything \
+(can happen due to overlap with another subquery): ${sq.toString()}`
+    );
+  }
+  assert(foundCase, 'Query does not match any cases');
+
+  // TODO: Contains lots of single-child subtrees. Consider cleaning those up (as postprocess?).
+  return tree;
+}
+
+function makeTreeForSuite(suite: string): TestSubtree<TestQueryMultiFile> {
+  return {
+    query: new TestQueryMultiFile(suite, []),
+    children: new Map(),
+    collapsible: false,
+  };
+}
+
+function addSubtreeForDirPath(
+  tree: TestSubtree<TestQueryMultiFile>,
+  file: string[]
+): TestSubtree<TestQueryMultiFile> {
+  const subqueryFile: string[] = [];
+  // To start, tree is suite:*
+  // This loop goes from that -> suite:a,* -> suite:a,b,*
+  for (const part of file) {
+    subqueryFile.push(part);
+    tree = getOrInsertSubtree(part, tree, () => {
+      const query = new TestQueryMultiFile(tree.query.suite, subqueryFile);
+      return { query, collapsible: false };
+    });
   }
   return tree;
 }
+
+function addSubtreeForFilePath(
+  tree: TestSubtree<TestQueryMultiFile>,
+  file: string[],
+  description: string,
+  checkCollapsible: (sq: TestQuery) => boolean
+): TestSubtree<TestQueryMultiTest> {
+  // To start, tree is suite:*
+  // This goes from that -> suite:a,* -> suite:a,b,*
+  tree = addSubtreeForDirPath(tree, file);
+  // This goes from that -> suite:a,b:*
+  const subtree = getOrInsertSubtree('', tree, () => {
+    const query = new TestQueryMultiTest(tree.query.suite, tree.query.filePathParts, []);
+    return { query, description, collapsible: checkCollapsible(query) };
+  });
+  return subtree;
+}
+
+function addSubtreeForTestPath(
+  tree: TestSubtree<TestQueryMultiTest>,
+  test: readonly string[],
+  isCollapsible: (sq: TestQuery) => boolean
+): TestSubtree<TestQueryMultiCase> {
+  const subqueryTest: string[] = [];
+  // To start, tree is suite:a,b:*
+  // This loop goes from that -> suite:a,b:c,* -> suite:a,b:c,d,*
+  for (const part of test) {
+    subqueryTest.push(part);
+    tree = getOrInsertSubtree(part, tree, () => {
+      const query = new TestQueryMultiTest(
+        tree.query.suite,
+        tree.query.filePathParts,
+        subqueryTest
+      );
+      return { query, collapsible: isCollapsible(query) };
+    });
+  }
+  // This goes from that -> suite:a,b:c,d:*
+  return getOrInsertSubtree('', tree, () => {
+    const query = new TestQueryMultiCase(
+      tree.query.suite,
+      tree.query.filePathParts,
+      subqueryTest,
+      {}
+    );
+    return { query, collapsible: isCollapsible(query) };
+  });
+}
+
+function addLeafForCase(
+  tree: TestSubtree<TestQueryMultiTest>,
+  t: RunCase,
+  checkCollapsible: (sq: TestQuery) => boolean
+): void {
+  const query = tree.query;
+  let name: string = '';
+  const subqueryParams: CaseParamsRW = {};
+
+  // To start, tree is suite:a,b:c,d:*
+  // This loop goes from that -> suite:a,b:c,d:x=1;* -> suite:a,b:c,d:x=1;y=2;*
+  for (const [k, v] of Object.entries(t.id.params)) {
+    name = stringifySingleParam(k, v);
+    subqueryParams[k] = v;
+
+    tree = getOrInsertSubtree(name, tree, () => {
+      const subquery = new TestQueryMultiCase(
+        query.suite,
+        query.filePathParts,
+        query.testPathParts,
+        subqueryParams
+      );
+      return { query: subquery, collapsible: checkCollapsible(subquery) };
+    });
+  }
+
+  // This goes from that -> suite:a,b:c,d:x=1;y=2
+  const subquery = new TestQuerySingleCase(
+    query.suite,
+    query.filePathParts,
+    query.testPathParts,
+    subqueryParams
+  );
+  checkCollapsible(subquery); // mark seenSubqueriesToExpand
+  insertLeaf(tree, subquery, t);
+}
+
+function getOrInsertSubtree<T extends TestQuery>(
+  key: string,
+  parent: TestSubtree,
+  createSubtree: () => Omit<TestSubtree<T>, 'children'>
+): TestSubtree<T> {
+  let v: TestSubtree<T>;
+  const child = parent.children.get(key);
+  if (child !== undefined) {
+    assert('children' in child); // Make sure cached subtree is not actually a leaf
+    v = child as TestSubtree<T>;
+  } else {
+    v = { ...createSubtree(), children: new Map() };
+    parent.children.set(key, v);
+  }
+  return v;
+}
+
+function insertLeaf(parent: TestSubtree, query: TestQuerySingleCase, t: RunCase) {
+  const key = '';
+  const leaf: TestTreeLeaf = {
+    query,
+    run: (rec: TestCaseRecorder) => t.run(rec),
+  };
+  assert(!parent.children.has(key));
+  parent.children.set(key, leaf);
+}

src/common/framework/url_query.ts

@@ -1,42 +0,0 @@
-import { TestCaseID, TestSpecID } from './id.js';
-import { ParamArgument, stringifyPublicParams } from './params_utils.js';
-import { unreachable } from './util/util.js';
-
-export function encodeSelectively(s: string): string {
-  let ret = encodeURIComponent(s);
-  ret = ret.replace(/%22/g, '"');
-  ret = ret.replace(/%2C/g, ',');
-  ret = ret.replace(/%2F/g, '/');
-  ret = ret.replace(/%3A/g, ':');
-  ret = ret.replace(/%3D/g, '=');
-  ret = ret.replace(/%5B/g, '[');
-  ret = ret.replace(/%5D/g, ']');
-  ret = ret.replace(/%7B/g, '{');
-  ret = ret.replace(/%7D/g, '}');
-  return ret;
-}
-
-export function checkPublicParamType(v: ParamArgument): void {
-  if (typeof v === 'number' || typeof v === 'string' || typeof v === 'boolean' || v === undefined) {
-    return;
-  }
-  if (v instanceof Array) {
-    for (const x of v) {
-      if (typeof x !== 'number') {
-        break;
-      }
-    }
-    return;
-  }
-  unreachable('Invalid type for test case params ' + v);
-}
-
-export function makeQueryString(spec: TestSpecID, testcase?: TestCaseID): string {
-  let s = spec.suite + ':';
-  s += spec.path + ':';
-  if (testcase !== undefined) {
-    s += testcase.test + '=';
-    s += stringifyPublicParams(testcase.params);
-  }
-  return encodeSelectively(s);
-}

src/common/framework/util/collect_garbage.ts

@@ -1,4 +1,4 @@
-import { resolveOnTimeout } from './util/util.js';
+import { resolveOnTimeout } from './util.js';
 
 /* eslint-disable-next-line @typescript-eslint/no-explicit-any */
 declare const Components: any;

src/common/framework/util/stack.ts

@@ -1,33 +1,17 @@
-// Takes a stack trace, and extracts only the first continuous range of lines
+// Returns the stack trace of an Error, but without the extra boilerplate at the bottom
-// containing '/(webgpu|unittests)/', which should provide only the useful part
+// (e.g. RunCaseSpecific, processTicksAndRejections, etc.), for logging.
-// of the stack to the caller (for logging).
+export function extractImportantStackTrace(e: Error): string {
-export function getStackTrace(e: Error): string {
   if (!e.stack) {
     return '';
   }
-
+  const lines = e.stack.split('\n');
-  const parts = e.stack.split('\n');
+  for (let i = lines.length - 1; i >= 0; --i) {
-
+    const line = lines[i];
-  const stack = [];
+    if (line.indexOf('.spec.') !== -1) {
-  const moreStack = [];
+      return lines.slice(0, i + 1).join('\n');
-  let found = false;
-  const commonRegex = /[/\\](webgpu|unittests)[/\\]/;
-  for (let i = 0; i < parts.length; ++i) {
-    const part = parts[i].trim();
-    const isSuites = commonRegex.test(part); // approximate
-    if (found && !isSuites) {
-      moreStack.push(part);
-    }
-    if (isSuites) {
-      if (moreStack.length) {
-        stack.push(...moreStack);
-        moreStack.length = 0;
-      }
-      stack.push(part);
-      found = true;
     }
   }
-  return stack.join('\n');
+  return e.stack;
 }
 
 // *** Examples ***

src/common/runtime/cmdline.ts

@@ -4,10 +4,9 @@
 import * as fs from 'fs';
 import * as process from 'process';
 
-import { TestSpecID } from '../framework/id.js';
+import { DefaultTestFileLoader } from '../framework/file_loader.js';
-import { TestLoader } from '../framework/loader.js';
+import { Logger } from '../framework/logging/logger.js';
-import { LiveTestCaseResult, Logger } from '../framework/logger.js';
+import { LiveTestCaseResult } from '../framework/logging/result.js';
-import { makeQueryString } from '../framework/url_query.js';
 import { assert, unreachable } from '../framework/util/util.js';
 
 function usage(rc: number): never {
@@ -29,7 +28,7 @@ if (!fs.existsSync('src/common/runtime/cmdline.ts')) {
 let verbose = false;
 let debug = false;
 let printJSON = false;
-const filterArgs = [];
+const filterArgs: string[] = [];
 for (const a of process.argv.slice(2)) {
   if (a.startsWith('-')) {
     if (a === '--verbose') {
@@ -52,44 +51,42 @@ if (filterArgs.length === 0) {
 
 (async () => {
   try {
-    const loader = new TestLoader();
+    const loader = new DefaultTestFileLoader();
-    const files = await loader.loadTestsFromCmdLine(filterArgs);
+    assert(filterArgs.length === 1, 'currently, there must be exactly one query on the cmd line');
+    const testcases = await loader.loadTests(filterArgs[0]);
 
-    const log = new Logger();
+    const log = new Logger(debug);
 
-    const failed: Array<[TestSpecID, LiveTestCaseResult]> = [];
+    const failed: Array<[string, LiveTestCaseResult]> = [];
-    const warned: Array<[TestSpecID, LiveTestCaseResult]> = [];
+    const warned: Array<[string, LiveTestCaseResult]> = [];
-    const skipped: Array<[TestSpecID, LiveTestCaseResult]> = [];
+    const skipped: Array<[string, LiveTestCaseResult]> = [];
 
     let total = 0;
-    for (const f of files) {
+
-      if (!('g' in f.spec)) {
+    for (const testcase of testcases) {
-        continue;
+      const name = testcase.query.toString();
+      const [rec, res] = log.record(name);
+      await testcase.run(rec);
+
+      if (verbose) {
+        printResults([[name, res]]);
       }
 
-      const [rec] = log.record(f.id);
+      total++;
-      for (const t of f.spec.g.iterate(rec)) {
+      switch (res.status) {
-        const res = await t.run(debug);
+        case 'pass':
-        if (verbose) {
+          break;
-          printResults([[f.id, res]]);
+        case 'fail':
-        }
+          failed.push([name, res]);
-
+          break;
-        total++;
+        case 'warn':
-        switch (res.status) {
+          warned.push([name, res]);
-          case 'pass':
+          break;
-            break;
+        case 'skip':
-          case 'fail':
+          skipped.push([name, res]);
-            failed.push([f.id, res]);
+          break;
-            break;
+        default:
-          case 'warn':
+          unreachable('unrecognized status');
-            warned.push([f.id, res]);
-            break;
-          case 'skip':
-            skipped.push([f.id, res]);
-            break;
-          default:
-            unreachable('unrecognized status');
-        }
       }
     }
 
@@ -138,9 +135,9 @@ Failed               = ${rpt(failed.length)}`);
   }
 })();
 
-function printResults(results: Array<[TestSpecID, LiveTestCaseResult]>): void {
+function printResults(results: Array<[string, LiveTestCaseResult]>): void {
-  for (const [id, r] of results) {
+  for (const [name, r] of results) {
-    console.log(`[${r.status}] ${makeQueryString(id, r)} (${r.timems}ms). Log:`);
+    console.log(`[${r.status}] ${name} (${r.timems}ms). Log:`);
     if (r.logs) {
       for (const l of r.logs) {
         console.log('  - ' + l.toJSON().replace(/\n/g, '\n    '));

src/common/runtime/helper/test_worker-worker.ts

@@ -1,28 +1,24 @@
-import { TestLoader } from '../../framework/loader.js';
+import { DefaultTestFileLoader } from '../../framework/file_loader.js';
-import { Logger } from '../../framework/logger.js';
+import { Logger } from '../../framework/logging/logger.js';
 import { assert } from '../../framework/util/util.js';
 
 /* eslint-disable-next-line @typescript-eslint/no-explicit-any */
 declare const self: any; // should be DedicatedWorkerGlobalScope
 
-const log = new Logger();
+const loader = new DefaultTestFileLoader();
-const loader = new TestLoader();
 
 self.onmessage = async (ev: MessageEvent) => {
-  const { query, debug } = ev.data;
+  const query: string = ev.data.query;
+  const debug: boolean = ev.data.debug;
 
-  const files = Array.from(await loader.loadTests([query]));
+  const log = new Logger(debug);
-  assert(files.length === 1, 'worker query resulted in != 1 files');
 
-  const f = files[0];
+  const testcases = Array.from(await loader.loadTests(query));
-  const [rec] = log.record(f.id);
+  assert(testcases.length === 1, 'worker query resulted in != 1 cases');
-  assert('g' in f.spec, 'worker query resulted in README');
 
-  const cases = Array.from(f.spec.g.iterate(rec));
+  const testcase = testcases[0];
-  assert(cases.length === 1, 'worker query resulted in != 1 cases');
+  const [rec, result] = log.record(testcase.query.toString());
-  const c = cases[0];
+  await testcase.run(rec);
-
-  const result = await c.run(debug);
 
   self.postMessage({ query, result });
 };

src/common/runtime/helper/test_worker.ts

@@ -1,14 +1,15 @@
-import {
+import { LogMessageWithStack } from '../../framework/logging/log_message.js';
-  LiveTestCaseResult,
+import { TransferredTestCaseResult, LiveTestCaseResult } from '../../framework/logging/result.js';
-  LogMessageWithStack,
+import { TestCaseRecorder } from '../../framework/logging/test_case_recorder.js';
-  TransferredTestCaseResult,
-} from '../../framework/logger.js';
 
 export class TestWorker {
-  private worker: Worker;
+  private readonly debug: boolean;
-  private resolvers = new Map<string, (result: LiveTestCaseResult) => void>();
+  private readonly worker: Worker;
+  private readonly resolvers = new Map<string, (result: LiveTestCaseResult) => void>();
+
+  constructor(debug: boolean) {
+    this.debug = debug;
 
-  constructor() {
     const selfPath = import.meta.url;
     const selfPathDir = selfPath.substring(0, selfPath.lastIndexOf('/'));
     const workerPath = selfPathDir + '/test_worker-worker.js';
@@ -28,10 +29,11 @@ export class TestWorker {
     };
   }
 
-  run(query: string, debug: boolean = false): Promise<LiveTestCaseResult> {
+  async run(rec: TestCaseRecorder, query: string): Promise<void> {
-    this.worker.postMessage({ query, debug });
+    this.worker.postMessage({ query, debug: this.debug });
-    return new Promise(resolve => {
+    const workerResult = await new Promise<LiveTestCaseResult>(resolve => {
       this.resolvers.set(query, resolve);
     });
+    rec.injectResult(workerResult);
   }
 }

src/common/runtime/standalone.ts

@@ -1,10 +1,10 @@
 // Implements the standalone test runner (see also: /standalone/index.html).
 
-import { TestLoader } from '../framework/loader.js';
+import { DefaultTestFileLoader } from '../framework/file_loader.js';
-import { LiveTestCaseResult, Logger } from '../framework/logger.js';
+import { Logger } from '../framework/logging/logger.js';
-import { RunCase } from '../framework/test_group.js';
+import { TestQuery } from '../framework/query/query.js';
-import { FilterResultTreeNode, treeFromFilterResults } from '../framework/tree.js';
+import { TestTreeNode, TestSubtree, TestTreeLeaf } from '../framework/tree.js';
-import { encodeSelectively } from '../framework/url_query.js';
+import { assert } from '../framework/util/util.js';
 
 import { optionEnabled } from './helper/options.js';
 import { TestWorker } from './helper/test_worker.js';
@@ -15,12 +15,13 @@ window.onbeforeunload = () => {
 };
 
 let haveSomeResults = false;
-const log = new Logger();
 
 const runnow = optionEnabled('runnow');
 const debug = optionEnabled('debug');
 
-const worker = optionEnabled('worker') ? new TestWorker() : undefined;
+const logger = new Logger(debug);
+
+const worker = optionEnabled('worker') ? new TestWorker(debug) : undefined;
 
 const resultsVis = document.getElementById('resultsVis')!;
 const resultsJSON = document.getElementById('resultsJSON')!;
@@ -29,25 +30,25 @@ type RunSubtree = () => Promise<void>;
 
 // DOM generation
 
-function makeTreeNodeHTML(name: string, tree: FilterResultTreeNode): [HTMLElement, RunSubtree] {
+function makeTreeNodeHTML(tree: TestTreeNode): [HTMLElement, RunSubtree] {
-  if (tree.children) {
+  if ('children' in tree) {
-    return makeSubtreeHTML(name, tree);
+    return makeSubtreeHTML(tree);
   } else {
-    return makeCaseHTML(name, tree.runCase!);
+    return makeCaseHTML(tree);
   }
 }
 
-function makeCaseHTML(name: string, t: RunCase): [HTMLElement, RunSubtree] {
+function makeCaseHTML(t: TestTreeLeaf): [HTMLElement, RunSubtree] {
   const div = $('<div>').addClass('testcase');
 
+  const name = t.query.toString();
   const runSubtree = async () => {
     haveSomeResults = true;
-    let res: LiveTestCaseResult;
+    const [rec, res] = logger.record(name);
     if (worker) {
-      res = await worker.run(name, debug);
+      await worker.run(rec, name);
-      t.injectResult(res);
     } else {
-      res = await t.run(debug);
+      await t.run(rec);
     }
 
     casetime.text(res.timems.toFixed(4) + ' ms');
@@ -72,7 +73,7 @@ function makeCaseHTML(name: string, t: RunCase): [HTMLElement, RunSubtree] {
     }
   };
 
-  const casehead = makeTreeNodeHeaderHTML(name, undefined, runSubtree);
+  const casehead = makeTreeNodeHeaderHTML(t.query, undefined, runSubtree, true);
   div.append(casehead);
   const casetime = $('<div>').addClass('testcasetime').html('ms').appendTo(casehead);
   const caselogs = $('<div>').addClass('testcaselogs').appendTo(div);
@@ -80,27 +81,22 @@ function makeCaseHTML(name: string, t: RunCase): [HTMLElement, RunSubtree] {
   return [div[0], runSubtree];
 }
 
-function makeSubtreeHTML(name: string, subtree: FilterResultTreeNode): [HTMLElement, RunSubtree] {
+function makeSubtreeHTML(t: TestSubtree): [HTMLElement, RunSubtree] {
   const div = $('<div>').addClass('subtree');
 
-  const header = makeTreeNodeHeaderHTML(name, subtree.description, () => {
+  const header = makeTreeNodeHeaderHTML(t.query, t.description, () => runSubtree(), false);
-    return runSubtree();
-  });
   div.append(header);
 
   const subtreeHTML = $('<div>').addClass('subtreechildren').appendTo(div);
-  const runSubtree = makeSubtreeChildrenHTML(subtreeHTML[0], subtree.children!);
+  const runSubtree = makeSubtreeChildrenHTML(subtreeHTML[0], t.children.values());
 
   return [div[0], runSubtree];
 }
 
-function makeSubtreeChildrenHTML(
+function makeSubtreeChildrenHTML(div: HTMLElement, children: Iterable<TestTreeNode>): RunSubtree {
-  div: HTMLElement,
-  children: Map<string, FilterResultTreeNode>
-): RunSubtree {
   const runSubtreeFns: RunSubtree[] = [];
-  for (const [name, subtree] of children) {
+  for (const subtree of children) {
-    const [subtreeHTML, runSubtree] = makeTreeNodeHTML(name, subtree);
+    const [subtreeHTML, runSubtree] = makeTreeNodeHTML(subtree);
     div.append(subtreeHTML);
     runSubtreeFns.push(runSubtree);
   }
@@ -113,24 +109,16 @@ function makeSubtreeChildrenHTML(
 }
 
 function makeTreeNodeHeaderHTML(
-  name: string,
+  query: TestQuery,
   description: string | undefined,
-  runSubtree: RunSubtree
+  runSubtree: RunSubtree,
+  isLeaf: boolean
 ): HTMLElement {
   const div = $('<div>').addClass('nodeheader');
 
-  const nameEncoded = encodeSelectively(name);
+  const href = `?${worker ? 'worker&' : ''}${debug ? 'debug&' : ''}q=${query.toString()}`;
-  let nameHTML;
-  {
-    const i = nameEncoded.indexOf('{');
-    const n1 = i === -1 ? nameEncoded : nameEncoded.slice(0, i + 1);
-    const n2 = i === -1 ? '' : nameEncoded.slice(i + 1);
-    nameHTML = n1.replace(/:/g, ':<wbr>') + '<wbr>' + n2.replace(/,/g, ',<wbr>');
-  }
-
-  const href = `?${worker ? 'worker&' : ''}${debug ? 'debug&' : ''}q=${nameEncoded}`;
   $('<button>')
-    .addClass('noderun')
+    .addClass(isLeaf ? 'leafrun' : 'subtreerun')
     .attr('alt', 'Run subtree')
     .attr('title', 'Run subtree')
     .on('click', async () => {
@@ -145,7 +133,7 @@ function makeTreeNodeHeaderHTML(
     .attr('title', 'Open')
     .appendTo(div);
   const nodetitle = $('<div>').addClass('nodetitle').appendTo(div);
-  $('<span>').addClass('nodename').html(nameHTML).appendTo(nodetitle);
+  $('<span>').addClass('nodename').html(query.toHTML()).appendTo(nodetitle);
   if (description) {
     $('<div>')
       .addClass('nodedescription')
@@ -156,18 +144,38 @@ function makeTreeNodeHeaderHTML(
 }
 
 function updateJSON(): void {
-  resultsJSON.textContent = log.asJSON(2);
+  resultsJSON.textContent = logger.asJSON(2);
 }
 
 (async () => {
-  const loader = new TestLoader();
+  const loader = new DefaultTestFileLoader();
 
-  // TODO: everything after this point is very similar across the three runtimes.
   // TODO: start populating page before waiting for everything to load?
-  const files = await loader.loadTestsFromQuery(window.location.search);
+  const qs = new URLSearchParams(window.location.search).getAll('q');
-  const tree = treeFromFilterResults(log, files);
+  if (qs.length === 0) {
+    qs.push('webgpu:*');
+  }
 
-  const runSubtree = makeSubtreeChildrenHTML(resultsVis, tree.children!);
+  // Update the URL bar to match the exact current options.
+  {
+    let url = window.location.protocol + '//' + window.location.host + window.location.pathname;
+    url +=
+      '?' +
+      new URLSearchParams([
+        ['runnow', runnow ? '1' : '0'],
+        ['worker', worker ? '1' : '0'],
+        ['debug', debug ? '1' : '0'],
+      ]).toString() +
+      '&' +
+      qs.map(q => 'q=' + q).join('&');
+    window.history.replaceState(null, '', url);
+  }
+
+  assert(qs.length === 1, 'currently, there must be exactly one ?q=');
+  const tree = await loader.loadTree(qs[0]);
+
+  const [el, runSubtree] = makeSubtreeHTML(tree.root);
+  resultsVis.append(el);
 
   if (runnow) {
     runSubtree();

src/common/runtime/wpt.ts

@@ -1,7 +1,8 @@
-import { TestLoader } from '../framework/loader.js';
+import { DefaultTestFileLoader } from '../framework/file_loader.js';
-import { LiveTestCaseResult, Logger } from '../framework/logger.js';
+import { Logger } from '../framework/logging/logger.js';
-import { makeQueryString } from '../framework/url_query.js';
+import { TestTreeLeaf } from '../framework/tree.js';
 import { AsyncMutex } from '../framework/util/async_mutex.js';
+import { assert } from '../framework/util/util.js';
 
 import { optionEnabled } from './helper/options.js';
 import { TestWorker } from './helper/test_worker.js';
@@ -15,51 +16,51 @@ declare interface WptTestObject {
 declare function async_test(f: (this: WptTestObject) => Promise<void>, name: string): void;
 
 (async () => {
-  const loader = new TestLoader();
+  const loader = new DefaultTestFileLoader();
-  const files = await loader.loadTestsFromQuery(window.location.search);
+  const qs = new URLSearchParams(window.location.search).getAll('q');
+  assert(qs.length === 1, 'currently, there must be exactly one ?q=');
+  const testcases = await loader.loadTests(qs[0]);
 
-  const worker = optionEnabled('worker') ? new TestWorker() : undefined;
+  const log = await addWPTTests(testcases);
 
-  const log = new Logger();
-  const mutex = new AsyncMutex();
-  const running: Array<Promise<void>> = [];
-
-  for (const f of files) {
-    if (!('g' in f.spec)) {
-      continue;
-    }
-
-    const [rec] = log.record(f.id);
-    for (const t of f.spec.g.iterate(rec)) {
-      const name = makeQueryString(f.id, t.id);
-
-      // Note: apparently, async_tests must ALL be added within the same task.
-      async_test(function (this: WptTestObject): Promise<void> {
-        const p = mutex.with(async () => {
-          let r: LiveTestCaseResult;
-          if (worker) {
-            r = await worker.run(name);
-            t.injectResult(r);
-          } else {
-            r = await t.run();
-          }
-
-          this.step(() => {
-            // Unfortunately, it seems not possible to surface any logs for warn/skip.
-            if (r.status === 'fail') {
-              throw (r.logs || []).map(s => s.toJSON()).join('\n\n');
-            }
-          });
-          this.done();
-        });
-
-        running.push(p);
-        return p;
-      }, name);
-    }
-  }
-
-  await Promise.all(running);
   const resultsElem = document.getElementById('results') as HTMLElement;
   resultsElem.textContent = log.asJSON(2);
 })();
+
+// Note: async_tests must ALL be added within the same task. This function *must not* be async.
+function addWPTTests(testcases: IterableIterator<TestTreeLeaf>): Promise<Logger> {
+  const worker = optionEnabled('worker') ? new TestWorker(false) : undefined;
+
+  const log = new Logger(false);
+  const mutex = new AsyncMutex();
+  const running: Array<Promise<void>> = [];
+
+  for (const testcase of testcases) {
+    const name = testcase.query.toString();
+    const wpt_fn = function (this: WptTestObject): Promise<void> {
+      const p = mutex.with(async () => {
+        const [rec, res] = log.record(name);
+        if (worker) {
+          await worker.run(rec, name);
+        } else {
+          await testcase.run(rec);
+        }
+
+        this.step(() => {
+          // Unfortunately, it seems not possible to surface any logs for warn/skip.
+          if (res.status === 'fail') {
+            throw (res.logs || []).map(s => s.toJSON()).join('\n\n');
+          }
+        });
+        this.done();
+      });
+
+      running.push(p);
+      return p;
+    };
+
+    async_test(wpt_fn, name);
+  }
+
+  return Promise.all(running).then(() => log);
+}

src/common/tools/crawl.ts

@@ -5,54 +5,53 @@
 import * as fs from 'fs';
 import * as path from 'path';
 
-import { TestSuiteListingEntry } from '../framework/listing.js';
+import { SpecFile } from '../framework/file_loader.js';
-import { TestSpec } from '../framework/loader.js';
+import { TestSuiteListingEntry, TestSuiteListing } from '../framework/test_suite_listing.js';
-import { assert } from '../framework/util/util.js';
+import { assert, unreachable } from '../framework/util/util.js';
 
 const fg = require('fast-glob');
 
-const specSuffix = '.spec.ts';
+const specFileSuffix = '.spec.ts';
 
 export async function crawl(suite: string): Promise<TestSuiteListingEntry[]> {
-  const specDir = 'src/' + suite;
+  const suiteDir = 'src/' + suite;
-  if (!fs.existsSync(specDir)) {
+  if (!fs.existsSync(suiteDir)) {
-    console.error(`Could not find ${specDir}`);
+    console.error(`Could not find ${suiteDir}`);
     process.exit(1);
   }
 
-  const specFiles = await fg(`${specDir}/**/{README.txt,*${specSuffix}}`, { onlyFiles: true });
+  const glob = `${suiteDir}/**/{README.txt,*${specFileSuffix}}`;
-  specFiles.sort();
+  const filesToEnumerate: string[] = await fg(glob, { onlyFiles: true });
+  filesToEnumerate.sort();
 
-  const groups: TestSuiteListingEntry[] = [];
+  const entries: TestSuiteListingEntry[] = [];
-  for (const file of specFiles) {
+  for (const file of filesToEnumerate) {
-    const f = file.substring((specDir + '/').length);
+    const f = file.substring((suiteDir + '/').length); // Suite-relative file path
-    if (f.endsWith(specSuffix)) {
+    if (f.endsWith(specFileSuffix)) {
-      const testPath = f.substring(0, f.length - specSuffix.length);
+      const filepathWithoutExtension = f.substring(0, f.length - specFileSuffix.length);
-      const filename = `../../../${specDir}/${testPath}.spec.js`;
+      const filename = `../../../${suiteDir}/${filepathWithoutExtension}.spec.js`;
-      const mod = (await import(filename)) as TestSpec;
+
+      const mod = (await import(filename)) as SpecFile;
       assert(mod.description !== undefined, 'Test spec file missing description: ' + filename);
       assert(mod.g !== undefined, 'Test spec file missing TestGroup definition: ' + filename);
-      groups.push({
+
-        path: testPath,
+      const path = filepathWithoutExtension.split('/');
-        description: mod.description.trim(),
+      entries.push({ file: path, description: mod.description.trim() });
-      });
     } else if (path.basename(file) === 'README.txt') {
-      const group = f.substring(0, f.length - 'README.txt'.length);
+      const filepathWithoutExtension = f.substring(0, f.length - '/README.txt'.length);
-      const description = fs.readFileSync(file, 'utf8').trim();
+      const readme = fs.readFileSync(file, 'utf8').trim();
-      groups.push({
+
-        path: group,
+      const path = filepathWithoutExtension ? filepathWithoutExtension.split('/') : [];
-        description,
+      entries.push({ file: path, readme });
-      });
     } else {
-      console.error('Unrecognized file: ' + file);
+      unreachable(`glob ${glob} matched an unrecognized filename ${file}`);
-      process.exit(1);
     }
   }
 
-  return groups;
+  return entries;
 }
 
-export function makeListing(filename: string): Promise<TestSuiteListingEntry[]> {
+export function makeListing(filename: string): Promise<TestSuiteListing> {
   const suite = path.basename(path.dirname(filename));
   return crawl(suite);
 }

src/common/tools/gen_wpt_cts_html.ts

@@ -1,9 +1,10 @@
 import { promises as fs } from 'fs';
 
 import { listing } from '../../webgpu/listing.js';
-import { generateMinimalQueryList } from '../framework/generate_minimal_query_list.js';
+import { DefaultTestFileLoader } from '../framework/file_loader.js';
-import { TestSuiteListingEntry } from '../framework/listing.js';
+import { TestQueryMultiTest } from '../framework/query/query.js';
-import { TestLoader } from '../framework/loader.js';
+import { kBigSeparator, kWildcard } from '../framework/query/separators.js';
+import { TestSuiteListingEntry } from '../framework/test_suite_listing.js';
 
 function printUsageAndExit(rc: number): void {
   console.error(`\
@@ -49,8 +50,8 @@ const [
     const entries = (await listing) as TestSuiteListingEntry[];
     const lines = entries
       // Exclude READMEs.
-      .filter(l => l.path.length !== 0 && !l.path.endsWith('/'))
+      .filter(l => !('readme' in l))
-      .map(l => '?q=webgpu:' + l.path);
+      .map(l => '?q=' + new TestQueryMultiTest('webgpu', l.file, []).toString());
     await generateFile(lines);
   } else {
     // Prefixes sorted from longest to shortest
@@ -79,14 +80,17 @@ const [
       throw new Error('All input lines must start with one of the prefixes. ' + exp);
     }
 
-    const loader = new TestLoader();
+    const loader = new DefaultTestFileLoader();
-    const files = Array.from(await loader.loadTestsFromCmdLine([suite + ':']));
+    const lines: Array<string | undefined> = [];
-
-    const lines = [];
     for (const prefix of argsPrefixes) {
-      lines.push(undefined);
+      const tree = await loader.loadTree(
-      for (const q of await generateMinimalQueryList(files, expectations.get(prefix)!)) {
+        suite + kBigSeparator + kWildcard,
-        lines.push(prefix + q);
+        expectations.get(prefix)!
+      );
+
+      lines.push(undefined); // output blank line between prefixes
+      for (const q of tree.iterateCollapsedQueries()) {
+        lines.push(prefix + q.toString());
       }
     }
     await generateFile(lines);

src/unittests/async_mutex.spec.ts

@@ -2,13 +2,13 @@ export const description = `
 Tests for AsyncMutex.
 `;
 
-import { TestGroup } from '../common/framework/test_group.js';
+import { makeTestGroup } from '../common/framework/test_group.js';
 import { AsyncMutex } from '../common/framework/util/async_mutex.js';
 import { objectEquals } from '../common/framework/util/util.js';
 
 import { UnitTest } from './unit_test.js';
 
-export const g = new TestGroup(UnitTest);
+export const g = makeTestGroup(UnitTest);
 
 /* eslint-disable-next-line  @typescript-eslint/no-unused-vars */
 g.test('basic').fn(async t => {
@@ -34,7 +34,7 @@ g.test('parallel').fn(async t => {
   ]);
 });
 
-g.test('parallel/many').fn(async t => {
+g.test('parallel,many').fn(async t => {
   const mutex = new AsyncMutex();
   const actual: number[] = [];
   const expected = [];
@@ -54,7 +54,7 @@ g.test('return').fn(async t => {
   t.expect(ret === 123);
 });
 
-g.test('return/parallel').fn(async t => {
+g.test('return,parallel').fn(async t => {
   const mutex = new AsyncMutex();
   const ret = await Promise.all([
     mutex.with(async () => 1),

src/unittests/basic.spec.ts

@@ -2,31 +2,31 @@ export const description = `
 Basic unit tests for test framework.
 `;
 
-import { TestGroup } from '../common/framework/test_group.js';
+import { makeTestGroup } from '../common/framework/test_group.js';
 
 import { UnitTest } from './unit_test.js';
 
-export const g = new TestGroup(UnitTest);
+export const g = makeTestGroup(UnitTest);
 
 /* eslint-disable-next-line  @typescript-eslint/no-unused-vars */
-g.test('test/sync').fn(t => {});
+g.test('test,sync').fn(t => {});
 
 /* eslint-disable-next-line  @typescript-eslint/no-unused-vars */
-g.test('test/async').fn(async t => {});
+g.test('test,async').fn(async t => {});
 
-g.test('testp/sync')
+g.test('test_with_params,sync')
   .params([{}])
   .fn(t => {
     t.debug(JSON.stringify(t.params));
   });
 
-g.test('testp/async')
+g.test('test_with_params,async')
   .params([{}])
   .fn(async t => {
     t.debug(JSON.stringify(t.params));
   });
 
-g.test('testp/private')
+g.test('test_with_params,private_params')
   .params([
     { a: 1, b: 2, _result: 3 }, //
     { a: 4, b: -3, _result: 1 },

src/unittests/getStackTrace.spec.ts

@@ -2,18 +2,18 @@ export const description = `
 Tests for getStackTrace.
 `;
 
-import { TestGroup } from '../common/framework/test_group.js';
+import { makeTestGroup } from '../common/framework/test_group.js';
-import { getStackTrace } from '../common/framework/util/stack.js';
+import { extractImportantStackTrace } from '../common/framework/util/stack.js';
 
 import { UnitTest } from './unit_test.js';
 
-export const g = new TestGroup(UnitTest);
+export const g = makeTestGroup(UnitTest);
 
 g.test('stacks')
   .params([
     {
       case: 'node_fail',
-      _expectedLines: 1,
+      _expectedLines: 3,
       _stack: `Error:
    at CaseRecorder.fail (/Users/kainino/src/cts/src/common/framework/logger.ts:99:30)
    at RunCaseSpecific.exports.g.test.t [as fn] (/Users/kainino/src/cts/src/unittests/logger.spec.ts:80:7)
@@ -23,7 +23,7 @@ g.test('stacks')
     {
       // TODO: make sure this test case actually matches what happens on windows
       case: 'node_fail_backslash',
-      _expectedLines: 1,
+      _expectedLines: 3,
       _stack: `Error:
    at CaseRecorder.fail (C:\\Users\\kainino\\src\\cts\\src\\common\\framework\\logger.ts:99:30)
    at RunCaseSpecific.exports.g.test.t [as fn] (C:\\Users\\kainino\\src\\cts\\src\\unittests\\logger.spec.ts:80:7)
@@ -32,7 +32,7 @@ g.test('stacks')
     },
     {
       case: 'node_fail_processTicksAndRejections',
-      _expectedLines: 3,
+      _expectedLines: 5,
       _stack: `Error: expectation had no effect: suite1:foo:
     at Object.generateMinimalQueryList (/Users/kainino/src/cts/src/common/framework/generate_minimal_query_list.ts:72:24)
     at testGenerateMinimalQueryList (/Users/kainino/src/cts/src/unittests/loading.spec.ts:289:25)
@@ -45,7 +45,7 @@ g.test('stacks')
     },
     {
       case: 'node_throw',
-      _expectedLines: 1,
+      _expectedLines: 2,
       _stack: `Error: hello
     at RunCaseSpecific.g.test.t [as fn] (/Users/kainino/src/cts/src/unittests/test_group.spec.ts:51:11)
     at RunCaseSpecific.run (/Users/kainino/src/cts/src/common/framework/test_group.ts:121:18)
@@ -53,7 +53,7 @@ g.test('stacks')
     },
     {
       case: 'firefox_fail',
-      _expectedLines: 1,
+      _expectedLines: 3,
       _stack: `fail@http://localhost:8080/out/common/framework/logger.js:104:30
 expect@http://localhost:8080/out/common/framework/default_fixture.js:59:16
 @http://localhost:8080/out/unittests/util.spec.js:35:5
@@ -67,7 +67,7 @@ run@http://localhost:8080/out/common/framework/test_group.js:119:18`,
     },
     {
       case: 'safari_fail',
-      _expectedLines: 1,
+      _expectedLines: 3,
       _stack: `fail@http://localhost:8080/out/common/framework/logger.js:104:39
 expect@http://localhost:8080/out/common/framework/default_fixture.js:59:20
 http://localhost:8080/out/unittests/util.spec.js:35:11
@@ -87,7 +87,7 @@ promiseReactionJob@[native code]`,
     },
     {
       case: 'chrome_fail',
-      _expectedLines: 1,
+      _expectedLines: 4,
       _stack: `Error
     at CaseRecorder.fail (http://localhost:8080/out/common/framework/logger.js:104:30)
     at DefaultFixture.expect (http://localhost:8080/out/common/framework/default_fixture.js:59:16)
@@ -98,7 +98,7 @@ promiseReactionJob@[native code]`,
     },
     {
       case: 'chrome_throw',
-      _expectedLines: 5,
+      _expectedLines: 6,
       _stack: `Error: hello
     at RunCaseSpecific.fn (http://localhost:8080/out/unittests/test_group.spec.js:48:11)
     at RunCaseSpecific.run (http://localhost:8080/out/common/framework/test_group.js:119:18)"
@@ -111,7 +111,7 @@ promiseReactionJob@[native code]`,
     },
     {
       case: 'multiple_lines',
-      _expectedLines: 7,
+      _expectedLines: 8,
       _stack: `Error: hello
     at RunCaseSpecific.fn (http://localhost:8080/out/unittests/test_group.spec.js:48:11)
     at RunCaseSpecific.fn (http://localhost:8080/out/unittests/test_group.spec.js:48:11)
@@ -126,17 +126,13 @@ promiseReactionJob@[native code]`,
     },
   ])
   .fn(t => {
-    const lines: number = t.params._expectedLines;
-
     const ex = new Error();
     ex.stack = t.params._stack;
     t.expect(ex.stack === t.params._stack);
-    const stringified = getStackTrace(ex);
+    const stringified = extractImportantStackTrace(ex);
     const parts = stringified.split('\n');
 
-    t.expect(parts.length === lines);
+    t.expect(parts.length === t.params._expectedLines);
-    const fst = parts[0];
+    const last = parts[parts.length - 1];
-    const lst = parts[parts.length - 1];
+    t.expect(last.indexOf('/unittests/') !== -1 || last.indexOf('\\unittests\\') !== -1);
-    t.expect(fst.indexOf('/unittests/') !== -1 || fst.indexOf('\\unittests\\') !== -1);
-    t.expect(lst.indexOf('/unittests/') !== -1 || lst.indexOf('\\unittests\\') !== -1);
   });

src/unittests/listing.ts

@@ -1,4 +1,4 @@
-import { TestSuiteListing } from '../common/framework/listing.js';
+import { TestSuiteListing } from '../common/framework/test_suite_listing.js';
 import { makeListing } from '../common/tools/crawl.js';
 
 export const listing: Promise<TestSuiteListing> = makeListing(__filename);

src/unittests/loaders_and_trees.spec.ts

@@ -0,0 +1,356 @@
+export const description = `
+Tests for queries/filtering, loading, and running.
+`;
+
+import { TestFileLoader, SpecFile } from '../common/framework/file_loader.js';
+import { Logger } from '../common/framework/logging/logger.js';
+import { Status } from '../common/framework/logging/result.js';
+import {
+  TestQuery,
+  TestQuerySingleCase,
+  TestQueryMultiCase,
+  TestQueryMultiTest,
+  TestQueryMultiFile,
+} from '../common/framework/query/query.js';
+import { makeTestGroup, makeTestGroupForUnitTesting } from '../common/framework/test_group.js';
+import { TestSuiteListing, TestSuiteListingEntry } from '../common/framework/test_suite_listing.js';
+import { TestTreeLeaf } from '../common/framework/tree.js';
+import { assert, objectEquals } from '../common/framework/util/util.js';
+
+import { UnitTest } from './unit_test.js';
+
+const listingData: { [k: string]: TestSuiteListingEntry[] } = {
+  suite1: [
+    { file: [], readme: 'desc 1a' },
+    { file: ['foo'], description: 'desc 1b' },
+    { file: ['bar'], readme: 'desc 1c' },
+    { file: ['bar', 'buzz', 'buzz'], description: 'desc 1d' },
+    { file: ['baz'], description: 'desc 1e' },
+  ],
+  suite2: [
+    { file: [], readme: 'desc 2a' },
+    { file: ['foof'], description: 'desc 2b' },
+  ],
+};
+
+const specsData: { [k: string]: SpecFile } = {
+  'suite1/foo.spec.js': {
+    description: 'desc 1b',
+    g: (() => {
+      const g = makeTestGroupForUnitTesting(UnitTest);
+      g.test('hello').fn(() => {});
+      g.test('bonjour').fn(() => {});
+      g.test('hola').fn(() => {});
+      return g;
+    })(),
+  },
+  'suite1/bar/biz.spec.js': {
+    description: 'desc 1f',
+    g: makeTestGroupForUnitTesting(UnitTest),
+  },
+  'suite1/bar/bez.spec.js': {
+    description: 'desc 1g',
+    g: makeTestGroupForUnitTesting(UnitTest),
+  },
+  'suite1/bar/buzz/buzz.spec.js': {
+    description: 'desc 1d',
+    g: (() => {
+      const g = makeTestGroupForUnitTesting(UnitTest);
+      g.test('zap').fn(() => {});
+      return g;
+    })(),
+  },
+  'suite1/baz.spec.js': {
+    description: 'desc 1e',
+    g: (() => {
+      const g = makeTestGroupForUnitTesting(UnitTest);
+      g.test('wye')
+        .params([{}, { x: 1 }])
+        .fn(() => {});
+      g.test('zed')
+        .params([
+          { a: 1, b: 2, _c: 0 },
+          { b: 3, a: 1, _c: 0 },
+        ])
+        .fn(() => {});
+      return g;
+    })(),
+  },
+  'suite2/foof.spec.js': {
+    description: 'desc 2b',
+    g: (() => {
+      const g = makeTestGroupForUnitTesting(UnitTest);
+      g.test('blah').fn(t => {
+        t.debug('OK');
+      });
+      g.test('bleh')
+        .params([{ a: 1 }])
+        .fn(t => {
+          t.debug('OK');
+          t.debug('OK');
+        });
+      g.test('bluh,a').fn(t => {
+        t.fail('bye');
+      });
+      return g;
+    })(),
+  },
+};
+
+class FakeTestFileLoader extends TestFileLoader {
+  async listing(suite: string): Promise<TestSuiteListing> {
+    return listingData[suite];
+  }
+
+  async import(path: string): Promise<SpecFile> {
+    assert(path in specsData, '[test] mock file ' + path + ' does not exist');
+    return specsData[path];
+  }
+}
+
+class LoadingTest extends UnitTest {
+  static readonly loader = new FakeTestFileLoader();
+
+  async load(filter: string): Promise<TestTreeLeaf[]> {
+    return Array.from(await LoadingTest.loader.loadTests(filter));
+  }
+
+  async loadNames(filter: string): Promise<string[]> {
+    return (await this.load(filter)).map(c => c.query.toString());
+  }
+}
+
+export const g = makeTestGroup(LoadingTest);
+
+g.test('suite').fn(async t => {
+  t.shouldReject('Error', t.load('suite1'));
+  t.shouldReject('Error', t.load('suite1:'));
+});
+
+g.test('group').fn(async t => {
+  t.expect((await t.load('suite1:*')).length === 8);
+  t.expect((await t.load('suite1:foo,*')).length === 3); // x:foo,* matches x:foo:
+  t.expect((await t.load('suite1:bar,*')).length === 1);
+  t.expect((await t.load('suite1:bar,buzz,buzz,*')).length === 1);
+
+  t.shouldReject('Error', t.load('suite1:f*'));
+
+  {
+    const s = new TestQueryMultiFile('suite1', ['bar', 'buzz']).toString();
+    t.expect((await t.load(s)).length === 1);
+  }
+});
+
+g.test('test').fn(async t => {
+  t.shouldReject('Error', t.load('suite1::'));
+  t.shouldReject('Error', t.load('suite1:bar:'));
+  t.shouldReject('Error', t.load('suite1:bar,:'));
+
+  t.shouldReject('Error', t.load('suite1::*'));
+  t.shouldReject('Error', t.load('suite1:bar,:*'));
+  t.shouldReject('Error', t.load('suite1:bar:*'));
+
+  t.expect((await t.load('suite1:foo:*')).length === 3);
+  t.expect((await t.load('suite1:bar,buzz,buzz:*')).length === 1);
+  t.expect((await t.load('suite1:baz:*')).length === 4);
+
+  t.expect((await t.load('suite2:foof:bluh,*')).length === 1);
+  t.expect((await t.load('suite2:foof:bluh,a,*')).length === 1);
+
+  {
+    const s = new TestQueryMultiTest('suite2', ['foof'], ['bluh']).toString();
+    t.expect((await t.load(s)).length === 1);
+  }
+});
+
+g.test('case').fn(async t => {
+  t.shouldReject('Error', t.load('suite1:foo::'));
+  t.shouldReject('Error', t.load('suite1:bar:zed,:'));
+
+  t.shouldReject('Error', t.load('suite1:foo:h*'));
+
+  t.shouldReject('Error', t.load('suite1:foo::*'));
+  t.shouldReject('Error', t.load('suite1:baz::*'));
+  t.shouldReject('Error', t.load('suite1:baz:zed,:*'));
+
+  t.shouldReject('Error', t.load('suite1:baz:zed:'));
+  t.shouldReject('Error', t.load('suite1:baz:zed:a=1;b=2*'));
+  t.shouldReject('Error', t.load('suite1:baz:zed:a=1;b=2;'));
+  t.shouldReject('SyntaxError', t.load('suite1:baz:zed:a=1;b=2,')); // tries to parse '2,' as JSON
+  t.shouldReject('Error', t.load('suite1:baz:zed:a=1,b=2')); // '=' not allowed in value '1,b=2'
+  t.shouldReject('Error', t.load('suite1:baz:zed:b=2*'));
+  t.shouldReject('Error', t.load('suite1:baz:zed:b=2;a=1;_c=0'));
+  t.shouldReject('Error', t.load('suite1:baz:zed:a=1,*'));
+
+  t.expect((await t.load('suite1:baz:zed:*')).length === 2);
+  t.expect((await t.load('suite1:baz:zed:a=1;*')).length === 2);
+  t.expect((await t.load('suite1:baz:zed:a=1;b=2')).length === 1);
+  t.expect((await t.load('suite1:baz:zed:a=1;b=2;*')).length === 1);
+  t.expect((await t.load('suite1:baz:zed:b=2;*')).length === 1);
+  t.expect((await t.load('suite1:baz:zed:b=2;a=1')).length === 1);
+  t.expect((await t.load('suite1:baz:zed:b=2;a=1;*')).length === 1);
+  t.expect((await t.load('suite1:baz:zed:b=3;a=1')).length === 1);
+  t.expect((await t.load('suite1:baz:zed:a=1;b=3')).length === 1);
+  t.expect((await t.load('suite1:foo:hello:')).length === 1);
+
+  {
+    const s = new TestQueryMultiCase('suite1', ['baz'], ['zed'], { a: 1, b: 2 }).toString();
+    t.expect((await t.load(s)).length === 1);
+  }
+  {
+    const s = new TestQuerySingleCase('suite1', ['baz'], ['zed'], { a: 1, b: 2 }).toString();
+    t.expect((await t.load(s)).length === 1);
+  }
+});
+
+g.test('end2end').fn(async t => {
+  const l = await t.load('suite2:foof:*');
+  assert(l.length === 3, 'listing length');
+
+  const log = new Logger(true);
+
+  const exp = async (
+    i: number,
+    query: TestQuery,
+    status: Status,
+    logs: (s: string[]) => boolean
+  ) => {
+    t.expect(objectEquals(l[i].query, query));
+    const name = l[i].query.toString();
+    const [rec, res] = log.record(name);
+    await l[i].run(rec);
+
+    t.expect(log.results.get(name) === res);
+    t.expect(res.status === status);
+    t.expect(res.timems > 0);
+    assert(res.logs !== undefined); // only undefined while pending
+    t.expect(logs(res.logs.map(l => JSON.stringify(l))));
+  };
+
+  await exp(0, new TestQuerySingleCase('suite2', ['foof'], ['blah'], {}), 'pass', logs =>
+    objectEquals(logs, ['"DEBUG: OK"'])
+  );
+  await exp(1, new TestQuerySingleCase('suite2', ['foof'], ['bleh'], { a: 1 }), 'pass', logs =>
+    objectEquals(logs, ['"DEBUG: OK"', '"DEBUG: OK"'])
+  );
+  await exp(
+    2,
+    new TestQuerySingleCase('suite2', ['foof'], ['bluh', 'a'], {}),
+    'fail',
+    logs =>
+      logs.length === 1 &&
+      logs[0].startsWith('"FAIL: Error: bye\\n') &&
+      logs[0].indexOf('loaders_and_trees.spec.') !== -1
+  );
+});
+
+async function testIterateCollapsed(
+  t: LoadingTest,
+  expectations: string[],
+  expectedResult: 'throws' | string[]
+) {
+  const treePromise = LoadingTest.loader.loadTree('suite1:*', expectations);
+  if (expectedResult === 'throws') {
+    t.shouldReject('Error', treePromise, 'loadTree should have thrown Error');
+    return;
+  }
+  const tree = await treePromise;
+  const actual = Array.from(tree.iterateCollapsedQueries(), q => q.toString());
+  if (!objectEquals(actual, expectedResult)) {
+    t.fail(
+      `iterateCollapsed failed:
+  got [${actual.join(', ')}]
+  exp [${expectedResult.join(', ')}]
+${tree.toString()}`
+    );
+  }
+}
+
+g.test('print').fn(async () => {
+  const tree = await LoadingTest.loader.loadTree('suite1:*');
+  tree.toString();
+});
+
+g.test('iterateCollapsed').fn(async t => {
+  // No effect
+  await testIterateCollapsed(t, [], ['suite1:foo:*', 'suite1:bar,buzz,buzz:*', 'suite1:baz:*']);
+  await testIterateCollapsed(
+    t,
+    ['suite1:*'],
+    ['suite1:foo:*', 'suite1:bar,buzz,buzz:*', 'suite1:baz:*']
+  );
+  await testIterateCollapsed(
+    t,
+    ['suite1:foo:*'],
+    ['suite1:foo:*', 'suite1:bar,buzz,buzz:*', 'suite1:baz:*']
+  );
+  await testIterateCollapsed(
+    t,
+    ['suite1:bar,buzz,buzz:*'],
+    ['suite1:foo:*', 'suite1:bar,buzz,buzz:*', 'suite1:baz:*']
+  );
+
+  // Some effect
+  await testIterateCollapsed(
+    t,
+    ['suite1:baz:wye:*'],
+    ['suite1:foo:*', 'suite1:bar,buzz,buzz:*', 'suite1:baz:wye:*', 'suite1:baz:zed,*']
+  );
+  await testIterateCollapsed(
+    t,
+    ['suite1:baz:zed:*'],
+    ['suite1:foo:*', 'suite1:bar,buzz,buzz:*', 'suite1:baz:wye,*', 'suite1:baz:zed:*']
+  );
+  await testIterateCollapsed(
+    t,
+    ['suite1:baz:wye:*', 'suite1:baz:zed:*'],
+    ['suite1:foo:*', 'suite1:bar,buzz,buzz:*', 'suite1:baz:wye:*', 'suite1:baz:zed:*']
+  );
+  await testIterateCollapsed(
+    t,
+    ['suite1:baz:wye:'],
+    [
+      'suite1:foo:*',
+      'suite1:bar,buzz,buzz:*',
+      'suite1:baz:wye:',
+      'suite1:baz:wye:x=1;*',
+      'suite1:baz:zed,*',
+    ]
+  );
+  await testIterateCollapsed(
+    t,
+    ['suite1:baz:wye:x=1'],
+    [
+      'suite1:foo:*',
+      'suite1:bar,buzz,buzz:*',
+      'suite1:baz:wye:',
+      'suite1:baz:wye:x=1',
+      'suite1:baz:zed,*',
+    ]
+  );
+  await testIterateCollapsed(
+    t,
+    ['suite1:foo:*', 'suite1:baz:wye:'],
+    [
+      'suite1:foo:*',
+      'suite1:bar,buzz,buzz:*',
+      'suite1:baz:wye:',
+      'suite1:baz:wye:x=1;*',
+      'suite1:baz:zed,*',
+    ]
+  );
+
+  // Invalid expectation queries
+  await testIterateCollapsed(t, ['garbage'], 'throws');
+  await testIterateCollapsed(t, ['garbage*'], 'throws');
+  await testIterateCollapsed(t, ['suite1*'], 'throws');
+  await testIterateCollapsed(t, ['suite1:foo*'], 'throws');
+  await testIterateCollapsed(t, ['suite1:foo:ba*'], 'throws');
+
+  // Valid expectation queries but they don't match anything
+  await testIterateCollapsed(t, ['garbage:*'], 'throws');
+  await testIterateCollapsed(t, ['suite1:doesntexist:*'], 'throws');
+  await testIterateCollapsed(t, ['suite2:foo:*'], 'throws');
+  // Doesn't match anything because we collapse this unnecessary node into just 'suite1:foo:*'
+  await testIterateCollapsed(t, ['suite1:foo,*'], 'throws');
+});

src/unittests/loading.spec.ts

@@ -1,354 +0,0 @@
-export const description = `
-Tests for queries/filtering, loading, and running.
-`;
-
-import { generateMinimalQueryList } from '../common/framework/generate_minimal_query_list.js';
-import { TestSuiteListing, TestSuiteListingEntry } from '../common/framework/listing.js';
-import { TestFileLoader, TestLoader, TestSpecOrReadme } from '../common/framework/loader.js';
-import { Logger } from '../common/framework/logger.js';
-import { paramsEquals } from '../common/framework/params_utils.js';
-import { TestFilterResult } from '../common/framework/test_filter/test_filter_result.js';
-import { RunCase, TestGroup } from '../common/framework/test_group.js';
-import { makeQueryString } from '../common/framework/url_query.js';
-import { assert, objectEquals } from '../common/framework/util/util.js';
-
-import { UnitTest } from './unit_test.js';
-
-const listingData: { [k: string]: TestSuiteListingEntry[] } = {
-  suite1: [
-    { path: '', description: 'desc 1a' },
-    { path: 'foo', description: 'desc 1b' },
-    { path: 'bar/', description: 'desc 1c' },
-    { path: 'bar/buzz', description: 'desc 1d' },
-    { path: 'baz', description: 'desc 1e' },
-  ],
-  suite2: [
-    { path: '', description: 'desc 2a' },
-    { path: 'foof', description: 'desc 2b' },
-  ],
-};
-
-const specsData: { [k: string]: TestSpecOrReadme } = {
-  'suite1/README.txt': { description: 'desc 1a' },
-  'suite1/foo.spec.js': {
-    description: 'desc 1b',
-    g: (() => {
-      const g = new TestGroup(UnitTest);
-      g.test('hello').fn(() => {});
-      g.test('bonjour').fn(() => {});
-      g.test('hola').fn(() => {});
-      return g;
-    })(),
-  },
-  'suite1/bar/README.txt': { description: 'desc 1c' },
-  'suite1/bar/biz.spec.js': {
-    description: 'desc 1f',
-    g: new TestGroup(UnitTest),
-  },
-  'suite1/bar/bez.spec.js': {
-    description: 'desc 1g',
-    g: new TestGroup(UnitTest),
-  },
-  'suite1/bar/buzz.spec.js': {
-    description: 'desc 1d',
-    g: (() => {
-      const g = new TestGroup(UnitTest);
-      g.test('zap').fn(() => {});
-      return g;
-    })(),
-  },
-  'suite1/baz.spec.js': {
-    description: 'desc 1e',
-    g: (() => {
-      const g = new TestGroup(UnitTest);
-      g.test('wye')
-        .params([
-          {}, //
-          { x: 1 },
-        ])
-        .fn(() => {});
-      g.test('zed')
-        .params([
-          { a: 1, b: 2, _c: 0 }, //
-          { a: 1, b: 3, _c: 0 },
-        ])
-        .fn(() => {});
-      return g;
-    })(),
-  },
-  'suite2/foof.spec.js': {
-    description: 'desc 2b',
-    g: (() => {
-      const g = new TestGroup(UnitTest);
-      g.test('blah').fn(t => {
-        t.debug('OK');
-      });
-      g.test('bleh')
-        .params([{}])
-        .fn(t => {
-          t.debug('OK');
-          t.debug('OK');
-        });
-      g.test('bluh').fn(t => {
-        t.fail('bye');
-      });
-      return g;
-    })(),
-  },
-};
-
-class FakeTestFileLoader implements TestFileLoader {
-  async listing(suite: string): Promise<TestSuiteListing> {
-    return listingData[suite];
-  }
-
-  async import(path: string): Promise<TestSpecOrReadme> {
-    assert(path in specsData, '[test] mock file ' + path + ' does not exist');
-    return specsData[path];
-  }
-}
-
-class LoadingTest extends UnitTest {
-  static readonly loader: TestLoader = new TestLoader(new FakeTestFileLoader());
-
-  async load(filters: string[]): Promise<TestFilterResult[]> {
-    return Array.from(await LoadingTest.loader.loadTestsFromCmdLine(filters));
-  }
-
-  async singleGroup(query: string): Promise<RunCase[]> {
-    const [rec] = new Logger().record({ suite: '', path: '' });
-    const a = await this.load([query]);
-    assert(a.length === 1, 'more than one group');
-    const spec = a[0].spec;
-    assert('g' in spec, 'group undefined');
-    return Array.from(spec.g.iterate(rec));
-  }
-}
-
-export const g = new TestGroup(LoadingTest);
-
-g.test('whole suite').fn(async t => {
-  t.shouldReject('Error', t.load(['suite1']));
-  t.expect((await t.load(['suite1:'])).length === 5);
-});
-
-g.test('partial suite').fn(async t => {
-  t.expect((await t.load(['suite1:f'])).length === 1);
-  t.expect((await t.load(['suite1:fo'])).length === 1);
-  t.expect((await t.load(['suite1:foo'])).length === 1);
-  t.expect((await t.load(['suite1:foof'])).length === 0);
-  t.expect((await t.load(['suite1:ba'])).length === 3);
-  t.expect((await t.load(['suite1:bar'])).length === 2);
-  t.expect((await t.load(['suite1:bar/'])).length === 2);
-  t.expect((await t.load(['suite1:bar/b'])).length === 1);
-});
-
-g.test('whole group').fn(async t => {
-  t.shouldReject('Error', t.load(['suite1::']));
-  t.shouldReject('Error', t.load(['suite1:bar:']));
-  t.shouldReject('Error', t.load(['suite1:bar/:']));
-  t.expect((await t.singleGroup('suite1:bar/buzz:')).length === 1);
-  t.expect((await t.singleGroup('suite1:baz:')).length === 4);
-
-  {
-    const foo = (await t.load(['suite1:foo:']))[0];
-    t.expect(foo.id.suite === 'suite1');
-    t.expect(foo.id.path === 'foo');
-    assert('g' in foo.spec, 'foo group');
-    const [rec] = new Logger().record({ suite: '', path: '' });
-    t.expect(Array.from(foo.spec.g.iterate(rec)).length === 3);
-  }
-});
-
-g.test('partial group').fn(async t => {
-  t.expect((await t.singleGroup('suite1:foo:h')).length === 2);
-  t.expect((await t.singleGroup('suite1:foo:he')).length === 1);
-  t.expect((await t.singleGroup('suite1:foo:hello')).length === 1);
-  t.expect((await t.singleGroup('suite1:baz:zed')).length === 2);
-});
-
-g.test('partial test/exact').fn(async t => {
-  t.expect((await t.singleGroup('suite1:foo:hello=')).length === 1);
-  t.expect((await t.singleGroup('suite1:baz:zed=')).length === 0);
-  t.expect((await t.singleGroup('suite1:baz:zed=')).length === 0);
-  t.expect((await t.singleGroup('suite1:baz:zed={}')).length === 0);
-  t.expect((await t.singleGroup('suite1:baz:zed={"a":1,"b":2}')).length === 1);
-  t.expect((await t.singleGroup('suite1:baz:zed={"b":2,"a":1}')).length === 1);
-  t.expect((await t.singleGroup('suite1:baz:zed={"a":1,"b":2,"_c":0}')).length === 0);
-});
-
-g.test('partial test/makeQueryString').fn(async t => {
-  const s = makeQueryString(
-    { suite: 'suite1', path: 'baz' },
-    { test: 'zed', params: { a: 1, b: 2 } }
-  );
-  t.expect((await t.singleGroup(s)).length === 1);
-});
-
-g.test('partial test/match').fn(async t => {
-  t.expect((await t.singleGroup('suite1:baz:zed~')).length === 2);
-  t.expect((await t.singleGroup('suite1:baz:zed~{}')).length === 2);
-  t.expect((await t.singleGroup('suite1:baz:zed~{"a":1}')).length === 2);
-  t.expect((await t.singleGroup('suite1:baz:zed~{"a":1,"b":2}')).length === 1);
-  t.expect((await t.singleGroup('suite1:baz:zed~{"b":2,"a":1}')).length === 1);
-  t.expect((await t.singleGroup('suite1:baz:zed~{"b":2}')).length === 1);
-  t.expect((await t.singleGroup('suite1:baz:zed~{"a":2}')).length === 0);
-  t.expect((await t.singleGroup('suite1:baz:zed~{"c":0}')).length === 0);
-  t.expect((await t.singleGroup('suite1:baz:zed~{"_c":0}')).length === 0);
-});
-
-g.test('end2end').fn(async t => {
-  const l = await t.load(['suite2:foof']);
-  assert(l.length === 1, 'listing length');
-
-  t.expect(l[0].id.suite === 'suite2');
-  t.expect(l[0].id.path === 'foof');
-  t.expect(l[0].spec.description === 'desc 2b');
-  assert('g' in l[0].spec);
-  t.expect(l[0].spec.g.iterate instanceof Function);
-
-  const log = new Logger();
-  const [rec, res] = log.record(l[0].id);
-  const rcs = Array.from(l[0].spec.g.iterate(rec));
-  assert(rcs.length === 3, 'iterate length');
-
-  t.expect(rcs[0].id.test === 'blah');
-  t.expect(rcs[0].id.params === null);
-
-  t.expect(rcs[1].id.test === 'bleh');
-  t.expect(paramsEquals(rcs[1].id.params, {}));
-
-  t.expect(log.results[0] === res);
-  t.expect(res.spec === 'suite2:foof:');
-  t.expect(res.cases.length === 0);
-
-  {
-    const cases = res.cases;
-    const res0 = await rcs[0].run(true);
-    assert(cases.length === 1, 'results cases length');
-    t.expect(res.cases[0] === res0);
-    t.expect(res0.test === 'blah');
-    t.expect(res0.params === null);
-    t.expect(res0.status === 'pass');
-    t.expect(res0.timems >= 0);
-    assert(res0.logs !== undefined, 'results case logs');
-    t.expect(objectEquals(JSON.stringify(res0.logs), '["DEBUG: OK"]'));
-  }
-  {
-    // Store cases off to a separate variable due to a typescript bug
-    // where it can't detect that res.cases.length might change between
-    // above and here.
-    const cases = res.cases;
-    const res1 = await rcs[1].run(true);
-    assert(cases.length === 2, 'results cases length');
-    t.expect(res.cases[1] === res1);
-    t.expect(res1.test === 'bleh');
-    t.expect(paramsEquals(res1.params, {}));
-    t.expect(res1.status === 'pass');
-    t.expect(res1.timems >= 0);
-    assert(res1.logs !== undefined, 'results case logs');
-    t.expect(objectEquals(JSON.stringify(res1.logs), '["DEBUG: OK","DEBUG: OK"]'));
-  }
-  {
-    // Store cases off to a separate variable due to a typescript bug
-    // where it can't detect that res.cases.length might change between
-    // above and here.
-    const cases = res.cases;
-    const res2 = await rcs[2].run(true);
-    assert(cases.length === 3, 'results cases length');
-    t.expect(res.cases[2] === res2);
-    t.expect(res2.test === 'bluh');
-    t.expect(res2.params === null);
-    t.expect(res2.status === 'fail');
-    t.expect(res2.timems >= 0);
-    assert(res2.logs !== undefined, 'results case logs');
-    t.expect(res2.logs.length === 1);
-    const l = res2.logs[0].toJSON();
-    t.expect(l.startsWith('FAIL: bye\n'));
-    t.expect(l.indexOf('loading.spec.') !== -1);
-  }
-});
-
-const testGenerateMinimalQueryList = async (
-  t: LoadingTest,
-  expectations: string[],
-  result: string[]
-) => {
-  const l = await t.load(['suite1:']);
-  const queries = await generateMinimalQueryList(l, expectations);
-  t.expect(objectEquals(queries, result));
-};
-
-g.test('generateMinimalQueryList/errors').fn(async t => {
-  t.shouldReject('Error', testGenerateMinimalQueryList(t, ['garbage'], []));
-  t.shouldReject('Error', testGenerateMinimalQueryList(t, ['suite1:'], []));
-  t.shouldReject('Error', testGenerateMinimalQueryList(t, ['suite1:foo'], []));
-  t.shouldReject('Error', testGenerateMinimalQueryList(t, ['suite1:foo:ba'], []));
-  t.shouldReject('Error', testGenerateMinimalQueryList(t, ['suite2:foo:'], []));
-});
-
-g.test('generateMinimalQueryList').fn(async t => {
-  await testGenerateMinimalQueryList(
-    t, //
-    [],
-    ['suite1:foo:', 'suite1:bar/buzz:', 'suite1:baz:']
-  );
-  await testGenerateMinimalQueryList(
-    t, //
-    ['suite1:foo:'],
-    ['suite1:foo:', 'suite1:bar/buzz:', 'suite1:baz:']
-  );
-  await testGenerateMinimalQueryList(
-    t, //
-    ['suite1:bar/buzz:'],
-    ['suite1:foo:', 'suite1:bar/buzz:', 'suite1:baz:']
-  );
-  await testGenerateMinimalQueryList(
-    t, //
-    ['suite1:baz:wye~'],
-    ['suite1:foo:', 'suite1:bar/buzz:', 'suite1:baz:wye~', 'suite1:baz:zed~']
-  );
-  await testGenerateMinimalQueryList(
-    t, //
-    ['suite1:baz:zed~'],
-    ['suite1:foo:', 'suite1:bar/buzz:', 'suite1:baz:wye~', 'suite1:baz:zed~']
-  );
-  await testGenerateMinimalQueryList(
-    t, //
-    ['suite1:baz:wye~', 'suite1:baz:zed~'],
-    ['suite1:foo:', 'suite1:bar/buzz:', 'suite1:baz:wye~', 'suite1:baz:zed~']
-  );
-  await testGenerateMinimalQueryList(
-    t, //
-    ['suite1:baz:wye='],
-    [
-      'suite1:foo:',
-      'suite1:bar/buzz:',
-      'suite1:baz:wye=',
-      'suite1:baz:wye={"x":1}',
-      'suite1:baz:zed~',
-    ]
-  );
-  await testGenerateMinimalQueryList(
-    t, //
-    ['suite1:baz:wye={"x":1}'],
-    [
-      'suite1:foo:',
-      'suite1:bar/buzz:',
-      'suite1:baz:wye=',
-      'suite1:baz:wye={"x":1}',
-      'suite1:baz:zed~',
-    ]
-  );
-  await testGenerateMinimalQueryList(
-    t, //
-    ['suite1:foo:', 'suite1:baz:wye='],
-    [
-      'suite1:foo:',
-      'suite1:bar/buzz:',
-      'suite1:baz:wye=',
-      'suite1:baz:wye={"x":1}',
-      'suite1:baz:zed~',
-    ]
-  );
-});

src/unittests/logger.spec.ts

@@ -5,82 +5,61 @@ Also serves as a larger test of async test functions, and of the logging system.
 `;
 
 import { SkipTestCase } from '../common/framework/fixture.js';
-import { Logger } from '../common/framework/logger.js';
+import { Logger } from '../common/framework/logging/logger.js';
-import { paramsEquals } from '../common/framework/params_utils.js';
+import { makeTestGroup } from '../common/framework/test_group.js';
-import { TestGroup } from '../common/framework/test_group.js';
+import { assert } from '../common/framework/util/util.js';
 
 import { UnitTest } from './unit_test.js';
 
-export const g = new TestGroup(UnitTest);
+export const g = makeTestGroup(UnitTest);
 
 g.test('construct').fn(t => {
-  const mylog = new Logger();
+  const mylog = new Logger(true);
-  const [testrec, testres] = mylog.record({ suite: 'a', path: 'foo/bar' });
+  const [, res1] = mylog.record('one');
-  const [, res1] = testrec.record('baz', null);
+  const [, res2] = mylog.record('two');
-  const params2 = {};
-  const [, res2] = testrec.record('qux', params2);
 
-  t.expect(testres.spec === 'a:foo/bar:');
+  t.expect(mylog.results.get('one') === res1);
-  t.expect(testres.cases.length === 2);
+  t.expect(mylog.results.get('two') === res2);
-  t.expect(testres.cases[0] === res1);
-  t.expect(testres.cases[1] === res2);
-  t.expect(res1.test === 'baz');
-  t.expect(res1.params === null);
   t.expect(res1.logs === undefined);
   t.expect(res1.status === 'running');
   t.expect(res1.timems < 0);
-  t.expect(res2.test === 'qux');
-  t.expect(paramsEquals(res2.params, params2));
   t.expect(res2.logs === undefined);
   t.expect(res2.status === 'running');
   t.expect(res2.timems < 0);
 });
 
-g.test('private params').fn(t => {
-  const mylog = new Logger();
-  const [testrec] = mylog.record({ suite: 'a', path: 'foo/bar' });
-  const [, res] = testrec.record('baz', { a: 1, _b: 2 });
-
-  t.expect(paramsEquals(res.params, { a: 1 }));
-});
-
 g.test('empty').fn(t => {
-  const mylog = new Logger();
+  const mylog = new Logger(true);
-  const [testrec] = mylog.record({ suite: '', path: '' });
+  const [rec, res] = mylog.record('one');
-  const [rec, res] = testrec.record('baz', null);
 
   rec.start();
   t.expect(res.status === 'running');
   rec.finish();
+
   t.expect(res.status === 'pass');
   t.expect(res.timems >= 0);
 });
 
 g.test('pass').fn(t => {
-  const mylog = new Logger();
+  const mylog = new Logger(true);
-  const [testrec] = mylog.record({ suite: '', path: '' });
+  const [rec, res] = mylog.record('one');
-  const [rec, res] = testrec.record('baz', null);
 
   rec.start();
   rec.debug(new Error('hello'));
   t.expect(res.status === 'running');
   rec.finish();
+
   t.expect(res.status === 'pass');
   t.expect(res.timems >= 0);
 });
 
 g.test('skip').fn(t => {
-  const mylog = new Logger();
+  const mylog = new Logger(true);
-  const [testrec] = mylog.record({ suite: '', path: '' });
+  const [rec, res] = mylog.record('one');
-  const [rec, res] = testrec.record('baz', null);
 
   rec.start();
-  t.expect(res.status === 'running');
-
   rec.skipped(new SkipTestCase());
   rec.debug(new Error('hello'));
-
-  t.expect(res.status === 'running');
   rec.finish();
 
   t.expect(res.status === 'skip');
@@ -88,17 +67,12 @@ g.test('skip').fn(t => {
 });
 
 g.test('warn').fn(t => {
-  const mylog = new Logger();
+  const mylog = new Logger(true);
-  const [testrec] = mylog.record({ suite: '', path: '' });
+  const [rec, res] = mylog.record('one');
-  const [rec, res] = testrec.record('baz', null);
 
   rec.start();
-  t.expect(res.status === 'running');
+  rec.warn(new Error('hello'));
-
-  rec.warn(new Error());
   rec.skipped(new SkipTestCase());
-
-  t.expect(res.status === 'running');
   rec.finish();
 
   t.expect(res.status === 'warn');
@@ -106,18 +80,13 @@ g.test('warn').fn(t => {
 });
 
 g.test('fail').fn(t => {
-  const mylog = new Logger();
+  const mylog = new Logger(true);
-  const [testrec] = mylog.record({ suite: '', path: '' });
+  const [rec, res] = mylog.record('one');
-  const [rec, res] = testrec.record('baz', null);
 
   rec.start();
-  t.expect(res.status === 'running');
-
   rec.fail(new Error('bye'));
   rec.warn(new Error());
   rec.skipped(new SkipTestCase());
-
-  t.expect(res.status === 'running');
   rec.finish();
 
   t.expect(res.status === 'fail');
@@ -126,20 +95,25 @@ g.test('fail').fn(t => {
 
 g.test('debug')
   .params([
-    { debug: true, _logsCount: 1 }, //
+    { debug: true, _logsCount: 5 }, //
-    { debug: false, _logsCount: 0 },
+    { debug: false, _logsCount: 3 },
   ])
   .fn(t => {
     const { debug, _logsCount } = t.params;
 
-    const mylog = new Logger();
+    const mylog = new Logger(debug);
-    const [testrec] = mylog.record({ suite: '', path: '' });
+    const [rec, res] = mylog.record('one');
-    const [rec, res] = testrec.record('baz', null);
 
-    rec.start(debug);
+    rec.start();
     rec.debug(new Error('hello'));
+    rec.fail(new Error('bye'));
+    rec.warn(new Error());
+    rec.skipped(new SkipTestCase());
+    rec.debug(new Error('foo'));
     rec.finish();
-    t.expect(res.status === 'pass');
+
+    t.expect(res.status === 'fail');
     t.expect(res.timems >= 0);
-    t.expect(res.logs!.length === _logsCount);
+    assert(res.logs !== undefined);
+    t.expect(res.logs.length === _logsCount);
   });

src/unittests/make_filter.spec.ts

@@ -1,11 +0,0 @@
-export const description = `
-Unit tests for makeFilter.
-`;
-
-import { TestGroup } from '../common/framework/test_group.js';
-
-import { UnitTest } from './unit_test.js';
-
-export const g = new TestGroup(UnitTest);
-
-// TODO: tests for makeFilter, in particular idIfSingle()

src/unittests/params_builder_and_utils.spec.ts

@@ -2,20 +2,24 @@ export const description = `
 Unit tests for parameterization helpers.
 `;
 
-import { poptions, params } from '../common/framework/params.js';
+import { poptions, params } from '../common/framework/params_builder.js';
-import { ParamSpec, ParamSpecIterable, paramsEquals } from '../common/framework/params_utils.js';
+import {
-import { TestGroup } from '../common/framework/test_group.js';
+  CaseParams,
+  CaseParamsIterable,
+  publicParamsEquals,
+} from '../common/framework/params_utils.js';
+import { makeTestGroup } from '../common/framework/test_group.js';
 
 import { UnitTest } from './unit_test.js';
 
 class ParamsTest extends UnitTest {
-  expectSpecEqual(act: ParamSpecIterable, exp: ParamSpec[]): void {
+  expectSpecEqual(act: CaseParamsIterable, exp: CaseParams[]): void {
     const a = Array.from(act);
-    this.expect(a.length === exp.length && a.every((x, i) => paramsEquals(x, exp[i])));
+    this.expect(a.length === exp.length && a.every((x, i) => publicParamsEquals(x, exp[i])));
   }
 }
 
-export const g = new TestGroup(ParamsTest);
+export const g = makeTestGroup(ParamsTest);
 
 g.test('options').fn(t => {
   t.expectSpecEqual(poptions('hello', [1, 2, 3]), [{ hello: 1 }, { hello: 2 }, { hello: 3 }]);
@@ -25,14 +29,14 @@ g.test('params').fn(t => {
   t.expectSpecEqual(params(), [{}]);
 });
 
-g.test('combine/zeroes and ones').fn(t => {
+g.test('combine,zeroes_and_ones').fn(t => {
   t.expectSpecEqual(params().combine([]).combine([]), []);
   t.expectSpecEqual(params().combine([]).combine([{}]), []);
   t.expectSpecEqual(params().combine([{}]).combine([]), []);
   t.expectSpecEqual(params().combine([{}]).combine([{}]), [{}]);
 });
 
-g.test('combine/mixed').fn(t => {
+g.test('combine,mixed').fn(t => {
   t.expectSpecEqual(
     params()
       .combine(poptions('x', [1, 2]))
@@ -110,7 +114,7 @@ g.test('expand').fn(t => {
   );
 });
 
-g.test('expand/invalid').fn(t => {
+g.test('expand,invalid').fn(t => {
   const p = params()
     .combine([{ x: 1 }])
     .expand(function* (p) {
@@ -122,8 +126,13 @@ g.test('expand/invalid').fn(t => {
 });
 
 g.test('undefined').fn(t => {
-  t.expectSpecEqual([{ a: undefined }], [{}]);
+  t.expect(!publicParamsEquals({ a: undefined }, {}));
-  t.expectSpecEqual([{}], [{ a: undefined }]);
+  t.expect(!publicParamsEquals({}, { a: undefined }));
+});
+
+g.test('private').fn(t => {
+  t.expect(publicParamsEquals({ _a: 0 }, {}));
+  t.expect(publicParamsEquals({}, { _a: 0 }));
 });
 
 g.test('arrays').fn(t => {

src/unittests/params_builder_toplevel.spec.ts

@@ -2,14 +2,14 @@ export const description = `
 Unit tests for parameterization.
 `;
 
-import { params } from '../common/framework/params.js';
+import { params } from '../common/framework/params_builder.js';
-import { ParamSpec } from '../common/framework/params_utils.js';
+import { CaseParams } from '../common/framework/params_utils.js';
-import { TestGroup } from '../common/framework/test_group.js';
+import { makeTestGroup, makeTestGroupForUnitTesting } from '../common/framework/test_group.js';
 
 import { TestGroupTest } from './test_group_test.js';
 import { UnitTest } from './unit_test.js';
 
-export const g = new TestGroup(TestGroupTest);
+export const g = makeTestGroup(TestGroupTest);
 
 g.test('none')
   .params([])
@@ -17,7 +17,7 @@ g.test('none')
     t.fail("this test shouldn't run");
   });
 
-g.test('combine none')
+g.test('combine_none')
   .params(params().combine([]))
   .fn(t => {
     t.fail("this test shouldn't run");
@@ -66,13 +66,13 @@ g.test('exclude')
   });
 
 g.test('generator').fn(t0 => {
-  const g = new TestGroup(UnitTest);
+  const g = makeTestGroupForUnitTesting(UnitTest);
 
-  const ran: ParamSpec[] = [];
+  const ran: CaseParams[] = [];
 
   g.test('generator')
     .params(
-      (function* (): IterableIterator<ParamSpec> {
+      (function* (): IterableIterator<CaseParams> {
         for (let x = 0; x < 3; ++x) {
           for (let y = 0; y < 2; ++y) {
             yield { x, y };
@@ -85,11 +85,11 @@ g.test('generator').fn(t0 => {
     });
 
   t0.expectCases(g, [
-    { test: 'generator', params: { x: 0, y: 0 } },
+    { test: ['generator'], params: { x: 0, y: 0 } },
-    { test: 'generator', params: { x: 0, y: 1 } },
+    { test: ['generator'], params: { x: 0, y: 1 } },
-    { test: 'generator', params: { x: 1, y: 0 } },
+    { test: ['generator'], params: { x: 1, y: 0 } },
-    { test: 'generator', params: { x: 1, y: 1 } },
+    { test: ['generator'], params: { x: 1, y: 1 } },
-    { test: 'generator', params: { x: 2, y: 0 } },
+    { test: ['generator'], params: { x: 2, y: 0 } },
-    { test: 'generator', params: { x: 2, y: 1 } },
+    { test: ['generator'], params: { x: 2, y: 1 } },
   ]);
 });

src/unittests/query_compare.spec.ts

@@ -0,0 +1,133 @@
+export const description = `
+Tests for TestQuery comparison
+`;
+
+import { compareQueries, Ordering } from '../common/framework/query/compare.js';
+import {
+  TestQuery,
+  TestQuerySingleCase,
+  TestQueryMultiFile,
+  TestQueryMultiTest,
+  TestQueryMultiCase,
+} from '../common/framework/query/query.js';
+import { makeTestGroup } from '../common/framework/test_group.js';
+
+import { UnitTest } from './unit_test.js';
+
+class F extends UnitTest {
+  expectQ(a: TestQuery, exp: '<' | '=' | '>' | '!', b: TestQuery) {
+    const [expOrdering, expInvOrdering] =
+      exp === '<'
+        ? [Ordering.StrictSubset, Ordering.StrictSuperset]
+        : exp === '='
+        ? [Ordering.Equal, Ordering.Equal]
+        : exp === '>'
+        ? [Ordering.StrictSuperset, Ordering.StrictSubset]
+        : [Ordering.Unordered, Ordering.Unordered];
+    {
+      const act = compareQueries(a, b);
+      this.expect(act === expOrdering, `${a} ${b}  got ${act}, exp ${expOrdering}`);
+    }
+    {
+      const act = compareQueries(a, b);
+      this.expect(act === expOrdering, `${b} ${a}  got ${act}, exp ${expInvOrdering}`);
+    }
+  }
+
+  expectWellOrdered(...qs: TestQuery[]) {
+    for (let i = 0; i < qs.length; ++i) {
+      this.expectQ(qs[i], '=', qs[i]);
+      for (let j = i + 1; j < qs.length; ++j) {
+        this.expectQ(qs[i], '>', qs[j]);
+      }
+    }
+  }
+
+  expectUnordered(...qs: TestQuery[]) {
+    for (let i = 0; i < qs.length; ++i) {
+      this.expectQ(qs[i], '=', qs[i]);
+      for (let j = i + 1; j < qs.length; ++j) {
+        this.expectQ(qs[i], '!', qs[j]);
+      }
+    }
+  }
+}
+
+export const g = makeTestGroup(F);
+
+// suite:*  >  suite:a,*  >  suite:a,b,*   >  suite:a,b:*
+// suite:a,b:*  >  suite:a,b:c,*  >  suite:a,b:c,d,*  >  suite:a,b:c,d:*
+// suite:a,b:c,d:*  >  suite:a,b:c,d:x=1;*  >  suite:a,b:c,d:x=1;y=2;*  >  suite:a,b:c,d:x=1;y=2
+// suite:a;* (unordered) suite:b;*
+g.test('well_ordered').fn(t => {
+  t.expectWellOrdered(
+    new TestQueryMultiFile('suite', []),
+    new TestQueryMultiFile('suite', ['a']),
+    new TestQueryMultiFile('suite', ['a', 'b']),
+    new TestQueryMultiTest('suite', ['a', 'b'], []),
+    new TestQueryMultiTest('suite', ['a', 'b'], ['c']),
+    new TestQueryMultiTest('suite', ['a', 'b'], ['c', 'd']),
+    new TestQueryMultiCase('suite', ['a', 'b'], ['c', 'd'], {}),
+    new TestQueryMultiCase('suite', ['a', 'b'], ['c', 'd'], { x: 1 }),
+    new TestQueryMultiCase('suite', ['a', 'b'], ['c', 'd'], { x: 1, y: 2 }),
+    new TestQuerySingleCase('suite', ['a', 'b'], ['c', 'd'], { x: 1, y: 2 })
+  );
+  t.expectWellOrdered(
+    new TestQueryMultiFile('suite', []),
+    new TestQueryMultiFile('suite', ['a']),
+    new TestQueryMultiFile('suite', ['a', 'b']),
+    new TestQueryMultiTest('suite', ['a', 'b'], []),
+    new TestQueryMultiTest('suite', ['a', 'b'], ['c']),
+    new TestQueryMultiTest('suite', ['a', 'b'], ['c', 'd']),
+    new TestQueryMultiCase('suite', ['a', 'b'], ['c', 'd'], {}),
+    new TestQuerySingleCase('suite', ['a', 'b'], ['c', 'd'], {})
+  );
+});
+
+g.test('unordered').fn(t => {
+  t.expectUnordered(
+    new TestQueryMultiFile('suite', ['a']), //
+    new TestQueryMultiFile('suite', ['x'])
+  );
+  t.expectUnordered(
+    new TestQueryMultiFile('suite', ['a', 'b']),
+    new TestQueryMultiFile('suite', ['a', 'x'])
+  );
+  t.expectUnordered(
+    new TestQueryMultiTest('suite', ['a', 'b'], ['c']),
+    new TestQueryMultiTest('suite', ['a', 'b'], ['x']),
+    new TestQueryMultiTest('suite', ['a'], []),
+    new TestQueryMultiTest('suite', ['a', 'x'], [])
+  );
+  t.expectUnordered(
+    new TestQueryMultiTest('suite', ['a', 'b'], ['c', 'd']),
+    new TestQueryMultiTest('suite', ['a', 'b'], ['c', 'x']),
+    new TestQueryMultiTest('suite', ['a'], []),
+    new TestQueryMultiTest('suite', ['a', 'x'], [])
+  );
+  t.expectUnordered(
+    new TestQueryMultiTest('suite', ['a', 'b'], ['c', 'd']),
+    new TestQueryMultiTest('suite', ['a', 'b'], ['c', 'x']),
+    new TestQueryMultiTest('suite', ['a'], []),
+    new TestQueryMultiTest('suite', ['a', 'x'], ['c'])
+  );
+  t.expectUnordered(
+    new TestQueryMultiCase('suite', ['a', 'b'], ['c', 'd'], { x: 1 }),
+    new TestQueryMultiCase('suite', ['a', 'b'], ['c', 'd'], { x: 9 }),
+    new TestQueryMultiCase('suite', ['a', 'b'], ['c'], { x: 9 })
+  );
+  t.expectUnordered(
+    new TestQueryMultiCase('suite', ['a', 'b'], ['c', 'd'], { x: 1, y: 2 }),
+    new TestQueryMultiCase('suite', ['a', 'b'], ['c', 'd'], { x: 1, y: 8 }),
+    new TestQueryMultiCase('suite', ['a', 'b'], ['c'], { x: 1, y: 8 })
+  );
+  t.expectUnordered(
+    new TestQuerySingleCase('suite', ['a', 'b'], ['c', 'd'], { x: 1, y: 2 }),
+    new TestQuerySingleCase('suite', ['a', 'b'], ['c', 'd'], { x: 1, y: 8 }),
+    new TestQuerySingleCase('suite', ['a', 'b'], ['c'], { x: 1, y: 8 })
+  );
+  t.expectUnordered(
+    new TestQuerySingleCase('suite1', ['bar', 'buzz', 'buzz'], ['zap'], {}),
+    new TestQueryMultiTest('suite1', ['bar'], [])
+  );
+});

src/unittests/test_filter.spec.ts

@@ -1,82 +0,0 @@
-export const description = `
-Unit tests for test_filter .matches() methods.
-`;
-
-import { FilterByGroup } from '../common/framework/test_filter/filter_by_group.js';
-import {
-  FilterByParamsExact,
-  FilterByParamsMatch,
-  FilterByTestMatch,
-} from '../common/framework/test_filter/filter_one_file.js';
-import { TestGroup } from '../common/framework/test_group.js';
-
-import { UnitTest } from './unit_test.js';
-
-export const g = new TestGroup(UnitTest);
-
-g.test('FilterByGroup/matches').fn(t => {
-  {
-    const f = new FilterByGroup('ab', 'de/f');
-    t.expect(f.matches({ spec: { suite: 'ab', path: 'de/fg' } }));
-    t.expect(f.matches({ spec: { suite: 'ab', path: 'de/fg' }, test: 'x' }));
-    t.expect(f.matches({ spec: { suite: 'ab', path: 'de/fg' }, test: 'x', params: null }));
-    t.expect(f.matches({ spec: { suite: 'ab', path: 'de/fg' }, test: 'x', params: {} }));
-    t.expect(f.matches({ spec: { suite: 'ab', path: 'de/fg/' }, test: 'x', params: null }));
-    t.expect(f.matches({ spec: { suite: 'ab', path: 'de/f/g' }, test: 'x', params: null }));
-    t.expect(f.matches({ spec: { suite: 'ab', path: 'de/f/g/' }, test: 'x', params: null }));
-  }
-  {
-    const f = new FilterByGroup('ab', 'de/f/');
-    t.expect(!f.matches({ spec: { suite: 'ab', path: 'de/fg' } }));
-    t.expect(!f.matches({ spec: { suite: 'ab', path: 'de/fg' }, test: 'x' }));
-    t.expect(!f.matches({ spec: { suite: 'ab', path: 'de/fg' }, test: 'x', params: null }));
-    t.expect(!f.matches({ spec: { suite: 'ab', path: 'de/fg' }, test: 'x', params: {} }));
-    t.expect(!f.matches({ spec: { suite: 'ab', path: 'de/fg/' }, test: 'x', params: null }));
-    t.expect(f.matches({ spec: { suite: 'ab', path: 'de/f/g' }, test: 'x', params: null }));
-    t.expect(f.matches({ spec: { suite: 'ab', path: 'de/f/g/' }, test: 'x', params: null }));
-  }
-});
-
-g.test('FilterByTestMatch/matches').fn(t => {
-  const f = new FilterByTestMatch({ suite: 'ab', path: 'de/f' }, 'xy');
-  t.expect(f.matches({ spec: { suite: 'ab', path: 'de/f' }, test: 'xy' }));
-  t.expect(f.matches({ spec: { suite: 'ab', path: 'de/f' }, test: 'xy', params: null }));
-  t.expect(f.matches({ spec: { suite: 'ab', path: 'de/f' }, test: 'xyz', params: null }));
-  t.expect(f.matches({ spec: { suite: 'ab', path: 'de/f' }, test: 'xyz', params: {} }));
-  t.expect(f.matches({ spec: { suite: 'ab', path: 'de/f' }, test: 'xyz', params: { z: 1 } }));
-  t.expect(!f.matches({ spec: { suite: 'ab', path: 'de/f' } }));
-  t.expect(!f.matches({ spec: { suite: 'ab', path: 'de/f/' }, test: 'xyz', params: null }));
-  t.expect(!f.matches({ spec: { suite: 'ab', path: 'de/fg' }, test: 'xyz', params: null }));
-});
-
-g.test('FilterByParamsMatch/matches').fn(t => {
-  {
-    const f = new FilterByParamsMatch({ suite: 'ab', path: 'de/f' }, 'xy', null);
-    t.expect(f.matches({ spec: { suite: 'ab', path: 'de/f' }, test: 'xy', params: null }));
-    t.expect(f.matches({ spec: { suite: 'ab', path: 'de/f' }, test: 'xy', params: {} }));
-    t.expect(f.matches({ spec: { suite: 'ab', path: 'de/f' }, test: 'xy', params: { z: 1 } }));
-    t.expect(!f.matches({ spec: { suite: 'ab', path: 'de/f' } }));
-    t.expect(!f.matches({ spec: { suite: 'ab', path: 'de/f' }, test: 'xy' }));
-    t.expect(!f.matches({ spec: { suite: 'ab', path: 'de/f' }, test: 'xy', params: undefined }));
-    t.expect(!f.matches({ spec: { suite: 'ab', path: 'de/f' }, test: 'xyz', params: {} }));
-    t.expect(!f.matches({ spec: { suite: 'ab', path: 'de/f/' }, test: 'xy', params: {} }));
-  }
-  {
-    const f = new FilterByParamsMatch({ suite: 'ab', path: 'de/f' }, 'xy', {});
-    t.expect(f.matches({ spec: { suite: 'ab', path: 'de/f' }, test: 'xy', params: null }));
-    t.expect(f.matches({ spec: { suite: 'ab', path: 'de/f' }, test: 'xy', params: {} }));
-    t.expect(!f.matches({ spec: { suite: 'ab', path: 'de/f' } }));
-    t.expect(!f.matches({ spec: { suite: 'ab', path: 'de/f' }, test: 'xy' }));
-  }
-});
-
-g.test('FilterByParamsExact').fn(t => {
-  const f = new FilterByParamsExact({ suite: 'ab', path: 'de/f' }, 'xy', {});
-  t.expect(f.matches({ spec: { suite: 'ab', path: 'de/f' }, test: 'xy', params: null }));
-  t.expect(f.matches({ spec: { suite: 'ab', path: 'de/f' }, test: 'xy', params: {} }));
-  t.expect(!f.matches({ spec: { suite: 'ab', path: 'de/f' }, test: 'xy', params: { z: 1 } }));
-  t.expect(!f.matches({ spec: { suite: 'ab', path: 'de/f' }, test: 'xyz', params: {} }));
-  t.expect(!f.matches({ spec: { suite: 'ab', path: 'de/f/' }, test: 'xy', params: {} }));
-  t.expect(!f.matches({ spec: { suite: 'ab', path: 'de/f' } }));
-  t.expect(!f.matches({ spec: { suite: 'ab', path: 'de/f' }, test: 'xy' }));
-});

src/unittests/test_group.spec.ts

@@ -3,23 +3,22 @@ Unit tests for TestGroup.
 `;
 
 import { Fixture } from '../common/framework/fixture.js';
-import { poptions } from '../common/framework/params.js';
+import { makeTestGroup, makeTestGroupForUnitTesting } from '../common/framework/test_group.js';
-import { TestGroup } from '../common/framework/test_group.js';
 import { assert } from '../common/framework/util/util.js';
 
 import { TestGroupTest } from './test_group_test.js';
 import { UnitTest } from './unit_test.js';
 
-export const g = new TestGroup(TestGroupTest);
+export const g = makeTestGroup(TestGroupTest);
 
-g.test('UnitTest fixture').fn(async t0 => {
+g.test('UnitTest_fixture').fn(async t0 => {
   let seen = 0;
   /* eslint-disable-next-line  @typescript-eslint/no-unused-vars */
   function count(t: Fixture): void {
     seen++;
   }
 
-  const g = new TestGroup(UnitTest);
+  const g = makeTestGroupForUnitTesting(UnitTest);
 
   g.test('test').fn(count);
   g.test('testp')
@@ -30,7 +29,7 @@ g.test('UnitTest fixture').fn(async t0 => {
   t0.expect(seen === 2);
 });
 
-g.test('custom fixture').fn(async t0 => {
+g.test('custom_fixture').fn(async t0 => {
   let seen = 0;
   class Counter extends UnitTest {
     count(): void {
@@ -38,7 +37,7 @@ g.test('custom fixture').fn(async t0 => {
     }
   }
 
-  const g = new TestGroup(Counter);
+  const g = makeTestGroupForUnitTesting(Counter);
 
   g.test('test').fn(t => {
     t.count();
@@ -54,7 +53,7 @@ g.test('custom fixture').fn(async t0 => {
 });
 
 g.test('stack').fn(async t0 => {
-  const g = new TestGroup(UnitTest);
+  const g = makeTestGroupForUnitTesting(UnitTest);
 
   const doNestedThrow1 = () => {
     throw new Error('goodbye');
@@ -70,24 +69,23 @@ g.test('stack').fn(async t0 => {
     throw new Error('hello');
   });
   /* eslint-disable-next-line  @typescript-eslint/no-unused-vars */
-  g.test('throw nested').fn(t => {
+  g.test('throw_nested').fn(t => {
     doNestedThrow2();
   });
 
   const res = await t0.run(g);
 
-  const search = /unittests[/\\]test_group\.spec\.[tj]s|suites[/\\]unittests[/\\]unit_test\.[tj]s/;
+  const search = /unittests[/\\]test_group\.spec\.[tj]s/;
-  for (const { logs } of res.cases) {
+  t0.expect(res.size > 0);
+  for (const { logs } of res.values()) {
     assert(logs !== undefined, 'expected logs');
-    const l = logs[0].toJSON();
+    t0.expect(logs.some(l => search.test(l.toJSON())));
-    t0.expect(search.test(l));
+    t0.expect(search.test(logs[logs.length - 1].toJSON()));
-    const st = l.split('\n');
-    t0.expect(search.test(st[st.length - 1]));
   }
 });
 
-g.test('duplicate test name').fn(t => {
+g.test('duplicate_test_name').fn(t => {
-  const g = new TestGroup(UnitTest);
+  const g = makeTestGroupForUnitTesting(UnitTest);
   g.test('abc').fn(() => {});
 
   t.shouldThrow('Error', () => {
@@ -95,8 +93,8 @@ g.test('duplicate test name').fn(t => {
   });
 });
 
-g.test('duplicate test params').fn(t => {
+g.test('duplicate_test_params').fn(t => {
-  const g = new TestGroup(UnitTest);
+  const g = makeTestGroupForUnitTesting(UnitTest);
 
   t.shouldThrow('Error', () => {
     g.test('abc')
@@ -110,8 +108,8 @@ g.test('duplicate test params').fn(t => {
   });
 });
 
-g.test('duplicate test params/with different private params').fn(t => {
+g.test('duplicate_test_params,with_different_private_params').fn(t => {
-  const g = new TestGroup(UnitTest);
+  const g = makeTestGroupForUnitTesting(UnitTest);
 
   t.shouldThrow('Error', () => {
     g.test('abc')
@@ -125,19 +123,29 @@ g.test('duplicate test params/with different private params').fn(t => {
   });
 });
 
-const badChars = Array.from('"`~@#$+=\\|!^&*[]<>{}-\'.,');
+g.test('invalid_test_name').fn(t => {
-g.test('invalid test name')
+  const g = makeTestGroupForUnitTesting(UnitTest);
-  .params(poptions('char', badChars))
-  .fn(t => {
-    const g = new TestGroup(UnitTest);
 
-    t.shouldThrow('Error', () => {
+  const badChars = Array.from('"`~@#$+=\\|!^&*[]<>{}-\'. ');
-      g.test('a' + t.params.char + 'b').fn(() => {});
+  for (const char of badChars) {
-    });
+    const name = 'a' + char + 'b';
-  });
+    t.shouldThrow(
+      'Error',
+      () => {
+        g.test(name).fn(() => {});
+      },
+      name
+    );
+  }
+});
+
+g.test('valid_param_value').fn(() => {
+  const g = makeTestGroup(UnitTest);
+  g.test('a').params([{ x: JSON.stringify({ a: 1, b: 2 }) }]);
+});
 
 g.test('throws').fn(async t0 => {
-  const g = new TestGroup(UnitTest);
+  const g = makeTestGroupForUnitTesting(UnitTest);
 
   /* eslint-disable-next-line  @typescript-eslint/no-unused-vars */
   g.test('a').fn(t => {
@@ -145,7 +153,9 @@ g.test('throws').fn(async t0 => {
   });
 
   const result = await t0.run(g);
-  t0.expect(result.cases[0].status === 'fail');
+  const values = Array.from(result.values());
+  t0.expect(values.length === 1);
+  t0.expect(values[0].status === 'fail');
 });
 
 g.test('shouldThrow').fn(async t0 => {
@@ -153,7 +163,7 @@ g.test('shouldThrow').fn(async t0 => {
     throw new TypeError();
   });
 
-  const g = new TestGroup(UnitTest);
+  const g = makeTestGroupForUnitTesting(UnitTest);
 
   g.test('a').fn(t => {
     t.shouldThrow('Error', () => {
@@ -162,7 +172,9 @@ g.test('shouldThrow').fn(async t0 => {
   });
 
   const result = await t0.run(g);
-  t0.expect(result.cases[0].status === 'fail');
+  const values = Array.from(result.values());
+  t0.expect(values.length === 1);
+  t0.expect(values[0].status === 'fail');
 });
 
 g.test('shouldReject').fn(async t0 => {
@@ -173,7 +185,7 @@ g.test('shouldReject').fn(async t0 => {
     })()
   );
 
-  const g = new TestGroup(UnitTest);
+  const g = makeTestGroupForUnitTesting(UnitTest);
 
   g.test('a').fn(async t => {
     t.shouldReject(
@@ -186,5 +198,7 @@ g.test('shouldReject').fn(async t0 => {
 
   const result = await t0.run(g);
   // Fails even though shouldReject doesn't fail until after the test function ends
-  t0.expect(result.cases[0].status === 'fail');
+  const values = Array.from(result.values());
+  t0.expect(values.length === 1);
+  t0.expect(values[0].status === 'fail');
 });

src/unittests/test_group_test.ts

@@ -1,35 +1,23 @@
-import { Fixture } from '../common/framework/fixture.js';
+import { Logger, LogResults } from '../common/framework/logging/logger.js';
-import { TestCaseID } from '../common/framework/id.js';
+import { TestQuerySingleCase } from '../common/framework/query/query.js';
-import { LiveTestSpecResult, Logger } from '../common/framework/logger.js';
+import { RunCaseIterable, TestCaseID } from '../common/framework/test_group.js';
-import { paramsEquals } from '../common/framework/params_utils.js';
+import { objectEquals } from '../common/framework/util/util.js';
-import { TestGroup } from '../common/framework/test_group.js';
 
 import { UnitTest } from './unit_test.js';
 
 export class TestGroupTest extends UnitTest {
-  async run<F extends Fixture>(g: TestGroup<F>): Promise<LiveTestSpecResult> {
+  async run(g: RunCaseIterable): Promise<LogResults> {
-    const [rec, res] = new Logger().record({ suite: '', path: '' });
+    const logger = new Logger(true);
-    await Promise.all(Array.from(g.iterate(rec)).map(test => test.run()));
+    for (const rc of await Promise.all(g.iterate())) {
-    return res;
+      const query = new TestQuerySingleCase('xx', ['yy'], rc.id.test, rc.id.params);
+      const [rec] = logger.record(query.toString());
+      await rc.run(rec);
+    }
+    return logger.results;
   }
 
-  enumerate<F extends Fixture>(g: TestGroup<F>): TestCaseID[] {
+  expectCases(g: RunCaseIterable, cases: TestCaseID[]): void {
-    const cases = [];
+    const gcases = Array.from(g.iterate()).map(rc => rc.id);
-    const [rec] = new Logger().record({ suite: '', path: '' });
+    this.expect(objectEquals(gcases, cases));
-    for (const test of g.iterate(rec)) {
-      cases.push(test.id);
-    }
-    return cases;
-  }
-
-  expectCases<F extends Fixture>(g: TestGroup<F>, cases: TestCaseID[]): void {
-    const gcases = this.enumerate(g);
-
-    if (this.expect(gcases.length === cases.length)) {
-      for (let i = 0; i < cases.length; ++i) {
-        this.expect(gcases[i].test === cases[i].test);
-        this.expect(paramsEquals(gcases[i].params, cases[i].params));
-      }
-    }
   }
 }

src/unittests/test_query.spec.ts

@@ -0,0 +1,56 @@
+export const description = `
+Tests for TestQuery
+`;
+import {
+  TestQueryMultiFile,
+  TestQueryMultiTest,
+  TestQueryMultiCase,
+  TestQuerySingleCase,
+  TestQuery,
+} from '../common/framework/query/query.js';
+import { makeTestGroup } from '../common/framework/test_group.js';
+
+import { UnitTest } from './unit_test.js';
+
+class F extends UnitTest {
+  expectToString(q: TestQuery, exp: string) {
+    this.expect(q.toString() === exp);
+  }
+}
+
+export const g = makeTestGroup(F);
+
+g.test('constructor').fn(t => {
+  t.shouldThrow('Error', () => new TestQueryMultiTest('suite', [], []));
+
+  t.shouldThrow('Error', () => new TestQueryMultiCase('suite', ['a'], [], {}));
+  t.shouldThrow('Error', () => new TestQueryMultiCase('suite', [], ['c'], {}));
+  t.shouldThrow('Error', () => new TestQueryMultiCase('suite', [], [], {}));
+
+  t.shouldThrow('Error', () => new TestQuerySingleCase('suite', ['a'], [], {}));
+  t.shouldThrow('Error', () => new TestQuerySingleCase('suite', [], ['c'], {}));
+  t.shouldThrow('Error', () => new TestQuerySingleCase('suite', [], [], {}));
+});
+
+g.test('toString').fn(t => {
+  t.expectToString(new TestQueryMultiFile('s', []), 's:*');
+  t.expectToString(new TestQueryMultiFile('s', ['a']), 's:a,*');
+  t.expectToString(new TestQueryMultiFile('s', ['a', 'b']), 's:a,b,*');
+  t.expectToString(new TestQueryMultiTest('s', ['a', 'b'], []), 's:a,b:*');
+  t.expectToString(new TestQueryMultiTest('s', ['a', 'b'], ['c']), 's:a,b:c,*');
+  t.expectToString(new TestQueryMultiTest('s', ['a', 'b'], ['c', 'd']), 's:a,b:c,d,*');
+  t.expectToString(new TestQueryMultiCase('s', ['a', 'b'], ['c', 'd'], {}), 's:a,b:c,d:*');
+  t.expectToString(
+    new TestQueryMultiCase('s', ['a', 'b'], ['c', 'd'], { x: 1 }),
+    's:a,b:c,d:x=1;*'
+  );
+  t.expectToString(
+    new TestQueryMultiCase('s', ['a', 'b'], ['c', 'd'], { x: 1, y: 2 }),
+    's:a,b:c,d:x=1;y=2;*'
+  );
+  t.expectToString(
+    new TestQuerySingleCase('s', ['a', 'b'], ['c', 'd'], { x: 1, y: 2 }),
+    's:a,b:c,d:x=1;y=2'
+  );
+  t.expectToString(new TestQuerySingleCase('s', ['a', 'b'], ['c', 'd'], {}), 's:a,b:c,d:');
+});

src/unittests/url_query.spec.ts

@@ -2,27 +2,75 @@ export const description = `
 Unit tests for URL queries.
 `;
 
-import { TestGroup } from '../common/framework/test_group.js';
+import {
-import { makeQueryString } from '../common/framework/url_query.js';
+  TestQuery,
+  TestQuerySingleCase,
+  TestQueryMultiCase,
+  TestQueryMultiTest,
+  TestQueryMultiFile,
+} from '../common/framework/query/query.js';
+import { makeTestGroup } from '../common/framework/test_group.js';
 
 import { UnitTest } from './unit_test.js';
 
-export const g = new TestGroup(UnitTest);
+class T extends UnitTest {
+  expectQueryString(q: TestQuery, exp: string): void {
+    const s = q.toString();
+    this.expect(s === exp, `got ${s} expected ${exp}`);
+  }
+}
 
-g.test('makeQueryString').fn(t => {
+export const g = makeTestGroup(T);
-  const s = makeQueryString({ suite: 'a', path: 'b/c' }, { test: 'd/e', params: { f: 'g', h: 3 } });
-  t.expect(s === 'a:b/c:d/e={"f":"g","h":3}');
-});
 
-g.test('makeQueryString/private').fn(t => {
+g.test('stringifyQuery,single_case').fn(t => {
-  const s = makeQueryString({ suite: 'a', path: 'b' }, { test: 'c', params: { pub: 2, _pri: 3 } });
+  t.expectQueryString(
-  t.expect(s === 'a:b:c={"pub":2}');
+    new TestQuerySingleCase('a', ['b_1', '2_c'], ['d_3', '4_e'], {
-});
+      f: 'g',
-
+      _pri1: 0,
-g.test('makeQueryString/undefined').fn(t => {
+      a: 3,
-  const s = makeQueryString(
+      _pri2: 1,
-    { suite: 'a', path: 'b' },
+    }),
-    { test: 'c', params: { f: undefined, h: 3 } }
+    'a:b_1,2_c:d_3,4_e:f="g";a=3'
+  );
+});
+
+g.test('stringifyQuery,multi_case').fn(t => {
+  t.expectQueryString(
+    new TestQueryMultiCase('a', ['b_1', '2_c'], ['d_3', '4_e'], {
+      f: 'g',
+      _pri1: 0,
+      a: 3,
+      _pri2: 1,
+    }),
+    'a:b_1,2_c:d_3,4_e:f="g";a=3;*'
+  );
+
+  t.expectQueryString(
+    new TestQueryMultiCase('a', ['b_1', '2_c'], ['d_3', '4_e'], {}),
+    'a:b_1,2_c:d_3,4_e:*'
+  );
+});
+
+g.test('stringifyQuery,multi_test').fn(t => {
+  t.expectQueryString(
+    new TestQueryMultiTest('a', ['b_1', '2_c'], ['d_3', '4_e']),
+    'a:b_1,2_c:d_3,4_e,*'
+  );
+
+  t.expectQueryString(
+    new TestQueryMultiTest('a', ['b_1', '2_c'], []), //
+    'a:b_1,2_c:*'
+  );
+});
+
+g.test('stringifyQuery,multi_file').fn(t => {
+  t.expectQueryString(
+    new TestQueryMultiFile('a', ['b_1', '2_c']), //
+    'a:b_1,2_c,*'
+  );
+
+  t.expectQueryString(
+    new TestQueryMultiFile('a', []), //
+    'a:*'
   );
-  t.expect(s === 'a:b:c={"h":3}');
 });

src/webgpu/api/operation/buffers/create_mapped.spec.ts

@@ -1,11 +1,11 @@
 export const description = '';
 
-import { params, pbool, poptions } from '../../../../common/framework/params.js';
+import { params, pbool, poptions } from '../../../../common/framework/params_builder.js';
-import { TestGroup } from '../../../../common/framework/test_group.js';
+import { makeTestGroup } from '../../../../common/framework/test_group.js';
 
 import { MappingTest } from './mapping_test.js';
 
-export const g = new TestGroup(MappingTest);
+export const g = makeTestGroup(MappingTest);
 
 g.test('createBufferMapped')
   .params(

src/webgpu/api/operation/buffers/map.spec.ts

@@ -1,11 +1,11 @@
 export const description = '';
 
-import { pbool, poptions, params } from '../../../../common/framework/params.js';
+import { pbool, poptions, params } from '../../../../common/framework/params_builder.js';
-import { TestGroup } from '../../../../common/framework/test_group.js';
+import { makeTestGroup } from '../../../../common/framework/test_group.js';
 
 import { MappingTest } from './mapping_test.js';
 
-export const g = new TestGroup(MappingTest);
+export const g = makeTestGroup(MappingTest);
 
 g.test('mapWriteAsync')
   .params(poptions('size', [12, 512 * 1024]))

src/webgpu/api/operation/buffers/map_detach.spec.ts

@@ -1,6 +1,6 @@
 export const description = '';
 
-import { TestGroup } from '../../../../common/framework/test_group.js';
+import { makeTestGroup } from '../../../../common/framework/test_group.js';
 import { GPUTest } from '../../../gpu_test.js';
 
 class F extends GPUTest {
@@ -17,7 +17,7 @@ class F extends GPUTest {
   }
 }
 
-export const g = new TestGroup(F);
+export const g = makeTestGroup(F);
 
 g.test('mapWriteAsync')
   .params([
@@ -43,7 +43,7 @@ g.test('mapReadAsync')
     t.checkDetach(buffer, arrayBuffer, t.params.unmap, t.params.destroy);
   });
 
-g.test('create mapped')
+g.test('create_mapped')
   .params([
     { unmap: true, destroy: false },
     { unmap: false, destroy: true },

src/webgpu/api/operation/buffers/map_oom.spec.ts

@@ -1,6 +1,6 @@
 export const description = '';
 
-import { TestGroup } from '../../../../common/framework/test_group.js';
+import { makeTestGroup } from '../../../../common/framework/test_group.js';
 import { GPUTest } from '../../../gpu_test.js';
 
 function getBufferDesc(): GPUBufferDescriptor {
@@ -10,7 +10,7 @@ function getBufferDesc(): GPUBufferDescriptor {
   };
 }
 
-export const g = new TestGroup(GPUTest);
+export const g = makeTestGroup(GPUTest);
 
 g.test('mapWriteAsync').fn(async t => {
   const buffer = t.device.createBuffer(getBufferDesc());

src/webgpu/api/operation/command_buffer/basic.spec.ts

@@ -2,10 +2,10 @@ export const description = `
 Basic tests.
 `;
 
-import { TestGroup } from '../../../../common/framework/test_group.js';
+import { makeTestGroup } from '../../../../common/framework/test_group.js';
 import { GPUTest } from '../../../gpu_test.js';
 
-export const g = new TestGroup(GPUTest);
+export const g = makeTestGroup(GPUTest);
 
 g.test('empty').fn(async t => {
   const encoder = t.device.createCommandEncoder();

src/webgpu/api/operation/command_buffer/copies.spec.ts

@@ -2,10 +2,10 @@ export const description = `
 copy{Buffer,Texture}To{Buffer,Texture} tests.
 `;
 
-import { TestGroup } from '../../../../common/framework/test_group.js';
+import { makeTestGroup } from '../../../../common/framework/test_group.js';
 import { GPUTest } from '../../../gpu_test.js';
 
-export const g = new TestGroup(GPUTest);
+export const g = makeTestGroup(GPUTest);
 
 g.test('b2b').fn(async t => {
   const data = new Uint32Array([0x01020304]);

src/webgpu/api/operation/command_buffer/render/basic.spec.ts

@@ -2,10 +2,10 @@ export const description = `
 Basic command buffer rendering tests.
 `;
 
-import { TestGroup } from '../../../../../common/framework/test_group.js';
+import { makeTestGroup } from '../../../../../common/framework/test_group.js';
 import { GPUTest } from '../../../../gpu_test.js';
 
-export const g = new TestGroup(GPUTest);
+export const g = makeTestGroup(GPUTest);
 
 g.test('clear').fn(async t => {
   const dst = t.device.createBuffer({

src/webgpu/api/operation/fences.spec.ts

@@ -1,29 +1,29 @@
 export const description = '';
 
-import { attemptGarbageCollection } from '../../../common/framework/collect_garbage.js';
+import { makeTestGroup } from '../../../common/framework/test_group.js';
-import { TestGroup } from '../../../common/framework/test_group.js';
+import { attemptGarbageCollection } from '../../../common/framework/util/collect_garbage.js';
 import { raceWithRejectOnTimeout } from '../../../common/framework/util/util.js';
 import { GPUTest } from '../../gpu_test.js';
 
-export const g = new TestGroup(GPUTest);
+export const g = makeTestGroup(GPUTest);
 
-g.test('initial/no descriptor').fn(t => {
+g.test('initial,no_descriptor').fn(t => {
   const fence = t.queue.createFence();
   t.expect(fence.getCompletedValue() === 0);
 });
 
-g.test('initial/empty descriptor').fn(t => {
+g.test('initial,empty_descriptor').fn(t => {
   const fence = t.queue.createFence({});
   t.expect(fence.getCompletedValue() === 0);
 });
 
-g.test('initial/descriptor with initialValue').fn(t => {
+g.test('initial,descriptor_with_initialValue').fn(t => {
   const fence = t.queue.createFence({ initialValue: 2 });
   t.expect(fence.getCompletedValue() === 2);
 });
 
 // Promise resolves when onCompletion value is less than signal value.
-g.test('wait/less than signaled').fn(async t => {
+g.test('wait,less_than_signaled').fn(async t => {
   const fence = t.queue.createFence();
   t.queue.signal(fence, 2);
   await fence.onCompletion(1);
@@ -31,7 +31,7 @@ g.test('wait/less than signaled').fn(async t => {
 });
 
 // Promise resolves when onCompletion value is equal to signal value.
-g.test('wait/equal to signaled').fn(async t => {
+g.test('wait,equal_to_signaled').fn(async t => {
   const fence = t.queue.createFence();
   t.queue.signal(fence, 2);
   await fence.onCompletion(2);
@@ -39,7 +39,7 @@ g.test('wait/equal to signaled').fn(async t => {
 });
 
 // All promises resolve when signal is called once.
-g.test('wait/signaled once').fn(async t => {
+g.test('wait,signaled_once').fn(async t => {
   const fence = t.queue.createFence();
   t.queue.signal(fence, 20);
   const promises = [];
@@ -54,7 +54,7 @@ g.test('wait/signaled once').fn(async t => {
 });
 
 // Promise resolves when signal is called multiple times.
-g.test('wait/signaled multiple times').fn(async t => {
+g.test('wait,signaled_multiple_times').fn(async t => {
   const fence = t.queue.createFence();
   t.queue.signal(fence, 1);
   t.queue.signal(fence, 2);
@@ -63,7 +63,7 @@ g.test('wait/signaled multiple times').fn(async t => {
 });
 
 // Promise resolves if fence has already completed.
-g.test('wait/already completed').fn(async t => {
+g.test('wait,already_completed').fn(async t => {
   const fence = t.queue.createFence();
   t.queue.signal(fence, 2);
 
@@ -81,7 +81,7 @@ g.test('wait/already completed').fn(async t => {
 });
 
 // Test many calls to signal and wait on fence values one at a time.
-g.test('wait/many/serially').fn(async t => {
+g.test('wait,many,serially').fn(async t => {
   const fence = t.queue.createFence();
   for (let i = 1; i <= 20; ++i) {
     t.queue.signal(fence, i);
@@ -91,7 +91,7 @@ g.test('wait/many/serially').fn(async t => {
 });
 
 // Test many calls to signal and wait on all fence values.
-g.test('wait/many/parallel').fn(async t => {
+g.test('wait,many,parallel').fn(async t => {
   const fence = t.queue.createFence();
   const promises = [];
   for (let i = 1; i <= 20; ++i) {
@@ -107,7 +107,7 @@ g.test('wait/many/parallel').fn(async t => {
 });
 
 // Test onCompletion promise resolves within a time limit.
-g.test('wait/resolves within timeout').fn(t => {
+g.test('wait,resolves_within_timeout').fn(t => {
   const fence = t.queue.createFence();
   t.queue.signal(fence, 2);
 
@@ -122,7 +122,7 @@ g.test('wait/resolves within timeout').fn(t => {
 });
 
 // Test dropping references to the fence and onCompletion promise does not crash.
-g.test('drop/fence and promise').fn(async t => {
+g.test('drop,fence_and_promise').fn(async t => {
   {
     const fence = t.queue.createFence();
     t.queue.signal(fence, 2);
@@ -132,7 +132,7 @@ g.test('drop/fence and promise').fn(async t => {
 });
 
 // Test dropping references to the fence and holding the promise does not crash.
-g.test('drop/promise').fn(async t => {
+g.test('drop,promise').fn(async t => {
   let promise;
   {
     const fence = t.queue.createFence();

src/webgpu/api/operation/resource_init/copied_texture_clear.spec.ts

@@ -1,7 +1,7 @@
 export const description = 'Test uninitialized textures are initialized to zero when copied.';
 
 import * as C from '../../../../common/constants.js';
-import { TestGroup } from '../../../../common/framework/test_group.js';
+import { makeTestGroup } from '../../../../common/framework/test_group.js';
 import { assert, unreachable } from '../../../../common/framework/util/util.js';
 import { SubresourceRange } from '../../../util/texture/subresource.js';
 
@@ -78,9 +78,9 @@ class CopiedTextureClearTest extends TextureZeroInitTest {
   }
 }
 
-export const g = new TestGroup(CopiedTextureClearTest);
+export const g = makeTestGroup(CopiedTextureClearTest);
 
-g.test('uninitialized texture is zero')
+g.test('uninitialized_texture_is_zero')
   .params(TextureZeroInitTest.generateParams([ReadMethod.CopyToBuffer, ReadMethod.CopyToTexture]))
   .fn(t => {
     t.run();

src/webgpu/api/operation/resource_init/texture_zero_init_test.ts

@@ -1,7 +1,7 @@
 import * as C from '../../../../common/constants.js';
-import { TestCaseRecorder } from '../../../../common/framework/logger.js';
+import { TestCaseRecorder } from '../../../../common/framework/logging/test_case_recorder.js';
-import { params, poptions, pbool } from '../../../../common/framework/params.js';
+import { params, poptions, pbool } from '../../../../common/framework/params_builder.js';
-import { ParamSpec } from '../../../../common/framework/params_utils.js';
+import { CaseParams } from '../../../../common/framework/params_utils.js';
 import { assert, unreachable } from '../../../../common/framework/util/util.js';
 import { kTextureAspects, kTextureFormatInfo, kTextureFormats } from '../../../capability_info.js';
 import { GPUTest } from '../../../gpu_test.js';
@@ -214,7 +214,7 @@ function getRequiredTextureUsage(
 export abstract class TextureZeroInitTest extends GPUTest {
   protected stateToTexelComponents: { [k in InitializedState]: PerTexelComponent<number> };
 
-  constructor(rec: TestCaseRecorder, params: ParamSpec) {
+  constructor(rec: TestCaseRecorder, params: CaseParams) {
     super(rec, params);
 
     const stateToTexelComponents = (state: InitializedState) => {

src/webgpu/api/validation/createBindGroup.spec.ts

@@ -3,8 +3,8 @@ createBindGroup validation tests.
 `;
 
 import * as C from '../../../common/constants.js';
-import { poptions, params } from '../../../common/framework/params.js';
+import { poptions, params } from '../../../common/framework/params_builder.js';
-import { TestGroup } from '../../../common/framework/test_group.js';
+import { makeTestGroup } from '../../../common/framework/test_group.js';
 import { unreachable } from '../../../common/framework/util/util.js';
 import {
   kBindingTypes,
@@ -21,9 +21,9 @@ function clone<T extends GPUTextureDescriptor>(descriptor: T): T {
   return JSON.parse(JSON.stringify(descriptor));
 }
 
-export const g = new TestGroup(ValidationTest);
+export const g = makeTestGroup(ValidationTest);
 
-g.test('binding count mismatch').fn(async t => {
+g.test('binding_count_mismatch').fn(async t => {
   const bindGroupLayout = t.device.createBindGroupLayout({
     entries: [{ binding: 0, visibility: GPUShaderStage.COMPUTE, type: 'storage-buffer' }],
   });
@@ -51,7 +51,7 @@ g.test('binding count mismatch').fn(async t => {
   });
 });
 
-g.test('binding must be present in layout').fn(async t => {
+g.test('binding_must_be_present_in_layout').fn(async t => {
   const bindGroupLayout = t.device.createBindGroupLayout({
     entries: [{ binding: 0, visibility: GPUShaderStage.COMPUTE, type: 'storage-buffer' }],
   });
@@ -75,7 +75,7 @@ g.test('binding must be present in layout').fn(async t => {
   });
 });
 
-g.test('buffer binding must contain exactly one buffer of its type')
+g.test('buffer_binding_must_contain_exactly_one_buffer_of_its_type')
   .params(
     params()
       .combine(poptions('bindingType', kBindingTypes))
@@ -100,7 +100,7 @@ g.test('buffer binding must contain exactly one buffer of its type')
     }, !resourceBindingMatches);
   });
 
-g.test('texture binding must have correct usage')
+g.test('texture_binding_must_have_correct_usage')
   .params(
     params()
       .combine(poptions('type', kTextureBindingTypes))
@@ -130,7 +130,7 @@ g.test('texture binding must have correct usage')
     }, shouldError);
   });
 
-g.test('texture must have correct component type')
+g.test('texture_must_have_correct_component_type')
   .params(
     poptions('textureComponentType', [
       C.TextureComponentType.Float,
@@ -208,7 +208,7 @@ g.test('texture must have correct component type')
   });
 
 // TODO: Write test for all dimensions.
-g.test('texture must have correct dimension').fn(async t => {
+g.test('texture_must_have_correct_dimension').fn(async t => {
   const bindGroupLayout = t.device.createBindGroupLayout({
     entries: [
       {
@@ -244,7 +244,7 @@ g.test('texture must have correct dimension').fn(async t => {
   });
 });
 
-g.test('buffer offset and size for bind groups match')
+g.test('buffer_offset_and_size_for_bind_groups_match')
   .params([
     { offset: 0, size: 512, _success: true }, // offset 0 is valid
     { offset: 256, size: 256, _success: true }, // offset 256 (aligned) is valid

src/webgpu/api/validation/createBindGroupLayout.spec.ts

@@ -3,8 +3,8 @@ createBindGroupLayout validation tests.
 `;
 
 import * as C from '../../../common/constants.js';
-import { poptions, params } from '../../../common/framework/params.js';
+import { poptions, params } from '../../../common/framework/params_builder.js';
-import { TestGroup } from '../../../common/framework/test_group.js';
+import { makeTestGroup } from '../../../common/framework/test_group.js';
 import {
   kBindingTypeInfo,
   kBindingTypes,
@@ -18,9 +18,9 @@ function clone<T extends GPUBindGroupLayoutDescriptor>(descriptor: T): T {
   return JSON.parse(JSON.stringify(descriptor));
 }
 
-export const g = new TestGroup(ValidationTest);
+export const g = makeTestGroup(ValidationTest);
 
-g.test('some binding index was specified more than once').fn(async t => {
+g.test('some_binding_index_was_specified_more_than_once').fn(async t => {
   const goodDescriptor = {
     entries: [
       { binding: 0, visibility: GPUShaderStage.COMPUTE, type: C.BindingType.StorageBuffer },
@@ -40,13 +40,13 @@ g.test('some binding index was specified more than once').fn(async t => {
   });
 });
 
-g.test('Visibility of bindings can be 0').fn(async t => {
+g.test('visibility_of_bindings_can_be_0').fn(async t => {
   t.device.createBindGroupLayout({
     entries: [{ binding: 0, visibility: 0, type: 'storage-buffer' }],
   });
 });
 
-g.test('number of dynamic buffers exceeds the maximum value')
+g.test('number_of_dynamic_buffers_exceeds_the_maximum_value')
   .params([
     { type: C.BindingType.StorageBuffer, maxDynamicBufferCount: 4 },
     { type: C.BindingType.UniformBuffer, maxDynamicBufferCount: 8 },
@@ -88,7 +88,7 @@ g.test('number of dynamic buffers exceeds the maximum value')
     });
   });
 
-g.test('dynamic set to true is allowed only for buffers')
+g.test('dynamic_set_to_true_is_allowed_only_for_buffers')
   .params(poptions('type', kBindingTypes))
   .fn(async t => {
     const { type } = t.params;
@@ -143,7 +143,7 @@ const kCasesForMaxResourcesPerStageTests = params()
 
 // Should never fail unless kMaxBindingsPerBindGroup is exceeded, because the validation for
 // resources-of-type-per-stage is in pipeline layout creation.
-g.test('max resources per stage/in bind group layout')
+g.test('max_resources_per_stage,in_bind_group_layout')
   .params(kCasesForMaxResourcesPerStageTests)
   .fn(async t => {
     const { maxedType, extraType, maxedVisibility, extraVisibility } = t.params;
@@ -184,7 +184,7 @@ g.test('max resources per stage/in bind group layout')
 // One pipeline layout can have a maximum number of each type of binding *per stage* (which is
 // different for each type). Test that the max works, then add one more binding of same-or-different
 // type and same-or-different visibility.
-g.test('max resources per stage/in pipeline layout')
+g.test('max_resources_per_stage,in_pipeline_layout')
   .params(kCasesForMaxResourcesPerStageTests)
   .fn(async t => {
     const { maxedType, extraType, maxedVisibility, extraVisibility } = t.params;

src/webgpu/api/validation/createPipelineLayout.spec.ts

@@ -3,8 +3,8 @@ createPipelineLayout validation tests.
 `;
 
 import * as C from '../../../common/constants.js';
-import { pbool, poptions, params } from '../../../common/framework/params.js';
+import { pbool, poptions, params } from '../../../common/framework/params_builder.js';
-import { TestGroup } from '../../../common/framework/test_group.js';
+import { makeTestGroup } from '../../../common/framework/test_group.js';
 import {
   kBindingTypeInfo,
   kBindingTypes,
@@ -17,9 +17,9 @@ function clone<T extends GPUBindGroupLayoutDescriptor>(descriptor: T): T {
   return JSON.parse(JSON.stringify(descriptor));
 }
 
-export const g = new TestGroup(ValidationTest);
+export const g = makeTestGroup(ValidationTest);
 
-g.test('number of dynamic buffers exceeds the maximum value')
+g.test('number_of_dynamic_buffers_exceeds_the_maximum_value')
   .params(
     params()
       .combine(poptions('visibility', [0, 2, 4, 6]))
@@ -74,7 +74,7 @@ g.test('number of dynamic buffers exceeds the maximum value')
     });
   });
 
-g.test('visibility and dynamic offsets')
+g.test('visibility_and_dynamic_offsets')
   .params(
     params()
       .combine(poptions('type', kBindingTypes))
@@ -101,7 +101,7 @@ g.test('visibility and dynamic offsets')
     }, !success);
   });
 
-g.test('number of bind group layouts exceeds the maximum value').fn(async t => {
+g.test('number_of_bind_group_layouts_exceeds_the_maximum_value').fn(async t => {
   const bindGroupLayoutDescriptor: GPUBindGroupLayoutDescriptor = {
     entries: [],
   };

src/webgpu/api/validation/createTexture.spec.ts

@@ -2,8 +2,8 @@ export const description = `
 createTexture validation tests.
 `;
 
-import { poptions } from '../../../common/framework/params.js';
+import { poptions } from '../../../common/framework/params_builder.js';
-import { TestGroup } from '../../../common/framework/test_group.js';
+import { makeTestGroup } from '../../../common/framework/test_group.js';
 import { kTextureFormatInfo, kTextureFormats } from '../../capability_info.js';
 
 import { ValidationTest } from './validation_test.js';
@@ -38,9 +38,9 @@ class F extends ValidationTest {
   }
 }
 
-export const g = new TestGroup(F);
+export const g = makeTestGroup(F);
 
-g.test('validation of sampleCount')
+g.test('validation_of_sampleCount')
   .params([
     // TODO: Consider making a list of "valid"+"invalid" texture descriptors in capability_info.
     { sampleCount: 0, _success: false }, // sampleCount of 0 is not allowed
@@ -63,7 +63,7 @@ g.test('validation of sampleCount')
     }, !_success);
   });
 
-g.test('validation of mipLevelCount')
+g.test('validation_of_mipLevelCount')
   .params([
     { width: 32, height: 32, mipLevelCount: 1, _success: true }, // mipLevelCount of 1 is allowed
     { width: 32, height: 32, mipLevelCount: 0, _success: false }, // mipLevelCount of 0 is not allowed
@@ -85,20 +85,20 @@ g.test('validation of mipLevelCount')
     }, !_success);
   });
 
-g.test('it is valid to destroy a texture').fn(t => {
+g.test('it_is_valid_to_destroy_a_texture').fn(t => {
   const descriptor = t.getDescriptor();
   const texture = t.device.createTexture(descriptor);
   texture.destroy();
 });
 
-g.test('it is valid to destroy a destroyed texture').fn(t => {
+g.test('it_is_valid_to_destroy_a_destroyed_texture').fn(t => {
   const descriptor = t.getDescriptor();
   const texture = t.device.createTexture(descriptor);
   texture.destroy();
   texture.destroy();
 });
 
-g.test('it is invalid to submit a destroyed texture before and after encode')
+g.test('it_is_invalid_to_submit_a_destroyed_texture_before_and_after_encode')
   .params([
     { destroyBeforeEncode: false, destroyAfterEncode: false, _success: true },
     { destroyBeforeEncode: true, destroyAfterEncode: false, _success: false },
@@ -136,7 +136,7 @@ g.test('it is invalid to submit a destroyed texture before and after encode')
     }, !_success);
   });
 
-g.test('it is invalid to have an output attachment texture with non renderable format')
+g.test('it_is_invalid_to_have_an_output_attachment_texture_with_non_renderable_format')
   .params(poptions('format', kTextureFormats))
   .fn(async t => {
     const format: GPUTextureFormat = t.params.format;

src/webgpu/api/validation/createView.spec.ts

@@ -3,7 +3,7 @@ createView validation tests.
 `;
 
 import * as C from '../../../common/constants.js';
-import { TestGroup } from '../../../common/framework/test_group.js';
+import { makeTestGroup } from '../../../common/framework/test_group.js';
 
 import { ValidationTest } from './validation_test.js';
 
@@ -68,9 +68,9 @@ class F extends ValidationTest {
   }
 }
 
-export const g = new TestGroup(F);
+export const g = makeTestGroup(F);
 
-g.test('creating texture view on a 2D non array texture')
+g.test('creating_texture_view_on_a_2D_non_array_texture')
   .params([
     { _success: true }, // default view works
     { arrayLayerCount: 1, _success: true }, // it is OK to create a 2D texture view on a 2D texture
@@ -107,7 +107,7 @@ g.test('creating texture view on a 2D non array texture')
     }, !_success);
   });
 
-g.test('creating texture view on a 2D array texture')
+g.test('creating_texture_view_on_a_2D_array_texture')
   .params([
     { _success: true }, // default view works
     { dimension: C.TextureViewDimension.E2d, arrayLayerCount: 1, _success: true }, // it is OK to create a 2D texture view on a 2D array texture
@@ -139,7 +139,7 @@ g.test('creating texture view on a 2D array texture')
     }, !_success);
   });
 
-g.test('Using defaults validates the same as setting values for more than 1 array layer')
+g.test('Using_defaults_validates_the_same_as_setting_values_for_more_than_1_array_layer')
   .params([
     { _success: true },
     { format: C.TextureFormat.RGBA8Unorm, _success: true },
@@ -171,7 +171,7 @@ g.test('Using defaults validates the same as setting values for more than 1 arra
     }, !_success);
   });
 
-g.test('Using defaults validates the same as setting values for only 1 array layer')
+g.test('Using_defaults_validates_the_same_as_setting_values_for_only_1_array_layer')
   .params([
     { _success: true },
     { format: C.TextureFormat.RGBA8Unorm, _success: true },
@@ -196,7 +196,7 @@ g.test('Using defaults validates the same as setting values for only 1 array lay
     }, !_success);
   });
 
-g.test('creating cube map texture view')
+g.test('creating_cube_map_texture_view')
   .params([
     { dimension: C.TextureViewDimension.Cube, arrayLayerCount: 6, _success: true }, // it is OK to create a cube map texture view with arrayLayerCount == 6
     // it is an error to create a cube map texture view with arrayLayerCount != 6
@@ -224,7 +224,7 @@ g.test('creating cube map texture view')
     }, !_success);
   });
 
-g.test('creating cube map texture view with a non square texture')
+g.test('creating_cube_map_texture_view_with_a_non_square_texture')
   .params([
     { dimension: C.TextureViewDimension.Cube, arrayLayerCount: 6 }, // it is an error to create a cube map texture view with width != height.
     { dimension: C.TextureViewDimension.CubeArray, arrayLayerCount: 12 }, // it is an error to create a cube map array texture view with width != height.
@@ -250,7 +250,7 @@ g.test('creating cube map texture view with a non square texture')
   });
 
 // TODO: add more tests when rules are fully implemented.
-g.test('test the format compatibility rules when creating a texture view').fn(async t => {
+g.test('test_the_format_compatibility_rules_when_creating_a_texture_view').fn(async t => {
   const texture = t.createTexture({ arrayLayerCount: 1 });
 
   const descriptor = t.getDescriptor({
@@ -263,7 +263,7 @@ g.test('test the format compatibility rules when creating a texture view').fn(as
   });
 });
 
-g.test('it is invalid to use a texture view created from a destroyed texture').fn(async t => {
+g.test('it_is_invalid_to_use_a_texture_view_created_from_a_destroyed_texture').fn(async t => {
   const texture = t.createTexture({ arrayLayerCount: 1 });
 
   const commandEncoder = t.device.createCommandEncoder();

src/webgpu/api/validation/error_scope.spec.ts

@@ -4,7 +4,7 @@ error scope validation tests.
 
 import { Fixture } from '../../../common/framework/fixture.js';
 import { getGPU } from '../../../common/framework/gpu/implementation.js';
-import { TestGroup } from '../../../common/framework/test_group.js';
+import { makeTestGroup } from '../../../common/framework/test_group.js';
 import { assert, raceWithRejectOnTimeout } from '../../../common/framework/util/util.js';
 
 class F extends Fixture {
@@ -58,9 +58,9 @@ class F extends Fixture {
   }
 }
 
-export const g = new TestGroup(F);
+export const g = makeTestGroup(F);
 
-g.test('simple case where the error scope catches an error').fn(async t => {
+g.test('simple_case_where_the_error_scope_catches_an_error').fn(async t => {
   t.device.pushErrorScope('validation');
 
   t.createErrorBuffer();
@@ -69,7 +69,7 @@ g.test('simple case where the error scope catches an error').fn(async t => {
   t.expect(error instanceof GPUValidationError);
 });
 
-g.test('errors bubble to the parent scope if not handled by the current scope').fn(async t => {
+g.test('errors_bubble_to_the_parent_scope_if_not_handled_by_the_current_scope').fn(async t => {
   t.device.pushErrorScope('validation');
   t.device.pushErrorScope('out-of-memory');
 
@@ -85,7 +85,7 @@ g.test('errors bubble to the parent scope if not handled by the current scope').
   }
 });
 
-g.test('if an error scope matches an error it does not bubble to the parent scope').fn(async t => {
+g.test('if_an_error_scope_matches_an_error_it_does_not_bubble_to_the_parent_scope').fn(async t => {
   t.device.pushErrorScope('validation');
   t.device.pushErrorScope('validation');
 
@@ -101,7 +101,7 @@ g.test('if an error scope matches an error it does not bubble to the parent scop
   }
 });
 
-g.test('if no error scope handles an error it fires an uncapturederror event').fn(async t => {
+g.test('if_no_error_scope_handles_an_error_it_fires_an_uncapturederror_event').fn(async t => {
   t.device.pushErrorScope('out-of-memory');
 
   const uncapturedErrorEvent = await t.expectUncapturedError(() => {
@@ -113,7 +113,7 @@ g.test('if no error scope handles an error it fires an uncapturederror event').f
   t.expect(error === null);
 });
 
-g.test('push/popping sibling error scopes must be balanced').fn(async t => {
+g.test('push,popping_sibling_error_scopes_must_be_balanced').fn(async t => {
   {
     const promise = t.device.popErrorScope();
     t.shouldReject('OperationError', promise);
@@ -133,7 +133,7 @@ g.test('push/popping sibling error scopes must be balanced').fn(async t => {
   }
 });
 
-g.test('push/popping nested error scopes must be balanced').fn(async t => {
+g.test('push,popping_nested_error_scopes_must_be_balanced').fn(async t => {
   {
     const promise = t.device.popErrorScope();
     t.shouldReject('OperationError', promise);

src/webgpu/api/validation/fences.spec.ts

@@ -2,14 +2,14 @@ export const description = `
 fences validation tests.
 `;
 
-import { TestGroup } from '../../../common/framework/test_group.js';
+import { makeTestGroup } from '../../../common/framework/test_group.js';
 
 import { ValidationTest } from './validation_test.js';
 
-export const g = new TestGroup(ValidationTest);
+export const g = makeTestGroup(ValidationTest);
 
 // TODO: Remove if https://github.com/gpuweb/gpuweb/issues/377 is decided
-g.test('wait on a fence without signaling the value is invalid').fn(async t => {
+g.test('wait_on_a_fence_without_signaling_the_value_is_invalid').fn(async t => {
   const fence = t.queue.createFence();
 
   t.expectValidationError(() => {
@@ -19,7 +19,7 @@ g.test('wait on a fence without signaling the value is invalid').fn(async t => {
 });
 
 // TODO: Remove if https://github.com/gpuweb/gpuweb/issues/377 is decided
-g.test('wait on a fence with a value greater than signaled value is invalid').fn(async t => {
+g.test('wait_on_a_fence_with_a_value_greater_than_signaled_value_is_invalid').fn(async t => {
   const fence = t.queue.createFence();
   t.queue.signal(fence, 2);
 
@@ -29,7 +29,7 @@ g.test('wait on a fence with a value greater than signaled value is invalid').fn
   });
 });
 
-g.test('signal a value lower than signaled value is invalid').fn(async t => {
+g.test('signal_a_value_lower_than_signaled_value_is_invalid').fn(async t => {
   const fence = t.queue.createFence({ initialValue: 1 });
 
   t.expectValidationError(() => {
@@ -37,7 +37,7 @@ g.test('signal a value lower than signaled value is invalid').fn(async t => {
   });
 });
 
-g.test('signal a value equal to signaled value is invalid').fn(async t => {
+g.test('signal_a_value_equal_to_signaled_value_is_invalid').fn(async t => {
   const fence = t.queue.createFence({ initialValue: 1 });
 
   t.expectValidationError(() => {
@@ -45,7 +45,7 @@ g.test('signal a value equal to signaled value is invalid').fn(async t => {
   });
 });
 
-g.test('increasing fence value by more than 1 succeeds').fn(async t => {
+g.test('increasing_fence_value_by_more_than_1_succeeds').fn(async t => {
   const fence = t.queue.createFence();
 
   t.queue.signal(fence, 2);
@@ -55,7 +55,7 @@ g.test('increasing fence value by more than 1 succeeds').fn(async t => {
   await fence.onCompletion(6);
 });
 
-g.test('signal a fence on a different device than it was created on is invalid').fn(async t => {
+g.test('signal_a_fence_on_a_different_device_than_it_was_created_on_is_invalid').fn(async t => {
   const fence = t.queue.createFence();
 
   const anotherDevice = await t.device.adapter.requestDevice();
@@ -66,7 +66,7 @@ g.test('signal a fence on a different device than it was created on is invalid')
   });
 });
 
-g.test('signal a fence on a different device does not update fence signaled value').fn(async t => {
+g.test('signal_a_fence_on_a_different_device_does_not_update_fence_signaled_value').fn(async t => {
   const fence = t.queue.createFence({ initialValue: 1 });
 
   const anotherDevice = await t.device.adapter.requestDevice();

src/webgpu/api/validation/queue_submit.spec.ts

@@ -2,13 +2,13 @@ export const description = `
 queue submit validation tests.
 `;
 
-import { TestGroup } from '../../../common/framework/test_group.js';
+import { makeTestGroup } from '../../../common/framework/test_group.js';
 
 import { ValidationTest } from './validation_test.js';
 
-export const g = new TestGroup(ValidationTest);
+export const g = makeTestGroup(ValidationTest);
 
-g.test('submitting with a mapped buffer is disallowed').fn(async t => {
+g.test('submitting_with_a_mapped_buffer_is_disallowed').fn(async t => {
   const buffer = t.device.createBuffer({
     size: 4,
     usage: GPUBufferUsage.MAP_WRITE | GPUBufferUsage.COPY_SRC,

src/webgpu/api/validation/render_pass_descriptor.spec.ts

@@ -2,7 +2,7 @@ export const description = `
 render pass descriptor validation tests.
 `;
 
-import { TestGroup } from '../../../common/framework/test_group.js';
+import { makeTestGroup } from '../../../common/framework/test_group.js';
 
 import { ValidationTest } from './validation_test.js';
 
@@ -75,9 +75,9 @@ class F extends ValidationTest {
   }
 }
 
-export const g = new TestGroup(F);
+export const g = makeTestGroup(F);
 
-g.test('a render pass with only one color is ok').fn(t => {
+g.test('a_render_pass_with_only_one_color_is_ok').fn(t => {
   const colorTexture = t.createTexture({ format: 'rgba8unorm' });
   const descriptor = {
     colorAttachments: [t.getColorAttachment(colorTexture)],
@@ -86,7 +86,7 @@ g.test('a render pass with only one color is ok').fn(t => {
   t.tryRenderPass(true, descriptor);
 });
 
-g.test('a render pass with only one depth attachment is ok').fn(t => {
+g.test('a_render_pass_with_only_one_depth_attachment_is_ok').fn(t => {
   const depthStencilTexture = t.createTexture({ format: 'depth24plus-stencil8' });
   const descriptor = {
     colorAttachments: [],
@@ -96,7 +96,7 @@ g.test('a render pass with only one depth attachment is ok').fn(t => {
   t.tryRenderPass(true, descriptor);
 });
 
-g.test('OOB color attachment indices are handled')
+g.test('OOB_color_attachment_indices_are_handled')
   .params([
     { colorAttachmentsCount: 4, _success: true }, // Control case
     { colorAttachmentsCount: 5, _success: false }, // Out of bounds
@@ -113,7 +113,7 @@ g.test('OOB color attachment indices are handled')
     await t.tryRenderPass(_success, { colorAttachments });
   });
 
-g.test('attachments must have the same size').fn(async t => {
+g.test('attachments_must_have_the_same_size').fn(async t => {
   const colorTexture1x1A = t.createTexture({ width: 1, height: 1, format: 'rgba8unorm' });
   const colorTexture1x1B = t.createTexture({ width: 1, height: 1, format: 'rgba8unorm' });
   const colorTexture2x2 = t.createTexture({ width: 2, height: 2, format: 'rgba8unorm' });
@@ -165,7 +165,7 @@ g.test('attachments must have the same size').fn(async t => {
   }
 });
 
-g.test('attachments must match whether they are used for color or depth stencil').fn(async t => {
+g.test('attachments_must_match_whether_they_are_used_for_color_or_depth_stencil').fn(async t => {
   const colorTexture = t.createTexture({ format: 'rgba8unorm' });
   const depthStencilTexture = t.createTexture({ format: 'depth24plus-stencil8' });
 
@@ -188,7 +188,7 @@ g.test('attachments must match whether they are used for color or depth stencil'
   }
 });
 
-g.test('check layer count for color or depth stencil')
+g.test('check_layer_count_for_color_or_depth_stencil')
   .params([
     { arrayLayerCount: 5, baseArrayLayer: 0, _success: false }, // using 2D array texture view with arrayLayerCount > 1 is not allowed
     { arrayLayerCount: 1, baseArrayLayer: 0, _success: true }, // using 2D array texture view that covers the first layer of the texture is OK
@@ -257,7 +257,7 @@ g.test('check layer count for color or depth stencil')
     }
   });
 
-g.test('check mip level count for color or depth stencil')
+g.test('check_mip_level_count_for_color_or_depth_stencil')
   .params([
     { mipLevelCount: 2, baseMipLevel: 0, _success: false }, // using 2D texture view with mipLevelCount > 1 is not allowed
     { mipLevelCount: 1, baseMipLevel: 0, _success: true }, // using 2D texture view that covers the first level of the texture is OK
@@ -326,7 +326,7 @@ g.test('check mip level count for color or depth stencil')
     }
   });
 
-g.test('it is invalid to set resolve target if color attachment is non multisampled').fn(
+g.test('it_is_invalid_to_set_resolve_target_if_color_attachment_is_non_multisampled').fn(
   async t => {
     const colorTexture = t.createTexture({ sampleCount: 1 });
     const resolveTargetTexture = t.createTexture({ sampleCount: 1 });
@@ -345,7 +345,7 @@ g.test('it is invalid to set resolve target if color attachment is non multisamp
   }
 );
 
-g.test('check the use of multisampled textures as color attachments').fn(async t => {
+g.test('check_the_use_of_multisampled_textures_as_color_attachments').fn(async t => {
   const colorTexture = t.createTexture({ sampleCount: 1 });
   const multisampledColorTexture = t.createTexture({ sampleCount: 4 });
 
@@ -369,7 +369,7 @@ g.test('check the use of multisampled textures as color attachments').fn(async t
   }
 });
 
-g.test('it is invalid to use a multisampled resolve target').fn(async t => {
+g.test('it_is_invalid_to_use_a_multisampled_resolve_target').fn(async t => {
   const multisampledColorTexture = t.createTexture({ sampleCount: 4 });
   const multisampledResolveTargetTexture = t.createTexture({ sampleCount: 4 });
 
@@ -383,7 +383,7 @@ g.test('it is invalid to use a multisampled resolve target').fn(async t => {
   await t.tryRenderPass(false, descriptor);
 });
 
-g.test('it is invalid to use a resolve target with array layer count greater than 1').fn(
+g.test('it_is_invalid_to_use_a_resolve_target_with_array_layer_count_greater_than_1').fn(
   async t => {
     const multisampledColorTexture = t.createTexture({ sampleCount: 4 });
     const resolveTargetTexture = t.createTexture({ arrayLayerCount: 2 });
@@ -399,7 +399,7 @@ g.test('it is invalid to use a resolve target with array layer count greater tha
   }
 );
 
-g.test('it is invalid to use a resolve target with mipmap level count greater than 1').fn(
+g.test('it_is_invalid_to_use_a_resolve_target_with_mipmap_level_count_greater_than_1').fn(
   async t => {
     const multisampledColorTexture = t.createTexture({ sampleCount: 4 });
     const resolveTargetTexture = t.createTexture({ mipLevelCount: 2 });
@@ -415,7 +415,7 @@ g.test('it is invalid to use a resolve target with mipmap level count greater th
   }
 );
 
-g.test('it is invalid to use a resolve target whose usage is not output attachment').fn(async t => {
+g.test('it_is_invalid_to_use_a_resolve_target_whose_usage_is_not_output_attachment').fn(async t => {
   const multisampledColorTexture = t.createTexture({ sampleCount: 4 });
   const resolveTargetTexture = t.createTexture({
     usage: GPUTextureUsage.COPY_SRC | GPUTextureUsage.COPY_DST,
@@ -431,7 +431,7 @@ g.test('it is invalid to use a resolve target whose usage is not output attachme
   await t.tryRenderPass(false, descriptor);
 });
 
-g.test('it is invalid to use a resolve target in error state').fn(async t => {
+g.test('it_is_invalid_to_use_a_resolve_target_in_error_state').fn(async t => {
   const ARRAY_LAYER_COUNT = 1;
 
   const multisampledColorTexture = t.createTexture({ sampleCount: 4 });
@@ -453,7 +453,7 @@ g.test('it is invalid to use a resolve target in error state').fn(async t => {
   await t.tryRenderPass(false, descriptor);
 });
 
-g.test('use of multisampled attachment and non multisampled resolve target is allowed').fn(
+g.test('use_of_multisampled_attachment_and_non_multisampled_resolve_target_is_allowed').fn(
   async t => {
     const multisampledColorTexture = t.createTexture({ sampleCount: 4 });
     const resolveTargetTexture = t.createTexture({ sampleCount: 1 });
@@ -469,7 +469,7 @@ g.test('use of multisampled attachment and non multisampled resolve target is al
   }
 );
 
-g.test('use a resolve target in a format different than the attachment is not allowed').fn(
+g.test('use_a_resolve_target_in_a_format_different_than_the_attachment_is_not_allowed').fn(
   async t => {
     const multisampledColorTexture = t.createTexture({ sampleCount: 4 });
     const resolveTargetTexture = t.createTexture({ format: 'bgra8unorm' });
@@ -485,7 +485,7 @@ g.test('use a resolve target in a format different than the attachment is not al
   }
 );
 
-g.test('size of the resolve target must be the same as the color attachment').fn(async t => {
+g.test('size_of_the_resolve_target_must_be_the_same_as_the_color_attachment').fn(async t => {
   const size = 16;
   const multisampledColorTexture = t.createTexture({ width: size, height: size, sampleCount: 4 });
   const resolveTargetTexture = t.createTexture({
@@ -523,7 +523,7 @@ g.test('size of the resolve target must be the same as the color attachment').fn
   }
 });
 
-g.test('check depth stencil attachment sample counts mismatch').fn(async t => {
+g.test('check_depth_stencil_attachment_sample_counts_mismatch').fn(async t => {
   const multisampledDepthStencilTexture = t.createTexture({
     sampleCount: 4,
     format: 'depth24plus-stencil8',

src/webgpu/api/validation/setBindGroup.spec.ts

@@ -2,8 +2,8 @@ export const description = `
 setBindGroup validation tests.
 `;
 
-import { poptions, params } from '../../../common/framework/params.js';
+import { poptions, params } from '../../../common/framework/params_builder.js';
-import { TestGroup } from '../../../common/framework/test_group.js';
+import { makeTestGroup } from '../../../common/framework/test_group.js';
 
 import { ValidationTest } from './validation_test.js';
 
@@ -48,9 +48,9 @@ class F extends ValidationTest {
   }
 }
 
-export const g = new TestGroup(F);
+export const g = makeTestGroup(F);
 
-g.test('dynamic offsets passed but not expected/compute pass')
+g.test('dynamic_offsets_passed_but_not_expected,compute_pass')
   .params(poptions('type', ['compute', 'renderpass', 'renderbundle']))
   .fn(async t => {
     const bindGroupLayout = t.device.createBindGroupLayout({ entries: [] });
@@ -91,7 +91,7 @@ g.test('dynamic offsets passed but not expected/compute pass')
     });
   });
 
-g.test('dynamic offsets match expectations in pass encoder')
+g.test('dynamic_offsets_match_expectations_in_pass_encoder')
   .params(
     params()
       .combine(poptions('type', ['compute', 'renderpass', 'renderbundle']))
@@ -107,12 +107,12 @@ g.test('dynamic offsets match expectations in pass encoder')
         // Dynamic uniform buffer out of bounds because of binding size
         { dynamicOffsets: [512, 0], _success: false },
         { dynamicOffsets: [1024, 0], _success: false },
-        { dynamicOffsets: [Number.MAX_SAFE_INTEGER, 0], _success: false },
+        { dynamicOffsets: [0xffffffff, 0], _success: false },
 
         // Dynamic storage buffer out of bounds because of binding size
         { dynamicOffsets: [0, 512], _success: false },
         { dynamicOffsets: [0, 1024], _success: false },
-        { dynamicOffsets: [0, Number.MAX_SAFE_INTEGER], _success: false },
+        { dynamicOffsets: [0, 0xffffffff], _success: false },
       ])
   )
   .fn(async t => {

src/webgpu/api/validation/setBlendColor.spec.ts

@@ -2,7 +2,7 @@ export const description = `
 setBlendColor validation tests.
 `;
 
-import { TestGroup } from '../../../common/framework/test_group.js';
+import { makeTestGroup } from '../../../common/framework/test_group.js';
 
 import { ValidationTest } from './validation_test.js';
 
@@ -26,9 +26,9 @@ class F extends ValidationTest {
   }
 }
 
-export const g = new TestGroup(F);
+export const g = makeTestGroup(F);
 
-g.test('basic use of setBlendColor').fn(t => {
+g.test('basic_use_of_setBlendColor').fn(t => {
   const commandEncoder = t.device.createCommandEncoder();
   const renderPass = t.beginRenderPass(commandEncoder);
   renderPass.setBlendColor({ r: 0, g: 0, b: 0, a: 0 });
@@ -36,7 +36,7 @@ g.test('basic use of setBlendColor').fn(t => {
   commandEncoder.finish();
 });
 
-g.test('setBlendColor allows any number value').fn(t => {
+g.test('setBlendColor_allows_any_number_value').fn(t => {
   const values = [Number.MIN_SAFE_INTEGER, Number.MAX_SAFE_INTEGER];
   for (const value of values) {
     const commandEncoder = t.device.createCommandEncoder();

src/webgpu/api/validation/setScissorRect.spec.ts

@@ -2,7 +2,7 @@ export const description = `
 setScissorRect validation tests.
 `;
 
-import { TestGroup } from '../../../common/framework/test_group.js';
+import { makeTestGroup } from '../../../common/framework/test_group.js';
 
 import { ValidationTest } from './validation_test.js';
 
@@ -29,9 +29,9 @@ class F extends ValidationTest {
   }
 }
 
-export const g = new TestGroup(F);
+export const g = makeTestGroup(F);
 
-g.test('use of setScissorRect')
+g.test('use_of_setScissorRect')
   .params([
     { x: 0, y: 0, width: 1, height: 1, _success: true }, // Basic use
     { x: 0, y: 0, width: 0, height: 1, _success: false }, // Width of zero is not allowed

src/webgpu/api/validation/setStencilReference.spec.ts

@@ -2,8 +2,8 @@ export const description = `
 setStencilReference validation tests.
 `;
 
-import { poptions } from '../../../common/framework/params.js';
+import { poptions } from '../../../common/framework/params_builder.js';
-import { TestGroup } from '../../../common/framework/test_group.js';
+import { makeTestGroup } from '../../../common/framework/test_group.js';
 
 import { ValidationTest } from './validation_test.js';
 
@@ -27,9 +27,9 @@ class F extends ValidationTest {
   }
 }
 
-export const g = new TestGroup(F);
+export const g = makeTestGroup(F);
 
-g.test('use of setStencilReference')
+g.test('use_of_setStencilReference')
   .params(poptions('reference', [0, 0xffffffff]))
   .fn(t => {
     const { reference } = t.params;

src/webgpu/api/validation/setViewport.spec.ts

@@ -2,7 +2,7 @@ export const description = `
 setViewport validation tests.
 `;
 
-import { TestGroup } from '../../../common/framework/test_group.js';
+import { makeTestGroup } from '../../../common/framework/test_group.js';
 
 import { ValidationTest } from './validation_test.js';
 
@@ -29,9 +29,9 @@ class F extends ValidationTest {
   }
 }
 
-export const g = new TestGroup(F);
+export const g = makeTestGroup(F);
 
-g.test('use of setViewport')
+g.test('use_of_setViewport')
   .params([
     { x: 0, y: 0, width: 1, height: 1, minDepth: 0, maxDepth: 1, _success: true }, // Basic use
     { x: 0, y: 0, width: 0, height: 1, minDepth: 0, maxDepth: 1, _success: false }, // Width of zero is not allowed

src/webgpu/examples.spec.ts

@@ -4,7 +4,7 @@ Examples of writing CTS tests with various features.
 Start here when looking for examples of basic framework usage.
 `;
 
-import { TestGroup } from '../common/framework/test_group.js';
+import { makeTestGroup } from '../common/framework/test_group.js';
 
 import { GPUTest } from './gpu_test.js';
 
@@ -18,11 +18,11 @@ import { GPUTest } from './gpu_test.js';
 // - ?q=webgpu:examples:basic/
 // - ?q=webgpu:examples:
 
-export const g = new TestGroup(GPUTest);
+export const g = makeTestGroup(GPUTest);
 
 // Note: spaces in test names are replaced with underscores: webgpu:examples:test_name=
 /* eslint-disable-next-line  @typescript-eslint/no-unused-vars */
-g.test('test name').fn(t => {});
+g.test('test_name').fn(t => {});
 
 g.test('basic').fn(t => {
   t.expect(true);
@@ -40,7 +40,7 @@ g.test('basic').fn(t => {
   );
 });
 
-g.test('basic/async').fn(async t => {
+g.test('basic,async').fn(async t => {
   // shouldReject must be awaited to ensure it can wait for the promise before the test ends.
   t.shouldReject(
     // The expected '.name' of the thrown error.
@@ -69,7 +69,7 @@ g.test('basic/async').fn(async t => {
 //
 // - webgpu:examples:basic/params={"x":2,"y":4}    runs with t.params = {x: 2, y: 5, _result: 6}.
 // - webgpu:examples:basic/params={"x":-10,"y":18} runs with t.params = {x: -10, y: 18, _result: 8}.
-g.test('basic/params')
+g.test('basic,params')
   .params([
     { x: 2, y: 4, _result: 6 }, //
     { x: -10, y: 18, _result: 8 },
@@ -79,14 +79,14 @@ g.test('basic/params')
   });
 // (note the blank comment above to enforce newlines on autoformat)
 
-g.test('gpu/async').fn(async t => {
+g.test('gpu,async').fn(async t => {
   const fence = t.queue.createFence();
   t.queue.signal(fence, 2);
   await fence.onCompletion(1);
   t.expect(fence.getCompletedValue() === 2);
 });
 
-g.test('gpu/buffers').fn(async t => {
+g.test('gpu,buffers').fn(async t => {
   const data = new Uint32Array([0, 1234, 0]);
   const [src, map] = t.device.createBufferMapped({
     size: 12,

src/webgpu/gpu_test.ts

@@ -1,6 +1,6 @@
-import { attemptGarbageCollection } from '../common/framework/collect_garbage.js';
 import { Fixture } from '../common/framework/fixture.js';
 import { DevicePool, TestOOMedShouldAttemptGC } from '../common/framework/gpu/device_pool.js';
+import { attemptGarbageCollection } from '../common/framework/util/collect_garbage.js';
 import { assert } from '../common/framework/util/util.js';
 
 import {

src/webgpu/listing.ts

@@ -1,4 +1,4 @@
-import { TestSuiteListing } from '../common/framework/listing.js';
+import { TestSuiteListing } from '../common/framework/test_suite_listing.js';
 import { makeListing } from '../common/tools/crawl.js';
 
 export const listing: Promise<TestSuiteListing> = makeListing(__filename);

src/webgpu/util/texture/texelData.ts

@@ -183,7 +183,7 @@ export interface TexelDataRepresentation {
 }
 
 class TexelDataRepresentationImpl implements TexelDataRepresentation {
-  // TODO: Determine endianess of the GPU data?
+  // TODO: Determine endianness of the GPU data?
   private isGPULittleEndian = true;
 
   constructor(

src/webgpu/web-platform/canvas/context_creation.spec.ts

@@ -1,11 +1,11 @@
 export const description = '';
 
 import { Fixture } from '../../../common/framework/fixture.js';
-import { TestGroup } from '../../../common/framework/test_group.js';
+import { makeTestGroup } from '../../../common/framework/test_group.js';
 
-export const g = new TestGroup(Fixture);
+export const g = makeTestGroup(Fixture);
 
-g.test('canvas element getContext returns GPUCanvasContext').fn(async t => {
+g.test('canvas_element_getContext_returns_GPUCanvasContext').fn(async t => {
   if (typeof document === 'undefined') {
     // Skip if there is no document (Workers, Node)
     t.skip('DOM is not available to create canvas element');

src/webgpu/web-platform/copyImageBitmapToTexture.spec.ts

@@ -2,8 +2,8 @@ export const description = `
 copy imageBitmap To texture tests.
 `;
 
-import { poptions, params } from '../../common/framework/params.js';
+import { poptions, params } from '../../common/framework/params_builder.js';
-import { TestGroup } from '../../common/framework/test_group.js';
+import { makeTestGroup } from '../../common/framework/test_group.js';
 import { GPUTest } from '../gpu_test.js';
 
 function calculateRowPitch(width: number, bytesPerPixel: number): number {
@@ -114,9 +114,9 @@ class F extends GPUTest {
   }
 }
 
-export const g = new TestGroup(F);
+export const g = makeTestGroup(F);
 
-g.test('from ImageData')
+g.test('from_ImageData')
   .params(
     params()
       .combine(poptions('width', [1, 2, 4, 15, 255, 256]))
@@ -195,7 +195,7 @@ g.test('from ImageData')
     );
   });
 
-g.test('from canvas')
+g.test('from_canvas')
   .params(
     params()
       .combine(poptions('width', [1, 2, 4, 15, 255, 256]))

standalone/index.html

@@ -1,6 +1,6 @@
 <html>
   <head>
-    <title></title>
+    <title>WebGPU CTS</title>
     <link
       id="favicon"
       rel="shortcut icon"
@@ -41,7 +41,8 @@
         width: 100%;
         padding: 1px 2px 1px 1px;
       }
-      .noderun,
+      .subtreerun,
+      .leafrun,
       .nodelink,
       .testcaselogbtn {
         display: inline-block;
@@ -54,7 +55,8 @@
         border: 1px solid #888;
       }
       @media (pointer: fine) {
-        .noderun,
+        .subtreerun,
+        .leafrun,
         .nodelink,
         .testcaselogbtn {
           flex-basis: 24px;
@@ -64,7 +66,8 @@
         }
       }
       @media (pointer: coarse) {
-        .noderun,
+        .subtreerun,
+        .leafrun,
         .nodelink,
         .testcaselogbtn {
           flex-basis: 36px;
@@ -73,9 +76,12 @@
           height: 36px;
         }
       }
-      .noderun {
+      .subtreerun {
         background-image: url(data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAkAAAAJCAYAAADgkQYQAAAANklEQVQoU2NkYGD4zwABjFAagwJJwBTBJDEUY1OEoRifIrhiYhSBHYvuJnSHM5LtJry+wxlOAGPTCQmAB/WwAAAAAElFTkSuQmCC);
       }
+      .leafrun {
+        background-image: url(data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAkAAAAJAQMAAADaX5RTAAAABlBMVEUAAAAAAAClZ7nPAAAAAXRSTlMAQObYZgAAAB5JREFUCNdjOMDAsIGBoYeBoZmBoaEBRPaARQ4wAABTfwX/l/WQvgAAAABJRU5ErkJggg==);
+      }
       .nodelink {
         background-image: url(data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAwAAAAMAQMAAABsu86kAAAABlBMVEUAAAAAAAClZ7nPAAAAAXRSTlMAQObYZgAAACRJREFUCNdjYGBg+P+BoUGAYesFhj4BhvsFDPYNDHwMCMTAAACqJwbp3VgbrAAAAABJRU5ErkJggg==);
       }
@@ -83,7 +89,6 @@
         flex: 10 0 4em;
       }
       .nodename {
-        user-select: all;
         word-break: break-word;
       }
       @media only screen and (max-width: 600px) {
@@ -174,11 +179,6 @@
         }
       }
     </style>
-    <script>
-      if (!window.location.search) {
-        window.location += '?runnow=0&worker=0&debug=0&q=webgpu:';
-      }
-    </script>
   </head>
   <body>
     <h1>WebGPU Conformance Test Suite</h1>