99 lines
4.8 KiB
Markdown
99 lines
4.8 KiB
Markdown
# Introduction
|
|
|
|
These documents contains guidelines for contributors to the WebGPU CTS (Conformance Test Suite)
|
|
on how to write effective tests, and on the testing philosophy to adopt.
|
|
|
|
The WebGPU CTS is arguably more important than the WebGPU specification itself, because
|
|
it is what forces implementation to be interoperable by checking they conform to the specification.
|
|
However writing a CTS is hard and requires a lot of effort to reach good coverage.
|
|
|
|
More than a collection of tests like regular end2end and unit tests for software artifacts, a CTS
|
|
needs to be exhaustive. Contrast for example the WebGL2 CTS with the ANGLE end2end tests: they
|
|
cover the same functionality (WebGL 2 / OpenGL ES 3) but are structured very differently:
|
|
|
|
- ANGLE's test suite has one or two tests per functionality to check it works correctly, plus
|
|
regression tests and special tests to cover implementation details.
|
|
- WebGL2's CTS can have thousands of tests per API aspect to cover every combination of
|
|
parameters (and global state) used by an operation.
|
|
|
|
Below are guidelines based on our collective experience with graphics API CTSes like WebGL's.
|
|
They are expected to evolve over time and have exceptions, but should give a general idea of what
|
|
to do.
|
|
|
|
## Contributing
|
|
|
|
Testing tasks are tracked in the [CTS project tracker](https://github.com/orgs/gpuweb/projects/3).
|
|
Go here if you're looking for tasks, or if you have a test idea that isn't already covered.
|
|
|
|
If contributing conformance tests, the directory you'll work in is [`src/webgpu/`](../src/webgpu/).
|
|
This directory is organized according to the goal of the test (API validation behavior vs
|
|
actual results) and its target (API entry points and spec areas, e.g. texture sampling).
|
|
|
|
The contents of a test file (`src/webgpu/**/*.spec.ts`) are twofold:
|
|
|
|
- Documentation ("test plans") on what tests do, how they do it, and what cases they cover.
|
|
Some test plans are fully or partially unimplemented:
|
|
they either contain "TODO" in a description or are `.unimplemented()`.
|
|
- Actual tests.
|
|
|
|
**Please read the following short documents before contributing.**
|
|
|
|
### 0. [Developing](developing.md)
|
|
|
|
- Reviewers should also read [Review Requirements](../reviews.md).
|
|
|
|
### 1. [Life of a Test Change](life_of.md)
|
|
|
|
### 2. [Adding or Editing Test Plans](plans.md)
|
|
|
|
### 3. [Implementing Tests](tests.md)
|
|
|
|
## [Additional Documentation](../)
|
|
|
|
## Examples
|
|
|
|
### Operation testing of vertex input id generation
|
|
|
|
This section provides an example of the planning process for a test.
|
|
It has not been refined into a set of final test plan descriptions.
|
|
(Note: this predates the actual implementation of these tests, so doesn't match the actual tests.)
|
|
|
|
Somewhere under the `api/operation` node are tests checking that running `GPURenderPipelines` on
|
|
the device using the `GPURenderEncoderBase.draw` family of functions works correctly. Render
|
|
pipelines are composed of several stages that are mostly independent so they can be split in
|
|
several parts such as `vertex_input`, `rasterization`, `blending`.
|
|
|
|
Vertex input itself has several parts that are mostly separate in hardware:
|
|
|
|
- generation of the vertex and instance indices to run for this draw
|
|
- fetching of vertex data from vertex buffers based on these indices
|
|
- conversion from the vertex attribute `GPUVertexFormat` to the datatype for the input variable
|
|
in the shader
|
|
|
|
Each of these are tested separately and have cases for each combination of the variables that may
|
|
affect them. This means that `api/operation/render/vertex_input/id_generation` checks that the
|
|
correct operation is performed for the cartesian product of all the following dimensions:
|
|
|
|
- for encoding in a `GPURenderPassEncoder` or a `GPURenderBundleEncoder`
|
|
- whether the draw is direct or indirect
|
|
- whether the draw is indexed or not
|
|
- for various values of the `firstInstance` argument
|
|
- for various values of the `instanceCount` argument
|
|
- if the draw is not indexed:
|
|
- for various values of the `firstVertex` argument
|
|
- for various values of the `vertexCount` argument
|
|
- if the draw is indexed:
|
|
- for each `GPUIndexFormat`
|
|
- for various values of the indices in the index buffer including the primitive restart values
|
|
- for various values for the `offset` argument to `setIndexBuffer`
|
|
- for various values of the `firstIndex` argument
|
|
- for various values of the `indexCount` argument
|
|
- for various values of the `baseVertex` argument
|
|
|
|
"Various values" above mean several small values, including `0` and the second smallest valid
|
|
value to check for corner cases, as well as some large value.
|
|
|
|
An instance of the test sets up a `draw*` call based on the parameters, using point rendering and
|
|
a fragment shader that outputs to a storage buffer. After the draw the test checks the content of
|
|
the storage buffer to make sure all expected vertex shader invocation, and only these ones have
|
|
been generated.
|