Add Measurement and Scoring details to README.md (#467)
Document in more detail how scoring and workload measurement works: - Add example timeline to explain which parts contribute to the score - Add basic formulas on how the test steps are combined into a final score
This commit is contained in:
Camillo Bruni
GitHub
parent
faf9fff140
commit
4dfa8faeea
3 changed files with 308 additions and 0 deletions
22
README.md
22
README.md
|
|
@ -14,3 +14,25 @@ benchmark, actual users should benefit. In order to achieve this, it should:
|
|||
- Be accessible to the public and useful to browser engineers. It should run in every modern
|
||||
browser by visiting a normal web page. It should run relatively quickly, while providing
|
||||
enough test coverage to be reflective of the real-world Web.
|
||||
|
||||
# Score and Test Measurement
|
||||
|
||||
Each test can contain several steps who contributed to the test duration. Note that the prepare step of a test is _unmeasured_ and thus does not contribute to the score. All following test
|
||||
steps are **measured** and summed up in the test step time.
|
||||
|
||||
<img src="./resources/measurement-timeline.svg" />
|
||||
|
||||
Each step consists of the following phases:
|
||||
|
||||
<dl>
|
||||
<dt>Sync:</dt>
|
||||
<dd>Time spent in synchronous JS execution.</dd>
|
||||
<dt>Async:</dt>
|
||||
<dd>Time spent in asynchronous JS like Promise or setTimeout tasks scheduled by the sync-step.</dd>
|
||||
</dl>
|
||||
|
||||
## Score
|
||||
|
||||
The final benchmark score is calculated based of the inverse of [geomean](https://en.wikipedia.org/wiki/Geometric_mean) of all tests. We average the score over multiple iterations with the [arithmetic mean](https://en.wikipedia.org/wiki/Arithmetic_mean). By using the geomean to combine the test durations we maintain the invariant that relative improvements are favoured equally amongst all tests, even though they can have vastly different durations.
|
||||
|
||||
<img src="./resources/score.svg" />
|
||||
|
|
|
|||
88
resources/measurement-timeline.svg
Normal file
88
resources/measurement-timeline.svg
Normal file
|
|
@ -0,0 +1,88 @@
|
|||
|
||||
<svg fill="none" viewBox="0 0 780 140" width="780" height="140" xmlns="http://www.w3.org/2000/svg">
|
||||
<style>
|
||||
:root {
|
||||
--foreground: rgb(235, 235, 235);
|
||||
--background: rgb(46, 51, 55);
|
||||
font-family: "system-ui";
|
||||
background-color: var(--background);
|
||||
color: var(--foreground);
|
||||
}
|
||||
|
||||
table {
|
||||
padding: 1em;
|
||||
}
|
||||
|
||||
th {
|
||||
text-align: left;
|
||||
}
|
||||
|
||||
td {
|
||||
white-space: nowrap;
|
||||
border: 1px solid var(--foreground);
|
||||
border-radius: 2px;
|
||||
padding: 0 2px 0 2px;
|
||||
}
|
||||
|
||||
.measured {
|
||||
padding: 0 2px 0 2px;
|
||||
border-radius: 2px;
|
||||
color: var(--background);
|
||||
background-color: var(--foreground);
|
||||
font-weight: bold;
|
||||
}
|
||||
|
||||
.unmeasured {
|
||||
padding: 0 2px 0 2px;
|
||||
border-radius: 2px;
|
||||
border-style: dashed !important;
|
||||
font-style: italic;
|
||||
}
|
||||
|
||||
td.hidden {
|
||||
border: none;
|
||||
}
|
||||
</style>
|
||||
<foreignObject width="100%" height="100%">
|
||||
<table xmlns="http://www.w3.org/1999/xhtml" width="100%">
|
||||
<tbody>
|
||||
<tr>
|
||||
<th>Iteration:</th>
|
||||
<td colspan="9">Iteration 1</td>
|
||||
<td>...</td>
|
||||
<td colspan="3">Iteration N</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>Test:</th>
|
||||
<td colspan="6">TodoMVC-JavaScript-ES5</td>
|
||||
<td>...</td>
|
||||
<td colspan="2">Perf-Dashboard</td>
|
||||
<td class="hidden"></td>
|
||||
<td colspan="2">TodoMVC-...</td>
|
||||
<td>...</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>Steps:</th>
|
||||
<td class="unmeasured">Prepare</td>
|
||||
<td colspan="2" class="measured">Adding</td>
|
||||
<td colspan="2" class="measured">Completing</td>
|
||||
<td class="measured">...</td>
|
||||
<td class="hidden"></td>
|
||||
<td class="unmeasured">Prepare</td>
|
||||
<td class="measured">...</td>
|
||||
<td class="hidden"></td>
|
||||
<td class="unmeasured">Prepare</td>
|
||||
<td class="measured">...</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<th></th>
|
||||
<td class="hidden"></td>
|
||||
<td class="measured">Sync</td>
|
||||
<td class="measured">Async</td>
|
||||
<td class="measured">Sync</td>
|
||||
<td class="measured">Async</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</foreignObject>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 2.5 KiB |
198
resources/score.svg
Normal file
198
resources/score.svg
Normal file
|
|
@ -0,0 +1,198 @@
|
|||
|
||||
<svg fill="none" viewBox="0 0 550 300" width="550" height="300" xmlns="http://www.w3.org/2000/svg">
|
||||
<style>
|
||||
:root {
|
||||
--foreground: rgb(235, 235, 235);
|
||||
--background: rgb(46, 51, 55);
|
||||
background-color: var(--background);
|
||||
color: var(--foreground);
|
||||
}
|
||||
|
||||
mtable {
|
||||
display: table;
|
||||
margin: 1em;
|
||||
}
|
||||
|
||||
mtr mtd {
|
||||
text-align: left;
|
||||
padding-bottom: 1em;
|
||||
}
|
||||
</style>
|
||||
<foreignObject width="100%" height="100%">
|
||||
<math xmlns="http://www.w3.org/1998/Math/MathML" width="100%" height="100%" >
|
||||
<mtable displaystyle="true" >
|
||||
<mtr>
|
||||
<mtd>
|
||||
<mrow>
|
||||
<mi>Score</mi>
|
||||
</mrow>
|
||||
</mtd>
|
||||
<mtd>
|
||||
<mrow>
|
||||
<mo>=</mo>
|
||||
<mfrac>
|
||||
<mn>1</mn>
|
||||
<mi>n</mi>
|
||||
</mfrac>
|
||||
<mrow>
|
||||
<munderover>
|
||||
<mo stretchy="true">∑</mo>
|
||||
<mrow>
|
||||
<mi>i</mi>
|
||||
<mo>=</mo>
|
||||
<mn>1</mn>
|
||||
</mrow>
|
||||
<mi>n</mi>
|
||||
</munderover>
|
||||
</mrow>
|
||||
<msub>
|
||||
<mrow>
|
||||
<mi>IterationScore</mi>
|
||||
</mrow>
|
||||
<mi>i</mi>
|
||||
</msub>
|
||||
</mrow>
|
||||
</mtd>
|
||||
</mtr>
|
||||
<mtr>
|
||||
<mtd>
|
||||
<msub>
|
||||
<mrow>
|
||||
<mi>IterationScore</mi>
|
||||
</mrow>
|
||||
<mi>i</mi>
|
||||
</msub>
|
||||
</mtd>
|
||||
<mtd>
|
||||
<mrow>
|
||||
<mo>=</mo>
|
||||
<mfrac>
|
||||
<mn>1000</mn>
|
||||
<msub>
|
||||
<mrow>
|
||||
<mi>IterationGeoMeanTime</mi>
|
||||
</mrow>
|
||||
<mi>i</mi>
|
||||
</msub>
|
||||
</mfrac>
|
||||
</mrow>
|
||||
</mtd>
|
||||
</mtr>
|
||||
<mtr>
|
||||
<mtd>
|
||||
<msub>
|
||||
<mrow>
|
||||
<mi>IterationGeoMeanTime</mi>
|
||||
</mrow>
|
||||
<mi>i</mi>
|
||||
</msub>
|
||||
</mtd>
|
||||
<mtd>
|
||||
<mrow>
|
||||
<mo>=</mo>
|
||||
<mroot>
|
||||
<mrow>
|
||||
<mrow>
|
||||
<munderover>
|
||||
<mo stretchy="true">∏</mo>
|
||||
<mrow>
|
||||
<mi>j</mi>
|
||||
<mo>=</mo>
|
||||
<mn>1</mn>
|
||||
</mrow>
|
||||
<mi>m</mi>
|
||||
</munderover>
|
||||
</mrow>
|
||||
<msub>
|
||||
<mrow>
|
||||
<mi>TestTime</mi>
|
||||
</mrow>
|
||||
<mrow>
|
||||
<mi>i</mi>
|
||||
<mi>j</mi>
|
||||
</mrow>
|
||||
</msub>
|
||||
</mrow>
|
||||
<mi>m</mi>
|
||||
</mroot>
|
||||
</mrow>
|
||||
</mtd>
|
||||
</mtr>
|
||||
<mtr>
|
||||
<mtd>
|
||||
<msub>
|
||||
<mrow>
|
||||
<mi>TestTime</mi>
|
||||
</mrow>
|
||||
<mrow>
|
||||
<mi>i</mi>
|
||||
<mi>j</mi>
|
||||
</mrow>
|
||||
</msub>
|
||||
</mtd>
|
||||
<mtd>
|
||||
<mrow>
|
||||
<mo>=</mo>
|
||||
<mrow>
|
||||
<munder>
|
||||
<mo stretchy="true">∑</mo>
|
||||
<mi>k</mi>
|
||||
</munder>
|
||||
</mrow>
|
||||
<msub>
|
||||
<mrow>
|
||||
<mi>TestStepTime</mi>
|
||||
</mrow>
|
||||
<mrow>
|
||||
<mi>i</mi>
|
||||
<mi>j</mi>
|
||||
<mi>k</mi>
|
||||
</mrow>
|
||||
</msub>
|
||||
</mrow>
|
||||
</mtd>
|
||||
</mtr>
|
||||
<mtr>
|
||||
<mtd>
|
||||
<msub>
|
||||
<mrow>
|
||||
<mi>TestStepTime</mi>
|
||||
</mrow>
|
||||
<mrow>
|
||||
<mi>i</mi>
|
||||
<mi>j</mi>
|
||||
<mi>k</mi>
|
||||
</mrow>
|
||||
</msub>
|
||||
</mtd>
|
||||
<mtd>
|
||||
<mrow>
|
||||
<mo>=</mo>
|
||||
<msub>
|
||||
<mrow>
|
||||
<mi>TestStepSyncTime</mi>
|
||||
</mrow>
|
||||
<mrow>
|
||||
<mi>i</mi>
|
||||
<mi>j</mi>
|
||||
<mi>k</mi>
|
||||
</mrow>
|
||||
</msub>
|
||||
<mo>+</mo>
|
||||
<msub>
|
||||
<mrow>
|
||||
<mi>TestStepAsyncTime</mi>
|
||||
</mrow>
|
||||
<mrow>
|
||||
<mi>i</mi>
|
||||
<mi>j</mi>
|
||||
<mi>k</mi>
|
||||
</mrow>
|
||||
</msub>
|
||||
</mrow>
|
||||
</mtd>
|
||||
</mtr>
|
||||
</mtable>
|
||||
</math>
|
||||
</foreignObject>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 6.6 KiB |
Loading…
Reference in a new issue