Add interleaved benchmark execution for test-suite #107
Conversation
This adds support for separating build and test phases, and running
benchmarks from multiple builds in an interleaved fashion to control
for environmental factors (ambient temperature, general system load etc).
New options:
* --build-only: Build tests without running them
* --test-prebuilt: Run tests from pre-built directory
* --build-dir: Specify build directory (used with --test-prebuilt)
* --exec-interleaved-builds: Comma-separated list of builds to interleave
Usage:
1. Build two compiler versions:
lnt runtest test-suite --build-only \
--sandbox /tmp/sandbox-a \
--cc /path/to/clang-a \
--test-suite ~/llvm-test-suite \
...
lnt runtest test-suite --build-only \
--sandbox /tmp/sandbox-b \
--cc /path/to/clang-b \
--test-suite ~/llvm-test-suite \
...
2. Run with interleaved execution:
lnt runtest test-suite \
--sandbox /tmp/results \
--exec-interleaved-builds /tmp/sandbox-a/build,/tmp/sandbox-b/build \
--exec-multisample 3
This runs tests in the pattern:
- Sample 0: build-a -> build-b
- Sample 1: build-a -> build-b
- Sample 2: build-a -> build-b
Temporal interleaving controls for environmental changes that could
bias results toward one build.
Or, test single build:
lnt runtest test-suite --test-prebuilt \
--build-dir /tmp/sandbox-a/build \
--exec-multisample 5
Reports are written to each build directory (report.json,
test-results.xunit.xml, test-results.csv).
This stack of pull requests is managed by Graphite. Learn more about stacking. |
There was a problem hiding this comment.
I'm not super familiar with the "run the test suite" part of LNT, so this might not be the most sensible question. But have you considered making this into a new subcommand of lnt runtest instead of adding "a new mode" for lnt runtest test-suite? This seems to be doing things quite a bit differently from the regular lnt runtest test-suite, and I (perhaps naively) wonder if it wouldn't be better to acknowledge that and not try to make it fit into lnt runtest test-suite.
Like I said, I don't have a strong opinion, but I wanted to at least ask the question.
|
|
||
| for build_idx, build_info in enumerate(build_infos): | ||
| logger.info(" Testing build %d/%d: %s" % | ||
| (build_idx + 1, len(build_infos), build_info['build_dir'])) |
There was a problem hiding this comment.
tox -e flake8 seems unhappy with this.
There was a problem hiding this comment.
Would it be possible to add unit tests for these new options? We have a bunch of existing ones for test_suite.py, for example https://github.com/llvm/llvm-lnt/blob/main/tests/runtest/test_suite-only-test.shtest.
2 participants
This adds support for separating build and test phases, and running
benchmarks from multiple builds in an interleaved fashion to control
for environmental factors (ambient temperature, general system load etc).
New options:
Usage:
Build two compiler versions:
lnt runtest test-suite --build-only
--sandbox /tmp/sandbox-a
--cc /path/to/clang-a
--test-suite ~/llvm-test-suite
...
lnt runtest test-suite --build-only
--sandbox /tmp/sandbox-b
--cc /path/to/clang-b
--test-suite ~/llvm-test-suite
...
Run with interleaved execution:
lnt runtest test-suite
--sandbox /tmp/results
--exec-interleaved-builds /tmp/sandbox-a/build,/tmp/sandbox-b/build
--exec-multisample 3
This runs tests in the pattern:
Temporal interleaving controls for environmental changes that could
bias results toward one build.
Or, test single build:
lnt runtest test-suite --test-prebuilt
--build-dir /tmp/sandbox-a/build
--exec-multisample 5
Reports are written to each build directory (report.json,
test-results.xunit.xml, test-results.csv).