Skip to content

Conversation

aemerson
Copy link

Add interleaved benchmark execution for test-suite

This adds support for separating build and test phases, and running
benchmarks from multiple builds in an interleaved fashion to control
for environmental factors (ambient temperature, general system load etc).

New options:

  • --build-only: Build tests without running them
  • --test-prebuilt: Run tests from pre-built directory
  • --build-dir: Specify build directory (used with --test-prebuilt)
  • --exec-interleaved-builds: Comma-separated list of builds to interleave

Usage:

  1. Build two compiler versions:
    lnt runtest test-suite --build-only
    --sandbox /tmp/sandbox-a
    --cc /path/to/clang-a
    --test-suite ~/llvm-test-suite
    ...

    lnt runtest test-suite --build-only
    --sandbox /tmp/sandbox-b
    --cc /path/to/clang-b
    --test-suite ~/llvm-test-suite
    ...

  2. Run with interleaved execution:
    lnt runtest test-suite
    --sandbox /tmp/results
    --exec-interleaved-builds /tmp/sandbox-a/build,/tmp/sandbox-b/build
    --exec-multisample 3

    This runs tests in the pattern:

    • Sample 0: build-a -> build-b
    • Sample 1: build-a -> build-b
    • Sample 2: build-a -> build-b

    Temporal interleaving controls for environmental changes that could
    bias results toward one build.

Or, test single build:
lnt runtest test-suite --test-prebuilt
--build-dir /tmp/sandbox-a/build
--exec-multisample 5

Reports are written to each build directory (report.json,
test-results.xunit.xml, test-results.csv).

Generalize --build-dir to work with all build modes

The --build-dir option now works with all build modes, not just
--test-prebuilt. It can be used to specify a custom build directory
location, overriding the default sandbox/build or timestamped directory.

This adds support for separating build and test phases, and running
benchmarks from multiple builds in an interleaved fashion to control
for environmental factors (ambient temperature, general system load etc).

New options:
* --build-only: Build tests without running them
* --test-prebuilt: Run tests from pre-built directory
* --build-dir: Specify build directory (used with --test-prebuilt)
* --exec-interleaved-builds: Comma-separated list of builds to interleave

Usage:

1. Build two compiler versions:
   lnt runtest test-suite --build-only \
     --sandbox /tmp/sandbox-a \
     --cc /path/to/clang-a \
     --test-suite ~/llvm-test-suite \
     ...

   lnt runtest test-suite --build-only \
     --sandbox /tmp/sandbox-b \
     --cc /path/to/clang-b \
     --test-suite ~/llvm-test-suite \
     ...

2. Run with interleaved execution:
   lnt runtest test-suite \
     --sandbox /tmp/results \
     --exec-interleaved-builds /tmp/sandbox-a/build,/tmp/sandbox-b/build \
     --exec-multisample 3

   This runs tests in the pattern:
   - Sample 0: build-a -> build-b
   - Sample 1: build-a -> build-b
   - Sample 2: build-a -> build-b

   Temporal interleaving controls for environmental changes that could
   bias results toward one build.

Or, test single build:
   lnt runtest test-suite --test-prebuilt \
     --build-dir /tmp/sandbox-a/build \
     --exec-multisample 5

Reports are written to each build directory (report.json,
test-results.xunit.xml, test-results.csv).
The --build-dir option now works with all build modes, not just
--test-prebuilt. It can be used to specify a custom build directory
location, overriding the default sandbox/build or timestamped directory.
Copy link
Author

This stack of pull requests is managed by Graphite. Learn more about stacking.

@aemerson aemerson closed this Oct 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant