Add shadow block builder stack for testing in production #20
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Introduces a shadow block building stack that runs in parallel with the production sequencer for testing purposes. The builder syncs from production but does not submit blocks to L1, making it safe for testing TIPS bundle integration.
Changes:
- Add builder-cl and builder services to docker-compose.yml
- builder-cl: op-node in sequencer mode syncing from production via P2P
- builder: op-rbuilder instance that queries TIPS datastore and builds
blocks with eligible bundles (blocks are not submitted to L1)
- Add build-rbuilder command to justfile for building op-rbuilder Docker
image with TIPS datastore integration
- Consolidate op-node environment configuration into .env.example and use
env_file in docker-compose.yml to share common config between
simulator-cl and builder-cl
- Update justfile commands to support multiple compose profiles
- Rename start-playground to start-builder
- Update stop-all/start-all to accept comma-separated profiles
- Document simulator and shadow block builder stack in README.md with
prerequisites and quick start instructions
Replace BundleWithLatestSimulation struct with a simple tuple (BundleWithMetadata, Simulation) for select_bundles_with_latest_simulation return values. This eliminates awkward naming in consuming code by allowing direct destructuring instead of verbose field access patterns like `result.bundle_with_metadata.bundle.block_number`.
Introduces shadow-boost, an Engine API proxy that sits between a non-sequencer op-node and shadow builder. Intercepts forkchoiceUpdated to force block building while keeping op-node synced with canonical chain via P2P. Solves P2P block rejection and L1-triggered reorg issues that occur when running shadow builders in sequencer mode.
When op-node sends FCU without payload attributes (follower mode), shadow-boost now synthesizes attributes from the last newPayload to trigger continuous shadow building. Tracks payload info (timestamp, gas_limit, randao, etc.) and injects them with updated timestamps to maintain building cadence. Improves logging throughout proxy and server with structured fields and clearer messages.
…al sync handling Fixes two critical issues with synthetic payload attribute injection: 1. Timestamp skew: Changed synthetic timestamp calculation from SystemTime::now() to last_block_timestamp + 2, eliminating large time differences (300-700+ seconds) that caused "FCU arrived too late" errors in the shadow builder. 2. Historical P2P sync: Added 30-second age threshold to detect and skip synthetic attribute injection for historical blocks during P2P sync. Proxy still forwards FCU to builder for chain state updates (head/safe/finalized) without triggering unnecessary block building. Changes: - Use chain-relative timestamps instead of wall clock time - Calculate block_age_seconds to detect historical sync - Skip synthetic attributes for blocks >30s old - Always forward FCU to builder (with or without attributes) - Log block age for monitoring and debugging - Suppress warnings when intentionally skipping builds Results: - Eliminated FCU timing errors during normal operation - Prevented unnecessary shadow building during catchup - Maintained proper chain state in builder during historical sync - Reduced resource usage during P2P sync
…injection Refactor shadow-boost to a minimal pass-through proxy that logs all Engine API traffic and injects synthetic payload attributes to trigger shadow building in non-sequencer mode. Key changes: - Switch to generic JSON Value handling for all Engine API methods - Remove response modification (except payload_id suppression for injected attrs) - Remove request modification of existing payload attributes - Remove async newPayload forwarding, historical sync detection, duplicate detection, and shadow block fetching - Store parent_beacon_block_root in LastPayloadInfo for synthetic attributes Behavior: - FCU without payload attributes: inject synthetic attributes from last newPayload to trigger shadow building, suppress payload_id in response - FCU with payload attributes: pass through unchanged - All other methods: pass through with request/response logging
wlawt
reviewed
Oct 16, 2025
| /// so op-node doesn't know shadow building occurred | ||
| #[derive(Clone)] | ||
| pub struct ShadowBuilderProxy { | ||
| pub builder_client: Arc<RwLock<HttpClient>>, |
There was a problem hiding this comment.
do we need a lock here? is the client endpoint expected to change frequently?
|
|
||
| 3. **Parallel Building**: The builder produces blocks in parallel with the canonical sequencer, but these blocks are never sent back to the op-node. | ||
|
|
||
| 4. **No Reorgs**: The op-node follows the canonical chain via P2P and L1 derivation normally, while the builder independently produces shadow blocks for comparison/analysis. |
There was a problem hiding this comment.
qq: is the shadow builder building the latest block, comparing, and then immediately discarding?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Introduces a shadow block building stack that runs in parallel with the production sequencer for testing purposes. The builder syncs from production but does not submit blocks to L1, making it safe for testing TIPS bundle integration.
Changes: