Boost provider not providing CIDs and timing out with no error in response to ad entries

[mishmosh]

Checklist

• This is not a question or a support request. If you have any boost related questions, please ask in the discussion forum.
• This is not a new feature request. If it is, please file a feature request instead.
• This is not an enhancement request. If it is, please file a improvement suggestion instead.
• I have searched on the issue tracker and the discussion forum, and there is no existing related issue or discussion.
• I am running the Latest release, or the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
• I did not make any code changes to boost.

Boost component

• boost daemon - storage providers
• boost client
• boost UI
• boost data-transfer
• boost index-provider
• Other

Boost Version

(2 separate boost nodes)

Describe the Bug

Filecoin users are reporting some instances of being unable to publish to IPNI. In both cases, IPNI is making a request for CIDs and the server is not returning after 15 minutes in 6 successive tries. @willscott believes this is a bug in the boost provider.

If this long delay is intentional and working as intended, it is incompatible with IPNI design and needs to be jointly resolved somehow. If the server returns 404 or some error, IPNI indexer can progress with ingest one way or another. But getting no response combined with linear ingestion of ads means that the indexer cannot progress until that URL resolves (result or error). Otherwise we end up with gap in index.

We have received 2 reports in the #ipni channel on Filecoin Slack in the past week. It looks like there are ~15-20 providers that are in this same state per cid.contact/providers.

First report - 02 October 2025

(thread in #ipni channel)

@TippyFlitsUK is seeing mixed results over the last few days/months with network indexing from Boost node f08403.

Deals over the last week do not appear to be indexing, or are doing so very slowly (these are Lighthouse deals) and I still have a ton of deals from months ago that remain in indexing state (these are mostly Solana deals). I have added a recent history covering the last few months to the F08403 - IPNI Status Google sheet with open access.

IPNI is seeing repeated timeouts when fetching ads from that endpoint.

2025-10-03T07:49:16.907Z    ERROR    indexer/ingest    ingest/ingest.go:1152    Error while ingesting ad. Bailing early, not ingesting later ads.    {"publisher": "12D3KooWMdvJ2PT8Rph2UYUneXwGKFu5DfUNbmkZBziJweXV6Nws", "worker": 39, "adCid": "baguqeeral2z6tmsgw7lgtw6z3d4kmx5lukl2sqcwmwixtan5mmk2g4d4xqza", "err": "syncEntriesErr: failed to sync first entry while checking entries type: sync handler failed: failed to traverse requested dag: failed to load node for root cid baguqeera5i6p63o6yqudksi2gov7yzckyqp3n6svowbxnibcgrlorqirar7q: failed to fetch block for cid baguqeera5i6p63o6yqudksi2gov7yzckyqp3n6svowbxnibcgrlorqirar7q: fetch request failed: Get \"http://yablufc.ddns.net:3104/ipni/v1/ad/baguqeera5i6p63o6yqudksi2gov7yzckyqp3n6svowbxnibcgrlorqirar7q\": GET http://yablufc.ddns.net:3104/ipni/v1/ad/baguqeera5i6p63o6yqudksi2gov7yzckyqp3n6svowbxnibcgrlorqirar7q giving up after 6 attempt(s): Get \"http://yablufc.ddns.net:3104/ipni/v1/ad/baguqeera5i6p63o6yqudksi2gov7yzckyqp3n6svowbxnibcgrlorqirar7q\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "adsLeftToProcess": 9840}
2025-10-03T07:49:22.668Z    ERROR    indexer/admin    admin/handler.go:369    Cannot sync with peer    {"peerID": "12D3KooWMdvJ2PT8Rph2UYUneXwGKFu5DfUNbmkZBziJweXV6Nws", "err": "syncEntriesErr: failed to sync first entry while checking entries type: sync handler failed: failed to traverse requested dag: failed to load node for root cid baguqeera5i6p63o6yqudksi2gov7yzckyqp3n6svowbxnibcgrlorqirar7q: failed to fetch block for cid baguqeera5i6p63o6yqudksi2gov7yzckyqp3n6svowbxnibcgrlorqirar7q: fetch request failed: Get \"http://yablufc.ddns.net:3104/ipni/v1/ad/baguqeera5i6p63o6yqudksi2gov7yzckyqp3n6svowbxnibcgrlorqirar7q\": GET http://yablufc.ddns.net:3104/ipni/v1/ad/baguqeera5i6p63o6yqudksi2gov7yzckyqp3n6svowbxnibcgrlorqirar7q giving up after 6 attempt(s): Get \"http://yablufc.ddns.net:3104/ipni/v1/ad/baguqeera5i6p63o6yqudksi2gov7yzckyqp3n6svowbxnibcgrlorqirar7q\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}

Second report - 08 October 2025

(thread in #ipni channel)
User Edward creates an online deal on Filecoin. That deal was supposed to advertise payload CID on IPNI but the publish did not happen.

Original Filecoin deal: https://filfox.info/en/deal/129424823
CID: https://cid.contact/ipfs/bafybeibsszq36h6fv4k6r4a4rnrz4tguubsgyfdp4y2qzk3rprfwzcenra

IPNI sees this response:
'error while ingesting ad baguqeeravhwxgdnvcvjxb34qn4jhwohqtibycsbwi347r6p7at54aufjx7pa: syncEntriesErr: failed to sync first entry while checking entries type: sync handler failed: failed to traverse requested dag: failed to load node for root cid baguqeera6e6lo53vmew6v5bxmls77mk55s3r34q4qgtd2rbqqemzvrbrsmna: failed to fetch block for cid baguqeera6e6lo53vmew6v5bxmls77mk55s3r34q4qgtd2rbqqemzvrbrsmna: fetch request failed: Get "http://79.104.211.73:3402/ipni/v1/ad/baguqeera6e6lo53vmew6v5bxmls77mk55s3r34q4qgtd2rbqqemzvrbrsmna": GET http://79.104.211.73:3402/ipni/v1/ad/baguqeera6e6lo53vmew6v5bxmls77mk55s3r34q4qgtd2rbqqemzvrbrsmna giving up after 6 attempt(s): Get "http://79.104.211.73:3402/ipni/v1/ad/baguqeera6e6lo53vmew6v5bxmls77mk55s3r34q4qgtd2rbqqemzvrbrsmna": net/http: request canceled (Client.Timeout exceeded while awaiting headers)'

Logging Information

See error logs inline

Repo Steps

No response

[LexLuthr]

Update: We have shipped a fix #2032 which fixes the issue with queued IPNI ad generation in Boost. Unfortunately, after debugging on Slack with @TippyFlitsUK, it was found that Boost's current design is not able to read back the indexes fast enough from DB to answer the IPNI requests.

Fixing this requires considerable efforts and redesign of index-provider and partially Boost. As Boost is set to EOL, I don't think it makes sense to invest a month worth of engineering time and money to fix this problem when we already has a working solution in Curio.

Easier mitigation for this is to migrate over to Curio where architecture allows for handling very dense car files and this problem is already fixed.