[CAS] gmodule support for caching build #11026

cachemeifyoucan · 2025-07-18T18:58:34Z

Prototype for gmodule support by switching splitDwarf references to CASIDs of the module/PCH

Also teach dsymutil to support reading from CAS as a prototype for lldb support.

adrian-prantl · 2025-08-06T19:37:03Z

lldb/scripts/clang-explicit-module-build.py

This would potentially fit better into llvm-project/lldb/packages/Python/lldbsuite/

This is a compiler launcher type of script and requires a just built clang (because it needs clang-scan-deps next to it). I don't what is a good way to hack into lldbsuite or driven from makefile. Let me know if you have any good idea.

cachemeifyoucan · 2025-09-23T17:02:01Z

@adrian-prantl @benlangmuir Ping for review

clang/include/clang/Basic/Module.h

clang/lib/Serialization/ASTReader.cpp

cachemeifyoucan · 2025-09-23T19:59:23Z

@swift-ci please test llvm

cachemeifyoucan · 2025-09-23T20:12:45Z

@swift-ci please test llvm

cachemeifyoucan · 2025-09-29T17:08:36Z

@swift-ci please test llvm

clang/lib/CodeGen/CGDebugInfo.cpp

llvm/tools/dsymutil/Options.td

benlangmuir

LGTM for the clang changes. Someone else should review the LLDB and dsymutil code.

adrian-prantl · 2025-10-16T20:59:34Z

clang/test/ClangScanDeps/gmodules.c

+/// Check debug info is correct.
+// RUN: %clang %t/tu.o -o %t/a.out
+// RUN: dsymutil -cas %t/cas %t/a.out -o %t/a.dSYM 2>&1 | FileCheck %s --check-prefix=WARN --allow-empty
+// RUN: dsymutil %t/a.out -o %t/a2.dSYM 2>&1 | FileCheck %s --check-prefix=WARN --allow-empty


Does this being dsymutil and not %dsymutil mean we might pick up the system one? Since there is also a dsymutil aspect to this, could that be problematic in CI?

Tools are always in the path.

NOTE: this is not the only test that calls dsymutil in clang tests and dsymutil wasn't even a build dependency for tests (added in this PR).

adrian-prantl · 2025-10-16T21:01:11Z

llvm/tools/dsymutil/Options.td

+
+def cas : Separate<["-", "--"], "cas">,
+  MetaVarName<"<path>">,
+  HelpText<"Specify CAS directory to load CAS references from.">,


Does this imply that there can only be one CAS per build?
Is that realistic? What if people use static archives and combine objects from different projects?

Yes, it has to be one CAS per build. It is a requirement for module imports and there is no way you can configure otherwise.

If you have multiple builds and uses some products from other build, the second CAS has to ingest everything from previous builds.

adrian-prantl · 2025-10-16T21:02:24Z

llvm/tools/dsymutil/BinaryHolder.cpp

+  WithColor::note() << "create CAS using configuration: " << Config->first
+                    << "'\n";
+  return DB->first;
+}


It would be good to find a way to mock this up in a dsymutil-only test that doesn't depend on clang.

Let me know if you have good ways to do it. It is very hard to do it since it needs to construct both object files (which can be committed) but also module files and CAS contents.

adrian-prantl · 2025-10-16T21:05:05Z

lldb/source/Core/ModuleList.cpp

+// START CAS
+FileSpec ModuleListProperties::GetCASOnDiskPath() const {
+  const uint32_t idx = ePropertyCASOnDiskPath;
+  return GetPropertyAtIndexAs<FileSpec>(idx, {});


How would these settings be communicated to LLDB in the typical use-case?
What if a debug session contains multiple dylibs built against different CAS storage? (I.e., in each project's DerivedData/)

See ConfigureCASStorage().

This option provides a global overwrite. If no global setting available, it will search for a CAS configuration from the path of the debugging binary upwards to find a CAS configuration file.

Teach clang to encode CASID as splitDwarfFilename for gmodule when clang caching is enabled. This allows the outputs from compiler do not contain paths to the clang module files, thus allows distributed caching without the need of a unified clang module cache directory path.

Teach dsymutil how to use CAS and how to load clang modules from CAS when building dSYM when gmodule is used.

Teach lldb to load clang modules when gmodule + clang caching is used.

Teach SwiftASTContext to load clang module dependencies from CAS instead of FileSystem.

cachemeifyoucan · 2025-10-28T02:52:10Z

Create PR on stable branch (so test is available) if you like to review over there: #11714

cachemeifyoucan requested review from adrian-prantl, akyrtzi, benlangmuir and jansvoboda11 July 18, 2025 18:58

cachemeifyoucan force-pushed the casify-module branch 2 times, most recently from 2bbc405 to 075be6d Compare July 31, 2025 18:32

cachemeifyoucan force-pushed the casify-module branch from 075be6d to 2a1ffca Compare August 5, 2025 22:38

cachemeifyoucan marked this pull request as ready for review August 5, 2025 22:39

cachemeifyoucan requested a review from JDevlieghere as a code owner August 5, 2025 22:39

cachemeifyoucan force-pushed the casify-module branch from 2a1ffca to d5d9a15 Compare August 6, 2025 18:57

adrian-prantl reviewed Aug 6, 2025

View reviewed changes

cachemeifyoucan force-pushed the casify-module branch from d5d9a15 to 0173e49 Compare August 8, 2025 20:49

cachemeifyoucan mentioned this pull request Aug 8, 2025

Update swift compiler after clang gmodule CAS build update swiftlang/swift#83621

Open

cachemeifyoucan force-pushed the casify-module branch 2 times, most recently from 8063aee to 512a886 Compare September 2, 2025 16:04

cachemeifyoucan force-pushed the casify-module branch from 512a886 to cb036bf Compare September 23, 2025 17:01

benlangmuir reviewed Sep 23, 2025

View reviewed changes

clang/include/clang/Basic/Module.h Outdated Show resolved Hide resolved

clang/lib/Serialization/ASTReader.cpp Outdated Show resolved Hide resolved

cachemeifyoucan force-pushed the casify-module branch from cb036bf to 4ae6ba2 Compare September 23, 2025 19:58

cachemeifyoucan force-pushed the casify-module branch from 4ae6ba2 to 5e96cdf Compare September 23, 2025 20:12

cachemeifyoucan force-pushed the casify-module branch 4 times, most recently from 479c33e to 5c6f640 Compare September 29, 2025 17:08

cachemeifyoucan requested review from adrian-prantl and benlangmuir September 29, 2025 20:07

benlangmuir reviewed Oct 3, 2025

View reviewed changes

clang/lib/CodeGen/CGDebugInfo.cpp Outdated Show resolved Hide resolved

llvm/tools/dsymutil/Options.td Show resolved Hide resolved

cachemeifyoucan force-pushed the casify-module branch 2 times, most recently from 11bba2c to 55c97e3 Compare October 6, 2025 22:13

cachemeifyoucan requested a review from benlangmuir October 6, 2025 23:29

benlangmuir approved these changes Oct 6, 2025

View reviewed changes

adrian-prantl reviewed Oct 16, 2025

View reviewed changes

cachemeifyoucan added 4 commits October 16, 2025 14:24

[dsymutil] Teach dsymutil about CAS and loading clang modules from CAS

36a921b

Teach dsymutil how to use CAS and how to load clang modules from CAS when building dSYM when gmodule is used.

[lldb] Teach LLDB to load gmodules from CAS

0d3f967

Teach lldb to load clang modules when gmodule + clang caching is used.

[lldb][SwiftASTContext] Teach clang module loading from CAS

a383255

Teach SwiftASTContext to load clang module dependencies from CAS instead of FileSystem.

cachemeifyoucan force-pushed the casify-module branch from 55c97e3 to a383255 Compare October 16, 2025 21:32

[CAS] gmodule support for caching build #11026

Are you sure you want to change the base?

[CAS] gmodule support for caching build #11026

Uh oh!

Conversation

cachemeifyoucan commented Jul 18, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cachemeifyoucan commented Sep 23, 2025

Uh oh!

Uh oh!

Uh oh!

cachemeifyoucan commented Sep 23, 2025

Uh oh!

cachemeifyoucan commented Sep 23, 2025

Uh oh!

cachemeifyoucan commented Sep 29, 2025

Uh oh!

Uh oh!

Uh oh!

benlangmuir left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cachemeifyoucan Oct 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cachemeifyoucan commented Oct 28, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

cachemeifyoucan Oct 16, 2025 •

edited

Loading