[amd-spirv-backend][Driver] Develop opt-in pipeline for AMDGPU

[lamb-j]

Currently, when --offload-arch=amdgcnspirv is specified, the clang driver first generates LLVM IR (bitcode), and then translates to SPIRV via the llvm-spirv translator

However, clang also supports generating SPIRV directly via the SPIRV backend.

We should add a clang option (off by default for now) and associated implementation to enable switching from the llvm-spirv translator to the SPIRV backend when users specify --offload-arch=amdgcnspirv.

This option and implementation should land in upstream llvm-project.

[jmmartinez]

Just some small cleanup PR before modifying the backend to call llc directly: llvm#162096

[jmmartinez]

Update from my side, week 41.

Since our last meeting I've worked on:

• adding a test documenting the backend pipeline for SPIRV
• add a get-pass-name method for the pass that did not implemented it
• For spirv, we use -fembed-bitcode-marker to embed the command line. This also creates a [0 x i8] variable for the marker. SPIRV's backend cannot handle [0 x i8] arrays (at least for kernel code, it's supported for shaders). A solution I explored is to extend the variable to be [1 x i8] (in [SPIRV] Support -fembed-bitcode=marker for non-shader modules llvm/llvm-project#162082), since the variable is a placeholder anyways, its content should not matter much. This variable is appended to @llvm.compiler.used and the address-spaces of the pointers in these variables are not what I expected, and the helper llvm::appendToCompilerUsed was leading to a crash. I closed the PR since I needed to gather to be sure what was going on first.
  • It seems that the type of @llvm.compiler.used/@llvm.used is incoherent between the frontend and the middle-end. llvm::appendToCompilerUsed expects the type to be an array of ptr addrspace(0) and the frontend emits ptr addrspace(default-language-as) which is generic addrspace(4) for spirv. Then, I filed a PR to document this crash and start harmonizing this [NFC][LLVM] Use @llvm.compiler.used/@llvm.used helpers in Clang and BitcodeWriter llvm/llvm-project#162660.
    • Side quest unblocked: "harmonizing the type llvm.*used":
      • First make the frontend set the element type of the llvm.*used variables to ptr addrspace(0) and see what it breaks.
      • Add a check to the llvm-ir verifier that enforces this.
  • I found that the SPIRV backend was lowering @llvm.compiler.used so I filed [SPIRV] Do not emit @llvm.compiler.used llvm/llvm-project#162678 .
• Landed [HIP][SPIRV] Create an intermediate file for the llvm-link output llvm/llvm-project#162096 before Manuel's [HIP][SPIRV] Enable the SPIRV backend instead of the translator through an experimental flag. llvm/llvm-project#162282
• The option -fembed-bitcode-marker doesn't take into account other -fembed-bitcode=... that could be passed. It's ok since we really do not expect the users to pass -fembed-bitcode=.... Currently, the -fembed-bitcode-marker option shadows any -fembed-bitcode=... passed by the user. It'd be better to print an error but... If we pass supported -fembed-bitcode=all, the @llvm.embeded.module variable would have a bigger than zero size which allows us to bypass the [0 x i8] issue and have something testeable. Thus, the horrible proposal in [HIP][SPIRV] Use OPT_fembed_bitcode_EQ instead of the alias OPT_fembed_bitcode_marker llvm/llvm-project#162258 .

[mgcarrasco]

I've worked on:

• [HIP][SPIRV] Enable the SPIRV backend instead of the translator through an experimental flag. llvm/llvm-project#162282
  • At the time of writing, the only pending review is about the new command-line option name.
• [NFC] Replace DenseMap with switch and fix build issues. SPIRV-LLVM-Translator#20
  • Fixes build error

[mgcarrasco]

Update on the new compilation path to use the SPIRV backend instead of the translator via an option. The new compilation path has to work for the new and old driver both available in upstream toggled via --new-offload-driver.

Old driver:

1. Support for the old driver has to be improved based on feedback in PR.

New driver:

1. Detected and fixed crash in the new driver in PR.
2. Opened refactoring PR to ease the landing of our required changes for the new driver.
3. Noticed that the new driver is not calling the SPIRV translator when compiling HIP without --cuda-device-only (upstream and prior to our changes).
   1. This looks like missing support. It does call the translator when --cuda-device-only is set. I've pinged Joseph about it.
4. Implemented our new compilation path, not yet under review because I found some already existing issues. Hopefully, this should be aligned with the reviews we received.
   1. It is working when --cuda-device-only is not set.
   2. We don't have a dependency with spirv-as or spriv-link and -S emits textual SPIRV. -save-temps also works.
   3. When --cuda-device-only is set, clang-link-wrapper is failing. This step is not called in the old driver at all. I've also asked Joseph about this to learn more.

---

Update: Joseph confirmed that it is very likely that this is indeed missing support in the new driver. So, we will have to implement it eventually.