[clang-scan-deps] Infer the tool locations from PATH #108539
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This allows the clang driver to know which tool is meant to be executed, which allows the clang driver to load the right clang config files, and allows clang to find colocated sysroots. This makes sure that doing `clang-scan-deps -- <tool> ...` looks up things in the same way as if one just would execute `<tool> ...`, when `<tool>` isn't an absolute or relative path.
|
@llvm/pr-subscribers-clang Author: Martin Storsjö (mstorsjo) ChangesThis allows the clang driver to know which tool is meant to be executed, which allows the clang driver to load the right clang config files, and allows clang to find colocated sysroots. This makes sure that doing Full diff: /~https://github.com/llvm/llvm-project/pull/108539.diff 12 Files Affected:
diff --git a/clang/include/clang/Tooling/CompilationDatabase.h b/clang/include/clang/Tooling/CompilationDatabase.h
index fee584acb48623..36fe0812ebe974 100644
--- a/clang/include/clang/Tooling/CompilationDatabase.h
+++ b/clang/include/clang/Tooling/CompilationDatabase.h
@@ -234,6 +234,12 @@ std::unique_ptr<CompilationDatabase>
std::unique_ptr<CompilationDatabase>
inferTargetAndDriverMode(std::unique_ptr<CompilationDatabase> Base);
+/// Returns a wrapped CompilationDatabase that will transform argv[0] to an
+/// absolute path, if it currently is a plain tool name, looking it up in
+/// PATH.
+std::unique_ptr<CompilationDatabase>
+inferToolLocation(std::unique_ptr<CompilationDatabase> Base);
+
/// Returns a wrapped CompilationDatabase that will expand all rsp(response)
/// files on commandline returned by underlying database.
std::unique_ptr<CompilationDatabase>
diff --git a/clang/lib/Tooling/CMakeLists.txt b/clang/lib/Tooling/CMakeLists.txt
index 93a9e707a134cf..fc1f1f9f9d367e 100644
--- a/clang/lib/Tooling/CMakeLists.txt
+++ b/clang/lib/Tooling/CMakeLists.txt
@@ -25,6 +25,7 @@ add_clang_library(clangTooling
GuessTargetAndModeCompilationDatabase.cpp
InterpolatingCompilationDatabase.cpp
JSONCompilationDatabase.cpp
+ LocateToolCompilationDatabase.cpp
Refactoring.cpp
RefactoringCallbacks.cpp
StandaloneExecution.cpp
diff --git a/clang/lib/Tooling/LocateToolCompilationDatabase.cpp b/clang/lib/Tooling/LocateToolCompilationDatabase.cpp
new file mode 100644
index 00000000000000..033f69f3760c6d
--- /dev/null
+++ b/clang/lib/Tooling/LocateToolCompilationDatabase.cpp
@@ -0,0 +1,71 @@
+//===- GuessTargetAndModeCompilationDatabase.cpp --------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/Tooling/CompilationDatabase.h"
+#include "clang/Tooling/Tooling.h"
+#include "llvm/Support/Path.h"
+#include "llvm/Support/Program.h"
+#include <memory>
+
+namespace clang {
+namespace tooling {
+
+namespace {
+class LocationAdderDatabase : public CompilationDatabase {
+public:
+ LocationAdderDatabase(std::unique_ptr<CompilationDatabase> Base)
+ : Base(std::move(Base)) {
+ assert(this->Base != nullptr);
+ }
+
+ std::vector<std::string> getAllFiles() const override {
+ return Base->getAllFiles();
+ }
+
+ std::vector<CompileCommand> getAllCompileCommands() const override {
+ return addLocation(Base->getAllCompileCommands());
+ }
+
+ std::vector<CompileCommand>
+ getCompileCommands(StringRef FilePath) const override {
+ return addLocation(Base->getCompileCommands(FilePath));
+ }
+
+private:
+ std::vector<CompileCommand>
+ addLocation(std::vector<CompileCommand> Cmds) const {
+ for (auto &Cmd : Cmds) {
+ if (Cmd.CommandLine.empty())
+ continue;
+ std::string &Driver = Cmd.CommandLine.front();
+ // If the driver name already is absolute, we don't need to do anything.
+ if (llvm::sys::path::is_absolute(Driver))
+ continue;
+ // If the name is a relative path, like bin/clang, we assume it's
+ // possible to resolve it and don't do anything about it either.
+ if (llvm::any_of(Driver,
+ [](char C) { return llvm::sys::path::is_separator(C); }))
+ continue;
+ auto Absolute = llvm::sys::findProgramByName(Driver);
+ // If we found it in path, update the entry in Cmd.CommandLine
+ if (Absolute && llvm::sys::path::is_absolute(*Absolute))
+ Driver = std::move(*Absolute);
+ }
+ return Cmds;
+ }
+ std::unique_ptr<CompilationDatabase> Base;
+};
+} // namespace
+
+std::unique_ptr<CompilationDatabase>
+inferToolLocation(std::unique_ptr<CompilationDatabase> Base) {
+ return std::make_unique<LocationAdderDatabase>(std::move(Base));
+}
+
+} // namespace tooling
+} // namespace clang
diff --git a/clang/test/ClangScanDeps/modules-extern-submodule.c b/clang/test/ClangScanDeps/modules-extern-submodule.c
index 92f2c749dd2c30..196a9e695f66f7 100644
--- a/clang/test/ClangScanDeps/modules-extern-submodule.c
+++ b/clang/test/ClangScanDeps/modules-extern-submodule.c
@@ -112,7 +112,7 @@ module third {}
// CHECK: "-fmodule-map-file=[[PREFIX]]/first/first/module.modulemap",
// CHECK: "-fmodule-file=first=[[PREFIX]]/cache/{{.*}}/first-{{.*}}.pcm",
// CHECK: ],
-// CHECK-NEXT: "executable": "clang",
+// CHECK-NEXT: "executable": "{{.*}}clang",
// CHECK-NEXT: "file-deps": [
// CHECK-NEXT: "[[PREFIX]]/tu.m"
// CHECK-NEXT: ],
diff --git a/clang/test/ClangScanDeps/modules-full-output-tu-order.c b/clang/test/ClangScanDeps/modules-full-output-tu-order.c
index 04939826817fc1..794503d6507269 100644
--- a/clang/test/ClangScanDeps/modules-full-output-tu-order.c
+++ b/clang/test/ClangScanDeps/modules-full-output-tu-order.c
@@ -35,7 +35,7 @@
// CHECK: "-D"
// CHECK-NEXT: "ONE"
// CHECK: ],
-// CHECK-NEXT: "executable": "clang",
+// CHECK-NEXT: "executable": "{{.*}}clang",
// CHECK-NEXT: "file-deps": [
// CHECK-NEXT: "[[PREFIX]]/tu.c"
// CHECK-NEXT: ],
@@ -52,7 +52,7 @@
// CHECK: "-D"
// CHECK-NEXT: "TWO"
// CHECK: ],
-// CHECK-NEXT: "executable": "clang",
+// CHECK-NEXT: "executable": "{{.*}}clang",
// CHECK-NEXT: "file-deps": [
// CHECK-NEXT: "[[PREFIX]]/tu.c"
// CHECK-NEXT: ],
diff --git a/clang/test/ClangScanDeps/modules-has-include-umbrella-header.c b/clang/test/ClangScanDeps/modules-has-include-umbrella-header.c
index e9363b2e14b07a..78c5ffe11d2374 100644
--- a/clang/test/ClangScanDeps/modules-has-include-umbrella-header.c
+++ b/clang/test/ClangScanDeps/modules-has-include-umbrella-header.c
@@ -64,7 +64,7 @@ module Dependency { header "dependency.h" }
// CHECK: ],
// CHECK-NEXT: "command-line": [
// CHECK: ],
-// CHECK-NEXT: "executable": "clang",
+// CHECK-NEXT: "executable": "{{.*}}clang",
// CHECK-NEXT: "file-deps": [
// CHECK-NEXT: "[[PREFIX]]/tu.c"
// CHECK-NEXT: ],
diff --git a/clang/test/ClangScanDeps/modules-header-sharing.m b/clang/test/ClangScanDeps/modules-header-sharing.m
index ec94923ae8eeee..60fdf3aafa42d3 100644
--- a/clang/test/ClangScanDeps/modules-header-sharing.m
+++ b/clang/test/ClangScanDeps/modules-header-sharing.m
@@ -77,7 +77,7 @@
// CHECK: "-fmodule-map-file=[[PREFIX]]/frameworks/A.framework/Modules/module.modulemap",
// CHECK: "-fmodule-name=A",
// CHECK: ],
-// CHECK-NEXT: "executable": "clang",
+// CHECK-NEXT: "executable": "{{.*}}clang",
// CHECK-NEXT: "file-deps": [
// CHECK-NEXT: "[[PREFIX]]/tu.m",
// CHECK-NEXT: "[[PREFIX]]/shared/H.h"
diff --git a/clang/test/ClangScanDeps/modules-implementation-module-map.c b/clang/test/ClangScanDeps/modules-implementation-module-map.c
index d76d3157004699..d988faa35ab23d 100644
--- a/clang/test/ClangScanDeps/modules-implementation-module-map.c
+++ b/clang/test/ClangScanDeps/modules-implementation-module-map.c
@@ -27,7 +27,7 @@ framework module FWPrivate { header "private.h" }
// CHECK: "-fmodule-map-file=[[PREFIX]]/frameworks/FW.framework/Modules/module.private.modulemap",
// CHECK: "-fmodule-name=FWPrivate",
// CHECK: ],
-// CHECK-NEXT: "executable": "clang",
+// CHECK-NEXT: "executable": "{{.*}}clang",
// CHECK-NEXT: "file-deps": [
// CHECK-NEXT: "[[PREFIX]]/tu.m"
// CHECK-NEXT: ],
diff --git a/clang/test/ClangScanDeps/modules-implementation-private.m b/clang/test/ClangScanDeps/modules-implementation-private.m
index acc01017d66403..236e6ba9f0b783 100644
--- a/clang/test/ClangScanDeps/modules-implementation-private.m
+++ b/clang/test/ClangScanDeps/modules-implementation-private.m
@@ -62,7 +62,7 @@
// CHECK-NEXT: ],
// CHECK-NEXT: "command-line": [
// CHECK: ],
-// CHECK-NEXT: "executable": "clang",
+// CHECK-NEXT: "executable": "{{.*}}clang",
// CHECK-NEXT: "file-deps": [
// CHECK-NEXT: "[[PREFIX]]/tu.m",
// CHECK-NEXT: "[[PREFIX]]/frameworks/FW.framework/PrivateHeaders/Missed.h",
diff --git a/clang/test/ClangScanDeps/modules-priv-fw-from-pub.m b/clang/test/ClangScanDeps/modules-priv-fw-from-pub.m
index 4847fedac3bf6f..a7ac19b49431d3 100644
--- a/clang/test/ClangScanDeps/modules-priv-fw-from-pub.m
+++ b/clang/test/ClangScanDeps/modules-priv-fw-from-pub.m
@@ -110,7 +110,7 @@
// CHECK-NEXT: ],
// CHECK-NEXT: "command-line": [
// CHECK: ],
-// CHECK-NEXT: "executable": "clang",
+// CHECK-NEXT: "executable": "{{.*}}clang",
// CHECK-NEXT: "file-deps": [
// CHECK-NEXT: "[[PREFIX]]/tu.m"
// CHECK-NEXT: ],
diff --git a/clang/test/ClangScanDeps/resolve-executable-path.c b/clang/test/ClangScanDeps/resolve-executable-path.c
new file mode 100644
index 00000000000000..63e34ca256a8b4
--- /dev/null
+++ b/clang/test/ClangScanDeps/resolve-executable-path.c
@@ -0,0 +1,32 @@
+// UNSUPPORTED: system-windows
+
+// Check that we expand the executable name to an absolute path, when invoked
+// with a plain executable name, which is implied to be found in PATH.
+// REQUIRES: x86-registered-target
+
+// RUN: rm -rf %t
+// RUN: mkdir -p %t/bin
+// RUN: ln -s %clang %t/bin/x86_64-w64-mingw32-clang
+// RUN: split-file %s %t
+// RUN: sed -e "s|DIR|%/t|g" %t/cdb.json.in > %t/cdb.json
+
+// Check that we can deduce this both when using a compilation database, and when using
+// a literal command line.
+
+// RUN: env "PATH=%t/bin:%PATH%" clang-scan-deps -format experimental-full -compilation-database %t/cdb.json | FileCheck %s -DBASE=%/t
+
+// RUN: env "PATH=%t/bin:%PATH%" clang-scan-deps -format experimental-full -- x86_64-w64-mingw32-clang %t/source.c -o %t/source.o | FileCheck %s -DBASE=%/t
+
+// CHECK: "executable": "[[BASE]]/bin/x86_64-w64-mingw32-clang"
+
+//--- cdb.json.in
+[
+ {
+ "directory": "DIR"
+ "command": "x86_64-w64-mingw32-clang -c DIR/source.c -o DIR/source.o"
+ "file": "DIR/source.c"
+ },
+]
+
+//--- source.c
+void func(void) {}
diff --git a/clang/tools/clang-scan-deps/ClangScanDeps.cpp b/clang/tools/clang-scan-deps/ClangScanDeps.cpp
index 1db7245390f557..ac68e3605a10cb 100644
--- a/clang/tools/clang-scan-deps/ClangScanDeps.cpp
+++ b/clang/tools/clang-scan-deps/ClangScanDeps.cpp
@@ -815,6 +815,8 @@ int clang_scan_deps_main(int argc, char **argv, const llvm::ToolContext &) {
Compilations = inferTargetAndDriverMode(std::move(Compilations));
+ Compilations = inferToolLocation(std::move(Compilations));
+
// The command options are rewritten to run Clang in preprocessor only mode.
auto AdjustingCompilations =
std::make_unique<tooling::ArgumentsAdjustingCompilations>(
|
|
Can you explain why exactly is this necessary/beneficial? If the compilation database got created with just the tool name, I'd expect the rest of the build to be able to use just the tool name when the scanner spits it out again, no? If this is about the scanner finding the right config, sysroot, resource directory, etc., I'd expect that to already happen, since the scanner needs to be colocated with the toolchain itself (since |
It's about the scanner finding the right config files and sysroot. Even if Now when |
|
And practically - in the cases I've seen where CMake generates calls to But in separate testing of these tools, I ran into this situation, where |
|
Ok, thanks for the explanation. This LGTM, but the checks are failing on Windows due to |
Ah, thanks. Yeah it's probably best that we remove those checks that don't carry much relevance there. (The main test that really exercise this functionality, that I'm adding here, has a |
jansvoboda11
approved these changes
Sep 13, 2024
|
LLVM Buildbot has detected a new failure on builder Full details are available at: https://lab.llvm.org/buildbot/#/builders/13/builds/2197 Here is the relevant piece of the build log for the reference |
cachemeifyoucan
added a commit
to cachemeifyoucan/llvm-project
that referenced
this pull request
Sep 16, 2024
Fix clang path output in clang-scan-deps output after upstream change makes the tool to return full path to clang.
cachemeifyoucan
added a commit
to swiftlang/llvm-project
that referenced
this pull request
Sep 17, 2024
Fix clang path output in clang-scan-deps output after upstream change makes the tool to return full path to clang.
llvmbot
pushed a commit
to llvmbot/llvm-project
that referenced
this pull request
Sep 24, 2024
This allows the clang driver to know which tool is meant to be executed, which allows the clang driver to load the right clang config files, and allows clang to find colocated sysroots. This makes sure that doing `clang-scan-deps -- <tool> ...` looks up things in the same way as if one just would execute `<tool> ...`, when `<tool>` isn't an absolute or relative path. (cherry picked from commit a26ec54)
tru
pushed a commit
to llvmbot/llvm-project
that referenced
this pull request
Oct 1, 2024
This allows the clang driver to know which tool is meant to be executed, which allows the clang driver to load the right clang config files, and allows clang to find colocated sysroots. This makes sure that doing `clang-scan-deps -- <tool> ...` looks up things in the same way as if one just would execute `<tool> ...`, when `<tool>` isn't an absolute or relative path. (cherry picked from commit a26ec54)
cachemeifyoucan
added a commit
to cachemeifyoucan/llvm-project
that referenced
this pull request
Oct 3, 2024
Fix clang path output in clang-scan-deps output after upstream change makes the tool to return full path to clang. (cherry picked from commit 4b9ce22)
cachemeifyoucan
added a commit
to swiftlang/llvm-project
that referenced
this pull request
Oct 3, 2024
Fix clang path output in clang-scan-deps output after upstream change makes the tool to return full path to clang. (cherry picked from commit 4b9ce22)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This allows the clang driver to know which tool is meant to be executed, which allows the clang driver to load the right clang config files, and allows clang to find colocated sysroots.
This makes sure that doing
clang-scan-deps -- <tool> ...looks up things in the same way as if one just would execute<tool> ..., when<tool>isn't an absolute or relative path.