Runtime windows verticals failing to build in the VMR #4335

ViktorHofer · 2024-04-17T15:41:48Z

CSC : error CS0016: Could not write to output file 'D:\a_work\1\s\src\runtime\artifacts\obj\cdacreader\Release\net9.0\win-x64\generated\Microsoft.Interop.ComInterfaceGenerator\Microsoft.Interop.ComClassGenerator\Microsoft.Diagnostics.DataContractReader.SOSDacImpl.cs' -- 'Could not find a part of the path 'D:\a_work\1\s\src\runtime\artifacts\obj\cdacreader\Release\net9.0\win-x64\generated\Microsoft.Interop.ComInterfaceGenerator\Microsoft.Interop.ComClassGenerator\Microsoft.Diagnostics.DataContractReader.SOSDacImpl.cs'.' [D:\a_work\1\s\src\runtime\src\native\managed\cdacreader\src\cdacreader.csproj]
CSC : error CS0016: Could not write to output file 'D:\a_work\1\s\src\runtime\artifacts\obj\cdacreader\Release\net9.0\win-x64\generated\Microsoft.Interop.ComInterfaceGenerator\Microsoft.Interop.ComInterfaceGenerator\Microsoft.Diagnostics.DataContractReader.ISOSDacInterface9.cs' -- 'Could not find a part of the path 'D:\a_work\1\s\src\runtime\artifacts\obj\cdacreader\Release\net9.0\win-x64\generated\Microsoft.Interop.ComInterfaceGenerator\Microsoft.Interop.ComInterfaceGenerator\Microsoft.Diagnostics.DataContractReader.ISOSDacInterface9.cs'.' [D:\a_work\1\s\src\runtime\src\native\managed\cdacreader\src\cdacreader.csproj]

https://dnceng-public.visualstudio.com/public/_build/results?buildId=646322&view=logs&j=9050e078-31bf-5111-d8ec-8b6fa95caf9c&t=4df1dbeb-294c-5d6e-97b8-581fe506c270

@elinor-fung

dotnet-issue-labeler · 2024-04-17T15:41:52Z

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

jeffschwMSFT · 2024-04-17T16:17:57Z

this is also impacting dotnet/runtime official build (though is non-deterministic)

https://dnceng.visualstudio.com/internal/_build/results?buildId=2430979&view=logs&j=bdec34cc-8c67-558e-657b-d36a4105119f&t=57c2fa6d-d2d0-5d36-e636-8657b10bc2de&l=599

elinor-fung · 2024-04-17T21:38:47Z

One thing I see between a working dotnet-runtime-official build and a failing one - for example 20240415.10 failure (binlog) versus
20240417.11 success (binlog) - is that the failing one has additional MSBuild command line arguments -p:UseSharedCompilation=false -p:EmitCompilerGeneratedFiles=true.

The build logs from the dotnet-unified-build 20240416.1 failure also shows that -p:EmitCompilerGeneratedFiles=true was passed on the command line.

The files in the error messages should only be written to disk by the source generator when EmitCompilerGeneratedFiles is true (defaults to false). I don't know what is (was?) including that property when invoking the build.

I'm not sure what the underlying cause is either. The cdacreader project is just using the COM source generator and those paths aren't long enough to be hitting any long path problems with source generators.

elinor-fung · 2024-04-17T21:52:30Z

@chsienki @dotnet/roslyn Any idea what could be leading to CS0016 during source generation with EmitCompilerGeneratedFiles=true? The path in the error message is under 256 characters, so I don't think it is hitting long path problems.

CSC : error CS0016: Could not write to output file 'D:\a_work\1\s\src\runtime\artifacts\obj\cdacreader\Release\net9.0\win-x64\generated\Microsoft.Interop.ComInterfaceGenerator\Microsoft.Interop.ComClassGenerator\Microsoft.Diagnostics.DataContractReader.SOSDacImpl.cs' -- 'Could not find a part of the path 'D:\a_work\1\s\src\runtime\artifacts\obj\cdacreader\Release\net9.0\win-x64\generated\Microsoft.Interop.ComInterfaceGenerator\Microsoft.Interop.ComClassGenerator\Microsoft.Diagnostics.DataContractReader.SOSDacImpl.cs'.'

Previous comment #4335 (comment) has links to binlogs.

RikkiGibson · 2024-04-17T22:12:04Z

Could not find a part of the path makes it sound like the output directory is not present. But the compiler and compiler tasks should create the directory (both the /generated directory and the individual generator output directories within it).

/~https://github.com/dotnet/roslyn/blob/d5b9aaa7e984890c9a8d967bde68d1c62d4858fa/src/Compilers/Core/MSBuildTask/Microsoft.Managed.Core.targets#L339-L340

/~https://github.com/dotnet/roslyn/blob/d5b9aaa7e984890c9a8d967bde68d1c62d4858fa/src/Compilers/Core/Portable/CommandLine/CommonCompiler.cs#L1173-L1176

However it looks like the task linked above didn't run in the cdacreader build. binlog says:

Target Name=CreateCompilerGeneratedFilesOutputPath Project=cdacreader.csproj
Target "CreateCompilerGeneratedFilesOutputPath" skipped, due to false condition; ('$(EmitCompilerGeneratedFiles)' == 'true' and !('$(DesignTimeBuild)' == 'true' OR '$(BuildingProject)' != 'true')) was evaluated as ('true' == 'true' and !('' == 'true' OR 'false' != 'true')).

I'm confused by this :) as 'true' == 'true' and !('' == 'true' OR 'false' != 'true') seems like it ought to evaluate to ~~true~~. false, my conditional logic brain wasn't working. I think the BuildingProject property should be true instead of false in order for the directory to be created here. What controls that?

I think maybe the compiler task has a bug here. Basically it feels like we should only be passing a /generatedfilesout argument to the compiler if we also ran target CreateCompilerGeneratedFilesOutputPath. But it looks like the current build task permits us to pass that argument without running the target.

chsienki · 2024-04-17T22:28:25Z

@RikkiGibson Lol I just spent like a genuine 5 minutes working through the logic of that condition, then did a git blame to see who wrote it and well, well we meet again past Chris.

Something in the authoring is off definitely, but it also seems odd that the compiler still tries to write the files out, even if the directory isn't there, so we should probably file a bug on Roslyn for that.

RikkiGibson · 2024-04-17T22:44:53Z

Opened dotnet/roslyn#73075

elinor-fung · 2024-04-18T17:38:06Z

The files in the error messages should only be written to disk by the source generator when EmitCompilerGeneratedFiles is true (defaults to false). I don't know what is (was?) including that property when invoking the build.

@ViktorHofer do you have any idea how to find out what would (sometimes) be passing this property? I'm not finding it anywhere in our yml or MSBuild logic for passing arguments when building the runtime. The binlogs are showing that it ends up as a command line argument for some build jobs, but not others (in the same build). I have no idea where to look to figure out where it is coming from.

dotnet-runtime-official

20240418.6 success

window-x64 release NativeAOT binlog
- does not have EmitCompilerGeneratedFiles set on the command line
same for other jobs/binlogs - EmitCompilerGeneratedFiles is not set as a command line argument

20240418.3 failure

windows-x64 release NativeAOT binlog
- has -p:EmitCompilerGeneratedFiles=true set on the command line
- the build step does not show it being explicitly passed when calling build.cmd
```
Script contents: shell
D:\a\_work\1\s\build.cmd -ci -arch x64 -os windows  -s clr.nativeaotlibs+clr.nativeaotruntime+libs+packs -c Release /p:BuildNativeAOTRuntimePack=true /p:SkipLibrariesNativeRuntimePackages=true /p:OfficialBuildId=20240418.3 
```
- other jobs also hit the same failure and their binlogs also show -p:EmitCompilerGeneratedFiles=true as a command line argument
- other jobs succeeded and their binlogs show that EmitCompilerGeneratedFiles was not set as a command line argument - for example, windows-arm64 release NativeAOT (binlog)

20240415.10 failure

windows-arm64 release NativeAOT binlog
- has -p:EmitCompilerGeneratedFiles=true set on the command line
- the build step does not show it being explicitly passed when calling build.cmd
```
Script contents: shell
D:\a\_work\1\s\build.cmd -ci -arch arm64 -os windows  -s clr.nativeaotlibs+clr.nativeaotruntime+libs+packs -c Release /p:BuildNativeAOTRuntimePack=true /p:SkipLibrariesNativeRuntimePackages=true /p:OfficialBuildId=20240415.10 
```
- again, for the same build, other jobs succeeded and their binlogs show that EmitCompilerGeneratedFiles was not set as a command line argument - for example, windows-x64 release NativeAOT job (binlog)

dotnet-unified-build

Looking at the logs at src\runtime\artifacts\buildLogs\log\Release\Build.binlog

20240417.2 success - win x64 logs

does not have EmitCompilerGeneratedFiles set on the command line

20240416.1 failure - win x64 logs

has EmitCompilerGeneratedFiles=true set on the command line

20240415.1 success - win x64 logs

does not have EmitCompilerGeneratedFiles set on the command line

akoeplinger · 2024-04-18T18:18:47Z

I think this is being injected by CodeQL. In both of the failing builds the 🛡 CodeQL Initialize step took about 40 seconds instead of 6s and it didn't print the There is no need to run CodeQL at this moment... so pretty sure it is active.

akoeplinger · 2024-04-18T18:22:12Z

Indeed: /~https://github.com/github/codeql/blob/a108fcd2b49eec5dacde7636cf1c781b04ee5aa9/csharp/tools/tracing-config.lua#L117-L121

akoeplinger · 2024-04-18T18:23:40Z

@jaredpar another case where CodeQL is messing with us :)

elinor-fung · 2024-04-18T23:19:19Z

The reason this is only being hit by cdacreader is because when we build that project as a NativeAOT-ed runtime component, we run the LinkNative target rather than Build or Publish (like the other projects in runtime that use source generators).
LinkNative depends on [Core]Compile, but not [Core]Build. This means the BuildOnlySettings target that sets BuildingProject=true doesn't run. So we hit dotnet/roslyn#73075 where CreateCompilerGeneratedFilesOutputPath doesn't run and the compiler errors trying to emit files.

elinor-fung · 2024-04-19T05:15:23Z

dotnet/runtime#101279 should work around the failure by always creating that output path when EmitCompilerGeneratedFiles=true

…01279) When we build `cdacreader`, we run the `LinkNative` target. This depends on `Compile` not `Build`, which means the `BuildOnlySettings` target that sets `BuildingProject=true` doesn't run. When `EmitCompilerGeneratedFiles=true`, this results in the generated output path not being created and the compiler erroring when trying to emit files to the path. See dotnet/roslyn#73075. CodeQL injects `-p:EmitCompilerGeneratedFiles=true` into the command line arguments. This was resulting in build failures in jobs where CodeQL is run. This change adds a workaround to our infrastructure for building NativeAOT-ed runtime components such that the generated output path is always created if `EmitCompilerGeneratedFiles=true`. Manually tested by explicitly passing that property on the command line. Fixes dotnet/source-build#4335

…tnet#101279) When we build `cdacreader`, we run the `LinkNative` target. This depends on `Compile` not `Build`, which means the `BuildOnlySettings` target that sets `BuildingProject=true` doesn't run. When `EmitCompilerGeneratedFiles=true`, this results in the generated output path not being created and the compiler erroring when trying to emit files to the path. See dotnet/roslyn#73075. CodeQL injects `-p:EmitCompilerGeneratedFiles=true` into the command line arguments. This was resulting in build failures in jobs where CodeQL is run. This change adds a workaround to our infrastructure for building NativeAOT-ed runtime components such that the generated output path is always created if `EmitCompilerGeneratedFiles=true`. Manually tested by explicitly passing that property on the command line. Fixes dotnet/source-build#4335

github-project-automation bot added this to .NET Source Build Apr 17, 2024

github-project-automation bot moved this to Backlog in .NET Source Build Apr 17, 2024

dotnet-issue-labeler bot added the untriaged label Apr 17, 2024

ViktorHofer added the area-unified-build label Apr 17, 2024

ViktorHofer removed this from .NET Source Build Apr 17, 2024

ViktorHofer added this to .NET Unified Build Apr 17, 2024

akoeplinger changed the title ~~Windows x64 verticals failing to build in the VMR~~ Windows verticals failing to build in the VMR Apr 17, 2024

github-project-automation bot added this to .NET Source Build Apr 17, 2024

github-project-automation bot moved this to Backlog in .NET Source Build Apr 17, 2024

RikkiGibson mentioned this issue Apr 17, 2024

Compiler task is passing /generatedfilesout even though it didn't create the directory dotnet/roslyn#73075

Open

ViktorHofer changed the title ~~Windows verticals failing to build in the VMR~~ Runtime windows verticals failing to build in the VMR Apr 18, 2024

elinor-fung mentioned this issue Apr 19, 2024

Workaround for build failure in cdacreader when EmitCompilerGeneratedFiles=true dotnet/runtime#101279

Merged

akoeplinger closed this as completed in dotnet/runtime#101279 Apr 19, 2024

github-project-automation bot moved this from Backlog to Done in .NET Source Build Apr 19, 2024

github-project-automation bot moved this to Done in .NET Unified Build Apr 19, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Runtime windows verticals failing to build in the VMR #4335

Runtime windows verticals failing to build in the VMR #4335

ViktorHofer commented Apr 17, 2024

dotnet-issue-labeler bot commented Apr 17, 2024

jeffschwMSFT commented Apr 17, 2024

elinor-fung commented Apr 17, 2024 •

edited

Loading

elinor-fung commented Apr 17, 2024 •

edited

Loading

RikkiGibson commented Apr 17, 2024 •

edited

Loading

chsienki commented Apr 17, 2024

RikkiGibson commented Apr 17, 2024

elinor-fung commented Apr 18, 2024

akoeplinger commented Apr 18, 2024

akoeplinger commented Apr 18, 2024

akoeplinger commented Apr 18, 2024 •

edited

Loading

elinor-fung commented Apr 18, 2024

elinor-fung commented Apr 19, 2024

Runtime windows verticals failing to build in the VMR #4335

Runtime windows verticals failing to build in the VMR #4335

Comments

ViktorHofer commented Apr 17, 2024

dotnet-issue-labeler bot commented Apr 17, 2024

jeffschwMSFT commented Apr 17, 2024

elinor-fung commented Apr 17, 2024 • edited Loading

elinor-fung commented Apr 17, 2024 • edited Loading

RikkiGibson commented Apr 17, 2024 • edited Loading

chsienki commented Apr 17, 2024

RikkiGibson commented Apr 17, 2024

elinor-fung commented Apr 18, 2024

dotnet-runtime-official

dotnet-unified-build

akoeplinger commented Apr 18, 2024

akoeplinger commented Apr 18, 2024

akoeplinger commented Apr 18, 2024 • edited Loading

elinor-fung commented Apr 18, 2024

elinor-fung commented Apr 19, 2024

elinor-fung commented Apr 17, 2024 •

edited

Loading

elinor-fung commented Apr 17, 2024 •

edited

Loading

RikkiGibson commented Apr 17, 2024 •

edited

Loading

akoeplinger commented Apr 18, 2024 •

edited

Loading