TBDGen: Introduce option to emit API descriptor as supplementary output #68994
Conversation
An "API descriptor" file is JSON describing the externally accessible symbols of a module and metadata associated with those symbols like availability and SPI status. This output was previously only generated by the `swift-api-extract` alias of `swift-frontend`, which is desgined to take an already built module as input. Post-processing a built module to extract this information is inefficient because the module and the module's dependencies need to be deserialized in order to visit the entire AST. We can generate this output more efficiently as a supplementary output of the -emit-module job that originally produced the module (since the AST is already available in-memory). The -emit-api-descriptor flag can be used to request this output. This change lays the groundwork by introducing frontend flags. Follow up changes are needed to make API descriptor emission during -emit-module functional. Part of rdar://110916764.
…ule. Make the changes to APIGenRecorder that are necessary to make it capable of emitting API descriptors during -emit-module jobs. The output in this mode differs from the output when run on an existing module in a couple of important ways: - The value for the `file` key in the descriptor JSON is now the path to the source file that defines the declaration responsible for the symbol. In `swift-api-extract` mode, the value for this key is the path to the module or swiftinterface which is unavailable during an -emit-module job since the module is usually not being emitted to its final installed location. - Some additional symbols may be included in the API descriptor JSON because more of the AST is available when emitting the module. Resolves rdar://110916764
Instead, use the `%validate-json` lit substitution to validate and format the API descriptor file before running it through FileCheck. This allows us to avoid needing to introduce a dedicated frontend option just to control whether the output of -emit-api-descriptor is pretty printed.
tshortli
requested review from
artemcm,
hborla,
slavapestov and
xedin
as code owners
|
@swift-ci please test |
tshortli
requested review from
cachemeifyoucan,
nkcsgexi,
ributzka,
xymus,
artemcm,
edymtt and
zixu-w
and removed request for
xedin,
slavapestov,
artemcm and
hborla
There was a problem hiding this comment.
Thanks Allan! 🎉
Some additional symbols may be included in the API descriptor JSON because more of the AST is available when emitting the module.
Do you have ideas of roughly what are the kinds of these additional symbols that were not available from serializing the module? I see in the test case one additional symbol for method lookup function for a class.
Yeah, the method lookup function is the example that prompted me to call this out. I haven't identified other examples, but that was enough to indicate to me that there could also be other examples that the test cases don't demonstrate. |
| @@ -98,6 +98,7 @@ TYPE("pch", PCH, "pch", "") | |||
| TYPE("none", Nothing, "", "") | |||
|
|
|||
| TYPE("abi-baseline-json", SwiftABIDescriptor, "abi.json", "") | |||
| TYPE("api-json", SwiftAPIDescriptor, "", "") | |||
There was a problem hiding this comment.
The extension right now for partial SDKDB files is .sdkdb . Should this be listed in the extension field?
There was a problem hiding this comment.
I could not identify a reason it was important to specify an extension here. There are some utilities in the compiler to take a filename and look up its corresponding file_types entry using the contents of this table, but AFAIK, we don't need to do that with an API descriptor since the path to one is always specified explicitly in the arguments to the frontend. We could certainly add .sdkdb here but it feels slightly over-specified to me since ultimately this output is just JSON to the compiler, it doesn't otherwise know about SDKDB as a concept.
@artemcm @xymus @nkcsgexi Am I missing any reasons this needs to be specified?
There was a problem hiding this comment.
Yeah, I don't remember any specific reasons for being explicit about file extensions other than inferring output kinds. However, it seems to be good documentation, no?
There was a problem hiding this comment.
I think I named it originally partial.sdkdb for the intermediate JSON format. Up for a better name.
There was a problem hiding this comment.
I think given what I've heard that we do not need to encode the suffix of this file in the Swift compiler. From the compiler's point of view it's just another type of supplementary file that it knows how to write to a given path when requested, but the filename can be opaque to the compiler.
| // CHECK-SPI-NEXT: "linkage": "exported", | ||
| // CHECK-SPI-NEXT: "introduced": "10.10" | ||
| // CHECK-SPI-NEXT: }, |
There was a problem hiding this comment.
It is concerning when a public symbol disappears. Does the symbol _main exist in the binary file?
There was a problem hiding this comment.
_main was erroneously added before because the swift-frontend invocation did not specify -parse-as-library (normally a library would not have a _main entry point).
|
See swiftlang/swift-driver#1460 for the driver. |
An "API descriptor" file is JSON describing the externally accessible symbols of a module and metadata associated with those symbols like availability and SPI status. This output was previously only generated by the
swift-api-extractalias ofswift-frontend, which is designed to take an already built module as input. Post-processing a built module to extract this information is inefficient because the module and the module's dependencies need to be deserialized in order to visit the entire AST. We can generate this output more efficiently as a supplementary output of the-emit-modulejob that originally produced the module (since the AST is already available in-memory). The new-emit-api-descriptorflag can be used to request this output.The output of
-emit-api-descriptordiffers from the output ofswift-api-extractrun on an existing module in a couple of important ways:filekey in the descriptor JSON is now the path to the source file that defines the declaration responsible for the symbol. Inswift-api-extractmode, the value for this key is the path to the module or swiftinterface which is unavailable during an-emit-modulejob since the module is usually not being emitted to its final installed location.Resolves rdar://110916764