Access to external state from Computable impls

[chklauser]

Maybe I missed it in the book, but what's the suggested approach for providing a reference to or accessing external state from within a Computable impl? Examples of state I can think of

• Configuration (e.g., which language features are enabled, things that might be command line options)
• Per-compilation-project caches (e.g., string interning)

Global state is always an answer, but it's not a good answer. While it works fine for CLI tools/batch compilers, it is not fun to manage for unit tests and servers.

[chklauser]

One way of doing it might be:

1. Create a global table ( Lazy<Table<Shared<Id, T>>> or lazy_static! { static ref MY_STATE : HashMap<Id, Arc<T>> = HashMap::new(); }
2. Create your per-compilation-project state let my_state = Arc::new(..) next to your analyzer;
3. Add document to analyzer (we receive a document ID in return)
4. Before you do anything else, add a reference to your shared state to the global table MY_STATE.insert(doc_id, my_state.clone());
   If you use multiple documents, the global table will hold multiple mappings that each point to the same shared state.
5. query the attrs that use the shared state
6. (Inside the Computable impl) query the global table with the doc id obtained from context.node_ref().id
7. After removing the document from the analyzer, remove the corresponding entry from the global table. Or when you drop the entire analyzer, remove the entries for all the documents managed by the analyzer

Given how document names are stored in a global table, I'm assuming that document IDs are globally unique and not re-used.

[Eliah-Lakhin]

@chklauser Well, the Analyzer object does not have built-in capabilities to enumerate or index documents because the file system mapping to documents is a job external to the analyzer's functional scope.

The current proposed way to reveal the mapping between the file system and the analyzer's documents is to manage a separate Lazy<Table<String, Id>> static state and keep it in sync with the documents.

Inside the computable function of the attribute, you can look up the document by, let's say, the name of the file (taken from, for example, a using MyClass; statement). Then, read the document using the read_doc function. This function automatically subscribes the attribute to the built-in event that triggers when the document is removed from the Analyzer, which simplifies synchronization with the external state. However, you can also subscribe to a custom user-defined event (e.g., context.subscribe(MY_FS_UPDATE_EVENT)) and then trigger it whenever the external state changes.

This approach should work fine for both compilers and language servers (recall that the LSP server, despite its name, typically manages just a single client), but binding the computable functions to the global static state is not good for testability.

I was thinking about introducing a more proper API to handle the external state but hesitated to include it in this release because the API is already quite complex. I wanted to hear your feedback first; maybe the global state is sufficient.

Alternatively, I could suggest the following API improvement.

In the language Node definition, you would define a type describing the external state's type for this language:

#[derive(Node)]
#[external(MyLangExtState)]
enum MyNode {
    //...
}

Then, in the mutation task, I will introduce a function to mutably access this state, and for the computable's context, a function to read this state (or a part of it) with automatic subscriptions to changes in the state.

The MyLangExtState object would merely be an analyzer-wide "attribute" with a value that you can read or write externally, or maybe a set of such attributes.

Given how document names are stored in a global table, I'm assuming that document IDs are globally unique and not re-used.

Yes, that's correct. You can rely on the uniqueness of document IDs within the current process.

[chklauser]

Lazy<Table<String, Id>> would probably need some kind of additional identifier so that I can have more than one instance of the analyzer per process. Lazy<Table<(InstanceId, String), Id>> or Lazy<Table<InstanceId, MyGlobalState>>. In general, I don't like solutions that involve global (static) state. It throws us back to 1998 era programming with manual lifetime management and leaks and dangling references and everything.

I wouldn't rush the API design here. Especially not just on my behalf. I think the API for the analysis framework is pretty cool (though I don't have enough experience with it to tell whether the automatic dependency inference works out in practice).

Things I like about your proposed API

1. The lifetime of MyLangExtState matches what I would want it to be (same as the analyzer)
2. External updates are integrated with the analyzer framework

Downsides I can see

1. MyLangExtState probably needs to be Default. It's not a deal-breaker, but for external input that is known before I create the analyzer (e.g., CLI options), its awkward to implement Default (the Default impl might not match the defaults that for instance Clap might initialize the struct to). Making such fields Optional is even more awkward.

I'm still wondering if there should be a global #[semantics] object. On the one hand, it sounds like a natural extension. On the other hand, it seems like an anti-pattern. As you say in the book, you'd want attribute dependencies to be as granular as possible. Global attributes that can change based on source code changes potentially invalidate many nodes.

[Eliah-Lakhin]

I don't like solutions that involve global (static) state. It throws us back to 1998 era programming with manual lifetime management and leaks and dangling references and everything.

Yes, I fully agree. I suggest it just as a temporary workaround that should be replaced with a more proper solution in the foreseeable future. This is an area where the current API could certainly be improved.

It's not a deal-breaker, but for external input that is known before I create the analyzer.

Perhaps this could be addressed by implementing an additional Analyzer constructor where one could set an initial value for the global semantics, or by using the current constructor if the object implements Default. There are more questions arising, and I need time to think about the new API design.

I'm still wondering if there should be a global #[semantics] object. On the one hand, it sounds like a natural extension. On the other hand, it seems like an anti-pattern.

I think it could become an anti-pattern if overused, but certain types of metadata (like file names and external configuration files) are naturally part of the semantic graph and don't belong in any document syntax tree. Perhaps this approach wouldn't contradict the granularity of the semantic graph in general. This metadata could be further distributed by the system of intermediary attributes, similar to how we handle scoped nodes' input attributes.

[Eliah-Lakhin]

Following our conversation, I opened Issue #15 with the proposed API changes.

[Eliah-Lakhin]

The feature is fully implemented.

For implementation details, see the Multi-File Analysis chapter of the User Guide, and issue #15.