default `resolve` should not throw `ERR_UNSUPPORTED_ESM_URL_SCHEME`

[giltayar]

While working on my ESM loaders talk for JSNation and NodeTLV conferences (repo here and draft slides here), I wrote a toy loader that supports HTTP[S]. This is the (obviously) simplistic code:

export async function load(url, context, nextLoad) {
  if (url.startsWith('http://') || url.startsWith('https://')) {
    const response = await fetch(url, {redirect: 'follow'})
    const source = await response.text()

    return {shortCircuit: true, format: 'module',source}
  }
  return await nextLoad(url, context)
}

Unfortunately, this won't work. If you use it, you will get an ERR_UNSUPPORTED_ESM_URL_SCHEME because the default Node.js resolver doesn't support HTTPS. So, unfortunately, I had to write (and demonstrate in my talk) that this loader needs a resolver, even though the only thing it does is support loading HTTPS:

export async function resolve(specifier, context, nextResolve) {
  if (isBareSpecifier(specifier))
    return await nextResolve(specifier, context)

  const url = new URL(specifier, context.parentURL).href

  if (url.startsWith('http://') || url.startsWith('https://'))
    return {url, shortCircuit: true}

  return await nextResolve(specifier, context)
}

Thinking about it more, I realized that the only reason I need to write this resolver is that Node.js doesn't recognize the HTTP scheme in its resolver. But if Node.js would have not recognized the scheme in the loader, then I wouldn't have needed to write this resolver at all.

The same logic goes for ERR_UNKNOWN_FILE_EXTENSION and writing, for example, a TypeScript loader.

So my point is this: both checks for unknown scheme and unknown file extension should move from the default Node.js resolver to the default Node.js loader. This will obviate the need for some loaders to implement a resolve which mostly replicates the Node.js logic, just for the sake of avoiding these exceptions.

I would gladly implement the change myself, but I wanted to check with the loaders team before tackling this task.

[arcanis]

I'm not sure that's correct - semantically, if the URL scheme is unsupported, then the resolution can't happen. For instance, let's say you have import 'my-protocol://hello', can you guarantee me that the untransformed my-protocol://hello is a valid resolution, if you don't know what my-protocol actually means?

For instance, let's say I write a tool validating that all the import statements in my project are resolvable, and this tool checks it by calling import.meta.resolve. With the change you propose, I wouldn't know that https:// urls are a problem - even though they'd cause the application to crash.

[aduh95]

both checks for unknown scheme and unknown file extension should move from the default Node.js resolver to the default Node.js loader

That would have a consequence on import.meta.resolve. Note that when writing a TS loader, I'm using to my advantage the fact that the default resolver throws:

export async function resolve(specifier, context, next) {
  try {
    return await next(specifier, context);
  } catch (err) {
    if (err?.code === "ERR_MODULE_NOT_FOUND" && specifier.endsWith(".js")) {
      try {
        return await next(specifier.replace(/\.js$/, ".ts"), context);
      } catch (err2) {
        if (err2) {
          err2.cause = err;
          throw err2;
        }
        throw err;
      }
    }
    throw err;
  }
}

The same logic goes for ERR_UNKNOWN_FILE_EXTENSION and writing, for example, a TypeScript loader.

To clarify, defaultResolve doesn't throw ERR_UNKNOWN_FILE_EXTENSION on unknown file extension. Would "whenver defaultResolve is given a non file: URL, it returns it immediately without any verification" be a good summary of your proposal?

[giltayar]

I'm not sure that's correct - semantically, if the URL scheme is unsupported, then the resolution can't happen. For instance, let's say you have import 'my-protocol://hello', can you guarantee me that the untransformed my-protocol://hello is a valid resolution, if you don't know what my-protocol actually means?

Why not accept all valid URL-s as valid during resolve? They will fail in the load phase anyway.

For instance, let's say I write a tool validating that all the import statements in my project are resolvable, and this tool checks it by calling import.meta.resolve. With the change you propose, I wouldn't know that https:// urls are a problem - even though they'd cause the application to crash.

That's a good point. Also @aduh95's point above, which points out that resolution checks for the existence of a file, and so could not be able to do that for an unknown extension. But @aduh95 proposes a solution to that, which makes sense: throw the exception in default resolve only for non file: extensions.

[aduh95]

I feel this discussion is quite similar to the one we had some time ago regarding import attributes, we might need another hoook for loaders to say what import attribute key they support, maybe we could add what protocols the support as well.

[giltayar]

To clarify, defaultResolve doesn't throw ERR_UNKNOWN_FILE_EXTENSION on unknown file extension.

You're right! I just modified my toy TS loader and removed the resolve and it works. Huh... I am absolutely sure that this was a problem! 😊 Oh, well. Thanks for pointing this out!

Would "whenver defaultResolve is given a non file: URL, it returns it immediately without any verification" be a good summary of your proposal?

Hmm.... yes! But @arcanis raises a valid concern: semantically, should import.meta.resolve always fail if the file does not exist? In this case, the HTTP loader would be forced to include a resolve that checks for the existence of the file (and maybe caches it knowing that it will be requested immediately after that).

That's a good question. In a way, it's a question for the JS standards group that is working on import.meta.resolve too.

If the answer is "yes", then import.meta.resolve MUST check for the existence, and @arcanis is right. But if the answer is "no", then @arcanis is not "allowed" to use import.meta.resolve to check for the existence of all modules.

[giltayar]

@arcanis just checked MDN on this: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/import.meta/resolve#description

Therefore, its return value is the same regardless of whether the returned path corresponds to a file that exists, and regardless of whether that file contains valid code for a module.

So you're actually not "allowed" to use import.meta.resolve to verify the existence of a module. This makes sense for the browser because import.meta.resolve is synchronous which means that they can't do any HTTP fetching to verify this.

Having seen this, I would submit that even Node.js shouldn't throw ERR_MODULE_NOT_FOUND in the resolution phase, but only in the loading phase (arguable, but would make sense).

This also means that @aduh95 code that checks for ERR_MODULE_NOT_FOUND in the resolution phase is incorrect when loaders are used, because they may not throw anything. This MUST be checked only in the loading phase.

[aduh95]

According to the HTML spec, browsers don't perform any check on the passed specifier string. I just tried to pass import('data:text/javascript,console.log(import.meta.resolve("my-protocol://hello"))') in Firefox, Safari, Chromium, and Deno, none of them threw an error.

[giltayar]

According to the HTML spec, browsers don't perform any check on the passed specifier string. I just tried to pass import('data:text/javascript,console.log(import.meta.resolve("my-protocol://hello"))') in Firefox, Safari, Chromium, and Deno, none of them threw an error.

Yup. The only thing they do is check the import map of the current HTML to see if the URL needs to be mapped.

[giltayar]

So given the above definition of import.meta.resolve, I believe we should amend the issue to be even stronger:

For any non-bare specifier, the default Node.js resolver should do only absolutization of the URL, and should not verify anything else about the final URL (e.g. it should not verify existence of the file, nor support of the URL scheme).

[GeoffreyBooth]

I feel like there’s a reason it currently happens in resolve, beyond the performance benefit of erroring before attempting a filesystem lookup (which isn’t all that important, since we don’t expect this error to happen often). Maybe we need the extension to find a valid translator, and that happens before load?

[giltayar]

@GeoffreyBooth as @aduh95 showed in his loader code above (and I can attest also happens in my babel-register-esm loader), people rely on the fact that the Node.js resolver checks for the existence of files! And, as @arcanis said, some devs assume that import.meta.resolve should check for the existence of files.

But this doesn't mean that it should be like that. In the browser world, it is explicitly not like that. And my guess is that most HTTP loaders won't check for the existence of the file in the resolve phase.

ESM in Node.js was always about aligning ourselves with the standardand not doing a separate thing. And the spec says not to check for existence of files in import.meta.resolve.

I suggest we align with the standard now, before it's too late and ESM loaders move out from experimental.

(and at least don't check the scheme is supported in the load phase and not the resolve phase 😊)

[giltayar]

BTW, not checking for existence of files in the resolve phase, will mean that for relative paths, loading will be faster, because one existSync will be avoided.

Moreover, this will mean that less loaders will have a resolve function, which will, again, improve performance when those loaders are used.

[GeoffreyBooth]

Okay, want to make a PR?

[giltayar]

Yes! I'll go all the way and include not checking for existence of files in the resolve phase, along with not checking if the scheme is supported in the resolve phase.

[giltayar]

For fairness' sake, I'd like to note that this will break loaders that use the existing node resolver to check for the existence of files. E.g. my babel-register-esm and @aduh95's loader.