[Meta] Extensions Framework

[saratvemulapalli]

Add the framework support for OpenSearch SDK to interface with extension points.

Issues

Steps to deliver extensions framework [P0]

• Discover Extensions: PluginsOrchestrator: Plugin Discovery for independent plugins #1972
• Add communication framework support for OpenSearch SDK Add transport support to connect with an extension #2019
• Proof of Concept: Add support for first extension point Design and support for initial extension point (IndicesModule) #2711
• Discover Extensions: Read extensions configuration during OpenSearch bootstrap [Feature/extensions] Add framework support to read extensions.yml #3084
• OpenSearch SDK: [META] Build OpenSearch SDK for extensions #1619
• Initial Benchmark Performance Extension Latency #2231 Performance Testing Set Up For Extensibility #3012
• Rest Handlers Extension point: [Feature/extensions] Add Rest Handlers support for extensions #3130
• [Feature/extensions] Add framework support for default extension points #3136
• [Feature/extensions] Add support to register a setting dynamically #3434
• [Feature/extensions] Unit Tests for ExtensionsOrchestrator #3408
• Run AD plugin as an Extension to create a detector - [Feature/extensions] Run AD backend plugin as an Extension to create a detector #3635

Back Burner[P1]

• Extend extensions framework support for additional extension points to migrate a plugin from opensearch-project [Feature/extensions] Add support for additional extension points to migrate SQL plugin #3011

Things we broke [P1]

• POC: Check for remote node and discovery node #2772
• [Feature/extensions] Gradle check failures #3282

Good to have [P2]

• [Feature/extensions] Modularize TransportService #2228
• Publish Netty module as a maven artifact #3118

Meta: #1422

[owaiskazi19]

Dependencies on PluginsService for repository-s3 plugin to run independently

1. pluginsService.updatedSettings()
2. pluginsService.filterPlugins(Plugin.class)
3. pluginsService.getPluginSettingsFilter()
4. p.getNamedXContent().stream() 

[saratvemulapalli]

Started working on PluginDiscovery:

1. Create new PluginOrchestrator (which will eventually replace PluginsService.java) to read through new plugins ("pluginsv2", didnt find a better name we could rename it later).
2. Update PluginInfo to understand a new field called Type (which will support LEGACY, INDEPENEDENT, REMOTE (in future)).
3. Define a new manifest file to expose a TCP port and PluginInfo.

[saratvemulapalli]

Feature branch on fork: /~https://github.com/saratvemulapalli/OpenSearch/tree/sarat-plugin-sandboxing
Seeing problems while reading new plugins folder:

[2022-01-12T22:13:52,984][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [3c22fb88f11c.ant.amazon.com] uncaught exception in thread [main]
org.opensearch.bootstrap.StartupException: java.security.AccessControlException: access denied ("java.io.FilePermission" "/Users/vemsarat/Desktop/workspace/Code/sarat-opensearch/distribution/archives/darwin-tar/build/install/opensearch-2.0.0-SNAPSHOT/pluginsv2" "read")
	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:182) ~[opensearch-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
	at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:169) ~[opensearch-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
	at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:100) ~[opensearch-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
	at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:135) ~[opensearch-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:101) ~[opensearch-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "/Users/vemsarat/Desktop/workspace/Code/sarat-opensearch/distribution/archives/darwin-tar/build/install/opensearch-2.0.0-SNAPSHOT/pluginsv2" "read")
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:?]
	at java.security.AccessController.checkPermission(AccessController.java:1036) ~[?:?]
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:408) ~[?:?]
	at java.lang.SecurityManager.checkRead(SecurityManager.java:747) ~[?:?]
	at sun.nio.fs.UnixPath.checkRead(UnixPath.java:818) ~[?:?]
	at sun.nio.fs.UnixFileSystemProvider.exists(UnixFileSystemProvider.java:525) ~[?:?]
	at java.nio.file.Files.exists(Files.java:2514) ~[?:?]
	at org.opensearch.plugins.PluginsService.findPluginDirs(PluginsService.java:365) ~[opensearch-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
	at org.opensearch.plugins.PluginsOrchestrator.pluginsDiscovery(PluginsOrchestrator.java:67) ~[opensearch-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
	at org.opensearch.plugins.PluginsOrchestrator.<init>(PluginsOrchestrator.java:52) ~[opensearch-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
	at org.opensearch.node.Node.<init>(Node.java:409) ~[opensearch-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
	at org.opensearch.node.Node.<init>(Node.java:329) ~[opensearch-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
	at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
	at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
	at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:412) ~[opensearch-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:178) ~[opensearch-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
	... 6 more
uncaught exception in thread [main]
java.security.AccessControlException: access denied ("java.io.FilePermission" "/Users/vemsarat/Desktop/workspace/Code/sarat-opensearch/distribution/archives/darwin-tar/build/install/opensearch-2.0.0-SNAPSHOT/pluginsv2" "read")
	at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
	at java.base/java.security.AccessController.checkPermission(AccessController.java:1036)
	at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:408)
	at java.base/java.lang.SecurityManager.checkRead(SecurityManager.java:747)
	at java.base/sun.nio.fs.UnixPath.checkRead(UnixPath.java:818)
	at java.base/sun.nio.fs.UnixFileSystemProvider.exists(UnixFileSystemProvider.java:525)
	at java.base/java.nio.file.Files.exists(Files.java:2514)
	at org.opensearch.plugins.PluginsService.findPluginDirs(PluginsService.java:365)
	at org.opensearch.plugins.PluginsOrchestrator.pluginsDiscovery(PluginsOrchestrator.java:67)
	at org.opensearch.plugins.PluginsOrchestrator.<init>(PluginsOrchestrator.java:52)
	at org.opensearch.node.Node.<init>(Node.java:409)
	at org.opensearch.node.Node.<init>(Node.java:329)
	at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242)
	at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242)
	at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:412)
	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:178)
	at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:169)
	at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:100)
	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
	at org.opensearch.cli.Command.main(Command.java:101)
	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:135)
	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:101)
For complete error details, refer to the log at /Users/vemsarat/Desktop/workspace/Code/sarat-opensearch/distribution/archives/darwin-tar/build/install/opensearch-2.0.0-SNAPSHOT/logs/opensearch.log