[BUG] Tenant Calculation(?) quite expensive

[HenryTheSir]

What is the bug?
After Upgrading from 2.18 to 2.19 we tried to drop our custom build security plugin which would return true as soon as it could determine that a user has all_access.

I'm currently unsure if this is a bottleneck of the optimized priviledge evaluation or a different root cause which occurs as we have arround 250 Tenants.

How can one reproduce the bug?
Steps to reproduce the behavior:

1. Create a high amount of tenants
2. Try to Index a huge amount of data
3. Observe slow ingestion and coordination nodes hot_threads showing attached thread stack

What is the expected behavior?
In a write requests tenants should not be evaluated.

   22.0% (109.9ms out of 500ms) cpu usage by thread 'opensearch[coordination-node][transport_worker][T#5]'
     10/10 snapshots sharing following 130 elements
       org.opensearch.security.securityconf.impl.SecurityDynamicConfiguration.getCEntries(SecurityDynamicConfiguration.java:288)
       org.opensearch.security.securityconf.ConfigModelV7$TenantHolder.lambda$mapTenants$2(ConfigModelV7.java:1149)
       org.opensearch.security.securityconf.ConfigModelV7$TenantHolder$$Lambda/0x00007fd2e9413748.accept(Unknown Source)
       java.base@21.0.6/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
       java.base@21.0.6/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
       java.base@21.0.6/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
       com.google.common.collect.CollectSpliterators$1.lambda$forEachRemaining$1(CollectSpliterators.java:128)
       com.google.common.collect.CollectSpliterators$1$$Lambda/0x00007fd2e9416d88.accept(Unknown Source)
       java.base@21.0.6/java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1715)
       com.google.common.collect.CollectSpliterators$1.forEachRemaining(CollectSpliterators.java:128)
       com.google.common.collect.CollectSpliterators$FlatMapSpliterator.lambda$forEachRemaining$1(CollectSpliterators.java:377)
       com.google.common.collect.CollectSpliterators$FlatMapSpliterator$$Lambda/0x00007fd2e9413dc0.accept(Unknown Source)
       java.base@21.0.6/java.util.HashMap$EntrySpliterator.forEachRemaining(HashMap.java:1858)
       com.google.common.collect.CollectSpliterators$FlatMapSpliterator.forEachRemaining(CollectSpliterators.java:373)
       java.base@21.0.6/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
       java.base@21.0.6/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
       java.base@21.0.6/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
       java.base@21.0.6/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
       java.base@21.0.6/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
       java.base@21.0.6/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
       org.opensearch.security.securityconf.ConfigModelV7$TenantHolder.mapTenants(ConfigModelV7.java:1135)
       org.opensearch.security.securityconf.ConfigModelV7.mapTenants(ConfigModelV7.java:1285)
       org.opensearch.security.privileges.PrivilegesEvaluatorImpl.mapTenants(PrivilegesEvaluatorImpl.java:598)
       org.opensearch.security.privileges.PrivilegesEvaluatorImpl.evaluate(PrivilegesEvaluatorImpl.java:520)
       org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:377)
       org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:166)
       app//org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218)
       org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:81)
       app//org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218)
       app//org.opensearch.action.support.TransportAction.execute(TransportAction.java:190)
       app//org.opensearch.action.support.TransportAction.execute(TransportAction.java:109)

[nibix]

Have the performance issues been present since the first time you have used OpenSearch, or was it a regression at some point?

Do you have figures regarding the slowdown comparing your custom plugin and the performance observed with OpenSearch 2.19?

[HenryTheSir]

Hi,

maybe I opened the issue too early. Sorry for disturbance.

We optimized the role assigned to our bulk index user to not have tenant_permissions '*' . This omits the mapTenants call as the user has no access to any tenant.

Your optimizations work quite fine - thanks a lot for this!

I'm unsure if this issue should stay open, as a user with tenant_permissions '*' in a environment with many tenants till has a slow(er) permission check or if we should close this as inperformant configuration for a bulk index user.

best regards and thanks again!
Henry

[nibix]

maybe I opened the issue too early. Sorry for disturbance.

No worries, I do think that the hot thread dump indicates an issue. However, at the moment I believe it is a separate issue. That's why I am curious about the actual impact and whether it was a regression at some point.

[cwperks]

@HenryTheSir FYI 2.19 has a separate optimization for all_access as well using the legacy authz code path: #4926

[derek-ho]

@HenryTheSir thank you for filing this issue. It sounds like possibly optimizing the privileges evaluation flow has uncovered other issues in terms of tenancy calculations. Someone will take a look.