Skip to content

Commit

Permalink
Merge branch 'opensearch-project:main' into fdp
Browse files Browse the repository at this point in the history
  • Loading branch information
mgodwan authored Jun 18, 2023
2 parents 802e679 + 90678c2 commit c4872d6
Show file tree
Hide file tree
Showing 856 changed files with 24,231 additions and 5,953 deletions.
2 changes: 2 additions & 0 deletions .ci/bwcVersions
Original file line number Diff line number Diff line change
Expand Up @@ -19,3 +19,5 @@ BWC_VERSION:
- "2.7.0"
- "2.7.1"
- "2.8.0"
- "2.8.1"
- "2.9.0"
2 changes: 1 addition & 1 deletion .github/CODEOWNERS
Original file line number Diff line number Diff line change
@@ -1 +1 @@
* @reta @anasalkouz @andrross @reta @Bukhtawar @CEHENKLE @dblock @gbbafna @setiah @kartg @kotwanikunal @mch2 @nknize @owaiskazi19 @Rishikesh1159 @ryanbogan @saratvemulapalli @shwetathareja @dreamer-89 @tlfeng @VachaShah @dbwiddis
* @reta @anasalkouz @andrross @reta @Bukhtawar @CEHENKLE @dblock @gbbafna @setiah @kartg @kotwanikunal @mch2 @nknize @owaiskazi19 @Rishikesh1159 @ryanbogan @saratvemulapalli @shwetathareja @dreamer-89 @tlfeng @VachaShah @dbwiddis @sachinpkale
49 changes: 49 additions & 0 deletions .github/workflows/poc-checklist.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
name: Add comment
on:
issues:
types:
- labeled
jobs:
add-comment:
if: github.event.label.name == 'poc'
runs-on: ubuntu-latest
permissions:
issues: write
steps:
- name: Add comment
uses: peter-evans/create-or-update-comment@v2
with:
issue-number: ${{ github.event.issue.number }}
body: |
### POC Checklist:
Please go through the following checklist to ensure these items are taken into account while designing the POC.
- [ ] Supports safe upgrade paths from all supported BWC versions to the current version
- [ ] Supports compatibility with all plugins
- [ ] opensearch-alerting
- [ ] opensearch-anomaly-detection
- [ ] opensearch-asynchronous-search
- [ ] opensearch-cross-cluster-replication
- [ ] opensearch-geospatial
- [ ] opensearch-index-management
- [ ] opensearch-job-scheduler
- [ ] opensearch-knn
- [ ] opensearch-ml
- [ ] opensearch-notifications
- [ ] opensearch-notifications-core
- [ ] opensearch-observability
- [ ] opensearch-performance-analyzer
- [ ] opensearch-reports-scheduler
- [ ] opensearch-security
- [ ] opensearch-sql
- [ ] Supports lucene upgrades across minor lucene versions
- [ ] Supports lucene upgrades across major lucene versions
- [ ] Supports lucene upgrades across underlying lucene codec bumps (Eg: Lucene95Codec -> Lucene96Codec)
- [ ] Supports wire compatibility of OpenSearch
- [ ] Plan to measure performance degradation/improvement (if any)
- [ ] Plan to document any user facing changes introduced by this feature
- [ ] Ensure working and passing CI
Thank you for your contribution!
2 changes: 1 addition & 1 deletion .idea/inspectionProfiles/Project_Default.xml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

40 changes: 40 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- Allow mmap to use new JDK-19 preview APIs in Apache Lucene 9.4+ ([#5151](/~https://github.com/opensearch-project/OpenSearch/pull/5151))
- Add events correlation engine plugin ([#6854](/~https://github.com/opensearch-project/OpenSearch/issues/6854))
- Add support for ignoring missing Javadoc on generated code using annotation ([#7604](/~https://github.com/opensearch-project/OpenSearch/pull/7604))
- Implement concurrent aggregations support without profile option ([#7514](/~https://github.com/opensearch-project/OpenSearch/pull/7514))
- Add dynamic index and cluster setting for concurrent segment search ([#7956](/~https://github.com/opensearch-project/OpenSearch/pull/7956))

### Dependencies
- Bump `log4j-core` from 2.18.0 to 2.19.0
Expand Down Expand Up @@ -41,6 +43,17 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- Bump `com.netflix.nebula:nebula-publishing-plugin` from 19.2.0 to 20.3.0
- Bump `com.diffplug.spotless` from 6.17.0 to 6.18.0
- Bump `io.opencensus:opencensus-api` from 0.18.0 to 0.31.1 ([#7291](/~https://github.com/opensearch-project/OpenSearch/pull/7291))
- Add `org.reactivestreams` 1.0.4 ([7372](/~https://github.com/opensearch-project/OpenSearch/pull/7372/))
- Add `com.github.luben:zstd-jni` version 1.5.5-3 ([#2996](/~https://github.com/opensearch-project/OpenSearch/pull/2996))
- OpenJDK Update (April 2023 Patch releases) ([#7344](/~https://github.com/opensearch-project/OpenSearch/pull/7344)
- Bump `com.amazonaws` 1.12.270 to `software.amazon.awssdk` 2.20.55 ([7372](/~https://github.com/opensearch-project/OpenSearch/pull/7372/))
- Bump `com.networknt:json-schema-validator` from 1.0.81 to 1.0.83 ([7933](/~https://github.com/opensearch-project/OpenSearch/pull/7933/))
- Bump `com.google.http-client:google-http-client:1.43.2` from 1.42.0 to 1.43.2 ([7928](/~https://github.com/opensearch-project/OpenSearch/pull/7928)))
- Bump `com.azure:azure-storage-blob` from 12.21.1 to 12.22.2 ([7930](/~https://github.com/opensearch-project/OpenSearch/pull/7930))
- Bump `io.projectreactor:reactor-core` from 3.5.1 to 3.5.6 in /plugins/repository-azure ([#8016](/~https://github.com/opensearch-project/OpenSearch/pull/8016))
- Bump `com.google.guava:guava` from 32.0.0-jre to 32.0.1-jre in /distribution/tools/upgrade-cli ([#8011](/~https://github.com/opensearch-project/OpenSearch/pull/8011))
- Bump `com.google.guava:guava` from 32.0.0-jre to 32.0.1-jre in /distribution/tools/plugin-cli ([#8012](/~https://github.com/opensearch-project/OpenSearch/pull/8012))
- Bump all `com.google.guava:guava` to 32.0.1-jre ([#8107](/~https://github.com/opensearch-project/OpenSearch/pull/8107))

### Changed
- [CCR] Add getHistoryOperationsFromTranslog method to fetch the history snapshot from translogs ([#3948](/~https://github.com/opensearch-project/OpenSearch/pull/3948))
Expand All @@ -50,6 +63,11 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- Change http code for DecommissioningFailedException from 500 to 400 ([#5283](/~https://github.com/opensearch-project/OpenSearch/pull/5283))
- Improve summary error message for invalid setting updates ([#4792](/~https://github.com/opensearch-project/OpenSearch/pull/4792))
- Reduce memory copy in zstd compression ([#7681](/~https://github.com/opensearch-project/OpenSearch/pull/7681))
- Add ZSTD compression for snapshotting ([#2996](/~https://github.com/opensearch-project/OpenSearch/pull/2996))
- Change `com.amazonaws.sdk.ec2MetadataServiceEndpointOverride` to `aws.ec2MetadataServiceEndpoint` ([7372](/~https://github.com/opensearch-project/OpenSearch/pull/7372/))
- Change `com.amazonaws.sdk.stsEndpointOverride` to `aws.stsEndpointOverride` ([7372](/~https://github.com/opensearch-project/OpenSearch/pull/7372/))
- Align range and default value for deletes_pct_allowed in merge policy ([#7730](/~https://github.com/opensearch-project/OpenSearch/pull/7730))
- Rename QueryPhase actors like Suggest, Rescore to be processors rather than phase ([#8025](/~https://github.com/opensearch-project/OpenSearch/pull/8025))

### Deprecated

Expand All @@ -68,30 +86,52 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- Remove LegacyESVersion.V_7_10_ Constants ([#5018](/~https://github.com/opensearch-project/OpenSearch/pull/5018))
- Remove Version.V_1_ Constants ([#5021](/~https://github.com/opensearch-project/OpenSearch/pull/5021))
- Remove custom Map, List and Set collection classes ([#6871](/~https://github.com/opensearch-project/OpenSearch/pull/6871))
- Remove COMPRESSOR variable from CompressorFactory and use DEFLATE_COMPRESSOR instead ([7907](/~https://github.com/opensearch-project/OpenSearch/pull/7907))

### Fixed
- Fix 'org.apache.hc.core5.http.ParseException: Invalid protocol version' under JDK 16+ ([#4827](/~https://github.com/opensearch-project/OpenSearch/pull/4827))
- Fix compression support for h2c protocol ([#4944](/~https://github.com/opensearch-project/OpenSearch/pull/4944))
- Support OpenSSL Provider with default Netty allocator ([#5460](/~https://github.com/opensearch-project/OpenSearch/pull/5460))
- Replaces ZipInputStream with ZipFile to fix Zip Slip vulnerability ([#7230](/~https://github.com/opensearch-project/OpenSearch/pull/7230))
- Add missing validation/parsing of SearchBackpressureMode of SearchBackpressureSettings ([#7541](/~https://github.com/opensearch-project/OpenSearch/pull/7541))
- Fix mapping char_filter when mapping a hashtag ([#7591](/~https://github.com/opensearch-project/OpenSearch/pull/7591))
- With only GlobalAggregation in request causes unnecessary wrapping with MultiCollector ([#8125](/~https://github.com/opensearch-project/OpenSearch/pull/8125))

### Security

## [Unreleased 2.x]
### Added
- Add task cancellation monitoring service ([#7642](/~https://github.com/opensearch-project/OpenSearch/pull/7642))
- Add TokenManager Interface ([#7452](/~https://github.com/opensearch-project/OpenSearch/pull/7452))
- Add Remote store as a segment replication source ([#7653](/~https://github.com/opensearch-project/OpenSearch/pull/7653))
- Add descending order search optimization through reverse segment read. ([#7967](/~https://github.com/opensearch-project/OpenSearch/pull/7967))
- Make remote cluster connection setup in async ([#8038](/~https://github.com/opensearch-project/OpenSearch/pull/8038))
- Update components of segrep backpressure to support remote store. ([#8020](/~https://github.com/opensearch-project/OpenSearch/pull/8020))

### Dependencies
- Bump `jackson` from 2.15.1 to 2.15.2 ([#7897](/~https://github.com/opensearch-project/OpenSearch/pull/7897))
- Bump `netty` from 4.1.91.Final to 4.1.93.Final ([#7901](/~https://github.com/opensearch-project/OpenSearch/pull/7901))
- Bump `com.netflix.nebula:gradle-extra-configurations-plugin` from 9.0.0 to 10.0.0 in /buildSrc ([#7068](/~https://github.com/opensearch-project/OpenSearch/pull/7068))
- Bump `com.google.guava:guava` from 32.0.0-jre to 32.0.1-jre (#8009)
- Bump `commons-io:commons-io` from 2.12.0 to 2.13.0 (#8014, #8013, #8010)
- Bump `com.diffplug.spotless` from 6.18.0 to 6.19.0 (#8007)
- Bump `org.jruby.joni:joni` from 2.1.48 to 2.2.1 (#8015)

### Changed
- Replace jboss-annotations-api_1.2_spec with jakarta.annotation-api ([#7836](/~https://github.com/opensearch-project/OpenSearch/pull/7836))
- Add min, max, average and thread info to resource stats in tasks API ([#7673](/~https://github.com/opensearch-project/OpenSearch/pull/7673))
- Compress and cache cluster state during validate join request ([#7321](/~https://github.com/opensearch-project/OpenSearch/pull/7321))
- [Snapshot Interop] Add Changes in Create Snapshot Flow for remote store interoperability. ([#7118](/~https://github.com/opensearch-project/OpenSearch/pull/7118))
- Add new query profile collector fields with concurrent search execution ([#7898](/~https://github.com/opensearch-project/OpenSearch/pull/7898))
- Allow insecure string settings to warn-log usage and advise to migration of a newer secure variant ([#5496](/~https://github.com/opensearch-project/OpenSearch/pull/5496))

### Deprecated

### Removed

### Fixed
- Fixing error: adding a new/forgotten parameter to the configuration for checking the config on startup in plugins/repository-s3 #7924
- Enforce 512 byte document ID limit in bulk updates ([#8039](/~https://github.com/opensearch-project/OpenSearch/pull/8039))

### Security

Expand Down
1 change: 1 addition & 0 deletions MAINTAINERS.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ This document contains a list of maintainers in this repo. See [opensearch-proje
| Owais Kazi | [owaiskazi19](/~https://github.com/owaiskazi19) | Amazon |
| Rishikesh Pasham | [Rishikesh1159](/~https://github.com/Rishikesh1159) | Amazon |
| Ryan Bogan | [ryanbogan](/~https://github.com/ryanbogan) | Amazon |
| Sachin Kale | [sachinpkale](/~https://github.com/sachinpkale) | Amazon |
| Sarat Vemulapalli | [saratvemulapalli](/~https://github.com/saratvemulapalli) | Amazon |
| Shweta Thareja | [shwetathareja](/~https://github.com/shwetathareja) | Amazon |
| Suraj Singh | [dreamer-89](/~https://github.com/dreamer-89) | Amazon |
Expand Down
2 changes: 1 addition & 1 deletion build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ plugins {
id 'lifecycle-base'
id 'opensearch.docker-support'
id 'opensearch.global-build-info'
id "com.diffplug.spotless" version "6.18.0" apply false
id "com.diffplug.spotless" version "6.19.0" apply false
id "org.gradle.test-retry" version "1.5.3" apply false
id "test-report-aggregation"
id 'jacoco-report-aggregation'
Expand Down
8 changes: 5 additions & 3 deletions buildSrc/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -105,11 +105,11 @@ dependencies {
api 'commons-codec:commons-codec:1.15'
api 'org.apache.commons:commons-compress:1.23.0'
api 'org.apache.ant:ant:1.10.13'
api 'com.netflix.nebula:gradle-extra-configurations-plugin:9.0.0'
api 'com.netflix.nebula:gradle-extra-configurations-plugin:10.0.0'
api 'com.netflix.nebula:nebula-publishing-plugin:20.3.0'
api 'com.netflix.nebula:gradle-info-plugin:12.1.3'
api 'org.apache.rat:apache-rat:0.15'
api 'commons-io:commons-io:2.11.0'
api 'commons-io:commons-io:2.13.0'
api "net.java.dev.jna:jna:5.13.0"
api 'gradle.plugin.com.github.johnrengelman:shadow:8.0.0'
api 'org.jdom:jdom2:2.0.6.1'
Expand All @@ -118,7 +118,9 @@ dependencies {
api 'com.avast.gradle:gradle-docker-compose-plugin:0.16.12'
api "org.yaml:snakeyaml:${props.getProperty('snakeyaml')}"
api 'org.apache.maven:maven-model:3.9.2'
api 'com.networknt:json-schema-validator:1.0.81'
api 'com.networknt:json-schema-validator:1.0.83'
api 'org.jruby.jcodings:jcodings:1.0.58'
api 'org.jruby.joni:joni:2.1.48'
api "com.fasterxml.jackson.core:jackson-databind:${props.getProperty('jackson_databind')}"

testFixturesApi "junit:junit:${props.getProperty('junit')}"
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,124 @@
/*
* SPDX-License-Identifier: Apache-2.0
*
* The OpenSearch Contributors require contributions made to
* this file be licensed under the Apache-2.0 license or a
*
* Modifications Copyright OpenSearch Contributors. See
* GitHub history for details.
*/

/*
* Copyright 2014-2016 Netflix, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.opensearch.gradle.plugin

import org.gradle.api.Plugin
import org.gradle.api.Project
import org.gradle.api.publish.PublishingExtension
import org.gradle.api.publish.ivy.IvyPublication
import org.gradle.api.publish.maven.MavenPublication
import org.gradle.api.publish.plugins.PublishingPlugin

/**
* Including the support of `optional` dependencies: /~https://github.com/nebula-plugins/gradle-extra-configurations-plugin/blob/v9.0.0/src/main/groovy/nebula/plugin/extraconfigurations/OptionalBasePlugin.groovy
*/
class OptionalDependenciesPlugin implements Plugin<Project> {
static final String OPTIONAL_IDENTIFIER = 'optional'

@Override
void apply(Project project) {
enhanceProjectModel(project)
configureMavenPublishPlugin(project)
configureIvyPublishPlugin(project)
}

/**
* Enhances the Project domain object by adding
*
* a) a extra property List that holds optional dependencies
* b) a extra method that can be executed as parameter when declaring dependencies
*
* @param project Project
*/
private void enhanceProjectModel(Project project) {
project.ext.optionalDeps = []

project.ext.optional = { dep ->
project.ext.optionalDeps << dep
}
}

/**
* Configures Maven Publishing plugin to ensure that published dependencies receive the optional element.
*
* @param project Project
*/
private void configureMavenPublishPlugin(Project project) {
project.plugins.withType(PublishingPlugin) {
project.publishing {
publications {
project.extensions.findByType(PublishingExtension)?.publications?.withType(MavenPublication) { MavenPublication pub ->
pub.pom.withXml {
project.ext.optionalDeps.each { dep ->
def foundDep = asNode().dependencies.dependency.find {
it.groupId.text() == dep.group && it.artifactId.text() == dep.name
}

if (foundDep) {
if (foundDep.optional) {
foundDep.optional.value = 'true'
} else {
foundDep.appendNode(OPTIONAL_IDENTIFIER, 'true')
}
}
}
}
}
}
}
}
}

/**
* Configures Ivy Publishing plugin to ensure that published dependencies receive the correct conf attribute value.
*
* @param project Project
*/
private void configureIvyPublishPlugin(Project project) {
project.plugins.withType(PublishingPlugin) {
project.publishing {
publications {
project.extensions.findByType(PublishingExtension)?.publications?.withType(IvyPublication) { IvyPublication pub ->
pub.descriptor.withXml {
def rootNode = asNode()

// Add optional configuration if it doesn't exist yet
if (!rootNode.configurations.find { it.@name == OPTIONAL_IDENTIFIER }) {
rootNode.configurations[0].appendNode('conf', [name: OPTIONAL_IDENTIFIER, visibility: 'public'])
}

// Replace dependency "runtime->default" conf attribute value with "optional"
project.ext.optionalDeps.each { dep ->
def foundDep = rootNode.dependencies.dependency.find { it.@name == dep.name }
foundDep?.@conf = OPTIONAL_IDENTIFIER
}
}
}
}
}
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -129,13 +129,9 @@ private void setupRepository(Project project, Jdk jdk) {
// To distinguish between those, the GA releases have only major version component (fe 17+32),
// the updates always have minor/patch components (fe 17.0.1+12), checking for the presence of
// version separator '.' should be enough.
artifactPattern = "jdk-"
+ jdk.getBaseVersion()
+ "+"
+ jdk.getBuild()
+ "/OpenJDK"
+ jdk.getMajor()
+ (jdk.getBaseVersion().contains(".") ? "U" : "")
artifactPattern = "jdk-" + jdk.getBaseVersion() + "+" + jdk.getBuild() + "/OpenJDK" + jdk.getMajor()
// JDK-20 does use 'U' suffix all the time, no matter it is update or GA release
+ (jdk.getBaseVersion().contains(".") || jdk.getBaseVersion().matches("^2\\d+$") ? "U" : "")
+ "-jdk_[classifier]_[module]_hotspot_"
+ jdk.getBaseVersion()
+ "_"
Expand Down
Loading

0 comments on commit c4872d6

Please sign in to comment.