Skip to content

Commit

Permalink
Add the shardingsphere-infra-expr-purelist module to replace the `s…
Browse files Browse the repository at this point in the history
…hardingsphere-infra-expr-espresso` module
  • Loading branch information
linghengqian committed Aug 9, 2023
1 parent 59767ba commit 92ab477
Show file tree
Hide file tree
Showing 16 changed files with 217 additions and 142 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/nightly-build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,7 @@ jobs:
ref: ${{ inputs.commit-id }}
- uses: graalvm/setup-graalvm@v1
with:
java-version: '17.0.7'
java-version: '17.0.8'
distribution: 'graalvm-community'
components: 'espresso,native-image'
github-token: ${{ secrets.GITHUB_TOKEN }}
Expand Down
5 changes: 1 addition & 4 deletions distribution/proxy-native/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -22,14 +22,11 @@ ENV LOCAL_PATH /opt/shardingsphere-proxy-native
ADD target/${APP_NAME}.tar.gz /opt
RUN mv /opt/${APP_NAME} ${LOCAL_PATH}

FROM ghcr.io/graalvm/graalvm-community:17.0.7-ol9
FROM oraclelinux:9-slim
MAINTAINER ShardingSphere "dev@shardingsphere.apache.org"

ARG NATIVE_IMAGE_NAME
ENV LOCAL_PATH /opt/shardingsphere-proxy-native

RUN $JAVA_HOME/bin/gu install espresso && \
$JAVA_HOME/bin/gu remove native-image

COPY --from=prepare ${LOCAL_PATH} ${LOCAL_PATH}
ENTRYPOINT ${LOCAL_PATH}/${NATIVE_IMAGE_NAME} 3307 ${LOCAL_PATH}/conf "0.0.0.0" false
3 changes: 0 additions & 3 deletions distribution/proxy-native/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,6 @@
<profile>
<id>release.native</id>
<properties>
<truffle-api.version>23.0.1</truffle-api.version>
<jacoco.skip>true</jacoco.skip>
<maven.javadoc.skip>true</maven.javadoc.skip>
<checkstyle.skip>true</checkstyle.skip>
Expand Down Expand Up @@ -96,8 +95,6 @@
<fallback>false</fallback>
<verbose>true</verbose>
<buildArgs>
<arg>--language:java</arg>
<arg>-H:+AllowDeprecatedBuilderClassesOnImageClasspath</arg>
<arg>--report-unsupported-elements-at-runtime</arg>
</buildArgs>
<metadataRepository>
Expand Down
10 changes: 0 additions & 10 deletions distribution/proxy-native/src/main/release-docs/LICENSE
Original file line number Diff line number Diff line change
Expand Up @@ -360,13 +360,3 @@ The text of each license is also included at licenses/LICENSE-[project].txt.
jul-to-slf4j 1.7.36: https://www.slf4j.org, MIT
slf4j-api 1.7.36: https://www.slf4j.org, MIT
jnanoid 2.0.0: /~https://github.com/aventrix/jnanoid, MIT

========================================================================
UPL licenses
========================================================================

The following components are provided under the UPL License. See project link for details.
The text of each license is also included at licenses/LICENSE-[project].txt.

graal-sdk 23.0.1: /~https://github.com/oracle/graal, UPL 1.0
truffle-api 23.0.1: /~https://github.com/oracle/graal/tree/master/truffle, UPL 1.0

This file was deleted.

10 changes: 0 additions & 10 deletions distribution/proxy/src/main/release-docs/LICENSE
Original file line number Diff line number Diff line change
Expand Up @@ -372,13 +372,3 @@ The text of each license is also included at licenses/LICENSE-[project].txt.
jul-to-slf4j 1.7.36: https://www.slf4j.org, MIT
slf4j-api 1.7.36: https://www.slf4j.org, MIT
jnanoid 2.0.0: /~https://github.com/aventrix/jnanoid, MIT

========================================================================
UPL licenses
========================================================================

The following components are provided under the UPL License. See project link for details.
The text of each license is also included at licenses/LICENSE-[project].txt.

graal-sdk 21.2.0: /~https://github.com/oracle/graal, UPL 1.0
truffle-api 21.2.0: /~https://github.com/oracle/graal/tree/master/truffle, UPL 1.0

This file was deleted.

Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,7 @@ weight = 2

## 背景信息

本节主要介绍如何通过 `GraalVM``native-image` 组件构建 ShardingSphere-Proxy 的 `Native Image` 和对应的 `Docker Image`
本节主要介绍如何通过 `GraalVM``native-image` 组件构建 ShardingSphere-Proxy 的 `Native Image` 和对应的 `Docker Image`

## 注意事项

Expand All @@ -30,20 +29,17 @@ services:
应当在 /~https://github.com/oracle/graalvm-reachability-metadata 打开新的 issue ,
并提交包含 ShardingSphere 自身或依赖的第三方库缺失的 GraalVM Reachability Metadata 的 PR。
- ShardingSphere 的 master 分支尚未准备好处理 Native Image 中的单元测试, 你总是需要在构建 GraalVM Native Image 的过程中,
- ShardingSphere 的 master 分支尚未准备好处理 Native Image 中的单元测试 你总是需要在构建 GraalVM Native Image 的过程中,
加上特定于 `GraalVM Native Build Tools` 的 `-DskipNativeTests` 或 `-DskipTests` 参数跳过 Native Image 中的单元测试。

- 如下 3 个算法类由于涉及到 GraalVM Truffle Espresso 不方便在 host JVM 和 guest JVM 之间交互的 `groovy.lang.Closure`
类,暂未可在 GraalVM Native Image 下使用。
- 如下的算法类由于涉及到 /~https://github.com/oracle/graal/issues/5522 , 暂未可在 GraalVM Native Image 下使用。
- `org.apache.shardingsphere.sharding.algorithm.sharding.inline.InlineShardingAlgorithm`
- `org.apache.shardingsphere.sharding.algorithm.sharding.inline.ComplexInlineShardingAlgorithm`
- `org.apache.shardingsphere.sharding.algorithm.sharding.hint.HintInlineShardingAlgorithm`

- 当前阶段,GraalVM Native Image 形态的 ShardingSphere Proxy 处于混合 AOT ( GraalVM Native Image ) 和 JIT ( GraalVM
Truffle Espresso ) 运行的阶段。由于 /~https://github.com/oracle/graal/issues/4555 尚未关闭,GraalVM Truffle Espresso
运行需要的 `.so` 文件并不会进入 GraalVM Native Image 内。因此如果你需要在 Docker Image 外运行 ShardingSphere Proxy
Native 的二进制文件,你需要确保系统环境变量 `JAVA_HOME` 指向 GraalVM 的目录,并且此 GraalVM
实例已经通过 `GraalVM Updater` 安装了 `espresso` 组件。
- 当前阶段,GraalVM Native Image 形态的 ShardingSphere Proxy 不支持使用带 Groovy
语法的 `行表达式`, 这首先导致 `数据分片` 功能的`actualDataNodes`属性只能使用纯列表来配置, 例如 `ds_0.t_order_0, ds_0.t_order_1`
或 `ds_0.t_user_0, ds_15.t_user_1023`。此问题在 /~https://github.com/oracle/graal/issues/5522 追踪。

- 本节假定处于 Linux(amd64,aarch64), MacOS(amd64)或 Windows(amd64)环境。
如果你位于 MacOS(aarch64/M1) 环境,你需要关注尚未关闭的 /~https://github.com/oracle/graal/issues/2666 。
Expand All @@ -53,20 +49,14 @@ services:
1. 根据 https://www.graalvm.org/downloads/ 要求安装和配置 JDK 17 对应的 `GraalVM Community Edition` 或 `Oracle GraalVM`。或者使用 `SDKMAN!`。

```shell
sdk install java 17.0.7-graalce
sdk install java 17.0.8-graalce
```

2. 通过 `GraalVM Updater` 工具安装 `espresso` 组件
2. 根据 https://www.graalvm.org/latest/reference-manual/native-image/#prerequisites 的要求安装本地工具链

```shell
gu install espresso
```

3. 根据 https://www.graalvm.org/latest/reference-manual/native-image/#prerequisites 的要求安装本地工具链。

4. 如果需要构建 Docker Image, 确保 `docker-ce` 已安装。
3. 如果需要构建 Docker Image, 确保 `docker-ce` 已安装。

5. 首先需要在项目的根目录下,执行如下命令以为所有子模块采集 Standard 形态的 GraalVM 可达性元数据。
4. 首先需要在项目的根目录下,执行如下命令以为所有子模块采集 Standard 形态的 GraalVM 可达性元数据。

```shell
./mvnw -PgenerateStandardMetadata -DskipNativeTests -B -T1C clean test
Expand Down Expand Up @@ -146,7 +136,7 @@ services:
- "3307:3307"
```

- 如果你不对 Git Source 做任何更改, 上文提及的命令将使用 `ghcr.io/graalvm/graalvm-community:17.0.7-ol9` 作为 Base Docker Image。
- 如果你不对 Git Source 做任何更改, 上文提及的命令将使用 `oraclelinux:9-slim` 作为 Base Docker Image。
但如果你希望使用 `busybox:glic`,`gcr.io/distroless/base` 或 `scratch` 等更小体积的 Docker Image 作为 Base Docker
Image,你需要根据 https://www.graalvm.org/latest/reference-manual/native-image/guides/build-static-executables/ 的要求,
做为 `pom.xml`的 `native profile` 添加 `-H:+StaticExecutableWithDynamicLibC` 的 `jvmArgs` 等操作。
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -36,19 +36,15 @@ services:
Plus `-DskipNativeTests` or `-DskipTests` parameter specific to `GraalVM Native Build Tools` to skip unit tests in
Native Image.

- The following three algorithm classes are not available under GraalVM Native Image because they involve
the `groovy.lang.Closure` class that is inconvenient for GraalVM Truffle Espresso to interact between the host JVM and
the guest JVM.
- The following algorithm classes are not available under GraalVM Native Image due to /~https://github.com/oracle/graal/issues/5522 involved.
- `org.apache.shardingsphere.sharding.algorithm.sharding.inline.InlineShardingAlgorithm`
- `org.apache.shardingsphere.sharding.algorithm.sharding.inline.ComplexInlineShardingAlgorithm`
- `org.apache.shardingsphere.sharding.algorithm.sharding.hint.HintInlineShardingAlgorithm`

- At the current stage, ShardingSphere Proxy in GraalVM Native Image is in the stage of mixed AOT ( GraalVM
Native Image ) and JIT ( GraalVM Truffle Espresso ) operation. Since /~https://github.com/oracle/graal/issues/4555 has
not been closed, the `.so` file required for GraalVM Truffle Espresso to run does not enter the GraalVM Native Image.
So if you need to run ShardingSphere Proxy outside of Docker Image For Native binaries,
you need to make sure that the system environment variable `JAVA_HOME` points to GraalVM's
directory and this GraalVM instance already has the `espresso` component installed via `GraalVM Updater`.
- At this stage, ShardingSphere Proxy in the form of GraalVM Native Image does not support the use of `Row Value Expressions`
with Groovy syntax, which first results in the `actualDataNodes` property of the `Sharding` feature being only
configurable using a pure list, such as `ds_0.t_order_0, ds_0.t_order_1` or `ds_0.t_user_0, ds_15.t_user_1023`. This
issue is tracked in /~https://github.com/oracle/graal/issues/5522 .

- This section assumes a Linux (amd64, aarch64), MacOS (amd64) or Windows (amd64) environment.
If you are on MacOS (aarch64/M1) environment, you need to follow /~https://github.com/oracle/graal/issues/2666 which is
Expand All @@ -59,20 +55,14 @@ services:
1. Install and configure `GraalVM Community Edition` or `Oracle GraalVM` for JDK 17 according to https://www.graalvm.org/downloads/. Or use `SDKMAN!`.

```shell
sdk install java 17.0.7-graalce
sdk install java 17.0.8-graalce
```

2. Install the `espresso` component via the `GraalVM Updater` tool.
2. Install the local toolchain as required by https://www.graalvm.org/latest/reference-manual/native-image/#prerequisites.

```shell
gu install espresso
```

3. Install the local toolchain as required by https://www.graalvm.org/latest/reference-manual/native-image/#prerequisites.

4. If you need to build a Docker Image, make sure `docker-ce` is installed.
3. If you need to build a Docker Image, make sure `docker-ce` is installed.

5. First, you need to execute the following command in the root directory of the project to collect the GraalVM
4. First, you need to execute the following command in the root directory of the project to collect the GraalVM
Reachability Metadata of the Standard form for all submodules.

```shell
Expand Down Expand Up @@ -158,8 +148,7 @@ services:
- "3307:3307"
```

- If you don't make any changes to the Git Source, the commands mentioned above will use `ghcr.io/graalvm/graalvm-community:17.0.7-ol9` as the
Base Docker Image.
- If you don't make any changes to the Git Source, the commands mentioned above will use `oraclelinux:9-slim` as the Base Docker Image.
But if you want to use a smaller Docker Image like `busybox:glic`, `gcr.io/distroless/base` or `scratch` as the Base Docker Image, you need according
to https://www.graalvm.org/latest/reference-manual/native-image/guides/build-static-executables/,
add operations such as `-H:+StaticExecutableWithDynamicLibC` to `jvmArgs` as the `native profile` of `pom.xml`.
Expand Down
2 changes: 1 addition & 1 deletion infra/expr/core/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@
</dependency>
<dependency>
<groupId>org.apache.shardingsphere</groupId>
<artifactId>shardingsphere-infra-expr-espresso</artifactId>
<artifactId>shardingsphere-infra-expr-purelist</artifactId>
<version>${project.version}</version>
</dependency>

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,6 @@ public final class InlineExpressionParserFactory {
* @return created instance
*/
public static InlineExpressionParser newInstance() {
return TypedSPILoader.getService(InlineExpressionParser.class, IS_SUBSTRATE_VM ? "ESPRESSO" : "HOTSPOT");
return TypedSPILoader.getService(InlineExpressionParser.class, IS_SUBSTRATE_VM ? "PURELIST" : "HOTSPOT");
}
}
1 change: 1 addition & 0 deletions infra/expr/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -32,5 +32,6 @@
<module>core</module>
<module>hotsopt</module>
<module>espresso</module>
<module>purelist</module>
</modules>
</project>
41 changes: 41 additions & 0 deletions infra/expr/purelist/pom.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
~ Licensed to the Apache Software Foundation (ASF) under one or more
~ contributor license agreements. See the NOTICE file distributed with
~ this work for additional information regarding copyright ownership.
~ The ASF licenses this file to You under the Apache License, Version 2.0
~ (the "License"); you may not use this file except in compliance with
~ the License. You may obtain a copy of the License at
~
~ http://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing, software
~ distributed under the License is distributed on an "AS IS" BASIS,
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~ See the License for the specific language governing permissions and
~ limitations under the License.
-->

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.apache.shardingsphere</groupId>
<artifactId>shardingsphere-infra-expr</artifactId>
<version>5.4.1-SNAPSHOT</version>
</parent>
<artifactId>shardingsphere-infra-expr-purelist</artifactId>
<name>${project.artifactId}</name>

<dependencies>
<dependency>
<groupId>org.apache.shardingsphere</groupId>
<artifactId>shardingsphere-infra-expr-spi</artifactId>
<version>${project.version}</version>
</dependency>

<dependency>
<groupId>org.apache.groovy</groupId>
<artifactId>groovy</artifactId>
</dependency>
</dependencies>
</project>
Loading

0 comments on commit 92ab477

Please sign in to comment.