Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update to 2.1.55 and refactor the build system #246

Merged
merged 28 commits into from
Dec 16, 2022
Merged
Show file tree
Hide file tree
Changes from 25 commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
5dc141e
Move anki submodule to top level
dae Nov 27, 2022
5b5dc6b
Add a missing drop() that recent Rust versions complain about
dae Nov 27, 2022
d8f2a0b
Update to 2.1.55 + ninja build
dae Nov 27, 2022
7223936
Refactor the build system
dae Nov 28, 2022
ef0aa36
Tweak print statements in build.gradle
dae Nov 30, 2022
622459a
Elaborate on prerequisites
dae Nov 30, 2022
3f9c8e2
Integrate HOWTO inside README; remove more obsolete instructions
dae Nov 30, 2022
7950791
Remove plantuml files
dae Nov 30, 2022
13e9923
Add platform guard to multi-arch build
dae Nov 30, 2022
d4d32de
Fix clippy lints; check in CI
dae Nov 30, 2022
f27a908
Include lint in check, and remind users they have to use build-aar first
dae Nov 30, 2022
5ef455b
Custom path no longer required for rsdroid-testing
dae Nov 30, 2022
7b4366d
Reduce verbosity of cargo build in aar
dae Nov 30, 2022
b594978
Mention python3 env
dae Dec 1, 2022
33b1bcc
Strip some obsolete stuff from doctor.sh
dae Dec 1, 2022
df58e36
Increase timeout to 10 seconds
dae Dec 1, 2022
3bd8060
Mention msys2
dae Dec 1, 2022
076b89a
Apply build-all.yml suggestions from Mike
dae Dec 1, 2022
d6848a7
Increase timeout of multi-arch build
dae Dec 2, 2022
8fc59cd
Tweak cache keys
dae Dec 2, 2022
8ae02fb
Confirm aar/jar output
dae Dec 2, 2022
a5459d5
Change Mani's PS script to a .bat file
dae Dec 2, 2022
f0d18c4
Fix incorrect build target in build-aar.bat
dae Dec 2, 2022
99ea300
Use desktop build's bundled Python on Windows
dae Dec 2, 2022
9540729
Add os suffix to artifacts
dae Dec 2, 2022
f8c23f0
Apply suggestions from code review
dae Dec 3, 2022
ea75f34
Only hash lockfiles
dae Dec 3, 2022
0eb47a8
Update to stable release commit
dae Dec 16, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 6 additions & 3 deletions .cargo/config.toml
Original file line number Diff line number Diff line change
@@ -1,4 +1,7 @@
[build]
# override any global customizations, as the gradle plugin requires a specific path
target-dir = "target"

[env]
RSLIB_FTL_ROOT = { value = "ftl/core/l10n.toml", relative = true }
BUILDINFO = { value = "rsdroid/build/generated/anki_artifacts/buildinfo.txt", relative = true }
BAZEL = "1"
STRINGS_JSON = { value = "anki/out/strings.json", relative = true }
PROTOC = { value = "anki/out/extracted/protoc/bin/protoc", relative = true }
4 changes: 0 additions & 4 deletions .github/scripts/install_rust_robolectric_targets.sh

This file was deleted.

9 changes: 0 additions & 9 deletions .github/scripts/install_rust_targets.sh

This file was deleted.

7 changes: 0 additions & 7 deletions .github/scripts/linux_install_macos_cross_compile.sh

This file was deleted.

15 changes: 0 additions & 15 deletions .github/scripts/linux_install_protobuf.sh

This file was deleted.

2 changes: 0 additions & 2 deletions .github/scripts/linux_install_x86_64-unknown-linux-gnu-gcc.sh

This file was deleted.

29 changes: 0 additions & 29 deletions .github/scripts/macos_install_protobuf_compiler.sh

This file was deleted.

7 changes: 0 additions & 7 deletions .github/scripts/macos_install_pyenv.sh

This file was deleted.

5 changes: 0 additions & 5 deletions .github/scripts/protoc_gen_deps.py

This file was deleted.

139 changes: 139 additions & 0 deletions .github/workflows/build-all.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,139 @@
name: Build AAR and Robo (all platforms)
on:
workflow_dispatch:
inputs:
mavenPublish:
description: "Publish artifacts to Maven Central? non-empty to publish, empty to skip publish"
required: false
type: string
push:
schedule:
- cron: "0 0 * * *"
pull_request:

env:
ALL_ARCHS: 1

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}-release
cancel-in-progress: true

jobs:
build:
runs-on: macos-latest
timeout-minutes: 120
steps:
- uses: actions/checkout@v2

- name: Fetch submodules
run: git submodule update --init

- name: Setup Ninja
uses: ashutoshvarma/setup-ninja@master
with:
version: 1.10.0

# The action puts a relative path on the PATH 🙄
- name: Make path absolute
run: echo $(pwd)/ninja_bin >> $GITHUB_PATH

- name: Configure JDK 1.11
uses: actions/setup-java@v3
with:
distribution: "adopt"
java-version: "11" # minimum for Android API31

- name: Install Android Command Line Tools
uses: android-actions/setup-android@v2

- name: Install Windows cross compiler
run: brew install mingw-w64 && x86_64-w64-mingw32-gcc -v

- name: Install Linux cross compiler
run: |
brew tap SergioBenitez/osxct
brew install x86_64-unknown-linux-gnu
x86_64-unknown-linux-gnu-gcc -v

- name: Install NDK
run: .github/scripts/install_ndk.sh 22.0.7026061

- name: Rust Cache
uses: actions/cache@v3
with:
path: |
~/.cargo/registry
~/.cargo/git
target
key: ${{ runner.os }}-rust-release-v2-${{ hashFiles('Cargo.lock', 'rslib-bridge/**', 'anki/rslib/**', 'anki/ftl/**', 'anki/proto/**') }}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you really want to hash every single file here instead of just the Cargo.lock files? That seems really expensive - it's already pretty expensive just doing the file scanning (in my experience it can add up to quite a bit of CI time). Best perhaps to focus these on just known Cargo.lock files that conclusively imply something has changed and warrants a cache upload?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't have a preference. The lockfiles should be sufficient of the majority of the cached content; changes to the other files only affect the compilation output of our first-party crates.

restore-keys: |
${{ runner.os }}-rust-release-v2
${{ runner.os }}-rust

- name: Anki cache
uses: actions/cache@v3
with:
path: |
anki/out/node_modules
anki/out/rust
anki/out/extracted
key: ${{ runner.os }}-anki-v4-${{ hashFiles('Cargo.lock', 'anki/package.json', 'anki/build/**', 'anki/rslib/**', 'anki/ftl/**', 'anki/proto/**') }}
restore-keys: |
${{ runner.os }}-anki-v4

- name: Build Android/All
run: ./build-aar.sh

- name: Build Robolectric/All
run: ./build-robo.sh

- name: Check Compiled Libraries
run: >
cd rsdroid-testing/assets &&
../../.github/scripts/check_robolectric_assets.sh

- name: Upload rsdroid AAR as artifact
uses: actions/upload-artifact@v2
with:
name: rsdroid-aar
if-no-files-found: error
path: rsdroid/build/outputs/aar

- name: Upload rsdroid-robo JAR as artifact
uses: actions/upload-artifact@v2
with:
name: rsdroid-robo
if-no-files-found: error
path: rsdroid-testing/build/libs

# following steps only run on workflow dispatch

- name: Publish AAR to Maven
if: "${{ github.event.inputs.mavenPublish != '' && github.event_name == 'workflow_dispatch'}}"
env:
ORG_GRADLE_PROJECT_SIGNING_PRIVATE_KEY: ${{ secrets.SIGNING_PRIVATE_KEY }}
ORG_GRADLE_PROJECT_SIGNING_PASSWORD: ${{ secrets.SIGNING_PASSWORD }}
SONATYPE_NEXUS_USERNAME: david-allison-1
SONATYPE_NEXUS_PASSWORD: ${{ secrets.SONATYPE_NEXUS_PASSWORD }}
run: |
./gradlew rsdroid:uploadArchives -DtestBuildType=release -Dorg.gradle.daemon=false -Dorg.gradle.console=plain

- name: Publish JAR to Maven
if: "${{ github.event.inputs.mavenPublish != '' && github.event_name == 'workflow_dispatch'}}"
env:
ORG_GRADLE_PROJECT_SIGNING_PRIVATE_KEY: ${{ secrets.SIGNING_PRIVATE_KEY }}
ORG_GRADLE_PROJECT_SIGNING_PASSWORD: ${{ secrets.SIGNING_PASSWORD }}
SONATYPE_NEXUS_USERNAME: david-allison-1
SONATYPE_NEXUS_PASSWORD: ${{ secrets.SONATYPE_NEXUS_PASSWORD }}
run: |
export ANKIDROID_LINUX_CC=x86_64-unknown-linux-gnu-gcc
export ANKIDROID_MACOS_CC=cc
export RUST_DEBUG=1
export RUST_BACKTRACE=1
export RUST_LOG=trace
export NO_CROSS=true
./gradlew rsdroid-testing:uploadArchives -Dorg.gradle.project.macCC=$ANKIDROID_MACOS_CC -DtestBuildType=debug -Dorg.gradle.daemon=false -Dorg.gradle.console=plain

- name: ℹ️Additional Release Instructions (requires human interaction)
if: "${{ github.event.inputs.mavenPublish != '' && github.event_name == 'workflow_dispatch'}}"
run: echo "Sign in to https://oss.sonatype.org/#stagingRepositories , close the repository, then release it"
126 changes: 126 additions & 0 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
name: Build AAR and Robo (single platform)
on:
workflow_dispatch:
push:
schedule:
- cron: "0 0 * * *"
pull_request:

env:
DEBUG: 1

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

jobs:
build:
strategy:
matrix:
os: [ubuntu-latest, windows-latest, macos-latest]
fail-fast: false
runs-on: ${{ matrix.os }}
timeout-minutes: 80
steps:
- uses: actions/checkout@v2

- name: Fetch submodules
run: git submodule update --init

- name: Setup Ninja
uses: ashutoshvarma/setup-ninja@master
with:
version: 1.10.0

# The action puts a relative path on the PATH 🙄
- name: Make path absolute
run: echo $(pwd)/ninja_bin >> $GITHUB_PATH

- name: Configure JDK 1.11
uses: actions/setup-java@v3
with:
distribution: "adopt"
java-version: "11" # minimum for Android API31

- name: Install Android Command Line Tools
uses: android-actions/setup-android@v2

- name: Install NDK
if: matrix.os != 'windows-latest'
run: .github/scripts/install_ndk.sh 22.0.7026061

- name: Install NDK (Windows)
if: matrix.os == 'windows-latest'
run: |
Write-Host "NDK Install Started"
(. sdkmanager.bat --install "ndk;22.0.7026061" --sdk_root="$Env:ANDROID_SDK_ROOT") | out-null
Write-Host "NDK Install Completed"

- name: Rust Cache
uses: actions/cache@v3
with:
path: |
~/.cargo/registry
~/.cargo/git
target
key: ${{ runner.os }}-rust-debug-v4-${{ hashFiles('rslib-bridge/**/Cargo.lock') }}
restore-keys: |
${{ runner.os }}-rust-debug-v4
${{ runner.os }}-rust-debug

- name: Anki cache
uses: actions/cache@v3
with:
path: |
anki/out/node_modules
anki/out/rust
anki/out/extracted
key: ${{ runner.os }}-anki-v4-${{ hashFiles('anki/package.json') }}
restore-keys: |
${{ runner.os }}-anki-v4
${{ runner.os }}-anki

- name: Build Android/X86_64
if: matrix.os != 'windows-latest'
run: ./build-aar.sh

- name: Build Android/X86_64 (Win)
if: matrix.os == 'windows-latest'
run: ./build-aar.bat

- name: Check Rust (Linux)
if: matrix.os == 'ubuntu-latest'
run: ./check-rust.sh

- name: Run tests (Mac)
uses: reactivecircus/android-emulator-runner@v2
if: matrix.os == 'macos-latest'
timeout-minutes: 30
with:
api-level: 21
target: default
arch: x86_64
profile: Nexus 6
script: ./check-droid.sh

- name: Build Robolectric/X86_64
if: matrix.os != 'windows-latest'
run: ./build-robo.sh

- name: Build Robolectric/X86_64 (Win)
if: matrix.os == 'windows-latest'
run: ./build-robo.bat

- name: Upload rsdroid AAR as artifact
uses: actions/upload-artifact@v2
with:
name: rsdroid-aar-${{ matrix.os }}
if-no-files-found: error
path: rsdroid/build/outputs/aar

- name: Upload rsdroid-robo JAR as artifact
uses: actions/upload-artifact@v2
with:
name: rsdroid-robo-${{ matrix.os }}
if-no-files-found: error
path: rsdroid-testing/build/libs
Loading