Enable ASan, TSan, UBSan for aarch64-apple-darwin. #79883

frewsxcv · 2020-12-10T04:56:46Z

I confirmed ASan, TSan, UBSan all work for me locally with clang on my new Macbook Air.

~~This requires rust-lang/llvm-project#86~~

rust-highfive · 2020-12-10T04:56:50Z

r? @lcnr

(rust-highfive has picked a reviewer for you, use r? to override)

frewsxcv · 2020-12-10T04:57:30Z

I'm compiling Rust locally with these changes to confirm.

frewsxcv · 2020-12-10T05:24:24Z

/Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/bin/rustc -Zsanitizer=address /Users/coreyf/dev/rust-lang/rust/src/test/ui/sanitize/new-llvm-pass-manager-thin-lto.rs
error: linking with `cc` failed: exit code: 1
  |
  = note: "cc" "-arch" "arm64" "-Wl,-rpath" "-Xlinker" "/Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib" "-lrustc-dev_rt.asan" "-L" "/Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib" "new-llvm-pass-manager-thin-lto.new_llvm_pass_manager_thin_lto.7rcbfp3g-cgu.0.rcgu.o" "new-llvm-pass-manager-thin-lto.new_llvm_pass_manager_thin_lto.7rcbfp3g-cgu.1.rcgu.o" "new-llvm-pass-manager-thin-lto.new_llvm_pass_manager_thin_lto.7rcbfp3g-cgu.2.rcgu.o" "new-llvm-pass-manager-thin-lto.new_llvm_pass_manager_thin_lto.7rcbfp3g-cgu.3.rcgu.o" "new-llvm-pass-manager-thin-lto.new_llvm_pass_manager_thin_lto.7rcbfp3g-cgu.4.rcgu.o" "new-llvm-pass-manager-thin-lto.new_llvm_pass_manager_thin_lto.7rcbfp3g-cgu.5.rcgu.o" "new-llvm-pass-manager-thin-lto.new_llvm_pass_manager_thin_lto.7rcbfp3g-cgu.6.rcgu.o" "new-llvm-pass-manager-thin-lto.new_llvm_pass_manager_thin_lto.7rcbfp3g-cgu.7.rcgu.o" "-o" "new-llvm-pass-manager-thin-lto" "new-llvm-pass-manager-thin-lto.3s7blind72jn2fhm.rcgu.o" "-Wl,-dead_strip" "-nodefaultlibs" "-L" "/Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib" "/Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib/libstd-65ab0e7d9cf0fb87.rlib" "/Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib/libpanic_unwind-4c9cd63d6513f642.rlib" "/Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib/libobject-a259f338f0b7b73d.rlib" "/Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib/libaddr2line-df5c21a60c2d9580.rlib" "/Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib/libgimli-912b769937cf57e6.rlib" "/Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib/librustc_demangle-50dd149dc108cbb5.rlib" "/Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib/libhashbrown-02799d7e76ed6237.rlib" "/Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib/librustc_std_workspace_alloc-4896ae0049a4a17c.rlib" "/Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib/libunwind-5cf7fb4e1b850582.rlib" "/Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib/libcfg_if-44d6c655565382dd.rlib" "/Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib/liblibc-83d4eff421d392fe.rlib" "/Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib/liballoc-e50135f33201856a.rlib" "/Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib/librustc_std_workspace_core-405f59a762fe09ad.rlib" "/Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib/libcore-cd4ecd8429490fa9.rlib" "/Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib/libcompiler_builtins-8f64d161accbb588.rlib" "-lSystem" "-lresolv" "-lc" "-lm"
  = note: ld: library not found for -lrustc-dev_rt.asan
          clang: error: linker command failed with exit code 1 (use -v to see invocation)
          

error: aborting due to previous error

I will investigate tomorrow

lcnr · 2020-12-10T15:47:05Z

can't really review this, as you've already requested a review by @shepmaster

r? @shepmaster

frewsxcv · 2020-12-10T17:40:34Z

Regarding my last comment, I forgot to enable sanitizers in my config.toml. I then had an issue where running a binary generated with -Zsanitizer=address would result in a sigkill. I then found this in the compiler build logs:

/Library/Developer/CommandLineTools/usr/bin/install_name_tool: warning: changes being made to the file will invalidate the code signature in: /Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib/librustc-dev_rt.asan.dylib (for architecture x86_64)
/Library/Developer/CommandLineTools/usr/bin/install_name_tool: warning: changes being made to the file will invalidate the code signature in: /Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib/librustc-dev_rt.asan.dylib (for architecture x86_64h)
/Library/Developer/CommandLineTools/usr/bin/install_name_tool: warning: changes being made to the file will invalidate the code signature in: /Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib/librustc-dev_rt.asan.dylib (for architecture arm64)
/Library/Developer/CommandLineTools/usr/bin/install_name_tool: warning: changes being made to the file will invalidate the code signature in: /Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib/librustc-dev_rt.lsan.dylib (for architecture x86_64)
/Library/Developer/CommandLineTools/usr/bin/install_name_tool: warning: changes being made to the file will invalidate the code signature in: /Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib/librustc-dev_rt.lsan.dylib (for architecture x86_64h)
/Library/Developer/CommandLineTools/usr/bin/install_name_tool: warning: changes being made to the file will invalidate the code signature in: /Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib/librustc-dev_rt.lsan.dylib (for architecture arm64)
/Library/Developer/CommandLineTools/usr/bin/install_name_tool: warning: changes being made to the file will invalidate the code signature in: /Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib/librustc-dev_rt.tsan.dylib (for architecture x86_64)
/Library/Developer/CommandLineTools/usr/bin/install_name_tool: warning: changes being made to the file will invalidate the code signature in: /Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib/librustc-dev_rt.tsan.dylib (for architecture x86_64h)
/Library/Developer/CommandLineTools/usr/bin/install_name_tool: warning: changes being made to the file will invalidate the code signature in: /Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib/librustc-dev_rt.tsan.dylib (for architecture arm64)

These warning occur when we run install_name_tool (e.g. install_name_tool -id @rpath/librustc-dev_rt.asan.dylib /Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib/librustc-dev_rt.asan.dylib).

So it seems that by changing the dynamic libraries' names, we invalidate the signature. If I manually re-sign via:

codesign -f -s - ./build/aarch64-apple-darwin/stage1/lib/rustlib/aarch64-apple-darwin/lib/librustc-dev_rt.asan.dylib

Then ASAN seems to work:

% cat badfree.rs
// needs-sanitizer-support
// needs-sanitizer-address
//
// compile-flags: -Z sanitizer=address -O
//
// run-fail
// error-pattern: AddressSanitizer: SEGV

use std::ffi::c_void;

extern "C" {
    fn free(ptr: *mut c_void);
}

fn main() {
    unsafe {
        free(1 as *mut c_void);
    }
}

% /Users/coreyf/dev/rust-lang/rust/build/aarch64-apple-darwin/stage1/bin/rustc -Zsanitizer=address badfree.rs

% ./badfree
==21477==ERROR: AddressSanitizer failed to allocate 0x200004000 (8589950976) bytes at address fffffc000 (errno: 12)
==21477==ReserveShadowMemoryRange failed while trying to map 0x200004000 bytes. Perhaps you're using ulimit -v
zsh: abort      ./badfree

alex · 2020-12-10T17:43:04Z

That's not really the ASAN error I'd expect from that code, FWIW -- what I get on x86 looks way more sensible.

frewsxcv · 2020-12-10T17:44:51Z

That's not really the ASAN error I'd expect from that code, FWIW -- what I get on x86 looks way more sensible.

I was just wondering about this. It seems like ASan is failing to allocate memory for itself and this is just the error log? Hm

frewsxcv · 2020-12-10T17:45:48Z

Here's the LLDB output, which maybe confirms that:

(lldb) target create "./badfree"
Current executable set to '/Users/coreyf/dev/rust-lang/rust/badfree' (arm64).

(lldb) run
Process 21550 launched: '/Users/coreyf/dev/rust-lang/rust/badfree' (arm64)
==21550==ERROR: AddressSanitizer failed to allocate 0x200004000 (8589950976) bytes at address fffffc000 (errno: 12)
==21550==ReserveShadowMemoryRange failed while trying to map 0x200004000 bytes. Perhaps you're using ulimit -v
Process 21550 stopped
* thread #1, stop reason = signal SIGABRT
    frame #0: 0x000000018be3fcec libsystem_kernel.dylib`__pthread_kill + 8
libsystem_kernel.dylib`__pthread_kill:
->  0x18be3fcec <+8>:  b.lo   0x18be3fd0c               ; <+40>
    0x18be3fcf0 <+12>: pacibsp 
    0x18be3fcf4 <+16>: stp    x29, x30, [sp, #-0x10]!
    0x18be3fcf8 <+20>: mov    x29, sp
Target 0: (badfree) stopped.

(lldb) thread backtrace
* thread #1, stop reason = signal SIGABRT
  * frame #0: 0x000000018be3fcec libsystem_kernel.dylib`__pthread_kill + 8
    frame #1: 0x000000018be70c24 libsystem_pthread.dylib`pthread_kill + 292
    frame #2: 0x000000018bdb8864 libsystem_c.dylib`abort + 104
    frame #3: 0x00000001001bf054 librustc-dev_rt.asan.dylib`::Abort() at sanitizer_posix_libcdep.cpp:155:3 [opt]
    frame #4: 0x00000001001ab200 librustc-dev_rt.asan.dylib`::ReserveShadowMemoryRange() at asan_shadow_setup.cpp:38:5 [opt]
    frame #5: 0x00000001001ab27c librustc-dev_rt.asan.dylib`::InitializeShadowMemory() at asan_shadow_setup.cpp:129:7 [opt]
    frame #6: 0x00000001001aa6f0 librustc-dev_rt.asan.dylib`::AsanInitInternal() at asan_rtl.cpp:463:3 [opt]
    frame #7: 0x00000001001aa5a0 librustc-dev_rt.asan.dylib`::AsanInitFromRtl() at asan_rtl.cpp:537:3 [opt] [artificial]
    frame #8: 0x00000001001a1fa0 librustc-dev_rt.asan.dylib`::wrap_malloc_default_zone() at sanitizer_malloc_mac.inc:86:3 [opt]
    frame #9: 0x000000018bca48c0 libsystem_malloc.dylib`__malloc_init + 896
    frame #10: 0x00000001954c27c4 libSystem.B.dylib`libSystem_initializer + 184
    frame #11: 0x000000010007f90c dyld`ImageLoaderMachO::doModInitFunctions(ImageLoader::LinkContext const&) + 868
    frame #12: 0x000000010007fb94 dyld`ImageLoaderMachO::doInitialization(ImageLoader::LinkContext const&) + 56
    frame #13: 0x000000010007984c dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, char const*, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 620
    frame #14: 0x0000000100079794 dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, char const*, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 436
    frame #15: 0x0000000100079794 dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, char const*, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 436
    frame #16: 0x0000000100079794 dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, char const*, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 436
    frame #17: 0x0000000100077300 dyld`ImageLoader::processInitializers(ImageLoader::LinkContext const&, unsigned int, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 192
    frame #18: 0x00000001000773cc dyld`ImageLoader::runInitializers(ImageLoader::LinkContext const&, ImageLoader::InitializerTimingList&) + 96
    frame #19: 0x000000010006284c dyld`dyld::initializeMainExecutable() + 220
    frame #20: 0x0000000100068b98 dyld`dyld::_main(macho_header const*, unsigned long, int, char const**, char const**, char const**, unsigned long*) + 7388
    frame #21: 0x0000000100061258 dyld`dyldbootstrap::start(dyld3::MachOLoaded const*, int, char const**, dyld3::MachOLoaded const*, unsigned long*) + 476
    frame #22: 0x0000000100061038 dyld`_dyld_start + 56

alex · 2020-12-10T17:49:17Z

I agree that it looks like ASAN is failing to set itself up.

frewsxcv · 2020-12-10T23:30:25Z

@saleemrashid pointed me to llvm/llvm-project@176a6e7 which I cherry-picked on top of Rust's current LLVM commit and it fixes the previous issue! d517333

frewsxcv · 2020-12-11T00:19:03Z

This is ready for review! Note that this pull request is blocked on rust-lang/llvm-project#86

frewsxcv · 2020-12-18T22:14:45Z

rust-lang/llvm-project#86 has merged so it's now just waiting for a review

bors · 2021-01-01T04:03:56Z

☔ The latest upstream changes (presumably #80566) made this pull request unmergeable. Please resolve the merge conflicts.

frewsxcv · 2021-01-01T04:28:03Z

Merge conflicts resolved

alex

One question, otherwise looks good to me (with the caveat that I'm far from a rustc expert!)

alex · 2021-01-01T16:26:37Z

compiler/rustc_session/src/session.rs

+        "aarch64-unknown-linux-gnu",
+        "x86_64-apple-darwin",
+        "x86_64-unknown-linux-gnu",
+    ];
    const MSAN_SUPPORTED_TARGETS: &[&str] =
        &["aarch64-unknown-linux-gnu", "x86_64-unknown-freebsd", "x86_64-unknown-linux-gnu"];


Why not MSAN :-)

I returned my new Macbook Air (need more memory and waiting for the new one to ship), so I can't easily confirm, but I recall there being an error message saying macOS is not supported by MSan, which is maybe why x86_64-apple-darwin is also not listed here. These docs also back up this claim

https://clang.llvm.org/docs/MemorySanitizer.html#supported-platforms

https://firefox-source-docs.mozilla.org/tools/sanitizer/memory_sanitizer.html#build-prerequisites

I clearly have the reading comprehension of a goldfish, I totally missed that no macOS platforms were there.

/~https://github.com/llvm/llvm-project/blob/62ec4ac90738a5f2d209ed28c822223e58aaaeb7/compiler-rt/cmake/config-ix.cmake#L696-L701

if (COMPILER_RT_HAS_SANITIZER_COMMON AND MSAN_SUPPORTED_ARCH AND OS_NAME MATCHES "Linux|FreeBSD|NetBSD") set(COMPILER_RT_HAS_MSAN TRUE) else() set(COMPILER_RT_HAS_MSAN FALSE) endif()

shepmaster · 2021-01-01T19:24:33Z

src/bootstrap/compile.rs

+            // Upon renaming the install name, the code signature of the file will invalidate,
+            // so we will sign it again.


I don't follow this. Renaming other signed files doesn't cause problems; why does this?

Renaming the file with mv doesn't invalidate the signature, but renaming the install name (within the contents of the file) with install_name_tool does invalidate.

Searching for "install_name_tool invalidate signature" on Google returns other projects that have hit this. For example: https://community.intel.com/t5/Intel-oneAPI-Threading-Building/Mac-install-location-conflicts-with-code-signing/m-p/1008483/highlight/true#M12505

shepmaster · 2021-01-01T20:08:46Z

@bors r+ rollup

My understanding is that these tools aren't critical so if we realize that this breaks them somehow, it won't be a big thing until we can fix them.

bors · 2021-01-01T20:08:47Z

📌 Commit d482de3 has been approved by shepmaster

frewsxcv · 2021-01-01T20:14:50Z

@bors r+ rollup

My understanding is that these tools aren't critical so if we realize that this breaks them somehow, it won't be a big thing until we can fix them.

Thanks for the review! What you said sounds right. The only functionality I'm changing for existing platforms is the addition of that extra code signing step for macOS machines. So if we saw an regressions on other platforms, I'd expect it to be just limited to macOS, and that code addition in particular.

bors · 2021-01-02T06:59:05Z

⌛ Testing commit d482de3 with merge 5986dd8...

bors · 2021-01-02T09:52:20Z

☀️ Test successful - checks-actions
Approved by: shepmaster
Pushing 5986dd8 to master...

Enable ASan, TSan, UBSan for aarch64-apple-darwin.

5940c19

rust-highfive assigned lcnr Dec 10, 2020

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Dec 10, 2020

frewsxcv requested a review from shepmaster December 10, 2020 04:56

frewsxcv added 2 commits December 10, 2020 00:18

rustfmt

7420776

If "aarch64-apple-darwin", copy sanitizers.

d1911dd

Fix sanitizer test output matching

01029e2

rust-highfive assigned shepmaster and unassigned lcnr Dec 10, 2020

Use fork of llvm-project that includes Apple Silicon fix

d517333

frewsxcv mentioned this pull request Dec 10, 2020

Cherry-pick commit that fixes a sanitizer bug on Apple Silicon rust-lang/llvm-project#86

Merged

Manually code-sign after running install_name_tool

25bfa15

frewsxcv added the A-sanitizers Area: Sanitizers for correctness and code quality label Dec 11, 2020

frewsxcv mentioned this pull request Dec 11, 2020

Tracking issue for supporting macOS on Apple Silicon (a.k.a arm64, M1, M2, aarch64) #73908

Closed

31 tasks

frewsxcv added 2 commits December 10, 2020 19:06

Fix typo in code comment

5ab1602

Update sanitizer supported docs to reflect reality

7ee606f

Switch back to rust-lang/llvm-project now that the patch has merged

26cc060

frewsxcv added the O-macos Operating system: macOS label Dec 18, 2020

tmiasko approved these changes Dec 23, 2020

View reviewed changes

Merge remote-tracking branch 'origin/master' into frewsxcv-san

d482de3

alex reviewed Jan 1, 2021

View reviewed changes

alex approved these changes Jan 1, 2021

View reviewed changes

shepmaster reviewed Jan 1, 2021

View reviewed changes

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jan 1, 2021

bors added the merged-by-bors This PR was explicitly merged by bors. label Jan 2, 2021

bors merged commit 5986dd8 into rust-lang:master Jan 2, 2021

rustbot added this to the 1.51.0 milestone Jan 2, 2021

frewsxcv deleted the frewsxcv-san branch January 2, 2021 14:51

petrochenkov mentioned this pull request Sep 22, 2021

Disable the leak sanitizer on Macos aarch64 for now #89170

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable ASan, TSan, UBSan for aarch64-apple-darwin. #79883

Enable ASan, TSan, UBSan for aarch64-apple-darwin. #79883

frewsxcv commented Dec 10, 2020 •

edited

Loading

rust-highfive commented Dec 10, 2020

frewsxcv commented Dec 10, 2020

frewsxcv commented Dec 10, 2020

lcnr commented Dec 10, 2020

frewsxcv commented Dec 10, 2020

alex commented Dec 10, 2020

frewsxcv commented Dec 10, 2020

frewsxcv commented Dec 10, 2020

alex commented Dec 10, 2020

frewsxcv commented Dec 10, 2020 •

edited

Loading

frewsxcv commented Dec 11, 2020

frewsxcv commented Dec 18, 2020

bors commented Jan 1, 2021

frewsxcv commented Jan 1, 2021

alex left a comment

alex Jan 1, 2021

frewsxcv Jan 1, 2021

alex Jan 1, 2021

frewsxcv Jan 1, 2021

shepmaster Jan 1, 2021

frewsxcv Jan 1, 2021

shepmaster commented Jan 1, 2021

bors commented Jan 1, 2021

frewsxcv commented Jan 1, 2021

bors commented Jan 2, 2021

bors commented Jan 2, 2021

		// Upon renaming the install name, the code signature of the file will invalidate,
		// so we will sign it again.

Enable ASan, TSan, UBSan for aarch64-apple-darwin. #79883

Enable ASan, TSan, UBSan for aarch64-apple-darwin. #79883

Conversation

frewsxcv commented Dec 10, 2020 • edited Loading

rust-highfive commented Dec 10, 2020

frewsxcv commented Dec 10, 2020

frewsxcv commented Dec 10, 2020

lcnr commented Dec 10, 2020

frewsxcv commented Dec 10, 2020

alex commented Dec 10, 2020

frewsxcv commented Dec 10, 2020

frewsxcv commented Dec 10, 2020

alex commented Dec 10, 2020

frewsxcv commented Dec 10, 2020 • edited Loading

frewsxcv commented Dec 11, 2020

frewsxcv commented Dec 18, 2020

bors commented Jan 1, 2021

frewsxcv commented Jan 1, 2021

alex left a comment

Choose a reason for hiding this comment

alex Jan 1, 2021

Choose a reason for hiding this comment

frewsxcv Jan 1, 2021

Choose a reason for hiding this comment

alex Jan 1, 2021

Choose a reason for hiding this comment

frewsxcv Jan 1, 2021

Choose a reason for hiding this comment

shepmaster Jan 1, 2021

Choose a reason for hiding this comment

frewsxcv Jan 1, 2021

Choose a reason for hiding this comment

shepmaster commented Jan 1, 2021

bors commented Jan 1, 2021

frewsxcv commented Jan 1, 2021

bors commented Jan 2, 2021

bors commented Jan 2, 2021

frewsxcv commented Dec 10, 2020 •

edited

Loading

frewsxcv commented Dec 10, 2020 •

edited

Loading