Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

lazy: Fix data race in TestGoodUpdateThenResolverError #8088

Merged
merged 1 commit into from
Feb 17, 2025

Conversation

arjan-bal
Copy link
Contributor

The test verifies that lazy's child get the latest resolver state followed by the latest resolver error. The test waits for an RPC to succeed before verifying the child has received the resolver error. The RPC can complete before the child get the resolver error. This PR uses a grpcsync.Event to wait till the child receives the resolver error.

Failing test run: /~https://github.com/grpc/grpc-go/actions/runs/13314475263/attempts/2?pr=8086

Failure stack trace
WARNING: DATA RACE
Read at 0x00c00019c57f by goroutine 191:
  google.golang.org/grpc/balancer/lazy_test.s.TestGoodUpdateThenResolverError()
      /home/runner/work/grpc-go/grpc-go/balancer/lazy/lazy_ext_test.go:289 +0x13b7
  google.golang.org/grpc/balancer/lazy_test.(*s).TestGoodUpdateThenResolverError()
      <autogenerated>:1 +0x3e
  runtime.call16()
      /opt/hostedtoolcache/go/1.23.6/x[64](/~https://github.com/grpc/grpc-go/actions/runs/13314475263/job/37208920450?pr=8086#step:8:65)/src/runtime/asm_amd64.s:775 +0x42
  reflect.callMethod()
      <autogenerated>:1 +0x56
  reflect.methodValueCall()
      /opt/hostedtoolcache/go/1.23.6/x64/src/reflect/asm_amd64.s:76 +0x6d
  testing.tRunner()
      /opt/hostedtoolcache/go/1.23.6/x64/src/testing/testing.go:1690 +0x226
  testing.(*T).Run.gowrap1()
      /opt/hostedtoolcache/go/1.23.6/x64/src/testing/testing.go:1743 +0x44

Previous write at 0x00c00019c57f by goroutine 200:
  google.golang.org/grpc/balancer/lazy_test.s.TestGoodUpdateThenResolverError.func3()
      /home/runner/work/grpc-go/grpc-go/balancer/lazy/lazy_ext_test.go:216 +0xa4
  google.golang.org/grpc/internal/balancer/stub.(*bal).ResolverError()
      /home/runner/work/grpc-go/grpc-go/internal/balancer/stub/stub.go:70 +0x94
  google.golang.org/grpc/balancer/lazy.(*lazyBalancer).ExitIdle()
      /home/runner/work/grpc-go/grpc-go/balancer/lazy/lazy.go:145 +0x586
  google.golang.org/grpc/balancer/lazy.NewBalancer.func1.gowrap1()
      /home/runner/work/grpc-go/grpc-go/balancer/lazy/lazy.go:[65](/~https://github.com/grpc/grpc-go/actions/runs/13314475263/job/37208920450?pr=8086#step:8:66) +0x33

Goroutine 191 (running) created at:
  testing.(*T).Run()
      /opt/hostedtoolcache/go/1.23.6/x64/src/testing/testing.go:1743 +0x825
  google.golang.org/grpc/internal/grpctest.RunSubTests()
      /home/runner/work/grpc-go/grpc-go/internal/grpctest/grpctest.go:114 +0x351
  google.golang.org/grpc/balancer/lazy_test.Test()
      /home/runner/work/grpc-go/grpc-go/balancer/lazy/lazy_ext_test.go:60 +0x34
  testing.tRunner()
      /opt/hostedtoolcache/go/1.23.6/x64/src/testing/testing.go:1690 +0x226
  testing.(*T).Run.gowrap1()
      /opt/hostedtoolcache/go/1.23.6/x64/src/testing/testing.go:1743 +0x44

Goroutine 200 (finished) created at:
  google.golang.org/grpc/balancer/lazy.NewBalancer.func1()
      /home/runner/work/grpc-go/grpc-go/balancer/lazy/lazy.go:65 +0x8d
  sync.OnceFunc.func1()
      /opt/hostedtoolcache/go/1.23.6/x64/src/sync/oncefunc.go:27 +0x7b
  sync.(*Once).doSlow()
      /opt/hostedtoolcache/go/1.23.6/x64/src/sync/once.go:76 +0xe1
  sync.(*Once).Do()
      /opt/hostedtoolcache/go/1.23.6/x64/src/sync/once.go:67 +0x44
  sync.OnceFunc.func2()
      /opt/hostedtoolcache/go/1.23.6/x64/src/sync/oncefunc.go:32 +0x4a
  google.golang.org/grpc/balancer/lazy.(*idlePicker).Pick()
      /home/runner/work/grpc-go/grpc-go/balancer/lazy/lazy.go:157 +0x35
  google.golang.org/grpc.(*pickerWrapper).pick()
      /home/runner/work/grpc-go/grpc-go/picker_wrapper.go:156 +0x948
  google.golang.org/grpc.(*ClientConn).getTransport()
      /home/runner/work/grpc-go/grpc-go/clientconn.go:1051 +0xae
  google.golang.org/grpc.(*csAttempt).getTransport()
      /home/runner/work/grpc-go/grpc-go/stream.go:470 +0x144
  google.golang.org/grpc.newClientStreamWithParams.func2()
      /home/runner/work/grpc-go/grpc-go/stream.go:353 +0x33
  google.golang.org/grpc.(*clientStream).withRetry()
      /home/runner/work/grpc-go/grpc-go/stream.go:790 +0x197
  google.golang.org/grpc.newClientStreamWithParams()
      /home/runner/work/grpc-go/grpc-go/stream.go:365 +0x1797
  google.golang.org/grpc.newClientStream.func3()
      /home/runner/work/grpc-go/grpc-go/stream.go:222 +0x191
  google.golang.org/grpc.newClientStream()
      /home/runner/work/grpc-go/grpc-go/stream.go:257 +0xc69
  google.golang.org/grpc.invoke()
      /home/runner/work/grpc-go/grpc-go/call.go:[66](/~https://github.com/grpc/grpc-go/actions/runs/13314475263/job/37208920450?pr=8086#step:8:67) +0xc6
  google.golang.org/grpc.(*ClientConn).Invoke()
      /home/runner/work/grpc-go/grpc-go/call.go:37 +0x2f0
  google.golang.org/grpc/interop/grpc_testing.(*testServiceClient).EmptyCall()
      /home/runner/work/grpc-go/grpc-go/interop/grpc_testing/test_grpc.pb.go:95 +0x1af
  google.golang.org/grpc/balancer/lazy_test.s.TestGoodUpdateThenResolverError()
      /home/runner/work/grpc-go/grpc-go/balancer/lazy/lazy_ext_test.go:281 +0x12dc
  google.golang.org/grpc/balancer/lazy_test.(*s).TestGoodUpdateThenResolverError()
      <autogenerated>:1 +0x3e
  runtime.call16()
      /opt/hostedtoolcache/go/1.23.6/x64/src/runtime/asm_amd64.s:775 +0x42
  reflect.callMethod()
      <autogenerated>:1 +0x56
  reflect.methodValueCall()
      /opt/hostedtoolcache/go/1.23.6/x64/src/reflect/asm_amd64.s:76 +0x6d
  testing.tRunner()
      /opt/hostedtoolcache/go/1.23.6/x64/src/testing/testing.go:1[69](/~https://github.com/grpc/grpc-go/actions/runs/13314475263/job/37208920450?pr=8086#step:8:70)0 +0x226
  testing.(*T).Run.gowrap1()
      /opt/hostedtoolcache/go/1.23.6/x64/src/testing/testing.go:1743 +0x44

RELEASE NOTES: N/A

@arjan-bal arjan-bal added Type: Testing Area: Resolvers/Balancers Includes LB policy & NR APIs, resolver/balancer/picker wrappers, LB policy impls and utilities. labels Feb 14, 2025
@arjan-bal arjan-bal added this to the 1.71 Release milestone Feb 14, 2025
@arjan-bal arjan-bal requested review from easwars and dfawley February 14, 2025 04:29
Copy link

codecov bot commented Feb 14, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 82.18%. Comparing base (fabe274) to head (87a41ab).
Report is 5 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #8088      +/-   ##
==========================================
- Coverage   82.20%   82.18%   -0.02%     
==========================================
  Files         387      387              
  Lines       38928    38943      +15     
==========================================
+ Hits        31999    32005       +6     
- Misses       5606     5615       +9     
  Partials     1323     1323              

see 23 files with indirect coverage changes

@dfawley dfawley assigned arjan-bal and unassigned easwars and dfawley Feb 14, 2025
@arjan-bal arjan-bal merged commit e55819e into grpc:master Feb 17, 2025
15 checks passed
@arjan-bal arjan-bal deleted the fix-race-lazy-test branch February 17, 2025 03:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area: Resolvers/Balancers Includes LB policy & NR APIs, resolver/balancer/picker wrappers, LB policy impls and utilities. Type: Bug
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants