Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

receive: Query failure on Seg fault #3497

Closed
bwplotka opened this issue Nov 23, 2020 · 12 comments
Closed

receive: Query failure on Seg fault #3497

bwplotka opened this issue Nov 23, 2020 · 12 comments
Labels
bug needs-investigation stale

Comments

@bwplotka
Copy link
Member

bwplotka commented Nov 23, 2020 •
edited
Loading

Looks like query uses ummaped piece of memory. Investigating.

level=error ts=2020-11-23T13:04:22.548388656Z caller=handler.go:331 component=receive component=receive-handler err="context deadline exceeded" msg="internal server error"
level=error ts=2020-11-23T13:05:22.832961821Z caller=handler.go:331 component=receive component=receive-handler err="context deadline exceeded" msg="internal server error"
level=error ts=2020-11-23T13:05:22.887742049Z caller=handler.go:331 component=receive component=receive-handler err="context deadline exceeded" msg="internal server error"
level=info ts=2020-11-23T13:06:06.449406977Z caller=db.go:1180 component=receive component=multi-tsdb tenant=FB870BF3-9F3A-44FF-9BF7-D7A047A52F43 msg="Deleting obsolete block" block=01EQG2JFVHM40T39RRBWNN7QQ9
level=info ts=2020-11-23T13:06:18.852083978Z caller=head.go:824 component=receive component=multi-tsdb tenant=FB870BF3-9F3A-44FF-9BF7-D7A047A52F43 msg="Head GC completed" duration=12.402633612s
level=info ts=2020-11-23T13:06:40.564396418Z caller=checkpoint.go:95 component=receive component=multi-tsdb tenant=FB870BF3-9F3A-44FF-9BF7-D7A047A52F43 msg="Creating checkpoint" from_segment=4406 to_segment=4408 mint=1606132800000
unexpected fault address 0x7fd9e74623a3
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x7fd9e74623a3 pc=0x470d27]

goroutine 247164100 [running]:
runtime.throw(0x1ae9f56, 0x5)
	/usr/local/go/src/runtime/panic.go:1116 +0x72 fp=0xc1a3cd1848 sp=0xc1a3cd1818 pc=0x4377d2
runtime.sigpanic()
	/usr/local/go/src/runtime/signal_unix.go:727 +0x405 fp=0xc1a3cd1878 sp=0xc1a3cd1848 pc=0x44df65
runtime.memmove(0xc152d1bf37, 0x7fd9e74623a3, 0x19)
	/usr/local/go/src/runtime/memmove_amd64.s:181 +0x147 fp=0xc1a3cd1880 sp=0xc1a3cd1878 pc=0x470d27
github.com/thanos-io/thanos/pkg/store/storepb.(*Chunk).MarshalToSizedBuffer(0xc449435f40, 0xc152d1bb00, 0x450, 0x450, 0xae4dc5, 0xc1a3cd19a0, 0xc44d2a7205)
	/go/src/github.com/thanos-io/thanos/pkg/store/storepb/types.pb.go:347 +0x13e fp=0xc1a3cd18c8 sp=0xc1a3cd1880 pc=0xbc521e
github.com/thanos-io/thanos/pkg/store/storepb.(*AggrChunk).MarshalToSizedBuffer(0xc449442340, 0xc152d1bb00, 0x450, 0x450, 0xc0006a2914, 0x24, 0x175f48f2a68)
	/go/src/github.com/thanos-io/thanos/pkg/store/storepb/types.pb.go:493 +0x357 fp=0xc1a3cd1938 sp=0xc1a3cd18c8 pc=0xbc5d37
github.com/thanos-io/thanos/pkg/store/storepb.(*Series).MarshalToSizedBuffer(0xc44e076120, 0xc152d1bb00, 0x450, 0x450, 0xbbb16d, 0xc44e076120, 0x44d)
	/go/src/github.com/thanos-io/thanos/pkg/store/storepb/types.pb.go:383 +0xf3 fp=0xc1a3cd19d0 sp=0xc1a3cd1938 pc=0xbc5513
github.com/thanos-io/thanos/pkg/store/storepb.(*SeriesResponse_Series).MarshalToSizedBuffer(0xc21b218de8, 0xc152d1bb00, 0x450, 0x450, 0xbbb16d, 0xc44e076120, 0x44d)
	/go/src/github.com/thanos-io/thanos/pkg/store/storepb/rpc.pb.go:1288 +0x79 fp=0xc1a3cd1a18 sp=0xc1a3cd19d0 pc=0xbb8fd9
github.com/thanos-io/thanos/pkg/store/storepb.(*SeriesResponse_Series).MarshalTo(0xc21b218de8, 0xc152d1bb00, 0x450, 0x450, 0x480, 0x17dedc0, 0xc21b218d01)
	/go/src/github.com/thanos-io/thanos/pkg/store/storepb/rpc.pb.go:1281 +0x5c fp=0xc1a3cd1a60 sp=0xc1a3cd1a18 pc=0xbb8f1c
github.com/thanos-io/thanos/pkg/store/storepb.(*SeriesResponse).MarshalToSizedBuffer(0xc44e056c40, 0xc152d1bb00, 0x450, 0x450, 0x40ba05, 0x188fb20, 0x1a5e3e0)
	/go/src/github.com/thanos-io/thanos/pkg/store/storepb/rpc.pb.go:1271 +0xc5 fp=0xc1a3cd1ab0 sp=0xc1a3cd1a60 pc=0xbb8e65
github.com/thanos-io/thanos/pkg/store/storepb.(*SeriesResponse).Marshal(0xc44e056c40, 0x1a5e3e0, 0xc44e056c40, 0x7fd772b9f218, 0xc44e056c40, 0xc1a3cd1b01)
	/go/src/github.com/thanos-io/thanos/pkg/store/storepb/rpc.pb.go:1250 +0x7a fp=0xc1a3cd1b08 sp=0xc1a3cd1ab0 pc=0xbb8c7a
google.golang.org/grpc/encoding/proto.codec.Marshal(0x1a5e3e0, 0xc44e056c40, 0xc0116cb140, 0xc0116cb080, 0x3, 0x3, 0x1d2c7a0)
	/go/pkg/mod/google.golang.org/grpc@v1.29.1/encoding/proto/proto.go:70 +0x199 fp=0xc1a3cd1b88 sp=0xc1a3cd1b08 pc=0xb67399
google.golang.org/grpc/encoding/proto.(*codec).Marshal(0x2a4b290, 0x1a5e3e0, 0xc44e056c40, 0xc00db924e0, 0xc1a3cd1c60, 0x8c172a, 0xc0116cb140, 0x1)
	<autogenerated>:1 +0x46 fp=0xc1a3cd1bd0 sp=0xc1a3cd1b88 pc=0xb67b86
google.golang.org/grpc.encode(0x7fd777ac8a08, 0x2a4b290, 0x1a5e3e0, 0xc44e056c40, 0x1, 0xc0116cb140, 0xc1a3cd1d38, 0x8c0dcc, 0xc002dae1e0)
	/go/pkg/mod/google.golang.org/grpc@v1.29.1/rpc_util.go:545 +0x52 fp=0xc1a3cd1c30 sp=0xc1a3cd1bd0 pc=0xb91712
google.golang.org/grpc.prepareMsg(0x1a5e3e0, 0xc44e056c40, 0x7fd777ac8a08, 0x2a4b290, 0x0, 0x0, 0x0, 0x0, 0x3, 0x0, ...)
	/go/pkg/mod/google.golang.org/grpc@v1.29.1/stream.go:1518 +0x107 fp=0xc1a3cd1cd0 sp=0xc1a3cd1c30 pc=0xba6dc7
google.golang.org/grpc.(*serverStream).SendMsg(0xc418f85140, 0x1a5e3e0, 0xc44e056c40, 0x0, 0x0)
	/go/pkg/mod/google.golang.org/grpc@v1.29.1/stream.go:1411 +0xfb fp=0xc1a3cd1e48 sp=0xc1a3cd1cd0 pc=0xba5f7b
github.com/grpc-ecosystem/go-grpc-prometheus.(*monitoredServerStream).SendMsg(0xc41ab01c00, 0x1a5e3e0, 0xc44e056c40, 0x0, 0x464620)
	/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-prometheus@v1.2.0/server_metrics.go:156 +0x4b fp=0xc1a3cd1e90 sp=0xc1a3cd1e48 pc=0x134612b
github.com/grpc-ecosystem/go-grpc-middleware.(*WrappedServerStream).SendMsg(0xc41ab01c20, 0x1a5e3e0, 0xc44e056c40, 0x1, 0x101)
	<autogenerated>:1 +0x50 fp=0xc1a3cd1ec8 sp=0xc1a3cd1e90 pc=0xbd8210
github.com/thanos-io/thanos/pkg/store/storepb.(*storeSeriesServer).Send(0xc41a83ffe0, 0xc44e056c40, 0x1, 0x0)
	/go/src/github.com/thanos-io/thanos/pkg/store/storepb/rpc.pb.go:839 +0x49 fp=0xc1a3cd1f00 sp=0xc1a3cd1ec8 pc=0xbb6d29
github.com/thanos-io/thanos/pkg/store.(*MultiTSDBStore).Series.func2(0xc4180f8718, 0x0)
	/go/src/github.com/thanos-io/thanos/pkg/store/multitsdb.go:258 +0x91 fp=0xc1a3cd1f78 sp=0xc1a3cd1f00 pc=0x11f9711
golang.org/x/sync/errgroup.(*Group).Go.func1(0xc41b4f2f30, 0xc41ab01d40)
	/go/pkg/mod/golang.org/x/sync@v0.0.0-20200930132711-30421366ff76/errgroup/errgroup.go:57 +0x59 fp=0xc1a3cd1fd0 sp=0xc1a3cd1f78 pc=0x90d8f9
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc1a3cd1fd8 sp=0xc1a3cd1fd0 pc=0x46fde1
created by golang.org/x/sync/errgroup.(*Group).Go
	/go/pkg/mod/golang.org/x/sync@v0.0.0-20200930132711-30421366ff76/errgroup/errgroup.go:54 +0x66
@bwplotka bwplotka added the bug label Nov 23, 2020
@bwplotka
Copy link
Member Author

BTW I would vote for setting https://golang.org/pkg/runtime/debug/#SetPanicOnFault so we can recover on those.

@bwplotka bwplotka mentioned this issue Nov 23, 2020
Merged
@bwplotka
Copy link
Member Author

Root cause: Prometheus TSDB Checkpoint unmaps chunk even though we hold chunk querier against it. Most likely bug on TSDB side.

Double checking still if we don't close prematurely on Thanos.

@bwplotka
Copy link
Member Author

Created: prometheus/prometheus#8217

@bwplotka bwplotka mentioned this issue Nov 23, 2020
Open
@arvidsnet
Copy link

not sure if it is the same issue, but I have SIGSEGV errors also in store 0.17. Attached logs.
store-0.17.SIGSEGV.log

@bwplotka
Copy link
Member Author

@arvidsnet nope, definitely different thing, let's create another issue (:

Anyway, both will now on v0.17.1 cause query error NOT segment fault, so at list damage is smaller (:

@bwplotka bwplotka changed the title receive v0.17.0: SIGSEGV receive: Query failure on Seg fault Nov 27, 2020
@bwplotka bwplotka mentioned this issue Nov 27, 2020
Closed
@bwplotka
Copy link
Member Author

Created #3511 @arvidsnet help wanted to find the root cause! 🤗

@stale
Copy link

stale bot commented Jan 26, 2021

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

@stale stale bot added the stale label Jan 26, 2021
@bwplotka
Copy link
Member Author

To investigate.

@stale stale bot removed the stale label Jan 26, 2021
@stale
Copy link

stale bot commented Apr 18, 2021

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

@stale stale bot added the stale label Apr 18, 2021
@stale
Copy link

stale bot commented Jun 3, 2021

Closing for now as promised, let us know if you need this to be reopened! 🤗

@stale stale bot closed this as completed Jun 3, 2021
@fpetkovski
Copy link
Contributor

fpetkovski commented Jan 26, 2023 •
edited
Loading

We are starting to experience this issue on regular basis now. Prometheus version is 0.40, Thanos version 0.29.

Attaching logs following the panic that crashes many receivers at the same time. Seems to occur during head compaction.

logs 
unexpected fault address 0x7ff2002d49b1
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x7ff2002d49b1 pc=0x46c7e1]

goroutine 1475338982 [running]:
runtime.throw({0x2259f0f?, 0x400?})
	/usr/local/go/src/runtime/panic.go:992 +0x71 fp=0xc38f8ca788 sp=0xc38f8ca758 pc=0x4386b1
runtime.sigpanic()
	/usr/local/go/src/runtime/signal_unix.go:825 +0x2ec fp=0xc38f8ca7d8 sp=0xc38f8ca788 pc=0x44e90c
runtime.memmove()
	/usr/local/go/src/runtime/memmove_amd64.s:200 +0x181 fp=0xc38f8ca7e0 sp=0xc38f8ca7d8 pc=0x46c7e1
github.com/thanos-io/thanos/pkg/store/storepb.(*Chunk).MarshalToSizedBuffer(0xc5c3006b70, {0xc7154dd000, 0x2?, 0xc38f8ca888?})
	/go/src/github.com/thanos-io/thanos/pkg/store/storepb/types.pb.go:354 +0xda fp=0xc38f8ca818 sp=0xc38f8ca7e0 pc=0x142ebda
github.com/thanos-io/thanos/pkg/store/storepb.(*AggrChunk).MarshalToSizedBuffer(0xc34a80c040, {0xc7154dd000, 0x271, 0x758})
	/go/src/github.com/thanos-io/thanos/pkg/store/storepb/types.pb.go:500 +0x4a5 fp=0xc38f8ca870 sp=0xc38f8ca818 pc=0x142f985
github.com/thanos-io/thanos/pkg/store/storepb.(*Series).MarshalToSizedBuffer(0xc5c3007b00, {0xc7154dd000, 0x758, 0x758})
	/go/src/github.com/thanos-io/thanos/pkg/store/storepb/types.pb.go:390 +0x2d1 fp=0xc38f8ca8f0 sp=0xc38f8ca870 pc=0x142f271
github.com/thanos-io/thanos/pkg/store/storepb.(*SeriesResponse_Series).MarshalToSizedBuffer(0x0?, {0xc7154dd000, 0x758, 0x1422c4e?})
	/go/src/github.com/thanos-io/thanos/pkg/store/storepb/rpc.pb.go:1806 +0x3c fp=0xc38f8ca920 sp=0xc38f8ca8f0 pc=0x141fc5c
github.com/thanos-io/thanos/pkg/store/storepb.(*SeriesResponse_Series).MarshalTo(0xc38f8ca980?, {0xc7154dd000, 0x758?, 0x758})
	/go/src/github.com/thanos-io/thanos/pkg/store/storepb/rpc.pb.go:1799 +0x47 fp=0xc38f8ca958 sp=0xc38f8ca920 pc=0x141fbc7
github.com/thanos-io/thanos/pkg/store/storepb.(*SeriesResponse).MarshalToSizedBuffer(0xc4992e1820, {0xc7154dd000, 0x758, 0x758})
	/go/src/github.com/thanos-io/thanos/pkg/store/storepb/rpc.pb.go:1789 +0x99 fp=0xc38f8ca990 sp=0xc38f8ca958 pc=0x141faf9
github.com/thanos-io/thanos/pkg/store/storepb.(*SeriesResponse).Marshal(0x10?)
	/go/src/github.com/thanos-io/thanos/pkg/store/storepb/rpc.pb.go:1768 +0x56 fp=0xc38f8ca9d8 sp=0xc38f8ca990 pc=0x141f956
google.golang.org/protobuf/internal/impl.legacyMarshal({{}, {0x2791d00, 0xc717edff00}, {0x0, 0x0, 0x0}, 0x0})
	/go/pkg/mod/google.golang.org/protobuf@v1.28.1/internal/impl/legacy_message.go:402 +0xa2 fp=0xc38f8caa60 sp=0xc38f8ca9d8 pc=0x835922
google.golang.org/protobuf/proto.MarshalOptions.marshal({{}, 0x38?, 0x0, 0x0}, {0x0, 0x0, 0x0}, {0x2791d00, 0xc717edff00})
	/go/pkg/mod/google.golang.org/protobuf@v1.28.1/proto/encode.go:166 +0x27b fp=0xc38f8cab00 sp=0xc38f8caa60 pc=0x7d15bb
google.golang.org/protobuf/proto.MarshalOptions.MarshalAppend({{}, 0x40?, 0x3d?, 0x1b?}, {0x0, 0x0, 0x0}, {0x2765de0?, 0xc717edff00?})
	/go/pkg/mod/google.golang.org/protobuf@v1.28.1/proto/encode.go:125 +0x79 fp=0xc38f8cab48 sp=0xc38f8cab00 pc=0x7d11f9
github.com/golang/protobuf/proto.marshalAppend({0x0, 0x0, 0x0}, {0x7fef25b21310?, 0xc4992e1820?}, 0x88?)
	/go/pkg/mod/github.com/golang/protobuf@v1.5.2/proto/wire.go:40 +0xa5 fp=0xc38f8cabc8 sp=0xc38f8cab48 pc=0x8620e5
github.com/golang/protobuf/proto.Marshal(...)
	/go/pkg/mod/github.com/golang/protobuf@v1.5.2/proto/wire.go:23
google.golang.org/grpc/encoding/proto.codec.Marshal({}, {0x21b3d40, 0xc4992e1820})
	/go/pkg/mod/google.golang.org/grpc@v1.45.0/encoding/proto/proto.go:45 +0x4e fp=0xc38f8cac18 sp=0xc38f8cabc8 pc=0xe8c00e
google.golang.org/grpc/encoding/proto.(*codec).Marshal(0x1?, {0x21b3d40?, 0xc4992e1820?})
	<autogenerated>:1 +0x37 fp=0xc38f8cac38 sp=0xc38f8cac18 pc=0xe8c1f7
google.golang.org/grpc.encode({0x7ff2c1b385d8?, 0x3a93398?}, {0x21b3d40?, 0xc4992e1820?})
	/go/pkg/mod/google.golang.org/grpc@v1.45.0/rpc_util.go:594 +0x44 fp=0xc38f8cac88 sp=0xc38f8cac38 pc=0xea4ba4
google.golang.org/grpc.prepareMsg({0x21b3d40?, 0xc4992e1820?}, {0x7ff2c1b385d8?, 0x3a93398?}, {0x0, 0x0}, {0x0, 0x0})
	/go/pkg/mod/google.golang.org/grpc@v1.45.0/stream.go:1610 +0xd2 fp=0xc38f8cad00 sp=0xc38f8cac88 pc=0xebbd92
google.golang.org/grpc.(*serverStream).SendMsg(0xc37debf380, {0x21b3d40?, 0xc4992e1820})
	/go/pkg/mod/google.golang.org/grpc@v1.45.0/stream.go:1503 +0xf3 fp=0xc38f8cae50 sp=0xc38f8cad00 pc=0xebabf3
github.com/grpc-ecosystem/go-grpc-prometheus.(*monitoredServerStream).SendMsg(0xc3929cf560, {0x21b3d40?, 0xc4992e1820?})
	/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-prometheus@v1.2.0/server_metrics.go:156 +0x33 fp=0xc38f8cae88 sp=0xc38f8cae50 pc=0x1a83e13
github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors.(*monitoredServerStream).SendMsg(0xc651572cc0, {0x21b3d40, 0xc4992e1820})
	/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/interceptors/server.go:64 +0x5a fp=0xc38f8caef0 sp=0xc38f8cae88 pc=0x133f6ba
github.com/grpc-ecosystem/go-grpc-middleware/v2.(*WrappedServerStream).SendMsg(0xc397f67710?, {0x21b3d40?, 0xc4992e1820?})
	<autogenerated>:1 +0x34 fp=0xc38f8caf18 sp=0xc38f8caef0 pc=0x1444674
github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors.(*monitoredServerStream).SendMsg(0xc651573020, {0x21b3d40, 0xc4992e1820})
	/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/interceptors/server.go:64 +0x5a fp=0xc38f8caf80 sp=0xc38f8caf18 pc=0x133f6ba
github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors.(*monitoredServerStream).SendMsg(0xc651573050, {0x21b3d40, 0xc4992e1820})
	/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/interceptors/server.go:64 +0x5a fp=0xc38f8cafe8 sp=0xc38f8caf80 pc=0x133f6ba
github.com/thanos-io/thanos/pkg/store/storepb.(*storeSeriesServer).Send(0x1edb440?, 0x101000000000e?)
	/go/src/github.com/thanos-io/thanos/pkg/store/storepb/rpc.pb.go:1098 +0x2b fp=0xc38f8cb010 sp=0xc38f8cafe8 pc=0x141bbab
github.com/thanos-io/thanos/pkg/store.(*sortedSeriesServer).Send(0xc38f8cb410, 0xc4992e1820)
	/go/src/github.com/thanos-io/thanos/pkg/store/sorted_series_server.go:44 +0x17a fp=0xc38f8cb0b0 sp=0xc38f8cb010 pc=0x17a367a
github.com/thanos-io/thanos/pkg/store.(*ProxyStore).Series(0xc0005c50e0, 0xc37d6d3d60, {0x27890f8, 0xc51a5cf880})
	/go/src/github.com/thanos-io/thanos/pkg/store/proxy.go:340 +0x16af fp=0xc38f8cb498 sp=0xc38f8cb0b0 pc=0x179b2af
github.com/thanos-io/thanos/pkg/store.(*ReadWriteTSDBStore).Series(0xc651573050?, 0x21871a0?, {0x27890f8?, 0xc51a5cf880?})
	<autogenerated>:1 +0x34 fp=0xc38f8cb4c8 sp=0xc38f8cb498 pc=0x17a9854
github.com/thanos-io/thanos/pkg/store/storepb._Store_Series_Handler({0x20f82e0?, 0xc0002c2f40}, {0x2785e78, 0xc651573050})
	/go/src/github.com/thanos-io/thanos/pkg/store/storepb/rpc.pb.go:1085 +0xd0 fp=0xc38f8cb508 sp=0xc38f8cb4c8 pc=0x141bb30
github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors.StreamServerInterceptor.func1({0x20f82e0, 0xc0002c2f40}, {0x2785e78, 0xc651573020}, 0xc3929cf548, 0x2351c78)
	/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/interceptors/server.go:35 +0x2ac fp=0xc38f8cb638 sp=0xc38f8cb508 pc=0x133f56c
github.com/grpc-ecosystem/go-grpc-middleware/v2.ChainStreamServer.func1.1.1({0x20f82e0?, 0xc0002c2f40?}, {0x2785e78?, 0xc651573020?})
	/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/chain.go:51 +0x3a fp=0xc38f8cb678 sp=0xc38f8cb638 pc=0x1443a5a
github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors.StreamServerInterceptor.func1({0x20f82e0, 0xc0002c2f40}, {0x2785de8, 0xc54fae8480}, 0xc3929cf548, 0xc54fae83e0)
	/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/interceptors/server.go:35 +0x2ac fp=0xc38f8cb7a8 sp=0xc38f8cb678 pc=0x133f56c
github.com/thanos-io/thanos/pkg/tracing.StreamServerInterceptor.func1({0x20f82e0, 0xc0002c2f40}, {0x2785e78?, 0xc651572cc0?}, 0x20dca00?, 0x2772e01?)
	/go/src/github.com/thanos-io/thanos/pkg/tracing/grpc.go:42 +0x19b fp=0xc38f8cb838 sp=0xc38f8cb7a8 pc=0x148135b
github.com/grpc-ecosystem/go-grpc-middleware/v2.ChainStreamServer.func1.1.1({0x20f82e0?, 0xc0002c2f40?}, {0x2785e78?, 0xc651572cc0?})
	/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/chain.go:51 +0x3a fp=0xc38f8cb878 sp=0xc38f8cb838 pc=0x1443a5a
github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors.StreamServerInterceptor.func1({0x20f82e0, 0xc0002c2f40}, {0x2785f08, 0xc3929cf560}, 0xc3929cf548, 0xc54fae8400)
	/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/interceptors/server.go:35 +0x2ac fp=0xc38f8cb9a8 sp=0xc38f8cb878 pc=0x133f56c
github.com/grpc-ecosystem/go-grpc-middleware/v2.ChainStreamServer.func1.1.1({0x20f82e0?, 0xc0002c2f40?}, {0x2785f08?, 0xc3929cf560?})
	/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/chain.go:51 +0x3a fp=0xc38f8cb9e8 sp=0xc38f8cb9a8 pc=0x1443a5a
github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).StreamServerInterceptor.func1({0x20f82e0, 0xc0002c2f40}, {0x2786730?, 0xc37debf380}, 0xc2c2b95ad0?, 0xc54fae8420)
	/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-prometheus@v1.2.0/server_metrics.go:121 +0x109 fp=0xc38f8cba38 sp=0xc38f8cb9e8 pc=0x1a836e9
github.com/grpc-ecosystem/go-grpc-middleware/v2.ChainStreamServer.func1.1.1({0x20f82e0?, 0xc0002c2f40?}, {0x2786730?, 0xc37debf380?})
	/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/chain.go:51 +0x3a fp=0xc38f8cba78 sp=0xc38f8cba38 pc=0x1443a5a
github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/recovery.StreamServerInterceptor.func1({0x20f82e0?, 0xc0002c2f40?}, {0x2786730?, 0xc37debf380?}, 0x7fef4fd5bf60?, 0xc3929cf548?)
	/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/interceptors/recovery/interceptors.go:45 +0x9e fp=0xc38f8cbaf0 sp=0xc38f8cba78 pc=0x1a8841e
github.com/grpc-ecosystem/go-grpc-middleware/v2.ChainStreamServer.func1.1.1({0x20f82e0?, 0xc0002c2f40?}, {0x2786730?, 0xc37debf380?})
	/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/chain.go:51 +0x3a fp=0xc38f8cbb30 sp=0xc38f8cbaf0 pc=0x1443a5a
github.com/grpc-ecosystem/go-grpc-middleware/v2.ChainStreamServer.func1({0x20f82e0, 0xc0002c2f40}, {0x2786730, 0xc37debf380}, 0x1dce0e0?, 0xc51a5cf620?)
	/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/chain.go:60 +0xbf fp=0xc38f8cbb88 sp=0xc38f8cbb30 pc=0x14438ff
google.golang.org/grpc.(*Server).processStreamingRPC(0xc000b23500, {0x278c7d8, 0xc071c70000}, 0xc3d2e78480, 0xc0005d89c0, 0x3a276a0, 0x0)
	/go/pkg/mod/google.golang.org/grpc@v1.45.0/server.go:1548 +0xeae fp=0xc38f8cbe48 sp=0xc38f8cbb88 pc=0xeae80e
google.golang.org/grpc.(*Server).handleStream(0xc000b23500, {0x278c7d8, 0xc071c70000}, 0xc3d2e78480, 0x0)
	/go/pkg/mod/google.golang.org/grpc@v1.45.0/server.go:1623 +0x9d6 fp=0xc38f8cbf68 sp=0xc38f8cbe48 pc=0xeb0036
google.golang.org/grpc.(*Server).serveStreams.func1.2()
	/go/pkg/mod/google.golang.org/grpc@v1.45.0/server.go:921 +0x98 fp=0xc38f8cbfe0 sp=0xc38f8cbf68 pc=0xea95b8
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc38f8cbfe8 sp=0xc38f8cbfe0 pc=0x46b7e1
created by google.golang.org/grpc.(*Server).serveStreams.func1
	/go/pkg/mod/google.golang.org/grpc@v1.45.0/server.go:919 +0x28a

@fpetkovski fpetkovski mentioned this issue Jan 27, 2023
Merged
2 tasks
@matej-g matej-g mentioned this issue Jan 27, 2023
Closed
@matej-g
Copy link
Collaborator

matej-g commented Jan 27, 2023

From the duplicate issue discussion (I'll redirect that discussion to here) - possible culprit found by @philipgough #4306 (comment)

looks like it could be related to gogo/protobuf#651? In which case I suppose #4557 is still the fix since we need to move away from the unmaintained gogoprotobuf

@dosubot dosubot bot mentioned this issue Aug 29, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug needs-investigation stale
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@kakkoyun @fpetkovski @bwplotka @matej-g @arvidsnet