receiver: Segfault on Chunk Marshalling

[bwplotka]

v0.20.1 version of receiver

level=info ts=2021-05-27T07:00:45.207242464Z caller=checkpoint.go:95 component=receive component=multi-tsdb tenant=default-tenant msg="Creating checkpoint" from_segment=49 to_segment=67 mint=1622095200000
unexpected fault address 0x7f49885dbb73
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x7f49885dbb73 pc=0x473325]

goroutine 131993508 [running]:
runtime.throw(0x1c9114c, 0x5)
	/usr/local/go/src/runtime/panic.go:1117 +0x72 fp=0xc1b61c2710 sp=0xc1b61c26e0 pc=0x438312
runtime.sigpanic()
	/usr/local/go/src/runtime/signal_unix.go:741 +0x268 fp=0xc1b61c2748 sp=0xc1b61c2710 pc=0x44fac8
runtime.memmove(0xc1e56e2753, 0x7f49885dbaf1, 0x102)
	/usr/local/go/src/runtime/memmove_amd64.s:332 +0x3c5 fp=0xc1b61c2750 sp=0xc1b61c2748 pc=0x473325
github.com/thanos-io/thanos/pkg/store/storepb.(*Chunk).MarshalToSizedBuffer(0xc158652220, 0xc1e56e2000, 0x855, 0x944, 0xdb, 0x0, 0x0)
	/home/circleci/project/pkg/store/storepb/types.pb.go:347 +0x1ce fp=0xc1b61c2780 sp=0xc1b61c2750 pc=0xc4888e
github.com/thanos-io/thanos/pkg/store/storepb.(*AggrChunk).MarshalToSizedBuffer(0xc1b36e1b80, 0xc1e56e2000, 0x855, 0x944, 0xec, 0x0, 0x0)
	/home/circleci/project/pkg/store/storepb/types.pb.go:493 +0x238 fp=0xc1b61c27f0 sp=0xc1b61c2780 pc=0xc49358
github.com/thanos-io/thanos/pkg/store/storepb.(*Series).MarshalToSizedBuffer(0xc1b37091d0, 0xc1e56e2000, 0x944, 0x944, 0xc3d26d, 0xc1b37091d0, 0x941)
	/home/circleci/project/pkg/store/storepb/types.pb.go:383 +0x133 fp=0xc1b61c2890 sp=0xc1b61c27f0 pc=0xc48bf3
github.com/thanos-io/thanos/pkg/store/storepb.(*SeriesResponse_Series).MarshalToSizedBuffer(0xc08556f270, 0xc1e56e2000, 0x944, 0x944, 0xc3d26d, 0xc1b37091d0, 0x941)
	/home/circleci/project/pkg/store/storepb/rpc.pb.go:1306 +0x79 fp=0xc1b61c28d8 sp=0xc1b61c2890 pc=0xc3a6f9
github.com/thanos-io/thanos/pkg/store/storepb.(*SeriesResponse_Series).MarshalTo(0xc08556f270, 0xc1e56e2000, 0x944, 0x944, 0x944, 0x1969900, 0xc08556f201)
	/home/circleci/project/pkg/store/storepb/rpc.pb.go:1299 +0x5c fp=0xc1b61c2920 sp=0xc1b61c28d8 pc=0xc3a63c
github.com/thanos-io/thanos/pkg/store/storepb.(*SeriesResponse).MarshalToSizedBuffer(0xc1b36f3a00, 0xc1e56e2000, 0x944, 0x944, 0x40bc45, 0x1a1c3e0, 0x1c00ce0)
	/home/circleci/project/pkg/store/storepb/rpc.pb.go:1289 +0xc5 fp=0xc1b61c2970 sp=0xc1b61c2920 pc=0xc3a585
github.com/thanos-io/thanos/pkg/store/storepb.(*SeriesResponse).Marshal(0xc1b36f3a00, 0x1c00ce0, 0xc1b36f3a00, 0x7f49ec3a9570, 0xc1b36f3a00, 0x1)
	/home/circleci/project/pkg/store/storepb/rpc.pb.go:1268 +0x7a fp=0xc1b61c29c8 sp=0xc1b61c2970 pc=0xc3a39a
google.golang.org/grpc/encoding/proto.codec.Marshal(0x1c00ce0, 0xc1b36f3a00, 0x0, 0x2d1de01, 0x0, 0xc0ca2c6090, 0xc0ca2c1fb0)
	/home/circleci/go/pkg/mod/google.golang.org/grpc@v1.29.1/encoding/proto/proto.go:70 +0x199 fp=0xc1b61c2a48 sp=0xc1b61c29c8 pc=0xa4a7f9
google.golang.org/grpc/encoding/proto.(*codec).Marshal(0x2d46418, 0x1c00ce0, 0xc1b36f3a00, 0x8a624a, 0xc0ca2c6090, 0x1, 0x1, 0xc086f8f020)
	<autogenerated>:1 +0x46 fp=0xc1b61c2a90 sp=0xc1b61c2a48 pc=0xa4afc6
google.golang.org/grpc.encode(0x7f4a0a35a008, 0x2d46418, 0x1c00ce0, 0xc1b36f3a00, 0xc1b61c2bf8, 0x8a58ec, 0xc0b0bf9980, 0x3063efd584807f2b, 0xc086f8f020)
	/home/circleci/go/pkg/mod/google.golang.org/grpc@v1.29.1/rpc_util.go:545 +0x52 fp=0xc1b61c2af0 sp=0xc1b61c2a90 pc=0xa8a8d2
google.golang.org/grpc.prepareMsg(0x1c00ce0, 0xc1b36f3a00, 0x7f4a0a35a008, 0x2d46418, 0x0, 0x0, 0x0, 0x0, 0x97, 0x1, ...)
	/home/circleci/go/pkg/mod/google.golang.org/grpc@v1.29.1/stream.go:1518 +0x10f fp=0xc1b61c2b98 sp=0xc1b61c2af0 pc=0xaa00af
google.golang.org/grpc.(*serverStream).SendMsg(0xc1af39e780, 0x1c00ce0, 0xc1b36f3a00, 0x0, 0x0)
	/home/circleci/go/pkg/mod/google.golang.org/grpc@v1.29.1/stream.go:1411 +0xfb fp=0xc1b61c2d10 sp=0xc1b61c2b98 pc=0xa9f23b
github.com/grpc-ecosystem/go-grpc-prometheus.(*monitoredServerStream).SendMsg(0xc1af728d80, 0x1c00ce0, 0xc1b36f3a00, 0xc1b526dad0, 0xc1b61c2dd0)
	/home/circleci/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-prometheus@v1.2.0/server_metrics.go:156 +0x4b fp=0xc1b61c2d58 sp=0xc1b61c2d10 pc=0x14be32b
github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors.(*monitoredServerStream).SendMsg(0xc1af73bb60, 0x1c00ce0, 0xc1b36f3a00, 0xc11f06b3ff, 0x12c87060169e)
	/home/circleci/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/interceptors/server.go:64 +0x6d fp=0xc1b61c2dc0 sp=0xc1b61c2d58 pc=0xaa89cd
github.com/grpc-ecosystem/go-grpc-middleware/v2.(*WrappedServerStream).SendMsg(0xc157541240, 0x1c00ce0, 0xc1b36f3a00, 0x1f06b3ad, 0x1f06b3ad89739d20)
	<autogenerated>:1 +0x50 fp=0xc1b61c2df8 sp=0xc1b61c2dc0 pc=0xca7cb0
github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors.(*monitoredServerStream).SendMsg(0xc1af73bbf0, 0x1c00ce0, 0xc1b36f3a00, 0x0, 0x0)

[GiedriusS]

Which version? Any way to reproduce?

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[stale]

Closing for now as promised, let us know if you need this to be reopened! 🤗

[matej-g]

This is not resolved, we just hit it on receiver (although I've seen this first time ever today). Does not seem easily reproducible.

[philipgough]

@matej-g looks like it could be related to gogo/protobuf#651? In which case I suppose #4557 is still the fix since we need to move away from the unmaintained gogoprotobuf

[matej-g]

Nice catch @philipgough, then the migration might really solve this 👍

[matej-g]

This is actually duplicate of #3497, I'll close this one and redirect the discussion there