Migrate away from gogoprotobuf to protobuf v2 API + vtprotobuf

[GiedriusS]

Is your proposal related to a problem?

gogoprotobuf is unmaintained: gogo/protobuf#691

So, we should move to the new protobuf v2 API. Also, we should try to use https://vitess.io/blog/2021-06-03-a-new-protobuf-generator-for-go/ to not make Thanos slower

Describe the solution you'd like

Move to the protobuf V2 API: https://blog.golang.org/protobuf-apiv2. Also, use vtprotobuf so that on Query side we could get all SeriesResponses from a sync.Pool

Describe alternatives you've considered

Alternative is to wait if gogoprotobuf will get a new maintainer and maybe then we wouldn't have to move but that probably will never happen.

Additional context

N/A

[yeya24]

Love this idea!
Can you explain the sentence below? Is this something done automatically by vtprotobuf?

Also, use vtprotobuf so that on Query side we could get all SeriesResponses from a sync.Pool

[squat]

I think he means that by switching to this proto generator we can benefit from using vtproto's memory pooling to greatly reduce the number of allocations needed for umarshaling when receiving message streams during querying, which could otherwise be allocation-heavy, and thus speed up the code path.

[kakkoyun]

Yes, this is a great idea! We had a quick chat about this with @bwplotka as well. We should definitely migrate to vtprotobuf. I'd be happy to help with this if @GiedriusS you're not already on it.

[GiedriusS]

I think he means that by switching to this proto generator we can benefit from using vtproto's memory pooling to greatly reduce the number of allocations needed for umarshaling when receiving message streams during querying, which could otherwise be allocation-heavy, and thus speed up the code path.

Yep, there's more info here: /~https://github.com/planetscale/vtprotobuf#available-features. We could call func YourProtoFromVTPool() *YourProto to get a SeriesResponse on the Querier side that comes from a sync.Pool instead of doing this: /~https://github.com/thanos-io/thanos/blob/main/pkg/store/storepb/rpc.pb.go#L755

It's quite a huge work, I haven't started anything 😄 I thought perhaps this could be a GSoC/LFX project

[squat]

It's quite a huge work, I haven't started anything 😄 I thought perhaps this could be a GSoC/LFX project

Yes i was thinking exactly the same. Shall we propose it? Id be happy to co-mentor

[squat]

Hi @iamrajiv we just submitted this project to the Linux foundation student mentorship program for the fall /~https://github.com/cncf/mentoring/blob/main/lfx-mentorship/2021/03-Fall/project_ideas.md#migrate-thanos-to-the-new-protocol-buffers-v2-api
Maybe we can figure out a way multiple people contributing to this though.

[pkj-m]

Hi folks! This looks like an interesting project, and in sync with my skillset. I would be happy to work on this and contribute to Thanos as part of the LFX Mentorship program.

I have prior experience with open-source projects through participating (and eventually mentoring) in GSoC. I'm quite proficient in Golang, and have worked with protobuf in the past during my summer internship at Uber, where I was tasked with writing a new proto service from scratch in Go.

I'll take a look at the repository and reach out soon with further questions. Thanks!

[GiedriusS]

Hi folks! This looks like an interesting project, and in sync with my skillset. I would be happy to work on this and contribute to Thanos as part of the LFX Mentorship program.

I have prior experience with open-source projects through participating (and eventually mentoring) in GSoC. I'm quite proficient in Golang, and have worked with protobuf in the past during my summer internship at Uber, where I was tasked with writing a new proto service from scratch in Go.

I'll take a look at the repository and reach out soon with further questions. Thanks!

Feel free to apply via the LFX website!

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[yeya24]

Still valid

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[GiedriusS]

Our work is here:
rahulii#1
rahulii#2

Need to continue with this work :/

[GiedriusS]

Hi @GiedriusS is there any doc that updates about how much work has to be done and how much has to be done? I am interested in working on this issue.

I'm afraid that the PRs are the only place you can find info. Also, I wrote a small post detailing the learnings from doing this: https://giedrius.blog/2022/02/04/things-learned-from-trying-to-migrate-to-protobuf-v2-api-from-gogoprotobuf-so-far/.

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[kakkoyun]

@GiedriusS What's the latest state on this? If there's still some work happy to help.

[GiedriusS]

@kakkoyun I am continuing work on this - already switched to upstream protobuf compiler. Just need to fix tests & add pooling now. My branch is available on /~https://github.com/GiedriusS/thanos/tree/gogo_remove.

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[matej-g]

Linking the current WIP for more visibility - #5421

[GiedriusS]

I finally pushed this over the line but it's not worth it: #7790. A better solution would be to focus on grpc/protobuf alternatives. See the PR for more.