API breakage in minor version breaks go install

[abergmeier]

What version of protobuf and what language are you using?
Version: v1.12

What did you do?
We pinned down go modules and vendor. In our build container, we do go install github.com/golang/protobuf/protoc-gen-go. In 1.12 it seems like install is not yet module + vendor aware. Thus it downloads the latest version of protoc-gen-go. It then uses this latest version to generate go code for protobuf.

What did you expect to see?
Since protobuf has increased minor version only I would expect everything to continue working.

What did you see instead?
The generated code no longer works with older (pre 1.20) versions of proto module, since the API was broken.

Anything else we should know about your project / environment?
For us the solution was to upgrade to golang 1.14, where install is module + vendor compatible.

Main complaint is that a new version of protoc was released with API breakages without increasing the major number.

[dsnet]

If I understand correctly, your build container builds the latest version of github.com/golang/protobuf/protoc-gen-go which v1.4.0, but you expected the code it generates to work with a older runtime (less than v1.4.0)? I don't think this is a reasonable expectation.

Code generated with version X of the generator should use at least version X of the runtime or later.

[abergmeier]

Code generated with version X of the generator should use at least version X of the runtime or later.

This leads me to the following:

1. I have not read it spelled out explicitly like that.
   More concrete - which guarantees is protoc-gen-go and proto giving with regards to versions?
2. previously this held up for years (I think all that changed were minor numbers)
3. IMO it might violate semantic versioning
   e.g. it should be ok for version 2 of the generator not to work with version 3 of the runtime

I have the feeling like there needs to be a general communication what guarantees/versioning schemes proto/protoc with go wants to support.

It would even be better to have this sementic then codified so that only supported combinations compile (without warnings).

[dsnet]

1. See /~https://github.com/protocolbuffers/protobuf-go#compatibility
2. Though we require that code generated by protoc-gen-go at version X use at least version X of the runtime, we do make an effort to not act upon this freedom. Just because this "held up for years" is not a valid reason why it must hold up forever. There must come a point in time when the generated code needs to rely on new features only available in a newer runtime. IIRC, the last time this happened was in mid-2018. It's rare, but it definitely happens.
3. I think you are confusing dependency management with semantic versioning. Functionally, using version X of protoc-gen-go is imposing a dependency constraint of at least version X of the protobuf module. This is no different than some Go library foo depending on bar being at least v1.9 or higher. If you try to build foo with bar@v1.8 a breakage is an entirely expected outcome since the dependency constraint was not satisfied.

It would even be better to have this sementic then codified so that only supported combinations compile (without warnings).

It is codified and the rule is simple: whatever version of protoc-gen-go you're using, you need to use at least the same version of the protobuf module.

[moloch--]

@abergmeier - I ran into a similar issue, the move to v1.4.0 completely broke our builds. Since go get can't specify a version our work around was to wget the source for the specific version and build it manually:

wget -O protoc-gen-go.tar.gz /~https://github.com/golang/protobuf/archive/v1.3.5.tar.gz
tar xvf protoc-gen-go.tar.gz
cd protobuf-1.3.5
make install

[dsnet]

Closing as there isn't anything actionable for us to do on our end.

Fundamentally, if the integration tests are building protoc-gen-go from HEAD, but testing the the generated .pb.go files against a specific version of the protobuf module, then it means that the integration tests are not hermetic. Any testing script that lacks the property of hermeticity is likely to be broken due to effects beyond itself.

See #763 (comment) for more or less the exact same issue from mid-2018.