perf(validators): reduced GC pressure

[fredbi]

This PR is a follow-up on #169, which cames with innocuous changes, but already
reduced the total number of allocations when validating a swagger spec.

We reduce further the pressure on go's GC by pooling temporarily
allocated objects: validators and results.

When recursively allocating validators, calls to "Validate()" redeem to the pool the allocated
validator. This means that a validator is usable only once.

A similar approach is taken for Results: upon merge, the merged result
is redeemed to the pool.

Eventually, temporarily allocated spec.Schemas are also pooled.

In order not to break existing interfaces, this is enabled by default
only when validating a spec (Spec()) or when using
AgainstSchema().

Another option at the spec level allows for skipping the gathering of schemas
into the result ("schemata"): these schemas must be cloned to remain
valid after their parent is redeemed to the pool.

Comment on the benchmark:

CPU profiling shows that about 30% of the CPU time is spent managing
garbage.

Memory profiling shows that our benchmark (validating the k8s API spec) goes
down from 25M allocated objects down to ~ 17M (#169 went down from 60M to 25M).

The Validate method generates only ~ 5M of those, out of which
~ 4M are caused by the schema unmarshaling & expansion and 1M by string
allocations for the validator "path".
Unfortunately, classic string interning methods don't work well with this dynamic "path",
so I leave it there.

Integration tests (go-swagger/go-swagger#3064)
show a significant improvement when running go-swagger validate in codegen.

• contributes: Swagger generate is using a very large amount of RAM go-swagger/go-swagger#2649

Further improvements would most likely come from optimizing the
validation of defaults and examples (which amount for 2M allocs).

Signed-off-by: Frederic BIDON fredbi@yahoo.com

[codecov]

Codecov Report

Attention: 42 lines in your changes are missing coverage. Please review.

Comparison is base (7c3e606) 92.88% compared to head (44a8ced) 93.30%.

Files	Patch %	Lines
validator.go	90.45%	13 Missing and 6 partials ⚠️
result.go	78.57%	6 Missing and 3 partials ⚠️
schema_props.go	89.65%	4 Missing and 2 partials ⚠️
pools.go	98.28%	2 Missing and 1 partial ⚠️
schema.go	94.23%	2 Missing and 1 partial ⚠️
example_validator.go	90.00%	1 Missing ⚠️
spec.go	94.73%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #170      +/-   ##
==========================================
+ Coverage   92.88%   93.30%   +0.41%     
==========================================
  Files          22       23       +1     
  Lines        3277     3839     +562     
==========================================
+ Hits         3044     3582     +538     
- Misses        190      203      +13     
- Partials       43       54      +11     

Flag	Coverage Δ
oldstable	93.30% <94.19%> (+0.41%)	⬆️
stable	93.25% <94.19%> (+0.36%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[casualjim]

Kubernetes uses an approach where they use an embedded memory allocator on objects that go to and from json frequently.

/~https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apimachinery/pkg/runtime/interfaces.go#L72-L88

[fredbi]

Kubernetes uses an approach where they use an embedded memory allocator on objects that go to and from json frequently.

/~https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apimachinery/pkg/runtime/interfaces.go#L72-L88
Fascinating. Thanks for the link.
On a similar note, I have had a few reads of K8's internal reuse of our pioneering efforts here. But I wasn't aware of this part.
Now they run their own version of the go-openapi stuff.