Tag Search is working incorrectly with badger

[zerda]

Problem - what in Jaeger blocks you from solving the requirement?

I'm using nginx ingress controller integrated with Jaeger (badger storage).

When searching with http.request_id tag, the result is incorrectly empty.

Here is an unit test to reproduce this issue.

func TestRandomTraceID(t *testing.T) {
	runFactoryTest(t, func(tb testing.TB, sw spanstore.Writer, sr spanstore.Reader) {
		s1 := model.Span{
			TraceID: model.TraceID{
				Low:  uint64(14767110704788176287),
				High: 0,
			},
			SpanID: model.SpanID(14976775253976086374),
			OperationName: "/",
			Process: &model.Process{
				ServiceName: "nginx",
			},
			Tags: model.KeyValues{
				model.KeyValue{
					Key:   "http.request_id",
					VStr:  "first",
					VType: model.StringType,
				},
			},
			StartTime: time.Now(),
			Duration:  1 * time.Second,
		}
		err := sw.WriteSpan(&s1)

		s2 := model.Span{
			TraceID: model.TraceID{
				Low:  uint64(4775132888371984950),
				High: 0,
			},
			SpanID: model.SpanID(13576481569227028654),
			OperationName: "/",
			Process: &model.Process{
				ServiceName: "nginx",
			},
			Tags: model.KeyValues{
				model.KeyValue{
					Key:   "http.request_id",
					VStr:  "second",
					VType: model.StringType,
				},
			},
			StartTime: time.Now(),
			Duration:  1 * time.Second,
		}
		err = sw.WriteSpan(&s2)

		params := &spanstore.TraceQueryParameters{
			StartTimeMin: time.Now().Add(-1 * time.Minute),
			StartTimeMax: time.Now(),
			ServiceName:  "nginx",
			Tags: map[string]string{
				"http.request_id": "second",
			},
		}
		traces, err := sr.FindTraces(context.Background(), params)
		assert.NoError(t, err)

                 // failed with `second` tag query, but success with `first`
		assert.Equal(t, 1, len(traces))
	})
}

And here are the traces produced by nginx. nginx-traces.zip

Proposal - what do you suggest to solve the problem or improve the existing situation?

Maybe it's about TraceID (the first traceID is large than the second one), and mergeJoinIds function is not handled this correclty.

[zerda]

@burmanm Any thoughts?

[burmanm]

@zerda It's not actually in the mergeJoinIds, but the index seek code. TraceIDs are correctly sorted, however the actual data is stored as sorted by timestamp first and not TraceID which causes this error.

And none of the unit tests catched this as they had time in the same order as ids (in tests that mattered). The index seek logging reveals this:

scanIndexKeys result:

[129 110 103 105 110 120 0 5 147 31 147 93 187 194 0 0 0 0 0 0 0 0 204 239 80 242 77 196 125 159]
[129 110 103 105 110 120 0 5 147 31 147 93 191 207 0 0 0 0 0 0 0 0 66 68 174 15 238 178 122 54]

Parsed traceIDs:

TraceID: [0 0 0 0 0 0 0 0 204 239 80 242 77 196 125 159]
TraceID: [0 0 0 0 0 0 0 0 66 68 174 15 238 178 122 54]

I'll have to add sort back to that scanIndexKeys method (after duplicate removal).

[burmanm]

Thanks for catching this, I'll fix it tomorrow.