Slice out of range error

[ajnavarro]

Calling to GetLanguage method, sometimes we receive this error:

panic: runtime error: slice bounds out of range
goroutine 17 [running, locked to thread]:
bytes.Count(0x7f627818c0d0, 0x4b, 0x0, 0x1c420038c20, 0x1, 0x20, 0x0)
/home/travis/.gimme/versions/go1.8.linux.amd64/src/bytes/bytes.go:62 +0x21d
gopkg.in/src-d/enry%2ev1.getHeaderAndFooter(0x7f627818c0d0, 0x4b, 0x0, 0x0, 0x0, 0x0)
/home/travis/gopath/src/gopkg.in/src-d/enry.v1/common.go:161 +0x9d
gopkg.in/src-d/enry%2ev1.GetLanguagesByModeline(0x7f627818f4a0, 0x0, 0x7f627818c0d0, 0x4b, 0x0, 0x7f62482e6c08, 0x0, 0x0, 0x0, 0x0, ...)
/home/travis/gopath/src/gopkg.in/src-d/enry.v1/common.go:142 +0x5c
gopkg.in/src-d/enry%2ev1.GetLanguages(0x7f627818f4a0, 0x0, 0x7f627818c0d0, 0x4b, 0x0, 0x0, 0x0, 0x0)
/home/travis/gopath/src/gopkg.in/src-d/enry.v1/common.go:126 +0x129
gopkg.in/src-d/enry%2ev1.GetLanguage(0x7f627818f4a0, 0x0, 0x7f627818c0d0, 0x4b, 0x0, 0x1c420038e48, 0x1c42001a500)
/home/travis/gopath/src/gopkg.in/src-d/enry.v1/common.go:38 +0x55
main.GetLanguage(0x7f627818f4a0, 0x0, 0x7f627818c0d0, 0x4b, 0x0, 0x0, 0x0)
/home/travis/build/src-d/enry/shared/enry.go:11 +0x55
main._cgoexpwrap_f7db11756761_GetLanguage(0x7f627818f4a0, 0x0, 0x7f627818c0d0, 0x4b, 0x0, 0x0, 0x0)
command-line-arguments/_obj/_cgo_gotypes.go:58 +0x9a

[mcarmonaa]

@ajnavarro how can I reproduce it?

[ajnavarro]

I would try to send to GetLanguagesByModeline an empty or nil content param.

[mcarmonaa]

Enry already has a test with nil content, as you can see here. It's for the general function GetLanguage, but GetLanguagesByModeline is the first strategy to run, so it should raise a fail if any problem.

Anyway, I added a couple of cases with nil and []byte{} as content to the test TestGetLanguagesByModeline and they pass without problems.

There is a weird thing, the function where the process is raising a panic (bytes.Count) works well with nil or []byte{] content, demonstration here

For the moment, I couldn't reproduce yet the panic in local.

Keep on it.

[mcarmonaa]

Eventually I couldn't reproduce the panic, so I didn't check if #130 fix the problem at all.

About the previous comment, some other things should be known about this issue:

• bytes.Count() function implementation depends on the platform it's running on. You can found these differences under $GOROOT/src/bytes in files bytes.go,bytes_generic.go,bytes_amd64.go,bytes_s390x.go
• Currently travis it's using go version 1.8, I changed it to 1.9.x to use the latest minor release of this version. Maybe it would fix something.
• Another weird behavior from bytes.Count(), sep argument it's always required except if s is nil. In that case, the function returns 1, check out here

Anyway, PR #130 try to fix this checking for empty or nil content before call bytes.Count() function

[vmarkovtsev]

I can reproduce it too and it blocks us.

panic: runtime error: slice bounds out of range

goroutine 17 [running, locked to thread]:
bytes.Count(0x7f23e02849b0, 0xa, 0x0, 0x1c42003cc20, 0x1, 0x20, 0x0)
	/home/travis/.gimme/versions/go1.8.linux.amd64/src/bytes/bytes.go:62 +0x21d
gopkg.in/src-d/enry%2ev1.getHeaderAndFooter(0x7f23e02849b0, 0xa, 0x0, 0x66, 0x6, 0x7f23b80bc5f0)
	/home/travis/gopath/src/gopkg.in/src-d/enry.v1/common.go:161 +0x9d
gopkg.in/src-d/enry%2ev1.GetLanguagesByModeline(0x7f23e02f4710, 0x2, 0x7f23e02849b0, 0xa, 0x0, 0x7f23b99cdc08, 0x0, 0x0, 0x0, 0x0, ...)
	/home/travis/gopath/src/gopkg.in/src-d/enry.v1/common.go:142 +0x5c
gopkg.in/src-d/enry%2ev1.GetLanguages(0x7f23e02f4710, 0x2, 0x7f23e02849b0, 0xa, 0x0, 0x7f23b8ef9a73, 0xc, 0x1c420020de0)
	/home/travis/gopath/src/gopkg.in/src-d/enry.v1/common.go:126 +0x129
gopkg.in/src-d/enry%2ev1.GetLanguage(0x7f23e02f4710, 0x2, 0x7f23e02849b0, 0xa, 0x0, 0x1c42003ce48, 0x1c420016f00)
	/home/travis/gopath/src/gopkg.in/src-d/enry.v1/common.go:38 +0x55
main.GetLanguage(0x7f23e02f4710, 0x2, 0x7f23e02849b0, 0xa, 0x0, 0x1c42001a1d8, 0x1c4200007e0)
	/home/travis/build/src-d/enry/shared/enry.go:11 +0x55
main._cgoexpwrap_f7db11756761_GetLanguage(0x7f23e02f4710, 0x2, 0x7f23e02849b0, 0xa, 0x0, 0x0, 0x0)
	command-line-arguments/_obj/_cgo_gotypes.go:58 +0x9a
panic: runtime error: slice bounds out of range

I will try to reproduce it because I am desperate and cannot work forward.

[abeaumont]

I've just released v1.6.0, which includes #130. Check if the error is solved with this release.

[vmarkovtsev]

@ajnavarro I think it makes sense to allow enry crash and just skip the corresponding file, recording it to the errors list together with stderr output. E.g. we used to do it in our Python pipeline.

[vmarkovtsev]

@abeaumont I confirm that there is no panic with the latest release.

[abeaumont]

great, closing then.

[EgorBu]

Hi,

it looks like that this issue still exists. It appears quite rarely, so it should be tricky to debug.
src-d/sourced-ce#243
should a new issue be created? Or reopen this?

@abeaumont , @mcarmonaa , what do you think?

[vmarkovtsev]

@EgorBu You need to prove that Engine uses the updated version. When I used to have similar issues, I ran Engine locally, without a container, and replaced libenry.so in the engine's JAR with the manually built binary. And for sure in that case you need to prove that your patched JAR is used - sometimes Maven decides to overwrite it.

[EgorBu]

@vmarkovtsev , from src-d/sourced-ce#243 (comment) - it should be enry 1.6.1

[abeaumont]

The issue can be reopened if needed. Do you have a file sample causing the crash? Does the problem also happen when enry cli is used?

[EgorBu]

@abeaumont, unfortunately - no. Because it rarely happens. Most of the times - everything works well, and sometimes it fails on the same dataset.
If you may tell which information could be useful for debugging - I may try to find the way how to collect it

[abeaumont]

Even if frequency is low, it must happen with some file... that would be best information for debugging. Apart from that, an updated stack trace would also help.

[EgorBu]

Do you want stack trace like in src-d/sourced-ce#243 (comment) ? Or something else?
BTW - does some kind of DEBUG version of enry exist? I may try to modify jar as @vmarkovtsev mentioned in #129 (comment)

[abeaumont]

@EgorBu yes, that one. Apparently you're passing an empty filename and that's causing the problem. I'll fix enry to avoid that crash, you may want to workaround it by filtering files without name in the meantime.

[EgorBu]

Interesting that it works most of the times if it always should be empty

[vmarkovtsev]

We can hit two different problems here: breaking on empty filenames and the engine probabilistically supplying empty file names due to some crazy race or whatever.

[EgorBu]

sounds reasonable.
What do you think, @ajnavarro? Can it happen?