enry core

[liuzhiwei]

I use Java bindings (maven version 1.6.2).

Here is the log:

panic: runtime error: slice bounds out of range
goroutine 26 [running, locked to thread]:
bytes.Count(0x7fc5dc00e3c0, 0x47, 0x0, 0x1c4206fbc20, 0x1, 0x20, 0x0)
/home/travis/.gimme/versions/go1.8.linux.amd64/src/bytes/bytes.go:62 +0x21d
gopkg.in/src-d/enry%2ev1.getHeaderAndFooter(0x7fc5dc00e3c0, 0x47, 0x0, 0x0, 0x0, 0x0)
/home/travis/gopath/src/gopkg.in/src-d/enry.v1/common.go:166 +0xae
gopkg.in/src-d/enry%2ev1.GetLanguagesByModeline(0x7fc5dc00e810, 0x7fc6f492d879, 0x7fc5dc00e3c0, 0x47, 0x0, 0x7fc5861f7c08, 0x0, 0x0, 0x0, 0x0
, ...)
/home/travis/gopath/src/gopkg.in/src-d/enry.v1/common.go:142 +0x5c
gopkg.in/src-d/enry%2ev1.GetLanguages(0x7fc5dc00e810, 0x7fc6f492d879, 0x7fc5dc00e3c0, 0x47, 0x0, 0x7fc585723aae, 0xc, 0x1c420024de0)
/home/travis/gopath/src/gopkg.in/src-d/enry.v1/common.go:126 +0x129
gopkg.in/src-d/enry%2ev1.GetLanguage(0x7fc5dc00e810, 0x7fc6f492d879, 0x7fc5dc00e3c0, 0x47, 0x0, 0x1c4206fbe48, 0x1c4209cc040)
/home/travis/gopath/src/gopkg.in/src-d/enry.v1/common.go:38 +0x55
main.GetLanguage(0x7fc5dc00e810, 0x7fc6f492d879, 0x7fc5dc00e3c0, 0x47, 0x0, 0x1c42012e1d8, 0x1c420499340)
/home/travis/build/src-d/enry/shared/enry.go:11 +0x55
main._cgoexpwrap_f7db11756761_GetLanguage(0x7fc5dc00e810, 0x7fc6f492d879, 0x7fc5dc00e3c0, 0x47, 0x0, 0x0, 0x0)
command-line-arguments/_obj/_cgo_gotypes.go:58 +0x9a
Aborted (core dumped)

[EgorBu]

Hi,
I got the same error during usage engine 0.3.4 with enry-java 1.6.2:

panic: runtime error: slice bounds out of range

goroutine 17 [running, locked to thread]:
bytes.Count(0x7f3b3042b4b0, 0x54, 0x0, 0x1c42005cc20, 0x1, 0x20, 0x0)
	/home/travis/.gimme/versions/go1.8.linux.amd64/src/bytes/bytes.go:62 +0x21d
gopkg.in/src-d/enry%2ev1.getHeaderAndFooter(0x7f3b3042b4b0, 0x54, 0x0, 0x1c42005cca8, 0x7f3dde378707, 0x7f3dec5c09b0)
	/home/travis/gopath/src/gopkg.in/src-d/enry.v1/common.go:166 +0xae
gopkg.in/src-d/enry%2ev1.GetLanguagesByModeline(0x7f3b3042e210, 0x2, 0x7f3b3042b4b0, 0x54, 0x0, 0x7f3ddeff8c08, 0x0, 0x0, 0x1c42004e000, 0x7f3dde524aae, ...)
	/home/travis/gopath/src/gopkg.in/src-d/enry.v1/common.go:142 +0x5c
gopkg.in/src-d/enry%2ev1.GetLanguages(0x7f3b3042e210, 0x2, 0x7f3b3042b4b0, 0x54, 0x0, 0x7f3dde524aae, 0xc, 0x1c42001cde0)
	/home/travis/gopath/src/gopkg.in/src-d/enry.v1/common.go:126 +0x129
gopkg.in/src-d/enry%2ev1.GetLanguage(0x7f3b3042e210, 0x2, 0x7f3b3042b4b0, 0x54, 0x0, 0x1c42005ce48, 0x1c420018500)
	/home/travis/gopath/src/gopkg.in/src-d/enry.v1/common.go:38 +0x55
main.GetLanguage(0x7f3b3042e210, 0x2, 0x7f3b3042b4b0, 0x54, 0x0, 0x1c420078098, 0x1c4200007e0)
	/home/travis/build/src-d/enry/shared/enry.go:11 +0x55
main._cgoexpwrap_f7db11756761_GetLanguage(0x7f3b3042e210, 0x2, 0x7f3b3042b4b0, 0x54, 0x0, 0x0, 0x0)
	command-line-arguments/_obj/_cgo_gotypes.go:58 +0x9a
panic: runtime error: slice bounds out of range

goroutine 50 [running, locked to thread]:
bytes.Count(0x7f3b280742c0, 0x5b, 0x0, 0x1c42005bc20, 0x1, 0x20, 0x0)
	/home/travis/.gimme/versions/go1.8.linux.amd64/src/bytes/bytes.go:62 +0x21d
gopkg.in/src-d/enry%2ev1.getHeaderAndFooter(0x7f3b280742c0, 0x5b, 0x0, 0x1c42005bca8, 0x7f3dde378707, 0x7f3dec4b00c0)
	/home/travis/gopath/src/gopkg.in/src-d/enry.v1/common.go:166 +0xae
gopkg.in/src-d/enry%2ev1.GetLanguagesByModeline(0x7f3b28074280, 0x2, 0x7f3b280742c0, 0x5b, 0x0, 0x7f3ddeff8c08, 0x0, 0x0, 0x1c4204c2400, 0x7f3dde524aae, ...)
	/home/travis/gopath/src/gopkg.in/src-d/enry.v1/common.go:142 +0x5c
gopkg.in/src-d/enry%2ev1.GetLanguages(0x7f3b28074280, 0x2, 0x7f3b280742c0, 0x5b, 0x0, 0x7f3dde524aae, 0xc, 0x1c4200240e0)
	/home/travis/gopath/src/gopkg.in/src-d/enry.v1/common.go:126 +0x129
gopkg.in/src-d/enry%2ev1.GetLanguage(0x7f3b28074280, 0x2, 0x7f3b280742c0, 0x5b, 0x0, 0x1c42005be48, 0x1c4200c8040)
	/home/travis/gopath/src/gopkg.in/src-d/enry.v1/common.go:38 +0x55
main.GetLanguage(0x7f3b28074280, 0x2, 0x7f3b280742c0, 0x5b, 0x0, 0x1c42085c048, 0x1c4204c47e0)
	/home/travis/build/src-d/enry/shared/enry.go:11 +0x55
main._cgoexpwrap_f7db11756761_GetLanguage(0x7f3b28074280, 0x2, 0x7f3b280742c0, 0x5b, 0x0, 0x0, 0x0)
	command-line-arguments/_obj/_cgo_gotypes.go:58 +0x9a
panic: runtime error: slice bounds out of range

goroutine 52 [running, locked to thread]:
bytes.Count(0x7f3b40044790, 0x43, 0x20, 0x1c42083fc20, 0x1, 0x20, 0x0)
	/home/travis/.gimme/versions/go1.8.linux.amd64/src/bytes/bytes.go:62 +0x21d
gopkg.in/src-d/enry%2ev1.getHeaderAndFooter(0x7f3b40044790, 0x43, 0x20, 0x66, 0x6, 0x7f3dec6a7c90)
	/home/travis/gopath/src/gopkg.in/src-d/enry.v1/common.go:166 +0xae
gopkg.in/src-d/enry%2ev1.GetLanguagesByModeline(0x7f3b4005ab40, 0x0, 0x7f3b40044790, 0x43, 0x20, 0x7f3ddeff8c08, 0x0, 0x0, 0x0, 0x0, ...)
	/home/travis/gopath/src/gopkg.in/src-d/enry.v1/common.go:142 +0x5c
gopkg.in/src-d/enry%2ev1.GetLanguages(0x7f3b4005ab40, 0x0, 0x7f3b40044790, 0x43, 0x20, 0x7f3dde524aae, 0xc, 0x1c42001e0e0)
	/home/travis/gopath/src/gopkg.in/src-d/enry.v1/common.go:126 +0x129
gopkg.in/src-d/enry%2ev1.GetLanguage(0x7f3b4005ab40, 0x0, 0x7f3b40044790, 0x43, 0x20, 0x1c42083fe48, 0x1c4200c6040)
	/home/travis/gopath/src/gopkg.in/src-d/enry.v1/common.go:38 +0x55
main.GetLanguage(0x7f3b4005ab40, 0x0, 0x7f3b40044790, 0x43, 0x20, 0x1c420148098, 0x1c4204aa7e0)
	/home/travis/build/src-d/enry/shared/enry.go:11 +0x55
main._cgoexpwrap_f7db11756761_GetLanguage(0x7f3b4005ab40, 0x0, 0x7f3b40044790, 0x43, 0x20, 0x0, 0x0)
	command-line-arguments/_obj/_cgo_gotypes.go:58 +0x9a

[abeaumont]

Hi @liuzhiwei, @EgorBu, thanks for reporting.
Could you provide the filename and content of the file that produces the crash?

[EgorBu]

I'm not sure if I can (easily or not) extract this information from engine - I will check.
Is it possible to add this logs to error report in enry? Because it's quite rare when enry fails.

[abeaumont]

I don't think it's possible to log a crash from enry when it crashes, it may be possible from the engine, i don't know.

[vmarkovtsev]

@abeaumont
This "slice out of range" error has been around for a few times already (#132 #129). We suffer much because we wait for 2 or 3 hours and then the classification suddenly crashes with a similar stack trace.

If we used enry as a Go library, we would wrap it with recover(), however, we use it throuh the JNI in the complex chain Java -> C -> Go. It is simply impossible to handle these panics on Java or C side - they are even not documented. Thus I propose the following: add an option to Enry to gracefully recover() in GetLanguage API call and return the corresponding error up. I think it is evident that no software is ideal and we will always panic in Enry from time to time. Given millions of repositories, the probability becomes close to 100%. If we had a chance to handle this panic, we could log it in the engine and ultimately report the bug (on either side) properly.

For sure this recover() should be optional and disabled by default; this implies adding another API to activate the "safe" mode.

[liuzhiwei]

Similar with @EgorBu and @vmarkovtsev . My Application has many thread and run many hours with Enry and then core happened. So I'm really hard to find out which file causing the core. As @vmarkovtsev said, it will be good if there is a recover API.

[bzz]

Second that - we might want to have an option on API side to enable "recover mode" for each individual call to enry, as that would imply having additional go-routine through defer.

I also find this article https://blog.golang.org/defer-panic-and-recover very nice in explanation how to use built-in recover function in Go.

[abeaumont]

enry should not panic in any case, no matter how many different inputs are (which is not very important). The bug has been around because it's not been properly fixed, primarily because there's been no input to reproduce it.

I think that extending the API for this purpose is overkill but will consider it if we're unable to solve it in a simpler way.

[vmarkovtsev]

@abeaumont we are not able to give a proper input because we are not able to handle the panic.

Again: enry will panic. This is big data. Even Spark itself crashes from time to time (Alex knows that). Yet we do not want to spend several hours to encounter another bug and then wait until somebody fixes it. We would rather log the offending input and move further.

[abeaumont]

Reviewing the code I'm pretty sure that it's not an enry problem but a problem with the java bindings. I'm quite suspicious of the Java GC at this point, but would need some environment to reproduce the problem. @EgorBu, @bzz could you give me some pointers about how you run the engine to produce this error (i.e. setup a valid testing environment)?

Apart from trying to debug and fix this error, I will convert the enry panic to an exception in Java so that it can be catched and recovered if needed, even if I think this is a mistake since errors will be silently ignored instead of reported and fixed, but it will prevent from interrupting its usage.

P.D. Discussing whether Spark itself crashes or not and what people knows about that is irrelevant to this bug, so please keep ontopic.

[vmarkovtsev]

That's awesome news @abeaumont thanks!

[mcarmonaa]

@abeaumont Running a simple code with engine by chance I stumbled upon this panic with just a set of three siva files, but the error is kind of random because I couldn't reproduce it again.

If you consider it helpful, I can provide you with those three siva files and the short code that raised the random panic that time.

[abeaumont]

@mcarmonaa that would be great indeed. The behavior you've seen matches with my suspicions, so it'd be great if I can validate them.

[mcarmonaa]

@abeaumont these are the steps to run the code, you must have spark downloaded, check out here for a quickstart guide.

Then run a spark-shell as follows:

$SPARK_HOME/bin/spark-shell --packages "tech.sourced:engine:0.3.4"

And run this code, you can type :paste on the spark-shell to paste it there:

import tech.sourced.engine._

val engine = Engine(spark, "/path/to/siva/directory", "siva")
val df = engine.getRepositories.getHEAD.getCommits.getFirstReferenceCommit.getBlobs.dropDuplicates("blob_id").classifyLanguages.extractUASTs.cache()
df.show()

Note that you must change /path/to/siva/directory to your own path.

You should be able to download a tar with the siva files from here

[abeaumont]

Thanks @mcarmonaa. I was able to reproduce the problem with your content (even if it's random). I also made some interesting findings. Instrumenting enry and enry-java I saw that the crash is produced when the files to check are duplicated. This, for some reason, seems to corruct the content of the filename (and I guess the content too) and when it reaches the go code in enry it's not a valid Go string anymore:

scala> import tech.sourced.engine._; val engine = Engine(spark, "/home/abeaumont/siva-examples", "siva"); val df = engine.getRepositories.getHEAD.getCommits.getFirstReferenceCommit.getBlobs.dropDuplicates("blob_id").classifyLanguages.extractUASTs.cache(); df.show()
get language for: gradle.properties
get language for: prime/is_prime_improved.py
Original pointer: 397118876
Original pointer: 323695518
allocated@0x7ff2c800b8b0 (26 bytes)
allocated@0x7ff3505507d0 (17 bytes)
Original pointer: 531756909
Original pointer: 596485868
allocated@0x7ff2bc028e20 (853 bytes)
allocated@0x7ff2c8015b70 (710 bytes)
get language for: prime/is_prime_improved.py
get language for: gradle.properties
Original pointer: 437814685
Original pointer: 1652934749
allocated@0x7ff2c8015a10 (26 bytes)
allocated@0x7ff2bc025ff0 (17 bytes)
Original pointer: 531756909
Original pointer: 596485868
allocated@0x7ff2bc0262a0 (853 bytes)
allocated@0x7ff2c8015e40 (710 bytes)
OK
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x2 addr=0x7ff2bc04217d pc=0x7ff2b46b07b5]

goroutine 17 [running, locked to thread]:
runtime.throw(0x7ff2b48b03cf, 0x2a)
	/usr/lib/go/src/runtime/panic.go:605 +0x97 fp=0x1c42003cb90 sp=0x1c42003cb70 pc=0x7ff2b46853c7
runtime.sigpanic()
	/usr/lib/go/src/runtime/signal_unix.go:351 +0x2bc fp=0x1c42003cbe0 sp=0x1c42003cb90 pc=0x7ff2b469861c
runtime.memmove(0x7ff2b52ffe40, 0x7ff2bc041ffd, 0x200)
	/usr/lib/go/src/runtime/memmove_amd64.s:329 +0x3c5 fp=0x1c42003cbe8 sp=0x1c42003cbe0 pc=0x7ff2b46b07b5
runtime.recordForPanic(0x7ff2bc026600, 0xf1ff4738, 0xf1ff4738)
	/usr/lib/go/src/runtime/print.go:45 +0x5c fp=0x1c42003cc20 sp=0x1c42003cbe8 pc=0x7ff2b4685c0c
runtime.gwrite(0x7ff2bc026600, 0xf1ff4738, 0xf1ff4738)
	/usr/lib/go/src/runtime/print.go:89 +0x4e fp=0x1c42003cc58 sp=0x1c42003cc20 pc=0x7ff2b4685e1e
runtime.printstring(0x7ff2bc026600, 0xf1ff4738)
	/usr/lib/go/src/runtime/print.go:236 +0x8d fp=0x1c42003cca8 sp=0x1c42003cc58 pc=0x7ff2b468671d
gopkg.in/src-d/enry%2ev1.GetLanguagesByFilename(0x7ff2bc026600, 0xf1ff4738, 0x7ff2bc025ff0, 0x11, 0x0, 0x7ff2b52ff8a8, 0x0, 0x0, 0x0, 0x0, ...)
	/home/abeaumont/go/src/gopkg.in/src-d/enry.v1/common.go:275 +0xb4 fp=0x1c42003ccf0 sp=0x1c42003cca8 pc=0x7ff2b47b1574
gopkg.in/src-d/enry%2ev1.GetLanguages(0x7ff2bc026600, 0xf1ff4738, 0x7ff2bc025ff0, 0x11, 0x0, 0x4, 0x1, 0x0)
	/home/abeaumont/go/src/gopkg.in/src-d/enry.v1/common.go:126 +0x15f fp=0x1c42003cda8 sp=0x1c42003ccf0 pc=0x7ff2b47b055f
gopkg.in/src-d/enry%2ev1.GetLanguage(0x7ff2bc026600, 0xf1ff4738, 0x7ff2bc025ff0, 0x11, 0x0, 0x1c42003ce48, 0x3)
	/home/abeaumont/go/src/gopkg.in/src-d/enry.v1/common.go:38 +0x55 fp=0x1c42003cdf8 sp=0x1c42003cda8 pc=0x7ff2b47afe45
main.GetLanguage(0x7ff2bc026600, 0xf1ff4738, 0x7ff2bc025ff0, 0x11, 0x0, 0x1c42003ce88, 0x38)
	/home/abeaumont/go/src/gopkg.in/src-d/enry.v1/shared/enry.go:11 +0x55 fp=0x1c42003ce40 sp=0x1c42003cdf8 pc=0x7ff2b47b5205
main._cgoexpwrap_f7db11756761_GetLanguage(0x7ff2bc026600, 0xf1ff4738, 0x7ff2bc025ff0, 0x11, 0x0, 0x0, 0x0)
	command-line-arguments/_obj/_cgo_gotypes.go:58 +0x91 fp=0x1c42003ce88 sp=0x1c42003ce40 pc=0x7ff2b47b3911
runtime.call64(0x0, 0x7ff2ba461818, 0x7ff2ba4618b0, 0x38)
	/usr/lib/go/src/runtime/asm_amd64.s:510 +0x3d fp=0x1c42003ced8 sp=0x1c42003ce88 pc=0x7ff2b46ad43d
runtime.cgocallbackg1(0x0)
	/usr/lib/go/src/runtime/cgocall.go:305 +0x1a0 fp=0x1c42003cf58 sp=0x1c42003ced8 pc=0x7ff2b465f0b0
runtime.cgocallbackg(0x0)
	/usr/lib/go/src/runtime/cgocall.go:187 +0x95 fp=0x1c42003cfc0 sp=0x1c42003cf58 pc=0x7ff2b465ee75
runtime.cgocallback_gofunc(0x0, 0x0, 0x0, 0x0)
	/usr/lib/go/src/runtime/asm_amd64.s:762 +0x9a fp=0x1c42003cfe0 sp=0x1c42003cfc0 pc=0x7ff2b46ae9ca
runtime.goexit()
	/usr/lib/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0x1c42003cfe8 sp=0x1c42003cfe0 pc=0x7ff2b46afc41

goroutine 18 [running, locked to thread]:
	goroutine running on other thread; stack unavailable
/usr/bin/spark-shell: line 44: 30689 Aborted                 (core dumped) "${SPARK_HOME}"/bin/spark-submit --class org.apache.spark.repl.Main --name "Spark shell" "$@"

So my guess is that there's some kind of concurrency problem in the engine while feeding enry and that spark somehow corrupts the memory in that case. This duplication is not deterministic, but when it happens, it produces the crash. Synchronizing the access to enry seems to solve the problem, but I cannot assure this yet.

So, this seems to point to a bug in the engine, but I'm not sure yet. @liuzhiwei could you please tell us whether you're using the engine or Spark?

[abeaumont]

I've being able to reduce the dataset that reproduces the crash and it still happens without file duplication, so that may be an unrelated problem. Anyway, somehow it seems that concurrency when calling enry is causing memory corruption. Synchronizing enry methods seem to solve the problem, so I'll release that fix for now.

[abeaumont]

I created a tentative fix at #138. I cannot reproduce the bug anymore with it, but since we don't know the root of the problem it may not be fixed yet, so please let me know if the problem disappears or not.

[liuzhiwei]

@abeaumont I don't use engine or spark. Just enry-java in java application.

And Thanks for changing. I'll try the enry-java 1.6.3 and will let you know the problem disappears or not.

[abeaumont]

Thanks @liuzhiwei, then it may not be related to engine at all. Let's see if that fix works, in the meantime I'll also add support to convert Go panics to exceptions so that it can mitigate the effects of these or related problems.

[liuzhiwei]

@abeaumont I tried enry-java 1.6.3 with big amount of files in multiple threads. No core happened. So the issue was fixed.

But it seems your solution is changing enry to be synchronized. Does it impact performance?

[juanjux]

@liuzhiwei Alfredo is not working on this project anymore. Since it seems that the fix worked I'll close it, please comment here if you find the problem again.