About Spark core update 3.4.1 and my working code stop

[aksh0369-smarten]

i update my spark core to 3.4.1 and my working code regarding jhownsnow nlp is not working and shows he below error
i am using latest version of spark nlp.

Can anyone help me?
java.io.InvalidClassException: com.johnsnowlabs.nlp.annotators.pos.perceptron.AveragedPerceptron; local class incompatible: stream classdesc serialVersionUID = -7114715142956979922, local class serialVersionUID = 6642857758815297725
at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:699) ~[?:1.8.0_231]
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1885) ~[?:1.8.0_231]
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1751) ~[?:1.8.0_231]

[maziyarpanahi]

Could you please share more info? Versions? Platform? Scala? Python? SBT? Any code? Is it a pipeline? model?

Just in case, if you are in a Scala/Java project, you must match the same exact Spark and Scala versions as our library or else some models/pipelines fail with that Java error (checking the sanity of the saved object.)

[LuchoTangorra]

hello! I have the same problem. I tried different configurations, this is what Ii'm currently trying still with errors:
SBT
scala version 2.12.12
Dependencies:
"com.amazonaws" % "aws-java-sdk" % 1.12.173 % "provided",
"org.apache.spark" %% "spark-sql" % "3.4.0",
"org.scalatest" %% "scalatest" % "3.2.11" % "test",
"joda-time" % "joda-time" % "2.10.13",
"com.typesafe.play" %% "play-json" % "2.9.2",
"com.typesafe" % "config" % "1.4.1",
"com.lihaoyi" %% "ujson" % "0.9.6",
"com.lihaoyi" %% "requests" % "0.7.0",
"com.johnsnowlabs.nlp" %% "spark-nlp" % "5.3.1",
"org.apache.spark" %% "spark-mllib" % "3.4.0"

I'm trying to run it on EMR emr-6.12.0 spark 3.4.0 (i also tried on EMR 7.2 with spark 3.5.1 and spark-nlp 5.5.2).

The process compiles without further complications (i'm doing a clean assembly), but it fails after a few minutes running in EMR.

Any help will be appreciated :)

[maziyarpanahi]

hi,
we have a documentation of how to use EMR: https://sparknlp.org/docs/en/install#emr-support

[LuchoTangorra]

i followed the instructions but it still raises the same error
25/02/20 14:23:50 WARN TaskSetManager: Lost task 6.0 in stage 35.0 (TID 1227) (ip-10-125-5-254.ec2.internal executor 2): java.io.InvalidClassException: com.johnsnowlabs.nlp.annotators.pos.perceptron.AveragedPerceptron; local class incompatible: stream classdesc serialVersionUID = -7114715142956979922, local class serialVersionUID = 6642857758815297725
at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:699)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:2005)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1852)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2186)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1669)
at java.io.ObjectInputStream.readArray(ObjectInputStream.java:2119)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1657)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:503)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:461)
at org.apache.spark.util.Utils$.deserialize(Utils.scala:147)
at org.apache.spark.SparkContext.$anonfun$objectFile$2(SparkContext.scala:1423)
at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:486)

with EMR 6.14 and this sbt config
  "com.amazonaws" % "aws-java-sdk" % awsVersion % "provided",

"org.apache.spark" %% "spark-sql" % "3.4.1" % "provided",
"com.typesafe" % "config" % "1.4.2",
"com.typesafe.play" %% "play-json" % "2.9.2",
"com.lihaoyi" %% "ujson" % "4.0.1",
"com.lihaoyi" %% "requests" % "0.8.0",
"com.johnsnowlabs.nlp" %% "spark-nlp" % "5.5.2",
"org.apache.spark" %% "spark-mllib" % "3.4.1" % "provided"