Discussion:
[Cucumber] Having issues running Cucumber testing in Scala/Spark -
(too old to reply)
Jeffrey Kang
2018-02-06 20:47:43 UTC
Permalink
I am writing a scala application using a maven build that i would like to
run in spark. I have no issues running this locally and just hitting play
will run the tests fine. However, when i run a spark-submit I get the error
displayed below.

Exception in thread "main" java.lang.NoClassDefFoundError:
cucumber/api/cli/Main

at cucumberFunctions.myTest$.main(myTest.scala:16)

at cucumberFunctions.myTest.main(myTest.scala)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:733)

at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:177)

at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:202)

at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:116)

at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Caused by: java.lang.ClassNotFoundException: cucumber.api.cli.Main

at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

... 11 more




These are the dependencies in my pom.xml


<dependency>

<groupId>info.cukes</groupId>

<artifactId>cucumber-scala_2.11</artifactId>

<version>1.2.4</version>

</dependency>


<dependency>

<groupId>info.cukes</groupId>

<artifactId>cucumber-junit</artifactId>

<version>1.2.4</version>

<scope>test</scope>

</dependency>

<dependency>

<groupId>info.cukes</groupId>

<artifactId>cucumber-core</artifactId>

<version>1.2.4</version>

</dependency>





Jeff
--
Posting rules: http://cukes.info/posting-rules.html
---
You received this message because you are subscribed to the Google Groups "Cukes" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cukes+***@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Paolo Ambrosio
2018-02-07 07:06:15 UTC
Permalink
Hi Jeff,

Technically, it looks like you are not packaging cucumber-core in your fat
jar. All dependencies needed by your job's main method need to be bundled
in the fat jar you upload for the Spark cluster, and sometimes you need
shading if Spark's own dependencies have conflicts with yours (the Kafka
client is a well-known example).

On the approach, I can't think of a reason why submitting a Spark job with
Cucumber tests in it would make sense. I'm happy to learn something new if
you have one: Spark jobs in the industry suffer from lack of testing and I
have not seen really good patterns yet. Let me explain...

You could be doing Integrated Acceptance Tests (black box), so you would
set up input data, submit your job, verify the output. In this kind of test
you'll have to painfully generate input and expected output for a large
amount of data.. and for each case described in Cucumber by pre- and
post-conditions (Given/Then). I find that Cucumber is not well suited for
this kind of Spark testing.

Otherwise you could be doing Decoupled Acceptance Tests (as in Aslak's
excellent talk [1]). In this case, you would run Spark locally (no submit)
and use something like Spark Testing Base [2], that lets you define and
make assertions on Datasets/RDDs/... without reading and writing from and
to your real data sources. I never tried, but I see how someone could use
Cucumber for this.

Cheers,
Paolo

Links:
[1]

[2] https://github.com/holdenk/spark-testing-base

On 6 Feb 2018 22:00, "Jeffrey Kang" <***@gmail.com> wrote:

I am writing a scala application using a maven build that i would like to
run in spark. I have no issues running this locally and just hitting play
will run the tests fine. However, when i run a spark-submit I get the error
displayed below.

Exception in thread "main" java.lang.NoClassDefFoundError:
cucumber/api/cli/Main

at cucumberFunctions.myTest$.main(myTest.scala:16)

at cucumberFunctions.myTest.main(myTest.scala)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce
ssorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
thodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy
$SparkSubmit$$runMain(SparkSubmit.scala:733)

at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:177)

at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:202)

at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:116)

at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Caused by: java.lang.ClassNotFoundException: cucumber.api.cli.Main

at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

... 11 more




These are the dependencies in my pom.xml


<dependency>

<groupId>info.cukes</groupId>

<artifactId>cucumber-scala_2.11</artifactId>

<version>1.2.4</version>

</dependency>


<dependency>

<groupId>info.cukes</groupId>

<artifactId>cucumber-junit</artifactId>

<version>1.2.4</version>

<scope>test</scope>

</dependency>

<dependency>

<groupId>info.cukes</groupId>

<artifactId>cucumber-core</artifactId>

<version>1.2.4</version>

</dependency>





Jeff
--
Posting rules: http://cukes.info/posting-rules.html
---
You received this message because you are subscribed to the Google Groups
"Cukes" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to cukes+***@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
Posting rules: http://cukes.info/posting-rules.html
---
You received this message because you are subscribed to the Google Groups "Cukes" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cukes+***@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Loading...