spark.read.format(“avro”).load(filepath)
I have a gradle test which loads an avro file using spark.
I was converting the project from a maven build to a grade build. It worked perfectly fine in the maven build, however in the gradle build it throws a java.lang.IllegalArgumentException error.
I have included the spark-avro dependency in the dependency list.
dependencies {
implementation group: ‘org.scala-lang’, name: ‘scala-library’, version:‘2.12.6’
implementation group: ‘org.apache.spark’, name: ‘spark-core_2.12’, version:‘2.4.1’
implementation group: ‘org.apache.spark’, name: ‘spark-yarn_2.12’, version:‘2.4.1’
implementation group: ‘org.apache.spark’, name: ‘spark-sql_2.12’, version:‘2.4.1’
implementation group: ‘org.apache.spark’, name: ‘spark-hive_2.12’, version:‘2.4.1’
implementation group: ‘org.apache.spark’, name: ‘spark-avro_2.12’, version:‘2.4.1’
implementation group: ‘com.typesafe’, name: ‘config’, version:‘1.4.1’
implementation group: ‘com.sun.mail’, name: ‘javax.mail’, version:‘1.5.6’
implementation group: ‘org.apache.avro’, name: ‘avro-tools’, version:‘1.10.2’
implementation group: ‘com.typesafe.play’, name: ‘play-json_2.12’, version:‘2.9.2’
testImplementation group: ‘junit’, name: ‘junit’, version:‘4.12’
testImplementation group: ‘org.scalatest’, name: ‘scalatest_2.12’, version:‘3.0.5’
testImplementation group: ‘org.specs2’, name: ‘specs2-core_2.12’, version:‘4.2.0’
testImplementation group: ‘org.specs2’, name: ‘specs2-junit_2.12’, version:‘4.2.0’
testImplementation group: ‘org.scalatestplus’, name: ‘mockito-3-4_2.12’, version:‘3.2.4.0’
testImplementation group: ‘org.mockito’, name: ‘mockito-core’, version:‘3.4.2’
testImplementation group: ‘org.mockito’, name: ‘mockito-inline’, version:‘4.0.0’
}