Caused by: java.lang.IllegalStateException: Could not read class: VirtualFile: Kotlin+ Apache Beam defined runner - kotlin

Question

I implemented an example using Kotlin + Apache Beam to define the Kotlin properties of the pipes but when I ran the project I got the error:
Caused by: java.lang.IllegalStateException: Could not read class: VirtualFile: /Users/duanybaro/.gradle/caches/modules-2/files-2.1/org.apache.beam/beam-runners-google-cloud-dataflow-java/2.27.0/3e551e54b23441cc58c9d01e6614ff67216a7e87/beam-runners-google-cloud-dataflow-java-2.27.0.jar!/org/apache/beam/runners/dataflow/DataflowPipelineJob.class
at org.jetbrains.kotlin.load.java.structure.impl.classFiles.BinaryJavaClass.<init>(BinaryJavaClass.kt:122)
at org.jetbrains.kotlin.load.java.structure.impl.classFiles.BinaryJavaClass.<init>(BinaryJavaClass.kt:34)
This error only occurs in Kotlin because, with the code made in java, it works perfectly. Can you give me any suggestions to solve the error?

Answer 1

I really recommend you using the latest version of Apache Beam, your version is very old.
You can also use the starter for Beam Kotlin.
I share with your an example of Kotlin Beam project from my Github repo based on Maven.
For the Beam pipeline options, can you try with instead of using DataflowPipelineOptions :
val options = PipelineOptionsFactory
.fromArgs(*args)
.withValidation()
.`as`(TeamLeagueOptions::class.java)
val pipeline = Pipeline.create(options)
Example of PipelineOptions :
import org.apache.beam.sdk.options.Description
import org.apache.beam.sdk.options.PipelineOptions
interface TeamLeagueOptions : PipelineOptions {
#get:Description("Path of the input Json file to read from")
var inputJsonFile: String
#get:Description("Path of the slogans file to read from")
var inputFileSlogans: String
#get:Description("Path of the file to write to")
var teamLeagueDataset: String
#get:Description("Team stats table")
var teamStatsTable: String
#get:Description("Job type")
var jobType: String
#get:Description("Failure output dataset")
var failureOutputDataset: String
#get:Description("Failure output table")
var failureOutputTable: String
#get:Description("Feature name for failures")
var failureFeatureName: String
}
And pass program argument in the mvn command line :
mvn compile exec:java \
-Dexec.mainClass=fr.groupbees.application.TeamLeagueApp \
-Dexec.args=" \
--project=my-project \
--runner=DataflowRunner \
--jobName=team-league-kotlin-job-$(date +'%Y-%m-%d-%H-%M-%S') \
--region=europe-west1 \
--streaming=false \
--zone=europe-west1-d \
--tempLocation=gs://mazlum_dev/dataflow/temp \
--gcpTempLocation=gs://mazlum_dev/dataflow/temp \
--stagingLocation=gs://mazlum_dev/dataflow/staging \
--inputJsonFile=gs://mazlum_dev/team_league/input/json/input_teams_stats_raw.json \
--inputFileSlogans=gs://mazlum_dev/team_league/input/json/input_team_slogans.json \
--teamLeagueDataset=mazlum_test \
--teamStatsTable=team_stat \
--jobType=team_league_kotlin_ingestion_job \
--failureOutputDataset=mazlum_test \
--failureOutputTable=job_failure \
--failureFeatureName=team_league \
" \
-Pdataflow-runner