'Insert SparkSession DataFrame' automatically converting data from integer to float in DSX - apache-spark-sql

Question

I have a csv file which I insert to IBM Data Science Experience using "SparkSession DataFrame". All content in the csv file (other than the headers) are integers.
The dataframe works as expected though certain Machine Learning models until trying to create a Linear Regression Classification where I get this error:
TypeError: Cannot cast array data from dtype('float64') to dtype('U32') according to the rule 'safe'
I believe this means that the data is no longer an integer and is being treated as a float.
How can I resolve this? Is there anything that can be done when you import the file to make sure it stays as an integer? See example below where I tried to add in a second option for format.
`from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
df = spark.read\
.format('org.apache.spark.sql.execution.datasources.csv.CSVFileFormat')\
.option('header', 'true')\
.option('format', 'int32')\
.load(bmos.url('name', 'name.csv'))
df.take(5)`

Answer 1

#charles-gomes is correct. Here's a complete example where my file tinyinttest.csv is in an objectstore container called TestingSandbox.
Contents of tinyinttest.csv is:
name,val
a,1
b,2
code:
from pyspark.sql import SparkSession
import ibmos2spark
credentials = {
'auth_url': 'https://identity.open.softlayer.com',
'project_id': 'xxx',
'region': 'xxx',
'user_id': 'xxx',
'username': 'xxx',
'password': 'xxx'
}
configuration_name = 'xxx'
bmos = ibmos2spark.bluemix(sc, credentials, configuration_name)
spark = SparkSession.builder.getOrCreate()
df = spark.read\
.format('org.apache.spark.sql.execution.datasources.csv.CSVFileFormat')\
.option('header', 'true')\
.option('inferSchema', 'true')\
.load(bmos.url('TestingSandbox', 'tinyinttest.csv'))
df.schema
output:
StructType(List(StructField(name,StringType,true),StructField(val,IntegerType,true)))