I have a 1000-line csv file containing tab-separated values, and want to read them into a data frame.
Here is my schema:
schema = StructType([StructField("TEST1", StringType(), True),
StructField("TEST2", StringType(), True),
StructField("TEST3", StringType(), True),
StructField("TEST4", StringType(), True),
StructField("TEST5", StringType(), True)])
Here is my readStream statement:
df = spark.readStream.format("cloudFiles")
.option("cloudFiles.format", "csv")
.option("delimiter", "\t")
.option("path", "/mnt/data/data.tsv")
.schema(schema).load()
Running the readStream cell results in:
df: pyspark.sql.dataframe.DataFrame
TEST1: string
TEST2: string
TEST3: string
TEST4: string
TEST5: string
In the next cell I executed
display(df)
and ended up with
Query returned no results.
I think there is something wrong with my schema or readStream statements. Is the tab delimiter set correctly?
I found out what the problem was. The path should not have been a specific file. Rather, it should have been a file path containing wild cards. After I changed the command to this:
df = spark.readStream.format("cloudFiles")
.option("cloudFiles.format", "csv")
.option("delimiter", "\t")
.option("path", "/mnt/data/*.tsv")
.schema(schema).load()
it worked just fine.
Related
I have a StructType as follows:
to_Schema = StructType([StructField('name', StringType(), True),
StructField('sales', IntegerType(), True)])
The dataframe_1 has both fields as StringType. So I created the above StructType so that I could use it to typecast the fields in dataframe_1.
I am able to do it in Scala:
val df2 = dataframe_1.selectExpr(to_Schema.map(
col => s"CAST ( ${col.name} As ${col.dataType.sql}) ${col.name}"
): _*)
I am not able to use the same map function in python as StructType has no map function.
I've tried using for loop but it doesn't work as expected.
I am looking for a PySpark equivalent of the above Scala code.
The below code will achieve the same thing in python:
for s in to_Schema:
df = df.withColumn(s.name, df[s.name].cast(s.dataType))
You can also create a new dataframe from the old one using the new schema as shown in this answer:
df2 = spark.createDataFrame(dataframe_1.rdd, to_Schema)
This would be the direct translation:
df2 = dataframe_1.selectExpr(*[f"CAST ({c.name} AS {c.dataType.simpleString()}) {c.name}" for c in to_Schema])
It could be simplified:
df2 = dataframe_1.select([col(c.name).cast(c.dataType).alias(c.name) for c in to_Schema])
However, I like this answer more ;)
When I perform a row count operation on my dataframe with/without cache(), I get a different result.
The example is the following:
videosDF = videosRawDF.withColumn("trending_date", F.to_date("trending_date", "yy.dd.MM"))\
.withColumn("publish_time", F.from_unixtime(F.unix_timestamp('publish_time', "yyyy-MM-dd'T'HH:mm:ss")).cast("timestamp"))\
.dropna(subset = ["trending_date", "publish_time"])
videosDF.count()
120746
On the other hand if I only add .cache() the result changes
videosDF = videosRawDF.withColumn("trending_date", F.to_date("trending_date", "yy.dd.MM"))\
.withColumn("publish_time", F.from_unixtime(F.unix_timestamp('publish_time', "yyyy-MM-dd'T'HH:mm:ss")).cast("timestamp"))\
.dropna(subset = ["trending_date", "publish_time"])
videosDF.cache()
videosDF.count()
0
The correct solution after performing the drop is 120746 rows, but I am forced to perform the cache. What I can do?
Thank you for reading.
EDIT:
The dataset can be found in: https://www.kaggle.com/datasets/datasnaek/youtube-new
and my code until now is the following:
from pyspark.sql.types import StructType, StructField, StringType, IntegerType,BooleanType
from pyspark.sql import functions as F
ruta = "gs://ucmbucketrafael/data/"
youtubeSchema = StructType([
StructField('video_id', StringType(), nullable = True),
StructField('trending_date', StringType(), nullable = True),
StructField('title', StringType(), nullable = True),
StructField('channel_title', StringType(), nullable = True),
StructField('category_id', StringType(), nullable = True),
StructField('publish_time', StringType(), nullable = True),
StructField('tags', StringType(), nullable = True),
StructField('views', IntegerType(), nullable = True),
StructField('likes', IntegerType(), nullable = True),
StructField('dislikes', IntegerType(), nullable = True),
StructField('comment_count', IntegerType(), nullable = True),
StructField('comments_disabled', BooleanType(), nullable = True),
StructField('ratings_disabled', BooleanType(), nullable = True),
StructField('video_error_or_removed', BooleanType(), nullable = True),
StructField('description', StringType(), nullable = True)
])
USvideosDF = spark.read.schema(youtubeSchema)\
.option("header", "true")\
.option("quote", "\"").option("escape", "\"")\
.option("mode","DROPMALFORMED")\
.csv(ruta+"youtube_USvideos.csv")\
.withColumn("pais",F.lit("EEUU"))\
.drop("description")
CAvideosDF = spark.read.schema(youtubeSchema)\
.option("header", "true")\
.option("quote", "\"").option("escape", "\"")\
.option("mode","DROPMALFORMED")\
.csv(ruta+"youtube_CAvideos.csv")\
.withColumn("pais",F.lit("CA"))\
.drop("description")
GBvideosDF = spark.read.schema(youtubeSchema)\
.option("header", "true")\
.option("quote", "\"").option("escape", "\"")\
.option("mode","DROPMALFORMED")\
.csv(ruta+"youtube_GBvideos.csv")\
.withColumn("pais",F.lit("GB"))\
.drop("description")
videosRawDF = USvideosDF.union(CAvideosDF).union(GBvideosDF)
videosDF = videosRawDF.withColumn("trending_date", F.to_date("trending_date", "yy.dd.MM"))\
.withColumn("publish_time", F.from_unixtime(F.unix_timestamp('publish_time', "yyyy-MM-dd'T'HH:mm:ss")).cast("timestamp"))\
.dropna(subset = ["trending_date", "publish_time"])
videosDF.cache()
videosDF.count()
I have already discovered the problem. It is the interaction between DROPMALFORMED and the scheme. If the schema is deleted when reading the csv it already works correctly.
How to read a csv file in Spark which has a structure like:
id,name,address
1,"ashu","building","street","area","city","state","pin"
When using a reader:
val df = spark.read.option("header",true).csv("input/input1.csv")
I am getting record till the third value in CSV.
+---+----+--------+
| id|name| address|
+---+----+--------+
| 1|ashu|building|
+---+----+--------+
How to ask Spark to read all the values starting from third value till the last one in single dataframe column address like:
+---+----+-----------------------------------------------+
| id|name| address |
+---+----+-----------------------------------------------+
| 1|ashu|"building","street","area","city","state","pin"|
+---+----+-----------------------------------------------+
I'm making my answer fit your requirements to use CSV. This is the least painful way to do what you want to do.
Modify your CSV file so that it use "|" to split fields instead of ",". This will allow you to have ',' inside your columns.
id,name,address
1|"ashu"|"building","street","area","city","state","pin"
Modify you code:
val df = spark.read
.option("header",true)
.option("delimiter", '|')
.csv("input/input1.csv")
If you can fix your input files to use another delimiter character than you should do that.
However, if you don't have that possibility, you can still read the file without header and specify a custom schema. Then, concatenate the 6 address columns to get the desired dataframe:
import org.apache.spark.sql.types._
val schema = StructType(
Array(
StructField("id", IntegerType, true),
StructField("name", StringType, true),
StructField("address1", StringType, true),
StructField("address2", StringType, true),
StructField("address3", StringType, true),
StructField("address4", StringType, true),
StructField("address5", StringType, true),
StructField("address6", StringType, true)
)
)
val input = spark.read.schema(schema).csv("input/input1.csv")
val df = input.filter("name != 'name'").withColumn(
"address",
concat_ws(", ", (1 to 6).map(n => col(s"address$n")):_*)
).select("id", "name", "address")
df.show(false)
//+---+----+----------------------------------------+
//|id |name|address |
//+---+----+----------------------------------------+
//|1 |ashu|building, street, area, city, state, pin|
//+---+----+----------------------------------------+
100000,20160214,93374987
100000,20160214,1925301
100000,20160216,1896542
100000,20160216,84167419
100000,20160216,77273616
100000,20160507,1303015
I want to read the csv file which has no column names in first row.
How to read it and name the columns with my specified names in the same time ?
for now, I just renamed the original columns with my specified names like this:
df = spark.read.csv("user_click_seq.csv",header=False)
df = df.withColumnRenamed("_c0", "member_srl")
df = df.withColumnRenamed("_c1", "click_day")
df = df.withColumnRenamed("_c2", "productid")
Any better way ?
You can import the csv file into a dataframe with a predefined schema. The way you define a schema is by using the StructType and StructField objects. Assuming your data is all IntegerType data:
from pyspark.sql.types import StructType, StructField, IntegerType
schema = StructType([
StructField("member_srl", IntegerType(), True),
StructField("click_day", IntegerType(), True),
StructField("productid", IntegerType(), True)])
df = spark.read.csv("user_click_seq.csv",header=False,schema=schema)
should work.
For those who would like to do this in scala and may not want to add types:
val df = spark.read.format("csv")
.option("header","false")
.load("hdfs_filepath")
.toDF("var0","var1","var2","var3")
You can read the data with header=False and then pass the column names with toDF as bellow:
data = spark.read.csv('data.csv', header=False)
data = data.toDF('name1', 'name2', 'name3')
In my case, it handled many columns and creating a schema was very tedious when, in addition, spark inferred the schema well.
So I opted to rename it using a select.
First I create a list with the new names:
val listNameColumns: List[String] = List("name1", "name2" , "name3")
Then I combine the column names of the original dataframe with the above list and create a list of Column elements:
import org.apache.spark.sql.Column
import org.apache.spark.sql.functions.col
val selectStament: Array[Column] = df.columns zip listNameColumns map { case(a, b) => col(a).as(b)}
Finally I make the select:
val dfRenamed = df.select(selectStament:_*)
Spark: 1.4.0
I have a flatfile from Amazon S3 which I loaded into HDFS (in the master node of my EC2 Spark cluster). The flatfile is a Hive output. Note: I couldn't change the context as it is already defined. The following codes are used in the pyspark shell:
Each 'row' corresponds to 1 row of data:
row = sc.textFile("/data/file")
row.first()
u'E8B98\x01John\x01Smith\x01Male\x01Gold\x0125''
Then I split each row using flatmap() since for some reason map() doesn't seem to delimit it (using '\x01' as the delimiter):
elements = row.flatMap(lambda x: x.split('\x01'))
elements.take(8)
[u'E8B98', u'John', u'Smith', u'Male', u'Gold', u'25', u'E8B99', u'Alice']
Since I know the data has 6 columns per row, how do I get the data into a dataframe? I'm intend to sort by attribute, sum etc.
I tried the following but it didn't work:
id = row.flatMap(lambda x: x.split('\x01')[0])
id.first()
E
There is many way to transform an rdd to a dataframe in python :
Considering the following rdd
rdd = sc.parallelize(list(["E8B98\x01John\x01Smith\x01Male\x01Gold\x0125","E8B2\x01Joe\x01Smith\x01Female\x01Gold\x0125"]))
rdd.first()
Output:
'E8B98\x01John\x01Smith\x01Male\x01Gold\x0125'
Let's now create an rdd of tuples :
rdd2 = rdd.map(lambda x : x.split("\x01"))
rdd2.first()
Output:
['E8B98', 'John', 'Smith', 'Male', 'Gold', '25']
We can now create a dataframe with one of the following ways :
Create it directly from the tuples rdd :
sqlContext.createDataFrame(rdd2).collect()
Output:
[Row(_1=u'E8B98', _2=u'John', _3=u'Smith', _4=u'Male', _5=u'Gold', _6=u'25'), Row(_1=u'E8B2', _2=u'Joe', _3=u'Smith', _4=u'Female', _5=u'Gold', _6=u'25')]
or create it with the same rdd specifying the name of the columns :
df = sqlContext.createDataFrame(rdd2, ['id', 'name', 'surname', 'gender', 'description', 'age'])
df.collect()
Output:
[Row(id=u'E8B98', name=u'John', surname=u'Smith', gender=u'Male', description=u'Gold', age=u'25'), Row(id=u'E8B2', name=u'Joe', surname=u'Smith', gender=u'Female', description=u'Gold', age=u'25')]
or create it with the inferred schema :
pyspark.sql.types import *
schema = StructType([
StructField("id", StringType(), True),
StructField("name", StringType(), True),
StructField("surname", StringType(), True),
StructField("gender", StringType(), True),
StructField("description", StringType(), True),
StructField("age", StringType(), True)])
df2 = sqlContext.createDataFrame(rdd2, schema)
df2.collect()
Output:
[Row(id=u'E8B98', name=u'John', surname=u'Smith', gender=u'Male', description=u'Gold', age=u'25'),Row(id=u'E8B2', name=u'Joe', surname=u'Smith', gender=u'Female', description=u'Gold', age=u'25')]
or yet specifying your row class schema as following :
from pyspark.sql import Row
Person = Row('id', 'name', 'surname', 'gender', 'description', 'age')
person = rdd2.map(lambda r: Person(*r))
df3 = sqlContext.createDataFrame(person)
df3.collect()
Output:
[Row(id=u'E8B98', name=u'John', surname=u'Smith', gender=u'Male', description=u'Gold', age=u'25'), Row(id=u'E8B2', name=u'Joe', surname=u'Smith', gender=u'Female', description=u'Gold', age=u'25')]
I hope this helps!
NB: Spark version >= 1.3.0