given a dataset with 2 columns:
| col1 | col2 |
| 1 | 2 |
| 2 | 2 |
| 1 | 2 |
| 1 | 2 |
I would like to add a column with the sum of col1 and col2
| col1 | col2 | col3 |
| 1 | 2 | 3 |
| 2 | 2 | 4 |
| 1 | 2 | 3 |
| 1 | 2 | 3 |
I have found this question which basically seems to do exactly the same but in Scala.
Any tip?
Assuming your data is present in df, the desired output can be obtained by using either of the below mentioned ways,
Using Dataframe operations
df.select("col1", "col2", (df3.col1 + df3.col2).alias("col3")).show()
Using Spark SQL
df.createOrReplaceTempView("temp_data")
spark.sql("select *, (col1 + col2) as col3 from temp_data").show()
Output:
+----+----+----+
|col1|col2|col3|
+----+----+----+
| 1| 2| 3|
| 2| 2| 4|
| 1| 2| 3|
| 1| 2| 3|
+----+----+----+
Please find the below answer to create a new column in df.
val df1 = df.withColumn("new col", col("col1") + col("col2"))
df1.show
Related
The data i've in my database:
| id| some_id| status|
| 1| 1 | SUCCESS|
| 2| 2 | SUCCESS|
| 3| 1 | SUCCESS|
| 4| 3 | SUCCESS|
| 5| 1 | SUCCESS|
| 6| 4 | FAILED |
| 7| 1 | SUCCESS|
| 8| 1 | FAILED |
| 9| 4 | FAILED |
| 10| 1 | FAILED |
.......
I ran a query to group by id and status to get the below result:
| some_id| count| status|
| 1 | 20| SUCCESS|
| 2 | 5 | SUCCESS|
| 3 | 10| SUCCESS|
| 2 | 15| FAILED |
| 3 | 12| FAILED |
| 4 | 25 | FAILED |
I want to use the above query as subquery to get the result below, where the distinct status are column name.
| some_id| SUCCESS| FAILED|
| 1 | 20 | null/0|
| 2 | 5 | 15 |
| 3 | 10 | 12 |
| 4 | null/0| 25 |
Any other approach to get the final data is also appreciated. Let me know if need more info.
Thanks
You may use a pivot query here with the help of FILTER:
SELECT
some_id,
COUNT(*) FILTER (WHERE status = 'SUCCESS') AS SUCCESS,
COUNT(*) FILTER (WHERE status = 'FAILED') AS FAILED
FROM yourTable
GROUP BY
some_id;
Demo
The input dataset looks like this:
org| id |step| value
1 | 1 | 1 | 12
1 | 1 | 2 | 13
1 | 1 | 3 | 14
1 | 1 | 4 | 15
1 | 2 | 1 | 16
1 | 2 | 2 | 17
2 | 1 | 1 | 1
2 | 1 | 2 | 2
for the output I want to add the missing steps per org group for example to id == 2 of org == 1
org| id |step| value
1 | 1 | 1 | 12
1 | 1 | 2 | 13
1 | 1 | 3 | 14
1 | 1 | 4 | 15
1 | 2 | 1 | 16
1 | 2 | 2 | 17
1 | 2 | 3 | null
1 | 2 | 4 | null
2 | 1 | 1 | 1
2 | 1 | 2 | 2
I tried this but doesn't work:
r = df.select("org", "step").distinct()
df.join(r, ["org", "step"], 'right_outer')
val l = df.select("org", "step");
val r = df.select("org", "id");
val right = l.join(r, "org");
val result = df.join(right, Seq("org", "id", "step"), "right_outer").distinct().orderBy("org", "id", "step");
result.show
Gives:
+---+---+----+-----+
|org| id|step|value|
+---+---+----+-----+
| 1| 1| 1| 12|
| 1| 1| 2| 13|
| 1| 1| 3| 14|
| 1| 1| 4| 15|
| 1| 2| 1| 16|
| 1| 2| 2| 17|
| 1| 2| 3| null|
| 1| 2| 4| null|
| 2| 1| 1| 1|
| 2| 1| 2| 2|
+---+---+----+-----+
Bonus: sql query for the table (orgs) reflecting the df contents
select distinct o_right."org", o_right."id", o_right."step", o_left."value"
from orgs as o_left
right outer join (
select o_in_left."org", o_in_right."id", o_in_left."step"
from orgs as o_in_right
join (select "org", "step" from orgs) as o_in_left
on o_in_right."org" = o_in_left."org"
order by "org", "id", "step"
) as o_right
on o_left."org" = o_right."org"
and o_left."step" = o_right."step"
and o_left."id" = o_right."id"
order by "org", "id", "step"
I have a pyspark dataframe with two id columns id and id2. Each id is repeated exactly n times. All id's have the same set of id2's. I'm trying to "flatten" the matrix resulting from each unique id into one row according to id2.
Here's an example to explain what I'm trying to achieve, my dataframe looks like this:
+----+-----+--------+--------+
| id | id2 | value1 | value2 |
+----+-----+--------+--------+
| 1 | 1 | 54 | 2 |
+----+-----+--------+--------+
| 1 | 2 | 0 | 6 |
+----+-----+--------+--------+
| 1 | 3 | 578 | 14 |
+----+-----+--------+--------+
| 2 | 1 | 10 | 1 |
+----+-----+--------+--------+
| 2 | 2 | 6 | 32 |
+----+-----+--------+--------+
| 2 | 3 | 0 | 0 |
+----+-----+--------+--------+
| 3 | 1 | 12 | 2 |
+----+-----+--------+--------+
| 3 | 2 | 20 | 5 |
+----+-----+--------+--------+
| 3 | 3 | 63 | 22 |
+----+-----+--------+--------+
The desired output is the following table:
+----+----------+----------+----------+----------+----------+----------+
| id | value1_1 | value1_2 | value1_3 | value2_1 | value2_2 | value2_3 |
+----+----------+----------+----------+----------+----------+----------+
| 1 | 54 | 0 | 578 | 2 | 6 | 14 |
+----+----------+----------+----------+----------+----------+----------+
| 2 | 10 | 6 | 0 | 1 | 32 | 0 |
+----+----------+----------+----------+----------+----------+----------+
| 3 | 12 | 20 | 63 | 2 | 5 | 22 |
+----+----------+----------+----------+----------+----------+----------+
So, basically, for each unique id and for each column col, I will have n new columns col_1,... for each of the n id2 values.
Any help would be appreciated!
In Spark 2.4 you can do this way
var df3 =Seq((1,1,54 , 2 ),(1,2,0 , 6 ),(1,3,578, 14),(2,1,10 , 1 ),(2,2,6 , 32),(2,3,0 , 0 ),(3,1,12 , 2 ),(3,2,20 , 5 ),(3,3,63 , 22)).toDF("id","id2","value1","value2")
scala> df3.show()
+---+---+------+------+
| id|id2|value1|value2|
+---+---+------+------+
| 1| 1| 54| 2|
| 1| 2| 0| 6|
| 1| 3| 578| 14|
| 2| 1| 10| 1|
| 2| 2| 6| 32|
| 2| 3| 0| 0|
| 3| 1| 12| 2|
| 3| 2| 20| 5|
| 3| 3| 63| 22|
+---+---+------+------+
using coalesce retrieve the first value of the id.
scala> var df4 = df3.groupBy("id").pivot("id2").agg(coalesce(first("value1")),coalesce(first("value2"))).orderBy(col("id"))
scala> val newNames = Seq("id","value1_1","value2_1","value1_2","value2_2","value1_3","value2_3")
Renaming columns
scala> df4.toDF(newNames: _*).show()
+---+--------+--------+--------+--------+--------+--------+
| id|value1_1|value2_1|value1_2|value2_2|value1_3|value2_3|
+---+--------+--------+--------+--------+--------+--------+
| 1| 54| 2| 0| 6| 578| 14|
| 2| 10| 1| 6| 32| 0| 0|
| 3| 12| 2| 20| 5| 63| 22|
+---+--------+--------+--------+--------+--------+--------+
rearranged column if needed. let me know if you have any question related to the same. HAppy HAdoop
It is similar to some other questions but it is different.
Let say we have a pyspark dataframe df as below:
+-----+------+-----+
|col1 | col2 | col3|
+-----+------+-----+
|A | 5 | 6 |
+-----+------+-----+
|A | 5 | 8 |
+-----+------+-----+
|A | 6 | 3 |
+-----+------+-----+
|A | 5 | 9 |
+-----+------+-----+
|B | 9 | 6 |
+-----+------+-----+
|B | 3 | 8 |
+-----+------+-----+
|B | 9 | 8 |
+-----+------+-----+
|C | 3 | 4 |
+-----+------+-----+
|C | 5 | 1 |
+-----+------+-----+
I want to add another column as new_col which contains the mean of col2 based on grouping on col1. So, the answer must be as follows
+-----+------+------+--------+
|col1 | col2 | col3 | new_col|
+-----+------+------+--------+
| A | 5 | 6 | 5.25 |
+-----+------+------+--------+
| A | 5 | 8 | 5.25 |
+-----+------+------+--------+
| A | 6 | 3 | 5.25 |
+-----+------+------+--------+
| A | 5 | 9 | 5.25 |
+-----+------+------+--------+
| B | 9 | 6 | 7 |
+-----+------+------+--------+
| B | 3 | 8 | 7 |
+-----+------+------+--------+
| B | 9 | 8 | 7 |
+-----+------+------+--------+
| C | 3 | 4 | 4 |
+-----+------+------+--------+
| C | 5 | 1 | 4 |
+-----+------+------+--------+
Any help will be appreciated.
Step 1: Creating the DataFrame.
from pyspark.sql.functions import avg, col
from pyspark.sql.window import Window
values = [('A',5,6),('A',5,8),('A',6,3),('A',5,9),('B',9,6),('B',3,8),('B',9,8),('C',3,4),('C',5,1)]
df = sqlContext.createDataFrame(values,['col1','col2','col3'])
df.show()
+----+----+----+
|col1|col2|col3|
+----+----+----+
| A| 5| 6|
| A| 5| 8|
| A| 6| 3|
| A| 5| 9|
| B| 9| 6|
| B| 3| 8|
| B| 9| 8|
| C| 3| 4|
| C| 5| 1|
+----+----+----+
Step 2: Creating another column having the mean, by grouping over column A.
w = Window().partitionBy('col1')
df = df.withColumn('new_col',avg(col('col2')).over(w))
df.show()
+----+----+----+-------+
|col1|col2|col3|new_col|
+----+----+----+-------+
| B| 9| 6| 7.0|
| B| 3| 8| 7.0|
| B| 9| 8| 7.0|
| C| 3| 4| 4.0|
| C| 5| 1| 4.0|
| A| 5| 6| 5.25|
| A| 5| 8| 5.25|
| A| 6| 3| 5.25|
| A| 5| 9| 5.25|
+----+----+----+-------+
Ok, after a lot of trying, I could answer the question myself. I post the answer here for anyone else with the similar question. The original file is a csv file here.
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
#reading the file
df = spark.read.csv('file's name.csv', header=True)
df.show()
output
+-----+------+-----+
|col1 | col2 | col3|
+-----+------+-----+
|A | 5 | 6 |
+-----+------+-----+
|A | 5 | 8 |
+-----+------+-----+
|A | 6 | 3 |
+-----+------+-----+
|A | 5 | 9 |
+-----+------+-----+
|B | 9 | 6 |
+-----+------+-----+
|B | 3 | 8 |
+-----+------+-----+
|B | 9 | 8 |
+-----+------+-----+
|C | 3 | 4 |
+-----+------+-----+
|C | 5 | 1 |
+-----+------+-----+
from pyspark.sql import functions as func
#Grouping the dataframe based on col1
col1group = df.groupBy('col1')
#Computing the average of col2 based on the grouping on col1
a= col1group.agg(func.avg("col2"))
a.show()
output
+-----+----------+
|col1 | avg(col2)|
+-----+----------+
| A | 5.25 |
+-----+----------+
| B | 7.0 |
+-----+----------+
| C | 4.0 |
+-----+----------+
Now, we join the last table with the initial dataframe to generate our desired dataframe:
df=test1.join(a, on = 'lable', how = 'inner')
df.show()
output
+-----+------+------+---------+
|col1 | col2 | col3 |avg(col2)|
+-----+------+------+---------+
| A | 5 | 6 | 5.25 |
+-----+------+------+---------+
| A | 5 | 8 | 5.25 |
+-----+------+------+---------+
| A | 6 | 3 | 5.25 |
+-----+------+------+---------+
| A | 5 | 9 | 5.25 |
+-----+------+------+---------+
| B | 9 | 6 | 7 |
+-----+------+------+---------+
| B | 3 | 8 | 7 |
+-----+------+------+---------+
| B | 9 | 8 | 7 |
+-----+------+------+---------+
| C | 3 | 4 | 4 |
+-----+------+------+---------+
| C | 5 | 1 | 4 |
+-----+------+------+---------+
Now change the name of the last column to whatever we want
df = df.withColumnRenamed('avg(val1)', 'new_col')
df.show()
output
+-----+------+------+--------+
|col1 | col2 | col3 | new_col|
+-----+------+------+--------+
| A | 5 | 6 | 5.25 |
+-----+------+------+--------+
| A | 5 | 8 | 5.25 |
+-----+------+------+--------+
| A | 6 | 3 | 5.25 |
+-----+------+------+--------+
| A | 5 | 9 | 5.25 |
+-----+------+------+--------+
| B | 9 | 6 | 7 |
+-----+------+------+--------+
| B | 3 | 8 | 7 |
+-----+------+------+--------+
| B | 9 | 8 | 7 |
+-----+------+------+--------+
| C | 3 | 4 | 4 |
+-----+------+------+--------+
| C | 5 | 1 | 4 |
+-----+------+------+--------+
I have a dataframe like this:
df =
--------------
|col1 | col2 |
--------------
| A | 1 |
| A | 5 |
| B | 0 |
| A | 2 |
| B | 6 |
| B | 8 |
--------------
I want to partition by col1, find the median of col2 in each partition, and append the result to form a new column. The result should look like this:
result =
---------------------
|col1 | col2 | col3 |
---------------------
| A | 1 | 2 |
| A | 5 | 2 |
| B | 0 | 6 |
| A | 2 | 2 |
| B | 6 | 6 |
| B | 8 | 8 |
---------------------
For now, I'm using this code:
val df2 = df
.withColumn("tmp", percent_rank over Window.partition('col1).orderBy('col2))
.where("tmp <= 0.5")
.groupBy("col1").agg(max(col2) as "col3")
val result = df.join(df2, df("col1") === df2("col1")).drop(df2("col1"))
But this takes too much time and space resources to run when the dataframe is big. Please help me find a way to do the above more efficiently!
Any help is much appreciated!
With the data you have, you can do a Spark DataFrame groupBy statement with percentile_approx to perform the calculation.
// Creating the `df` dataset
val df = Seq(("A", 1), ("A", 5), ("B", 0), ("A", 2), ("B", 6), ("B", 8)).toDF("col1", "col2")
df.createOrReplaceTempView("df")
Use percentile_approx with groupBy to perform median calculation:
val df2 = spark.sql("select col1, percentile_approx(col2, 0.5) as median from df group by col1 order by col1")
df2.show()
with the output of df2 being:
+----+------+
|col1|median|
+----+------+
| A| 2.0|
| B| 6.0|
+----+------+
And now running the join to recreate the final result:
val result = df.join(df2, df("col1") === df2("col1"))
result.show()
//// output
+----+----+----+------+
|col1|col2|col1|median|
+----+----+----+------+
| A| 1| A| 2.0|
| A| 5| A| 2.0|
| B| 0| B| 6.0|
| A| 2| A| 2.0|
| B| 6| B| 6.0|
| B| 8| B| 6.0|
+----+----+----+------+