This question already has answers here:
How to add a constant column in a Spark DataFrame?
(3 answers)
Closed 10 days ago.
I am trying to query a dataframe and add a column with a set value but I'm not sure how to get it to work. I know how it works in SQL but I could use help converting it to pyspark.
PL/SQL Example:
SELECT 1 AS column1
,2 AS column2
FROM dual;
pyspark:
empDF.select("name", col("").alias("nullColumn")).display()
Please have a look at the withColumn() function. Can be used in conjunction with lit()
https://spark.apache.org/docs/3.1.3/api/python/reference/api/pyspark.sql.DataFrame.withColumn.html
A new column can be added to an existing dataframe using this option.
Sample Example:
df.withColumn("Country", lit("USA")).show()
df.withColumn("Country", lit("USA")) \
.withColumn("anotherColumn",lit("anotherValue")) \
.show()
Example Source: Google led to https://azurelib.com/withcolumn-usage-in-databricks-with-examples/
Hope it helps...
Related
This question already has answers here:
What does the colon sign ":" do in a SQL query?
(8 answers)
Closed 10 months ago.
likely a very simple SQL (oracle) question for you. See here a simplified example of my problem:
DELETE FROM schema.tablename
WHERE
col1 = :v0
AND col2 = :v1
AND col3 = :v2
;
I don't know what :vo...:vo3 means and I have no idea what to google. Can you either explain it or give me some good literature on that ?
it's called bind-variable , just think that those start with colon(:) is different variable for binding value used in sql command.
Try imagine the use case of it.
"I want to delete records from table schema.tablename by using where clause so I will remove those record that have col1 = value1 and col2 = value2 and col3 = value3 "
Well ok then,how I write those above sql statement by not fixing the value of value1,value2,value3 ?
the answer is using variable for binding the value!
:v0 - the variable that has name v0
:v1 - the variable that has name v1
:v2 - the variable that has name v2
ref.
https://www.oracletutorial.com/python-oracle/bind-variables/
This question already has answers here:
PostgreSQL substring get string between brackets
(2 answers)
Closed 1 year ago.
I have the following details column, with varying parameters. How can I get only joblib values? "The Place of joblib is not always the same, so I may bot be able to use substring count"
date:01/12/2014--**--joblib:[snbsd]--**--date_type:no_date--**--max_feat_values:ss,group_filters:[]--**--no_imp_phrases:1--**--variable_facets:auto
date:01/12/2014--**--joblib:[jinxthin]--**--date_type:no_date--**--max_feat_values:ss,group_filters:[]--**--no_imp_phrases:1--**--variable_facets:auto
date:01/12/2014--**--joblib:[snbserv]--**--date_type:no_date--**--max_feat_values:ss,group_filters:[]--**--no_imp_phrases:1--**--variable_facets:auto
date:01/12/2016--**--joblib:[sql12server]--**--date_type:no_date--**--max_feat_values:ss,group_filters:[]--**--no_imp_phrases:1--**--variable_facets:auto
date:01/12/2015--**--joblib:[stfmbinserx]--**--date_type:no_date--**--max_feat_values:ss,group_filters:[]--**--no_imp_phrases:1--**--variable_facets:auto
date:01/12/2011--**--joblib:[ftplikes]--**--date_type:no_date--**--max_feat_values:ss,group_filters:[]--**--no_imp_phrases:1--**--variable_facets:auto
Desired result:
snbsd
jinxthin
snbserv
sql12server
stfmbinserx
ftplikes
Here You go:
substring(substring(var1, position('joblib:' in var1)+8), 1, position(']' in substring(var1, position('joblib:' in var1)+8))-1)
replace var1 with column name containing Your string
With below You can try it out directly on PostgreSQL:
WITH myconstants (var1) as (
values ('date:01/12/2014--**--joblib:[snbsd]--**--date_type:no_date--**--max_feat_values:ss,group_filters:[]--**--no_imp_phrases:1--**--variable_facets:auto')
)
SELECT substring(substring(var1, position('joblib:' in var1)+8), 1, position(']' in substring(var1, position('joblib:' in var1)+8))-1)
FROM myconstants
This question already has answers here:
SQL Query Where Column = '' returning Emoji characters 🎃 and 🍰
(4 answers)
Closed 3 years ago.
I currently have a query where I want to select cake:
SELECT '🍰'
But upon executing query, it gives me an output of '??'
How to output the real cake?
set as unicode using the N'' notation
SELECT N'🍰'
This question already has answers here:
How to filter Pandas dataframe using 'in' and 'not in' like in SQL
(11 answers)
Closed 3 years ago.
I'm trying to remove rows from a dataframe if a particular column value does not appear in a previously defined dictionary
dff= dff[dff['network'] in net_dic]
Each value of 'network' is a string. and net_dic looks like this:
{ 'abc' : 1
'def' : 2
.
.
.}
It errors:
TypeError: 'Series' objects are mutable, thus they cannot be hashed
Ah this works:
dff= dff[dff['network'].isin(net_dic.keys())]
This question already has answers here:
Equals(=) vs. LIKE
(16 answers)
WHERE clause on SQL Server "Text" data type
(7 answers)
Closed 7 years ago.
what is the differences between = to like ?
thanks for help :)
by the way if it's help , the error i get is: the data types text and varchar are incompatible in the equal to operator [msg 402]
1:
Like
The LIKE operator is used to search for a specified pattern in a column.
Where
The WHERE clause is used to extract only those records that fulfill a specified criterion.
In your case the first query do not work is because no records match the criteria you are searching.