How to get the number of empty cells for each column in spark dataframe - dataframe

I'd like to get the number of each column's empty value, so I tried
ele_df.where(ele_df['Shipment_ID'].isNotNull()).select('Shipment_ID').show()
But it returns me the empty value, it seems it consider the empty value as a non-null value.
+------------------+
|Shipment_ID|
+------------------+
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
+------------------+
Could you guys help me with this?

Related

SQL, iif excel cell is null then fill another excel cell

I have the following part of my query within excel that is not working.
iif(master.[Canada] is null or master.[USA] is null ,'USER','' ) as [Stackoverflow]
Am I doing the nulls correctly?
The logic should
1) If there is No Canada or No Usa data, put "USER" in Stackoverflow column.
2) If either Canada OR USA has data then Stackoverflow should be empty.
Currently what Im getting:
+-----------+--------------+---------------+
| Canada | USA | Stackoverflow |
+-----------+--------------+---------------+
| | | |
| | | |
| 912796NZ8 | | |
| | | |
| | US912796NZ81 | |
| | | |
| 912796NZ8 | US912796NZ81 | |
| 912796NZ8 | US912796NZ81 | |
| 912796qd4 | US912796QD43 | |
| 298785HB5 | US298785HB50 | |
+-----------+--------------+---------------+
What I am expecting:
+-----------+--------------+---------------+
| Canada | USA | Stackoverflow |
+-----------+--------------+---------------+
| | | USER |
| | | USER |
| 912796NZ8 | | |
| | | USER |
| | US912796NZ81 | |
| | | USER |
| 912796NZ8 | US912796NZ81 | |
| 912796NZ8 | US912796NZ81 | |
| 912796qd4 | US912796QD43 | |
| 298785HB5 | US298785HB50 | |
+-----------+--------------+---------------+
After changing query to iif(TRIM(master.[Camada]) = '' OR TRIM(master.[USA]) = '','USER', '') as [Stackoverflow]
It does a good job except now I still have some canada and USA data that gives me USER.
+-----------+-----+---------------+
| Canada | USA | Stackoverflow |
+-----------+-----+---------------+
| 62941ZPA6 | | USER |
| 62943Z4R0 | | USER |
| 62945ZLQ1 | | USER |
| 62950ZZE5 | | USER |
| 75585RLK9 | | USER |
| 00433JAA3 | | USER |
| 13509PEV1 | | USER |
| 13509PEZ2 | | USER |
| 62931ZLX2 | | USER |
| 62941Z8M9 | | USER |
| 62941ZYK4 | | USER |
| 62942ZV42 | | USER |
| 62943Z6T4 | | USER |
| 62946Z6Y0 | | USER |
| 62947ZWC8 | | USER |
| 62948ZTJ6 | | USER |
| 62949ZE51 | | USER |
| 75585RLK9 | | USER |
| 75585RMB8 | | USER |
| 75585RMW2 | | USER |
+-----------+-----+---------------+
Should not have USER for these 20 records.
Any help would be appreciated, thanks.
The Jet/ACE SQL dialect does support IS NULL. However, as your current results suggest, empty strings ('') are not the same as the NULL entity. This is especially true in Excel (a non-database application where empty cells may not default to NULL). In fact, you are actually assigning empty strings in the falsepart of your IIF() call where records without 'USER' value in [Stackoverflow] will be empty string and not NULL.
Consider extending your IIF expressions to account for zero-length strings and assigning NULL to non-matches:
IIF((master.[Canada] IS NULL AND master.[USA] IS NULL) OR
(master.[Canada] = '' AND master.[USA] IS NULL) OR
(master.[Canada] IS NULL AND master.[USA] = '') OR
(master.[Canada] = '' AND master.[USA] = ''), 'USER', NULL) As [Stackoverflow]
Even account for invisible whitespace by using TRIM():
IIF((master.[Canada] IS NULL AND master.[USA] IS NULL) OR
(TRIM(master.[Canada]) = '' AND master.[USA] IS NULL) OR
(master.[Canada] IS NULL AND TRIM(master.[USA]) = '') OR
(TRIM(master.[Canada]) = '' AND TRIM(master.[USA]) = ''), 'USER', NULL) As [Stackoverflow]
I think, Jet uses the IsNull() function instead of the IS NULL operator:
iif(IsNull(master.[Canada]) or IsNull(master.[USA]),'USER','' ) as [Stackoverflow]

What is the best way to apply SQL Pivot to a table?

I'm trying to use the pivot function but not sure how to replace the Letters in the bi-weekly table to a persons name instead of the query result. Here is my query.
SELECT a.OCD, a.[f_name],a.[l_name],a.[ads_phone], b.wk1m, b.wk1t, b.wk1w, b.wk1r, b.wk1f, b.wk2m, b.wk2t, b.wk2w, b.wk2r, b.wk2f
FROM [dbo].[tbl_teleworkers] as a
inner join [dbo].[tbl_scheduled] as b ON a.pin = b.pin
where ocd = '022'
here is my result:
OCD|f_name|l_name|ads_phone|wk1m|wk1t|wk1w|wk1r|wk1f|wk2m|wk2t|wk2w|wk2r|wk2f
022|John |smith |111-1111 |M | | | R | | | | | |
022|Jane |smith |222-2222 | | | | | | | | W | |
022|Joe |smith |333-3333 |M | | | | F | | | | |
022|Jim |smith |444-4444 | | T | | | | M | | | |
022|Jill |smith |555-5555 |M | | W | | | | | | R |
Here is what I'm looking to get: So instead of the letters of the week. I'm trying to display the person's name and phone number.
wk1m |wk1t|wk1w|wk1r |wk1f|wk2m|wk2t|wk2w |wk2r|wk2f
John | | |John | | | | | |
phone| | |phone| | | | | |
| | | | | | |Jane S| |
| | | | | | |phone | |
Joe | |
phone| |
Any help would be greatly appreciated.
Thanks!
I'm going to ignore the question in your subject line because your sample data and desired output do not require a PIVOT at all.
You can get your desired output by making each column a CASE expression. In each wk** column, if the value is not empty, return the value of the first name, last initial (if desired) and phone number concatenated together.

Need to shift the data to next column, unfortunately added data in wrong column

I have a table test
+----+--+------+--+--+--------------+--+--------------+
| ID | | Name1 | | | Name2 |
+----+--+------+--+--+--------------+--+--------------+
| 1 | | Andy | | | NULL |
| 2 | | Kevin | | | NULL |
| 3 | | Phil | | | NULL |
| 4 | | Maria | | | NULL |
| 5 | | Jackson | | | NULL |
+----+--+------+--+--+----------+--+--
I am expecting output like
+----+--+------+--+--+----------+--
| ID | | Name1 | | | Name2 |
+----+--+------+--+--+----------+--
| 1 | | NULL | | | Andy |
| 2 | | NULL | | | Kevin |
| 3 | | NULL | | | Phil |
| 4 | | NULL | | | Maria |
| 5 | | NULL | | | Jackson |
+----+--+------+--+--+----------+--
I unfortunately inserted data in wrong column and now I want to shift the data to the next column.
You can use an UPDATE statement with no WHERE condition, to cover the entire table.
UPDATE test
SET Name2 = Name1,
Name1 = NULL

Changing the orientation of a SplitView

I'm making a UWP (Windows 10) app. I'd like to know, is it possible to change the orientation of a SplitView? Typically, it's ordered like this:
______________________________________________
| | |
| | |
| | |
| | |
| | |
| Pane | Content |
| | |
| | |
| | |
| | |
| | |
----------------------------------------------
Is it possible to change the orientation to:
______________________________________________
| |
| |
| Pane |
| |
| |
| |
----------------------------------------------
| |
| |
| |
| |
| Content |
| |
| |
| |
----------------------------------------------
It is not supported by the platform (SplitVew.PanePlacement property can only be left or right).
You can likely achieve a somewhat similar affect by placing a command bar at the top of your application.

Setting the position of a tab Scrollbar when updaing the VerticalScroll Value

I am updating a number of controls dynamically on a tabpage within a c# application. The tab page has autoscroll enabled. As you may be aware, when the user has scrolled down it sets the location x=0,y=0 to be at the current x=0,y=0.
so:
+-----------------------------------+
| X |---| X is now 0,0 in terms of setting locations.
| | - |
| |---|
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| TEXT | |
| | |
+-----------------------------------+
AND
+-----------------------------------+
| X | | X is now 0,0 in terms of setting locations.
| TEXT | |
| | |
| | |
| | |
| |---|
| | - |
| |---|
| | |
| | |
| | |
| | |
+-----------------------------------+
Therefore I have to save the VerticalScroll.Value, set it to 0, update the controls and then set the VerticalScroll.Value back to the original value.
This works to a degree, the position of the tab form is correct, however the scrollbar doesn't update, so it looks like this:
+-----------------------------------+
| |---|
| TEXT | - |
| |---|
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
+-----------------------------------+
Is there any easy way to update the position of the scrollbar?
If I leave it as it is, when the user clicks on the scrollbar again it moves the position back to the top of the tab form. Thanks for any help you can provide.
To answer my own question:
http://msdn.microsoft.com/en-us/library/system.windows.forms.scrollablecontrol.autoscrollposition.aspx