Carefully select and retrieve one from a large set of data - apache-spark-sql

I don't know what to do when I want to retrieve one item in each column from data like the following.
|name|power|
test1|15
test1|120
test1|40
test2|50
test2|80
test2|100
test3|1
test3|4
test3|10
The results I want.
|name|power|
test1|120
test2|100
test3|10
What I tried.
pickup_one = (
beaconPickUp_three
.withColumn('r_max_date', f.max('power').over(w.partitionBy()))
.filter(f.col('r_max_date') == f.col('power'))
)
or
PickUp_one = beaconPickUp_three.groupBy(f.col("name")).agg({"power": "max"})
I need your help.

Related

find diffrences between 2 tables sql and how can i get the changed value?

i have this query
insert into changes (id_registro)
select d2.id_registro
from daily2 d2
where exists (
select 1
from daily d1
where
d1.id_registro = d2.id_registro
and (d2.origen, d2.sector, d2.entidad_um, d2.sexo, d2.entidad_nac, d2.entidad_res,
d2.municipio_res, d2.tipo_paciente,d2.fecha_ingreso, d2.fecha_sintomas,
d2.fecha_def, d2.intubado, d2.neumonia, d2.edad, d2.nacionalidad, d2.embarazo,
d2.habla_lengua_indig, d2.diabetes, d2.epoc, d2.asma, d2.inmusupr, d2.hipertension,
d2.otra_com, d2.cardiovascular, d2.obesidad,
d2.renal_cronica, d2.tabaquismo, d2.otro_caso, d2.resultado, d2.migrante,
d2.pais_nacionalidad, d2.pais_origen, d2.uci )
<>
(d1.origen, d1.sector, d1.entidad_um, d1.sexo, d1.entidad_nac, d1.entidad_res,
d1.municipio_res, d1.tipo_paciente, d1.fecha_ingreso, d1.fecha_sintomas,
d1.fecha_def, d1.intubado, d1.neumonia, d1.edad, d1.nacionalidad, d1.embarazo,
d1.habla_lengua_indig, d1.diabetes, d1.epoc, d1.asma, d1.inmusupr, d1.hipertension,
d1.otra_com, d1.cardiovascular, d1.obesidad,
d1.renal_cronica, d1.tabaquismo, d1.otro_caso, d1.resultado, d1.migrante,
d1.pais_nacionalidad, d1.pais_origen, d1.uci ))
it results in an insersion data that doesn't exist in another table, that's fine. but i want know exactly which field has changed to store it in a log table
You don't mention precisely what you expect to see in your output but basically to accomplish what you're after you'll need a long sequence of CASE clauses, one for each column
e.g. one approach might be to create a comma-separated list of the column names that have changed:
INSERT INTO changes (id_registro, column_diffs)
SELECT d2.id_registro,
CONCAT(
CASE WHEN d1.origen <> d2.origen THEN 'Origen,' ELSE '' END,
CASE WHEN d1.sector <> d2.sector THEN 'Sector,' ELSE '' END,
etc.
Within the THEN part of the CASE you can build whatever detail you want to show
e.g. a string showing before and after values of the columns CONCAT('Origen: Was==> ', d1.origen, ' Now==>', d2.origen). Presumably though you'll also need to record the times of these changes if there can be multiple updates to the same record throughout the day.
Essentially you'll need to decide what information you want to show in your logfile, but based on your example query you should have all the information you need.

UPDATE QUERY - Sum up a value from form with value from table

I've just started using microsoft access so I don't really know how to solve this. I would like to use an update query to add a value from a form to a value on a table.
I originally used the SUM expression which gave me an error saying it was an aggregate function.
I also tried to add the two values together (e.g [field1] + [field2]) which as a result gave me a value with both numbers together instead of adding them together.
The following is the SQL I'm using:
UPDATE Votes
SET Votes.NumVotes = [Votes]![NumVotes]+[Forms]![frmVote]![txtnumvotes]
WHERE (((Votes.ActID) = [Forms]![frmVote]![combacts])
AND ((Votes.RoundNum) = [Forms]![frmVote]![combrndnum]))
I want to add a value [txtnumvotes] a form to a field [NumVotes] from the table [Votes].
Could someone please help me?
You can specify the expected data type with parameters:
PARAMETERS
[Forms]![frmVote]![txtnumvotes] Short,
[Forms]![frmVote]![combacts] Long,
[Forms]![frmVote]![combrndnum] Long;
UPDATE
Votes
SET
Votes.NumVotes = [Votes]![NumVotes]+[Forms]![frmVote]![txtnumvotes]
WHERE
(((Votes.ActID) = [Forms]![frmVote]![combacts])
AND
((Votes.RoundNum) = [Forms]![frmVote]![combrndnum]))
Without the specification, Access has to guess, and that sometimes fails.

Using WHERE clause to only retrieve results where array field has only ONE element inside

I have a table with various arrays within it like this:
{10574664,10574665,10574679,10574724}
{8616204,10574643,10574644,10574645,10574651,10574688,10574690,10574696,10574708}
{8616208}
{9830397}
{8616203}
{8616204,10574643,10574644,10574645,10574651,10574688,10574690,10574696,10574708}
{8616204,10574643,10574644,10574645,10574651,10574688,10574690,10574696,10574708}
{8616208}
{10574680}
{8616203}
Is there a way to only pull back the records where there is only one element in the array.
The results would look like this:
{8616208}
{9830397}
{8616203}
{8616208}
{10574680}
{8616203}
My query would look something like this
Select * from attr_lookup where target_tcode = help with query here
Use array_length():
SELECT *
FROM attr_lookup
WHERE array_length(target_tcode, 1) = 1;

Check if a value exists in a collection stored in XML data type column

I have an XML data type column called "tags".
In that, I am storing a collection, like so:
<ArrayOfString>
<string>personal</string>
<string>travel</string>
<string>gadgets</string>
<string>parenting</string>
</ArrayOfString>
I want to select all the rows, that have one of the values that I am looking for: for example, I want to select all rows in the table that have a tag "travel".
I know that this works, if I know the index of the value I am looking for:
select * from posts
where tags.value('(/ArrayOfString/string)[1]', 'nvarchar(1000)') = 'travel'
but this query works only if the tag "travel" is the 2nd item in the nodes. How do I check if a value exists, irrespective of the position it is in?
select *
from tags
where tags.exist('/ArrayOfString/string[. = "travel"]') = 1
Or like this if you want to check against a variable.
declare #Val varchar(10)
set #Val = 'travel'
select *
from tags
where tags.exist('/ArrayOfString/string[. = sql:variable("#Val")]') = 1
You can try something like this:
SELECT
*
FROM
dbo.Posts
WHERE
tags.exist('/ArrayOfString/string/text()[. = "travel"]') = 1
This will list all the rows that have "travel" in one of the strings in your XML

Storing Select Query result in Variable Ruby on Rails

I wanted to know how we could store the result of my select query on a variable.
#ppt2 = Ppt.select('slide_name').where('id=?',4)
#ppt1 = Ppt.update_all({:time2=>#ppt2},['id like ?','1'])
Here, slide_name and time2 are both text attributes of the same table ppt.
What happens on the above execution is that the time2 field in id=1 gets the value "#ppt2" whereas I want it to get the value of slide_name from id=4 which does not get stored in #ppt1.
In other words, how do I store the value of the select query in #ppt2 so that it can be used in the next line?
Any help is appreciated.
Call the slide_name method on your first result.
#ppt2 = Ppt.select('slide_name').find(4).slide_name