Getting table(records) to update properply using the MERGE Statement - sql

Good morning everyone!
Below is a piece of code I stitched together: I used a CTE to grab the records(data) from a link table and than convert strings to dates, than use the merge statement to get the data into a local table:
I am having a problem with the column(field) LAST_RACE_DATE this field is set to NULL and is not required but it does not update with my current set up. What I am trying to accomplished is for this field to populate when data is entered but also update, meaning it should also update with NULL.
So if the field has a specific date, and a new date is entered in the remote database, this field should update as well, even if the data is deleted in the back end, it should also remove the local table data for this field.
WITH CTE AS(
SELECT MEMBER_ID
,[MEMBER_DATE] = MAX(CONVERT(DATE, MEMBER_DATE))
,RACE_DATE = MAX(CONVERT(DATE, RACE_DATE))
,LAST_RACE_DATE = MAX(CONVERT(DATE, LAST_RACE_DATE))
FROM [EXAMPLE].[dbo].[LINKED_MEMBER_DATA]
WHERE (MEMBER_DATE IS NOT NULL) AND (ISDATE(MEMBER_DATE)<> 0) AND (RACE_DATE IS NOT NULL) AND (ISDATE(RACE_DATE)<> 0)
AND (LAST_RACE_DATE IS NULL) OR (ISDATE(LAST_RACE_DATE)<> 0)
GROUP BY MEMBER_ID)
MERGE dbo.LINKED_MEMBER_DATA AS Target
USING (SELECT
MEMBER_ID, MEMBER_DATE, RACE_DATE, LAST_RACE_DATE
FROM CTE
GROUP BY MEMBER_ID, RACE_DATE, LAST_RACE_DATE)AS SOURCE ON (Target.MEMBER_ID = SOURCE.MEMBER_ID)
WHEN MATCHED AND
(Target.MEMBER_DATE) <> (SOURCE.MEMBER_DATE)
OR (Target.RACE_DATE) <> (SOURCE.RACE_DATE)
OR ISNULL(TARGET.LAST_RACE_DATE , Target.LAST_RACE_DATE) <> ISNULL(SOURCE.LAST_RACE_DATE, SOURCE.LAST_RACE_DATE)
THEN UPDATE SET
Target.MEMBER_DATE = SOURCE.MEMBER_DATE
,Target.RACE_DATE = SOURCE.RACE_DATE
,Target.LAST_RACE_DATE = SOURCE.LAST_RACE_DATE
WHEN NOT MATCHED BY TARGET THEN
INSERT(
MEMBER_ID, MEMBER_DATE, RACE_DATE, LAST_RACE_DATE)
VALUES (Source.MEMBER_ID, Source.MEMBER_DATE, Source.RACE_DATE, Source.LAST_RACE_DATE);
I also tried this:
ISNULL(Target.LAST_RACE_DATE,'N/A') <> ISNULL(SOURCE.LAST_RACE_DATE,'N/A')
But it generates the below error for dates conversion:
Conversion failed when converting date and/or time from character string.
Thanks a Million!!

Your current statement is failing because the ISNULLs that you have don't do anything (if one of the values is NULL the expression will evaluate to NULL), and NULL values don't compare. Your second attempt doesn't work because ISNULL requires the data types of the two values to be the same, so you could try eg ISNULL(Target.LAST_RACE_DATE, '1970-01-01') <> ISNULL(Source.LAST_RACE_DATE, '1970-01-01').
Another option would be to simply enumerate the different cases (eg, (((Source.LAST_RACE_DATE IS NULL AND Target.LAST_RACE_DATE IS NOT NULL) OR (Source.LAST_RACE_DATE IS NOT NULL AND Target.LAST_RACE_DATE IS NULL) OR (Source.LAST_RACE_DATE <> Target.LAST_RACE_DATE))). Enumerating the different situations makes the code a bit more verbose, but it can result in better performance (whether it is measurably better really depends on how much data you are processing).

Related

Snowflake ignores statement in where clause where I'm comparing timestamps

so I'm building a SCD type 2 in snowflake, but it ignores the where clause in which is comparision between "to_timestamp" and "expiry_date". Expiry_date is a variable that is set to '9999-08-17 07:31:29.901000000' (as infinity) and To_timestamp is a column in table. I want to query only the rows that have to_timestamp set to infinity (they are still active) but snowflake seems to ignore this part of where clause. Below is some of the code (it should update the rows that are expired - that means change their "to_timestamp" to current time. and it does but it does to rows with timestamps of all kind - it ignores last line)
SET EXPIRY_DATE_NTZ = '9999-08-17 07:31:29.901000000';
SET CURRENT_DATE_NTZ = TO_TIMESTAMP_NTZ(CURRENT_TIMESTAMP());
UPDATE CUSTOMER_TARGET CT
SET CT.TO_TIMESTAMP = $CURRENT_DATE_NTZ
FROM POC.SNOWFLAKE_POC.CUSTOMER_STAGE CS
WHERE CT.C_CUSTOMER_ID = CS.C_CUSTOMER_ID
AND (CT.C_FIRST_NAME <> CS.C_FIRST_NAME OR CT.C_LAST_NAME <> CS.C_LAST_NAME OR CT.C_BIRTH_YEAR
<> CS.C_BIRTH_YEAR OR CT.C_BIRTH_COUNTRY <> CS.C_BIRTH_COUNTRY OR CT.C_LAST_REVIEW_DATE<>CS.C_LAST_REVIEW_DATE)
AND CT.TO_TIMESTAMP = $EXPIRY_DATE_NTZ;
I have two of these update statements (one for updates and one for deletes) and a merge statement for inserts. And it ignores the comparision in every single one, updating the rows that have "to_timestamp" set to something like "2021-08-24 07:11:53.510000000". I've tried every combination possible (between ... and ..., >= ... <=, <=, >=, comparing in "case" statement of update,...) - nothing. What could be the cause/solution?
As we do not know the structure of CUSTOMER_TARGET I would suggest to explicitly set the data type of EXPIRY_DATE_NTZ variable to match the column data type:
SET EXPIRY_DATE_NTZ = '9999-08-17 07:31:29.901000000';
SELECT $EXPIRY_DATE_NTZ;
DESCRIBE RESULT LAST_QUERY_ID();
to:
-- TIMESTAMP_NTZ as an example
SET EXPIRY_DATE_NTZ = '9999-08-17 07:31:29.901000000'::TIMESTAMP_NTZ;
SELECT $EXPIRY_DATE_NTZ;
DESCRIBE RESULT LAST_QUERY_ID();
By doing that way there are no "implicit conversions" involved in the process.
Another advice is usage of IS DISTINCT FROM instead of <>. IS DISTINCT FROM is NULL safe, which is important if columns are defined as nullable.
UPDATE CUSTOMER_TARGET CT
SET CT.TO_TIMESTAMP = $CURRENT_DATE_NTZ
FROM POC.SNOWFLAKE_POC.CUSTOMER_STAGE CS
WHERE CT.C_CUSTOMER_ID = CS.C_CUSTOMER_ID
AND (CT.C_FIRST_NAME IS DISTINCT FROM CS.C_FIRST_NAME
OR CT.C_LAST_NAME IS DISTINCT FROM CS.C_LAST_NAME
OR CT.C_BIRTH_YEAR IS DISTINCT FROM CS.C_BIRTH_YEAR
OR CT.C_BIRTH_COUNTRY IS DISTINCT FROM CS.C_BIRTH_COUNTRY
OR CT.C_LAST_REVIEW_DATE IS DISTINCT FROM CS.C_LAST_REVIEW_DATE)
AND CT.TO_TIMESTAMP = $EXPIRY_DATE_NTZ;
Your SQL does not have any issues with the filters (ORs are surrounded by the brackets etc). I assume that you have checked the execution profile, and did not see your filter (CT.TO_TIMESTAMP = '9999-08-17 07:31:29.901000000'). In this case, all rows in the target table should have this value in the column TO_TIMESTAMP.
I highly recommend you check the data first. If you are running multiple UPDATE/MERGE commands, you may miss that the data has already updated with this value.

find diffrences between 2 tables sql and how can i get the changed value?

i have this query
insert into changes (id_registro)
select d2.id_registro
from daily2 d2
where exists (
select 1
from daily d1
where
d1.id_registro = d2.id_registro
and (d2.origen, d2.sector, d2.entidad_um, d2.sexo, d2.entidad_nac, d2.entidad_res,
d2.municipio_res, d2.tipo_paciente,d2.fecha_ingreso, d2.fecha_sintomas,
d2.fecha_def, d2.intubado, d2.neumonia, d2.edad, d2.nacionalidad, d2.embarazo,
d2.habla_lengua_indig, d2.diabetes, d2.epoc, d2.asma, d2.inmusupr, d2.hipertension,
d2.otra_com, d2.cardiovascular, d2.obesidad,
d2.renal_cronica, d2.tabaquismo, d2.otro_caso, d2.resultado, d2.migrante,
d2.pais_nacionalidad, d2.pais_origen, d2.uci )
<>
(d1.origen, d1.sector, d1.entidad_um, d1.sexo, d1.entidad_nac, d1.entidad_res,
d1.municipio_res, d1.tipo_paciente, d1.fecha_ingreso, d1.fecha_sintomas,
d1.fecha_def, d1.intubado, d1.neumonia, d1.edad, d1.nacionalidad, d1.embarazo,
d1.habla_lengua_indig, d1.diabetes, d1.epoc, d1.asma, d1.inmusupr, d1.hipertension,
d1.otra_com, d1.cardiovascular, d1.obesidad,
d1.renal_cronica, d1.tabaquismo, d1.otro_caso, d1.resultado, d1.migrante,
d1.pais_nacionalidad, d1.pais_origen, d1.uci ))
it results in an insersion data that doesn't exist in another table, that's fine. but i want know exactly which field has changed to store it in a log table
You don't mention precisely what you expect to see in your output but basically to accomplish what you're after you'll need a long sequence of CASE clauses, one for each column
e.g. one approach might be to create a comma-separated list of the column names that have changed:
INSERT INTO changes (id_registro, column_diffs)
SELECT d2.id_registro,
CONCAT(
CASE WHEN d1.origen <> d2.origen THEN 'Origen,' ELSE '' END,
CASE WHEN d1.sector <> d2.sector THEN 'Sector,' ELSE '' END,
etc.
Within the THEN part of the CASE you can build whatever detail you want to show
e.g. a string showing before and after values of the columns CONCAT('Origen: Was==> ', d1.origen, ' Now==>', d2.origen). Presumably though you'll also need to record the times of these changes if there can be multiple updates to the same record throughout the day.
Essentially you'll need to decide what information you want to show in your logfile, but based on your example query you should have all the information you need.

SQL Server 403 Error When Setting a Geography Type for Update

All I need to do is simply get one geography value from a table and store it in another table. There is some logic for which row to take from the origin table so it's not just a straight select.
In any of 50 possible variants of this, I get this error when hitting the update to the target table:
Msg 403, Level 16, State 1, Line 1
Invalid operator for data type. Operator equals not equal to, type equals geography.
My SQL looks like this at the moment:
declare
#EquipmentId int
, #CurrentLocationId int
, #CurrentGeoLocation geography
, #LastUpdated datetime
select #EquipmentId =
(
select top 1 EquipmentId
from Equipment
order by EquipmentId
)
select #CurrentLocationId = (select top 1 EquipmentLocationId from EquipmentLocation where EquipmentId = #EquipmentId order by LastUpdated desc)
select #LastUpdated = (select top 1 LastUpdated from EquipmentLocation where EquipmentId = #EquipmentId order by LastUpdated desc)
UPDATE
dbo.Equipment
SET
CurrentLocationDateTime = #LastUpdated
, CurrentGeoLocation = (select GeoLocation from EquipmentLocation where EquipmentLocationId = #CurrentLocationId)
, ModifiedBy = 'system'
, ModifiedByUserId = -1
, ModifiedDate = getdate()
WHERE
EquipmentId = #EquipmentId
I have had CurrentGeoLocation set in a variable of the same type, selected into by the same statement you see in the update.
I have had an #CurrentGeoLocation variable populated by a geography::STGeomFromText as well as geography::Point() function call.
I've used Lat and Long variables to call Point and FromText functions.
All the same result, the above 403 error. I could understand it somewhat when I was concatenating various permutations of the GeomFromText function that needs well known text format for the point parameter, but field value to field value is killing me, as is the fact that I get this error no matter how I try to give the origin point data to the target table.
Thoughts?
Update:
I've been experimenting a little and found that the following works just fine:
declare #GL geography
select #GL = (select GeoLocation from EquipmentLocation where EquipmentLocationId = 25482766)
print convert(varchar, #GL.Lat)
print convert(varchar, #GL.Long)
update Equipment set CurrentGeoLocation = geography::Point(#GL.Lat, #GL.Long, 4326)-- #NewGL where EquipmentId = 10518
But then when I apply this plan to the original script, I'm back to the same error.
The data in the test is working off the exact same records as in the original script. The original script is working off a collection of EquipmentIds, on the first one, I encounter this problem. The short test script uses the same EquipmentLocationId and EquipemntId that are the selected values used to update the first Equipment record in my collection.
Solved!
The error had nothing to do with the geography type as SQL reported. By pulling items in and out of the update statement in an effort to isolate why I still get the error even if I save everything but CurrentGeoLocation and then another update for the geography, I found that CurrentLocationDateTime (datetime, null) was the culprit. Deleted the column, added it back. Problem solved. Original script works as expected.
Don't know what happened to that datetime column that caused it to throw errors against a geometry type, but it's fixed.

SQL regex and field

I want to change the query to return multiply values in extra_fields, how can I change the regex? Also I don't understand what extra_fields is - is it a field? If so why it is not called with the table prefix like i.extra_fields?
SELECT i.*,
CASE WHEN i.modified = 0 THEN i.created ELSE i.modified END AS lastChanged,
c.name AS categoryname,
c.id AS categoryid,
c.alias AS categoryalias,
c.params AS categoryparams
FROM #__k2_items AS i
LEFT JOIN #__k2_categories AS c ON c.id = i.catid
WHERE i.published = 1
AND i.access IN(1,1)
AND i.trash = 0
AND c.published = 1
AND c.access IN(1,1)
AND c.trash = 0
AND (i.publish_up = '0000-00-00 00:00:00'
OR i.publish_up <= '2013-06-12 22:45:19'
)
AND (i.publish_down = '0000-00-00 00:00:00'
OR i.publish_down >= '2013-06-12 22:45:19'
)
AND extra_fields REGEXP BINARY '(.*{"id":"2","value":\["[^\"]*1[^\"]*","[^\"]*2[^\"]*","[^\"]*3[^\"]*"\]}.*)'
ORDER BY i.id DESC
The extra_fields is a column of the #__k2_items table. The table qualifier can be omitted, because it is not ambiguous in this query. The column is JSON encoded. That is a serialization format used to store information which is not searchable by design. Applying a RegExp may work one day, but fail another day, since there is no guarantee for id preceeding value (as in your example).
The right way
The right way to filter this is to ignore the extra_fields condition in the SQL query an evaluate in the resultset instead. Example:
$rows = $db->loadObjectList('id');
foreach ($rows as $id => $row) {
$extra_fields = json_decode($row->extra_fields);
if ($extra_fields->id != 2) {
unset($rows[$id]);
}
}
The short way
If you can't change the database layout (which is true for extensions you want to keep updateable), you must split the condition into two, because there is no guarantee for a certain order of the subfields. For some reason, one day value may occur before id. So change your query to
...
AND extra_fields LIKE '%"id":"2"%'
AND extra_fields REGEXP BINARY '"value":\[("[^\"]*[123][^\"]*",?)+\]'
Prepare an intermediate table to hold the contents of extra_fields. Each extra_fields field will be converted into a series of records. Then do a join.
Create a trigger and cronjob to keep the temp table in sync.
Another way is to write UDF in Perl that will decode the field, but AFAIK it is not indexable in mysql.
Using an external search engine is out of scope.
Ok, i didnt want to change the db strucure, i gost some help and changed the regex intoAND extra_fields REGEXP BINARY '(.*{"id":"2","value":\[("[^\"]*[123][^\"]*",?)+\]}.*)'
and i got the right resaults
Thanks

Comparing Date Values in Access - Data Type Mismatch in Criteria Expression

i'm having an issue comparing a date in an access database. basically i'm parsing out a date from a text field, then trying to compare that date to another to only pull newer/older records.
so far i have everything working, but when i try to add the expression to the where clause, it's acting like it's not a date value.
here's the full SQL:
SELECT
Switch(Isdate(TRIM(LEFT(bc_testingtickets.notes, Instr(bc_testingtickets.notes, ' ')))) = false, 'NOT ASSIGNED!!!') AS [Assigned Status],
TRIM(LEFT(bc_testingtickets.notes, Instr(bc_testingtickets.notes, ' '))) AS [Last Updated Date],
bc_testingtickets.notes AS [Work Diary],
bc_testingtickets.ticket_id,
clients.client_code,
bc_profilemain.SYSTEM,
list_picklists.TEXT,
list_picklists_1.TEXT,
list_picklists_2.TEXT,
list_picklists_3.TEXT,
bc_testingtickets.createdate,
bc_testingtickets.completedate,
Datevalue(TRIM(LEFT([bc_TestingTickets].[notes], Instr([bc_TestingTickets].[notes], ' ')))) AS datetest
FROM list_picklists AS list_picklists_3
RIGHT JOIN (list_picklists AS list_picklists_2
RIGHT JOIN (list_picklists AS list_picklists_1
RIGHT JOIN (bc_profilemain
RIGHT JOIN (((bc_testingtickets
LEFT JOIN clients
ON
bc_testingtickets.broker = clients.client_id)
LEFT JOIN list_picklists
ON
bc_testingtickets.status = list_picklists.id)
LEFT JOIN bc_profile2ticketmapping
ON bc_testingtickets.ticket_id =
bc_profile2ticketmapping.ticket_id)
ON bc_profilemain.id =
bc_profile2ticketmapping.profile_id)
ON list_picklists_1.id = bc_testingtickets.purpose)
ON list_picklists_2.id = bc_profilemain.destination)
ON list_picklists_3.id = bc_profilemain.security_type
WHERE ( ( ( list_picklists.TEXT ) <> 'Passed'
AND ( list_picklists.TEXT ) <> 'Failed'
AND ( list_picklists.TEXT ) <> 'Rejected' )
AND ( ( bc_testingtickets.ticket_id ) <> 4386 ) )
GROUP BY bc_testingtickets.notes,
bc_testingtickets.ticket_id,
clients.client_code,
bc_profilemain.SYSTEM,
list_picklists.TEXT,
list_picklists_1.TEXT,
list_picklists_2.TEXT,
list_picklists_3.TEXT,
bc_testingtickets.createdate,
bc_testingtickets.completedate,
DateValue(TRIM(LEFT([bc_TestingTickets].[notes], Instr([bc_TestingTickets].[notes], ' '))))
ORDER BY Datevalue(TRIM(LEFT([bc_TestingTickets].[notes], Instr([bc_TestingTickets].[notes], ' '))));
the value i'm trying to compare against a various date is this:
DateValue(Trim(Left([bc_TestingTickets].[notes],InStr([bc_TestingTickets].[notes],' '))))
if i add a section to the where clause like below, i get the Data Type Mismatch error:
WHERE DateValue(Trim(Left([bc_TestingTickets].[notes],InStr([bc_TestingTickets].[notes],' ')))) > #4/1/2012#
i've even tried using the DateValue function around the manual date i'm testing with but i still get the mismatch error:
WHERE DateValue(Trim(Left([bc_TestingTickets].[notes],InStr([bc_TestingTickets].[notes],' ')))) > DateValue("4/1/2012")
any tips on how i can compare a date in this method? i can't change any fields in the database, ect, that's why i'm parsing the date in SQL and trying to manipulate it so i can run reports against it.
i've tried googling but nothing specifically talks about parsing a date from text and converting it to a date object. i think it may be a bug or the way the date is being returned from the left/trim functions. you can see i've added a column to the end of the SELECT statement called DateTest and it's obvious access is treating it like a date (when the query is run, it asks to sort by oldest to newest/newest to oldest instead of A-Z or Z-A), unlike the second column in the select.
thanks in advance for any tips/clues on how i can query based on the date.
edit:
i just tried the following statements in my where clause and still getting a mismatch:
CDate(Trim(Left([bc_TestingTickets].[notes],InStr([bc_TestingTickets].[notes],' ')))) > #4/1/2012#
CDate(Trim(Left([bc_TestingTickets].[notes],InStr([bc_TestingTickets].[notes],' ')))) >
CDate("4/1/2012") CDate(DateValue(Trim(Left([bc_TestingTickets].[notes],InStr([bc_TestingTickets].[‌​notes],' '))))) > #4/1/2012#
i tried with all the various combinations i could think of regarding putting CDate inside of DateValue, outside, ect. the CDate function does look like what i should be using though. not sure why it's still throwing the error.
here's a link to a screenshot showing the results of the query http://ramonecung.com/access.jpg. there's two screenshots in one image.
You reported you get Data Type Mismatch error with this WHERE clause.
WHERE DateValue(Trim(Left([bc_TestingTickets].[notes],
InStr([bc_TestingTickets].[notes],' ')))) > #4/1/2012#
That makes me wonder whether [bc_TestingTickets].[notes] can ever be Null, either because the table design allows Null for that field, or Nulls are prohibited by the design but are present in the query's set of candidate rows as the result of a LEFT or RIGHT JOIN.
If Nulls are present, your situation may be similar to this simple query which also triggers the data type mismatch error:
SELECT DateValue(Trim(Left(Null,InStr(Null,' '))));
If that proves to be the cause of your problem, you will have to design around it somehow. I can't offer a suggestion about how you should do that. Trying to analyze your query scared me away. :-(
It seems like you are having a problem with the type conversion. In this case, I believe that you are looking for the CDate function.
A problem might be the order of the date parts. A test in the Immediate window shows this
?cdate(#4/1/2012#)
01.04.2012
?cdate(#2012/1/4#)
04.01.2012
Write the dates backwards in the format yyyy/MM/dd and thus avoiding inadverted swapping of days and months!
DateValue("2012/1/4")
and
CDate(#2012/1/4#)