Need to copy data to SQL based on conditions - sql

Question

I have a dataset that comes in an excel file that I need to copy to a SQL table. Before copying it I need to make sure of the following two conditions:
If the entire row is already in the SQL table then I do not need to copy that particular row.
If the value in column 'Step_API' is already in that column in the SQL table and column 'Source' is equal to 'PIDM' then I do not need to copy that row either.
I solved the first part with SELECT EXCEPT and it works. I do not know how to make the second part work. Please help me. Thanks
I tried the code below for the 1st part:
# write panda into a different table as a placeholder
col_options = dict(
dtype={
'Step_ID': sqlalchemy.types.INTEGER(),
'Step_Level': sqlalchemy.types.VARCHAR(length=50),
'Step_API': sqlalchemy.types.VARCHAR(length=50),
'Source_ID': sqlalchemy.types.VARCHAR(length=150),
'Source_Well_Name': sqlalchemy.VARCHAR(length=150),
'Start_Date': sqlalchemy.types.Date(),
'Stop_Date': sqlalchemy.types.Date(),
'Source': sqlalchemy.types.VARCHAR(length=150),
'Created_By': sqlalchemy.types.VARCHAR(length=50),
'Created_Dt': sqlalchemy.types.Date(),
'Updated_By': sqlalchemy.types.VARCHAR(length=50),
'Updated_Dt': sqlalchemy.types.Date(),
'ETL_Load_Date': sqlalchemy.types.Date(),
'Comment':sqlalchemy.types.VARCHAR(length=150)
}
)
df.to_sql(name="obo_external_xref_temp", con=engine, schema= 'MDM', if_exists='replace', index=False, **col_options)
# retrieve only new records comparing placeholder table and table where I intend to write
query = """
SELECT Step_ID, Step_Level, Step_API, Source_ID, Source_Well_Name, Start_Date, Stop_Date, Source, Created_By, Created_Dt, Updated_By, Updated_Dt, ETL_Load_Date, Comment FROM MDM.obo_external_xref_temp
EXCEPT
SELECT Step_ID, Step_Level, Step_API, Source_ID, Source_Well_Name, Start_Date, Stop_Date, Source, Created_By, Created_Dt, Updated_By, Updated_Dt, ETL_Load_Date, Comment FROM MDM.obo_external_xref;
"""
new_entries = pd.read_sql(query, con=engine)
# append only new records in the table
new_entries.to_sql(name="obo_external_xref", con=engine, schema= 'MDM', if_exists='append', index=False, **col_options)

Answer 1

I think this could work if you partition the two incoming data sets by the Source.
Use SELECT EXCEPT SELECT but with SELECT WHERE Source != 'PIDM' EXCEPT.
Then INSERT from the other partition Source = 'PIDM' but for Step_API values not in the existing dataset by using a subquery like this : INSERT INTO MDM.obo_external_xref SELECT Step_ID, Step_Level, Step_API, Source_ID, Source_Well_Name, Start_Date, Stop_Date, Source, Created_By, Created_Dt, Updated_By, Updated_Dt, ETL_Load_Date, Comment FROM MDM.obo_external_xref_temp WHERE Source = 'PIDM' AND Step_API NOT IN (SELECT Step_API FROM MDM.obo_external_xref WHERE Source = 'PIDM')