Does there exist a MSBuild property that allows for in-place column type update? My team would like our MSBuild to work that way rather than dropping the table and recreating the updated version.
We're ok with data loss.
--Sarah
Trying /p:BlockOnPossibleDataLoss=False as per reading about this here:
https://dba.stackexchange.com/questions/139478/how-to-prevent-ssdt-publishing-from-dropping-columns
Related
I'm creating a lot of stuff based on the manifest.json that dbt generates for me. But for whatever reason the "data_type" property for each column is always None in the manifest.json, even though I can see it in the catalog.json, I believe the data type is generated from the database.
How do I get the data_type attribute populated in my manifest.json file ?
Some helpful answers from this dbt Slack thread:
first reply (h/t Daniel Luftspring)
not sure if this is the only way but i'm running dbt version 0.20.1 you can specify the data_type as a column property in your schema.yml and it will show up in the manifest like so:
columns:
- name: city
data_type: string
If you have a big project and wanted to automate this you could probably pull together a script to edit your schema files in place and sync the data types with your db using the information schema
second reply (h/t Jonathon Talmi)
FYI catalog.json has data type becaause it queries the metadata data
tables in your dwh (e.g. info schema in snowflake) to contruct the
catalog, but your traditional dbt compile/run/etc which. generates a
manifest does not have such queries
I am using select statement in excel source to select just specific columns data from excel for import.
But I am wondering, is it possible to select data such way when I select for example column with name: Column_1, but if this column is not exists in excel then it will try to select column with name Column_2? Currently if Column_1 is missing, then data flow task fails.
Use a Script task and write .net code to read the excel file and then perform the check for the Column_1 availability in the file. If the column does not present then use Column_2 as input. Script Task in SSIS can act as a source.
SSIS is metadata based and will not support dynamic metadata, however you can use Script Component as #nitin-raj suggested to handle all known source columns. There is a good post below on how it can be done.
Dynamic File Connections
If you have many such files that can have varying columns then it is better to create a custom component.However, you cannot have dynamic metadata even with custom component, the set of columns should be known upfront to SSIS.
If the list of columns keep changing and you cannot know in advance what are expected columns then you are better off handling the entire thing in C#/VB.Net using Script Task of control flow
As a best practice, because SSIS meta data is static, any data quality and formatting issues in source files should be corrected before ssis data flow task runs.
I have seen this situation before and there is a very simple fix. In the beginning of your ssis package, using a file task to create copy of the source excel file and then run a c# script or execute a powershell to rename the columns so that if column 1 does not exist, it is either added at the appropriate spot in excel file or in case the column name is wrong is it corrected.
As a result of this, you will not need to refresh your ssis meta data every time it fails. This is a standard data standardization practice.
The easiest way is to add two data flow tasks, one data flow for each Excel source select statement and use precedence constraints to execute the second data flow when the first one fails.
The disadvantage of this approach is that if the first data flow task fails for another reason, it will also try to execute the second one. You will need some advanced error handling to check if the error is thrown due to missing columns or not.
But if have a similar situation, I will use a Script Task to check if the column exists and build the SQL command dynamically. Note that this SQL command must always return the same metadata (you must use aliases).
Helpful links
Overview of SSIS Precedence Constraints
Working with Precedence Constraints in SQL Server Integration Services
Precedence Constraints
I have a problem when building .wix MST difference file. I get the following error:
"The table definition of target database does not match the table definition updated database. A transform requires that the target database schema match the update database schema".
I tried finding solution on internet for almost 2 hours but no luck. I know It is probably caused by difference in MSI tables schema but I have no idea how to fix that.
I am attempting to completely rewrite footer.liquid. Unfortunately, there are pre-existing fields which I am no longer using which are causing errors to be thrown when I attempt to save my file. For example:
Error: New schema is incompatible with the current setting value. Invalid type value for block 'linklist'. Type must be defined in schema.
I understand that this is because I am not using an "id": "linklist" in my new schema. Is there a way to flush out these pre-existing schema types so that I don't have to include it in my schema?
If you delete the section in the theme editor and then readd the section after, that should solve the problem.
Alternatively, you could find the settings_data.json and remove the current values associated with your footer section.
I've exported records to a flat file delimited by "|" and it seems that when I import those records into a new database , SQL Server treats the NULL values as empty fields. IMy queries worked properly when the records/fields were NULL and so I want to either find a way to retain the NULL values in the data or convert the blank fields to NULL values. I'm assuming the former would be easier, but I don't know how to do that. Any help would be appreciated.
I just had the same problem. I resolved it by Changing the RetainNulls property in the properties of the Flat File Source in the Data Flow Task.
In your destination connection in the dataflow, there is a property that you can chceck that says Keep nulls, JUst check that. Why that isn't the default I'll never know.
Hmmm something stange going on there. I can suggest that you then clean the data and change it to null, you can either do this as part of the dataflow or do two dataflows, one which inserts the data into a staging table, then run an exectue SQl task to do the clean up and then create a dataflow to run fromthe staging table to the real table.
in case anyone is looking how to do this when building the package programatically you need to set the variable in your CManagedComponentWrapper object
CManagedComponentWrapper instanceSource = ComponentSource
...
instanceSource.SetComponentProperty("RetainNulls", true);