I have an import which imports data into a staging table. Once the data is inserted in to staging it is moved to either Main table or the error table depending on the data. For example Name field is a text fields with length 25. If the record is having name more than 25 characters the data is moved to error table.
Currently the VBA logic tries to move the data from Staging to Main table and if it fails (coz of primary key violation or missing data for a mndatory field) the record is moved to error table.
Problem is for data which exceed the maximum length of the field. I was hoping that when data is moved from staging to Main table and data exceeds the text field length the insert SQl will fail. Instead access truncates the data and inserts it in to main table.
Is there something which stops access from auto truncating data when an insert is attempted?
Regards
Dev
Related
To preface -- I am as green as at gets.
I am tasked with building an app for internal org use. We have a DB with patient data, and in interface with a hospital electronic medical records system. patient data entered into the EMR is sent to us via interface to update the patient profile in our database. Partial matches require manual intervention.
Message is received in a table within a staging schema
attempts to match to existing patient
if there are only 'partial matches' a status is set to 'mismatch'
I want to:
Display a summary of all 'mismatch' status records. I want to use an interactive grid to select individual records.
Set ROWID of interactive grid rows to the respective primary key of the record in the staging table.
Use the selected Interactive Grid ROWID (user selects a specific record in the interactive grid) to select the matching primary key of the corresponding record in staging table to create SQL query to find potential matches within our DB. Potential matches are displayed in a second table.
Questions:
How do I set the rowID of an Interactive grid to the unique key column of the staging table?
--Some research shows I need a hidden item, is this correct?
How do I access a rowID that has been selected in the Interactive grid to use in a SQL query?
My humble thanks and appreciation
So, your question is a bit confusing, but as far as I understand it. You are getting some data from table A, trying to match it with table B. If it matches, it irrelevant for us. If a match is not found you want to show it so that it can be manually matched.
In apex in order to update a table, you need to select what is the primary key by which it will update the data. That is usually a column in the table, but it can also be rowid(just include it in the SQL like any other column).
What I would suggest for you from what I understand of your situation.
Display the mismatched rows in an interactive grid, with rowid as primary key. Then you will need to have a column by which you match, if these entries already have some sort of key by which you tried to match but failed, display that. And have that column be a PopupLOV so the user can edit what value is in that field and set it to the appropriate match. One thing you will need to be careful about. You are editing a Unique key, or perhaps even Primary key, you might get conflicts here. Even if you only display unmatched data in the LOV, you can still have a user editing multiple rows and trying to match two rows to the same value, that will fail with an error that isnt particularly user friendly.
We have many tables with lots of fields that have data validation on them. When running an SQL insert statement to append imported data to these tables in the event of a data validation error they fail with the extremely helpful error message "27 records could not be added do to validation rule violations".
Tracking down these violations is very time consuming, so I am wondering if there is a way to do ANY of the following:
1) Find out the record and field(s) that caused the validation error programmatically
2) Override data validation during import so that people can afterwards look through the table to find the issues highlighted
3) Disable and then re enable validation for the same effect (short of looping through all columns, disabling data validation while saving the settings, then after the insert loop through again and re enable it for each column).
4) Some other suggestion I haven't thought of because it is past 10:00 pm here.
Any help would be very much appreciated.
No, Access does not allow validation rules to be disabled or overridden.
A separate import table is the best alternative that would facilitate possibilities #1 and #2.
Copy the destination table structure to a new table. You can do this with Copy (Ctrl+C) and Paste (Ctrl+V) within the navigation panel.
Alter the copied table definition by removing all validation rules and constraints.
Pay special attention to the primary key for the destination table.
If the primary key is an AutoNumber column, then go ahead and keep this column in the import table as the primary key. Just be aware that this value will be an arbitrary, temporary number used only for import row identification. Eventually the destination table will have a new unique value assigned.
If the primary key is composed of meaningful data columns, I suggest removing the primary key and unique constraint. Instead, add an additional AutoNumber primary key column just for importing.
If there is no primary key, you probably want to consider adding one and reviewing proper table normalization. As before, at least add an AutoNumber primary key as already described.
Define an import process. Some or all of this could be automated via code / macros.
Clear import table of existing data.
Import external data into the import table.
Copy rows from the import table to the permanent table.
The easiest method is probably an SQL INSERT INTO statement. Within the Access Query designer, an Append Query is the same thing as the INSERT INTO statement.
Alternative methods are to use VBA to loop through rows via RecordSet objects.
Run one or more comparison queries to detect and report rows which failed the copy operation.
This assumes that the data has a unique field or fields that can be used to distinguish rows from each other, both for the imported data as well as existing data in permanent table.
If the data does NOT have identifying fields, the only viable option is probably to loop through a RecordSet object in VBA code since a RecordSets can be used to select and operate on individual rows. But SQL statements operate on entire sets of data and rely on unique values to distinguish between rows of data.
Aside from problems verifying imported data, one should probably question the general usefulness and/or validity of the data rows if they have no individual identity.
For any import row which fails the copy operation, do one or more of the following, recursively improving the process and building a set of validations (queries and/or code) that you can automate.
Manually inspect data and try to determine failure reasons.
Write a validation query or set of validation queries which identify import rows that do not satisfy validation rules or constraints of the permanent table.
Possibly write additional queries which "massage" data into the permanent table, or which report on bad data to be fixed manually or researched further.
I have 2 tables that are linked to 2 different Excel spread sheet. My primary key for table A is stored as a text in Excel therefore the data type in Access is Short Text. The foreign key in Table B is also stored as a text in Excel and shows up as a Short Text data type in Access.
When I run a select query from these table I get "Type mismatch expression" error. If I go back and look at Table A my primary key field has now changed to a Number data type. Therefore my tables will not properly relate or query together. How can I get my linked table to stop changing the data type when I run a query?
How do I implement, or what steps do I use to create a transformation that compares a table and a list . For example a database table name Schools and an excel file with a huge list of names of Schools.
if the entry in the excel is not seen in the database, it should then be added to the database table.
I'm not quite sure if I can use the database lookup step, it does not tell if a lookup fails. insert update step doesn't seem a solution as well, for it requires some ID value but no ID is present on the list of schools in the excel file
Based on the information that you provided a simple join with table insert step will do your task. You can use the Merge rows step for comparing both the data stream (excel and database). The merge rows step uses the key to compare two streams and add a flag field which marks the row as new, identical, changed, deleted. In your case you would like to insert all the rows that are marked as new by using table insert step.
Please check the below links for more reference.
Merge rows, Synchronize after merge
This what worked for me,
excel file -->
select values (to delete unnecessary fields) -->
database lookup (this will create a new field, and will set null if not found) -->
filter rows (get the fields with null output from lookup) -->
table output (insert the filtered records)
Currently, developing an app that needs to synchronize its data between a server. The app heavily uses SQLite, so we moved to raw sqlite and decided to use FMDatabase. Our tables are denormalized and without keys because we do not manage data state, just gathering the data and sending it. We are updating the tables at some time interval (getting from a server and inserting new content) but sending data to the server is more frequent case.
Data from a server comes as full records with id values (char type GUID value), so we have made PK keys for the tables to consist of those id columns. The thing is, when getting data from a server, I need just to insert new records and currently, I'm making inserts with those id values. Some records get inserted (new) and some fail due to unique PK constraint on id columns. For me, it's perfectly fine, just annoys messages in console (unique key violated) and not sure is it affecting performance and etc. I could loop through records before inserting new, to find existing ones and to insert only really new records but I think it's a waste.
Also, I could make explicit internal int id (faster for PK index) but our tables are not related with FK, so it would be redundant. Any thoughts on that?
The INSERT statement comes with optional ON CONFLICT clause, so if you want to suppress the warnings, simply use INSERT OR IGNORE and the duplicated PKs will be ignored.