Ok, I've got a database table where data gets dumped by this horrid little program that I despise, but can't change at the moment. It has merchant data in there, names, addresses, and a set of categories that are pipe-delimited. What I need is a clean way to split these out, so I have one row for each merchant/category pair. From there, I can easily get it into the new data structure. This will need to be a repeatable process for a short period of time. I realize the optimal solution is to rid myself of this structure, but I've wracked my brain trying to figure out how to do this cleanly in sql.
I already have a function in the database that will split a delimited string and return a table.
This is in sql server 2008, btw.
Edit (for clarity_
Basically, the following might be a merchant (with the categories attached - other fields redacted for simplicity. Using commas for field delimiters here).
Jimbo's Bait Shoppe, Bait|Sports Gear|Sandwiches
What I need is:
Jimbo's Bait Shoppe, Bait
Jimbo's Bait Shoppe, Sports Gear
Jimbo's Bait Shoppe, Sandwiches
If you have already written a function that splits the string and returns the table you can use a trigger.
Create a trigger on INSERT on the table where the "horrid" program spits the data. The trigger will then take the unformatted data and populate two clean tables (I think in your case you should have two tables: one is a merchant, and another one is products, that are linked using one-to-many relationship using MerchantID).
In this case you can use the table with unformatted data as a "dirty" table. You can cleanse straight after the "horrid" program imported a file.
Please comment if you need help with the triggers
Related
I have an Azure SQL database, and my records inside table Spiderfood_RITMData in that database includes 13 different fields. Lots of stuff. I have confirmed in SQL-SMS that the records have data in each field.
There are way more items in the database than PowerApps can see using LOOKUP (1600-9000 records or more). However, I know FOR A FACT that there is only ONE record that has any given value in the NUMBER column. It's not a primary key, but it is unique in the table.
In PowerApps, I am trying to pull that field so that I can eventually parse out the individual items.
So, the commands I'm trying are:
ClearCollect(MLE_test1, Filter('Spiderfood_RITMData', "RITM2170467" in Number));
ClearCollect(MLE_test2, Search('Spiderfood_RITMData',"RITM2170467", "Number"));
However, the Collection results for MLE_test1 and MLE_test2 both are empty EXCEPT for the value of NUMBER. Say what?!
I'm trying to use the examples posted on https://learn.microsoft.com/en-us/powerapps/maker/canvas-apps/functions/function-filter-lookup but I am honestly getting baffled by this.
How should I be formatting this call such that I can pull the whole record?
Big picture explanation: I need to do a lot of data LOOKUPS into my table Spiderfood_RITMData table, but it has way more than 2000 rows, and PowerApps will not perform the Lookup correctly. So my presumably smart idea is to create a MUCH SMALLER "version" of Spiderfood_RITMData as a local collection, using a more delegateable function (such as FILTER or IN). If I filter by all records containing the values of NUMBER, then I go from, say a 10,000-record SQL table to a 10-record Collection. And I can do LOOKUPS against that collection for the rest of the function (uh, I think -- I'm still trying to experiment accordingly). Please let me know if this is crazy or not.
LookUp is just used to get one record, instead try this:
ClearCollect(MLE_test1, Filter('Spiderfood_RITMData', "RITM2170467" = Number));
This gets a collection with all the items where Number is = to "RITM2170467"
Collections are limited to only 2000 records in each collections.
I had same issue. Go to App settings. Under Upcoming Features make sure Explicit column selection is turned off. Hope this does it for you.
I work in a sales based environment and our data consists of 'leads'.
Let's say we record CompanyName, PhoneNumber, Address1 & PostCode(ZIP). These rows a seeded with a unique ID in the schema.
The leads come in from various sources and are compiled onto a spread sheet and then imported into SQL 2012 using SSIS.
After a validation check to see if a file exists we then use a simple data flow which consists of an Excel source, Derived Column, Data Conversion and finally an OLE DB Destination.
My requirement I'm sure has a relatively simple solution. I understand what I need to achieve is the first step. I need to take a sample of data from the last rolling two months, if 2 or more fields in the source excel file match the corresponding field in the destination sql table then I want to redirect to another table.
I am unsure of which combination of components I could use to achieve this. I believe that Fuzzy lookup may not be what I am looking for as I am looking to find exact field matches, I have looked at the lookup component but I am unsure if this is the way to go.
Could anyone please provide some advice on how I can best achieve this as simply as possible.
You can use the Lookup to check for matches in your existing table. However, it will be fairly complicated to implement the requirement of checking for any two or more fields matching. Your expression would be long and complex basically consisting of:
(using pseudo code for readability)
IIF((a=a AND b=b) OR (a=a AND c=c) OR (b=b AND c=c) OR ...and so on
for every combination of two columns you want to test
I would do this by importing the entire spreadsheet to a staging table, and doing the existing rows check in a SQL stored proc that moves the data to the desired destination table.
Fist, let me say I know very little about SQL language and am trying to learn (albeit very slowly). I have created a database table with columns for
ECOREGION_ID
ECOREGION_NAME
SPECIES_NAME
CLASS
so that there is one row for each species name in each ecoregion. My end goal is to create a form in which I can enter in multiple species names and search for the ecoregions they share. For example, if I enter into the 4 different search boxes "Tiger", "Red Panda", "Sloth Bear", and "Rhino" it would bring up a list of all the Ecoregions in which these four species share. I am wondering a few things:
Is my data set up in the correct way in order to do this or is there a more efficient way to set i t up?
What statement should I use to create an sql statement to perform the search I want?
What is the technical term for what I am wanting to do? I have tried many different searches on different forums and can't seem to find what I am looking for, mostly because I probably don't know what to search, lol.
Thanks,
-Drew
You have ECORegion_ID and ECORegion_Name in the same table. I would suggest create a separate table to hold ECORegions. This table would have both an ID and Name. The search table would then only have the ECORegion_ID. This process is called normalization. It basically reduces redundant data in your database.
You are looking for a SELECT statement, which is used to pull data out of one or more tables. The statement has a WHERE option to restrict which rows you bring back and an IN expression as part of the WHERE to allow you to look for multiple keywords.
Search for Normalization to see why to put region name in a separate table. Look up SQL Select to get syntax for the select statement you should get off to a good start
I have problems with my records within my database, so I have a template with about 260,000 records and for each record they have 3 identification columns to determine what time period the record is from and location: one for year, one for month, and one for region. Then the information for identifying the specific item is TagName, and Description. The Problem I am having is when someone entered data into this database they entered different description for the same device, I know this because the tag name is the same. Can I write code that will go through the data base find the items with the same tag name and use one of the descriptions to replace the ones that are different to have a more uniform database. Also some devices do not have tag names so we would want to avoid the "" Case.
Also moving forward into the future I have added more columns to the database to allow for more information to be retrieved, is there a way that I can back fill the data to older records once I know that they have the same tag name and Description once the database is cleaned up? Thanks in advance for the information it is much appreciated.
I assume that this will have to be done with VBA of some sort to modify records by looking for the first record with that description and using a variable to assign that description to all the other items with the same tag name? I just am not sure of the correct VBA syntax to go about this. I assume a similar method would be used for the backfilling process?
Your question is rather broad and multifaceted, so I'll answer key parts in steps:
The Problem I am having is when someone entered data into this
database they entered different description for the same device, I
know this because the tag name is the same.
While you could fix up those inconsistencies easily enough with a bit of SQL code, it would be better to avoid those inconsistencies being possible in the first place:
Create a new table, let's call it 'Tags', with TagName and TagDescription fields, and with TagName set as the primary key. Ensure both fields have their Required setting to True and Allow Zero Length to False.
Populate this new table with all possible tags - you can do this with a one-off 'append query' in Access jargon (INSERT INTO statement in SQL).
Delete the tag description column from the main table.
Go into the Relationships view and add a one-to-many relation between the two tables, linking the TagName field in the main table to the TagName field in the Tags table.
As required, create a query that aggregates data from the two tables.
Also some devices do not have tag names so we would want to avoid the
"" Case.
In Access, the concept of an empty string ("") is different from the concept of a true blank or 'null'. As such, it would be a good idea to replace all empty strings (if there are any) with nulls -
UPDATE MyTable SET TagName = Null WHERE TagName = '';
You can then set the TagName field's Allow Zero Length property to False in the table designer.
Also moving forward into the future I have added more columns to the
database to allow for more information to be retrieved
Think less in terms of more columns than more tables.
I assume that this will have to be done with VBA of some sort to modify records
Either VBA, SQL, or the Access query designers (which create SQL code behind the scenes). In terms of being able to crunch through data the quickest, SQL is best, though pure VBA (and in particular, using the DAO object library) can be easier to understand and follow.
Actually I have a new client & their Database has no standard naming conventions & the application is in classic asp.I have a form in which a form there are many values in the different textboxes, it it very difficult to trace the value come from which table.& also there is no erd.
I need a query from which I can get the table name with column name by giving Value.
Let's suppose I have a value having label name abc#= '6599912268'
& the new project has no ERD no standard of naming conventions... I need a fast way to know the abc# ='6599912268' is taking from which table & which column name.... like this the UI has many values which is time taken to trace manually
Is there any way to trace it?
The simple answer is no. There is no way to trace table/column it comes from by mere inspection of the value.
I suggest the following.
Find out what type of db your product is using. Where it is situatede, do you have access to it.
If you have access to the database, get to know the db structure. What each table is meant to store, the relationships etc. Speak to the db administrator or the business analayst to increase your knowledge on the product domain.
Once you have the db structure, try and compare the table to the page. Eg. The user details will most like be stored in a db table named 'Users' or 'Membership' Catch my drift?
Then have a look at the web sites source code. Look at the specific page you are at. Is the sql code embedded in the source code (asp page) or does it call a COM server or something similar? If you are "lucky" (and I say lucky for on the purpose of your problem that you are having) you fill find the sql code in the asp page.
If it calls a COM object or something similar, then you will have to dig up the source code for that, and that is most likely where you sql will reside.
There is no easy way to do this, you have to use a stored procedure to loop over all the tables in the database and search for the value, and it will probably take a while.
There's a stored procedure and examples here: Search all columns in all the tables in a database for a specific value. You'll see there are stored procedures for finding dates, strings, numbers.
Not possible, and If you search the column with the value, there is a possible chance that you get multiple columns with the same value, so how would you differentiate them and the same case is for the table.