I don't know the best way to describe my problem and I'm just looking for a push in the right direction, or where to start. I'd be perfectly happy with an answer that's a very useful link or pseudo code.
My problem, I have a database that's about to hit the MS Access hard coded 2 GB database limit and I don't want to split the database.
What I think is a possible solution - make the database more efficient in it's data storage. I think, but don't know if this is true, that I could do this by turning some string fields into numeric fields. Stay with me...
For instance:
My database has several million records of a field we'll call TooLongString
Each value is about 50 characters
Every record has a value for this field
There's only 9 possible values for TooLongString
Would it decrease my database size to instead store a number that
represents one of the 9 possible values and store the text value in a small table? (So go from 50 characters to 1 character several million times)
Did I explain my issue correctly? Is my potential solution actually a solution? How would I go about doing this?
Thanks!
The short answer is yes, that would reduce the size of your database. You could have a second table that holds the nine possible values for "TooLongString" and just store the ID of the appropriate answer in the main table, as you suggested. You would then need to join these tables when pulling the data out in order to retrieve the actual text instead of the ID.
I would set up your new table first, then add a new column for the ID into your existing one. As there are only nine possible values, I'd be tempted to just manually run an UPDATE query nine times, e.g. if the first string in your new table is "MyFirstString" with ID 1, you could run "UPDATE existingTableName SET newColumn = 1 WHERE oldColumn = 'MyFirstString'". Do this for each of the nine values then you can remove the old string column from your table at the end.
Related
I have an Azure SQL database, and my records inside table Spiderfood_RITMData in that database includes 13 different fields. Lots of stuff. I have confirmed in SQL-SMS that the records have data in each field.
There are way more items in the database than PowerApps can see using LOOKUP (1600-9000 records or more). However, I know FOR A FACT that there is only ONE record that has any given value in the NUMBER column. It's not a primary key, but it is unique in the table.
In PowerApps, I am trying to pull that field so that I can eventually parse out the individual items.
So, the commands I'm trying are:
ClearCollect(MLE_test1, Filter('Spiderfood_RITMData', "RITM2170467" in Number));
ClearCollect(MLE_test2, Search('Spiderfood_RITMData',"RITM2170467", "Number"));
However, the Collection results for MLE_test1 and MLE_test2 both are empty EXCEPT for the value of NUMBER. Say what?!
I'm trying to use the examples posted on https://learn.microsoft.com/en-us/powerapps/maker/canvas-apps/functions/function-filter-lookup but I am honestly getting baffled by this.
How should I be formatting this call such that I can pull the whole record?
Big picture explanation: I need to do a lot of data LOOKUPS into my table Spiderfood_RITMData table, but it has way more than 2000 rows, and PowerApps will not perform the Lookup correctly. So my presumably smart idea is to create a MUCH SMALLER "version" of Spiderfood_RITMData as a local collection, using a more delegateable function (such as FILTER or IN). If I filter by all records containing the values of NUMBER, then I go from, say a 10,000-record SQL table to a 10-record Collection. And I can do LOOKUPS against that collection for the rest of the function (uh, I think -- I'm still trying to experiment accordingly). Please let me know if this is crazy or not.
LookUp is just used to get one record, instead try this:
ClearCollect(MLE_test1, Filter('Spiderfood_RITMData', "RITM2170467" = Number));
This gets a collection with all the items where Number is = to "RITM2170467"
Collections are limited to only 2000 records in each collections.
I had same issue. Go to App settings. Under Upcoming Features make sure Explicit column selection is turned off. Hope this does it for you.
I need some help and I know I am not the only one to deal with this issue but I am wondering if you might have some ideas on how to handle the situation of comparing two rows of data filling out start and end dates.
To give you some context, we have a huge hierarchy (approx 8,000 rows and about 12 columns wide) that is updated each year. Sometimes the values change and sometimes they don’t. When the values don’t change, then I don’t need to adjust the dates. When the values do change and a new row is added, I need to change the data.
I have attached some fake data to try and illustrate my data. I am building this in MS Access, so I think this is more of a DBA type question that is going to be manipulated via a recordset type method.
In my example I have two tables – Old Table and New Table. In each table there is a routing code field that represents my join field and primary key for this table.
The Old table represents existing data - tblMain. The New Table represents the data to be appended - tblTemp.
To append the data, I have an append query set up in Access. I perform a left join between the Old and New tables, joining on every field and append the rows that are null in the Old table. That’s fine and that is not where my issue is.
What is causing me issue is how to fill out the start and end dates.
So as you can see from my tables, we are running a zoo. Let’s just say for the sake of the argument, our zoo started off pretty simple and has become more sophisticated. We now want our hierarchy to expand out and become a bit more detailed as we are now capturing the type of animal (Level 4) and the native location (Level 5).
As you can see when comparing one table to another the routing codes are the same, so the append query has to have a join on each field. When you do this, you return the Result Table which is essentially the Old and New tables stacked on top of each other. You might think about a Union query but this is going to give me duplicates and I don’t want that.
If you notice in the Result Table there is a Start and End Date. Let’s just say I get the start and end dates via message box that pops up upon the import of the data and is held in a variable. I think there are dates in my real data but still trying to verify this.
So how do I compare (pseudo code for the logic needed)?
• For each routing code:
Compare Levels 1-5
If the routing code is the same but Levels 1 -5 are not the same
fill out the end date of the old record
fill out the start date of the new record
This idea of comparing two records and filling out a data is quite prevalent in my organization but I haven’t found a way of creating the logic that consistently works so any help or suggestions would be appreciated.
Old Table
New Table
Result Table
Recently I have been using Microsoft SQL for creating databases that are referred to using an excel document. There have been a number of instances when I needed to make a small changes to my tables and ended up "DROP"-ing all my current tables and re-creating them using an updated query. I understand you can use UPDATE to change the values of records within a table, but I'm looking to manipulate a data type so that I can change the number of decimals in one record of my tables from 2 to 3. Code for creating the table looks something like this:
CREATE TABLE WIRE_INDEX
--"Field" "Data Type" "Null or Not"
(...
...
DENSITY decimal(18,2) Not Null);
I don't know if the solution is something obvious, but I have been unable to find anything useful. I'm not sure know how to refer to the data type of a field in SQL.
When I populate the database I use numbers like 0.283 and 0.164, but when I SELECT the record I only get the first two decimals. I'd like the first 3 decimals to appear in the way I enter them into the table.
(edit didn't show up properly)
(not sure if I'm supposed to post my solution), but credit to TEEKAY and Apurav for answering my question. I used the code posted by Apurav which looks like this:
ALTER TABLE WIRE_INDEX
ALTER COLUMN DENSITY decimal(18,3) Not Null
When I pulled the table, using a SELECT statement the precision showed three decimal places, but I lost the precision of my input and had to re-enter my values using UPDATE. Not sure if this is more effective than just starting over, but it worked for me and now I know.
I have a table I imported from excel. Everything was imported as strings to get started and gradually the data types were fixed.
I am now going through the process of normalisation and to start I done a select distinct on a given row to extract out the columns that were repeated to another table with an id.
Now I am trying to replace every occurrence of the string in the original table, with the id of the corresponding string in the other table.
I can do it one by one with something like this no prob...
UPDATE myTable
SET myCol = REPLACE(myCol, 'String', 'Num')
and later convert the myCol to an int.
But having to run this, and various variations on it for every string in the system is very error prone.
For example, if I accidently use the same number twice, replacing every string...I cant tell anymore which recordset originally belonged to the first update, and which were the new ones. This is a problem when the dataset is as large as mine.
Is there some way I can combine this with a join on the tables and have the system automate the process for me. I can imagine with normalisation being such a big thing for the last 30 or more years, this issue must have been met and resolved by now.
Any help would be appreciated.
I am exporting a file from a system as .csv. My aim is to link to this file as a table (which matches the output field for field) and then run the queries and export.
The problem I am having is that, upon import, all the fields are 255 bytes wide rather than what they need to be.
Here's what I've tried so far:
I've looked at ALTER TABLE but I cannot run multiple ALTER TABLE statements in one macro.
I've also tried appending the table into another table with the correct structure but it seems to overwrite the structure.
I've also tried using the Left function with the appropriate field length, but when I try to export, I pretty much just see 5 bytes per column.
What I would like is a suggestion as to what is the best path to take given my situation. I am not able to amend the initial .csv export, and I would like to avoid VBA if possible, as I am not at all familiar with it.
You don't really need to worry about the size of Text fields in an Access linked table that is connected to a CSV file. Access simply assigns each Text field the largest possible maximum size: 255. It does not mean that every value is actually 255 characters long, it just means that any values in those fields can be at most 255 characters long.
Even if you could change the structure of the linked table (which you can't), it wouldn't make any real difference except to possibly truncate longer Text values, and you could easily do that with a String function. For example, if a particular field had to be restricted to 15 characters then you could simply use Left([fieldName], 15) as a query column or as the control source in a report.
In the end, as the data set is not that large, I have set this up to append from my source data into a table with the correct structure. I can now run my processes against this table as per normal.