MS Access: Code order gets mixed up after sorting to another table - ms-access-2007
I've been struggling with this for quite some hours now.
I've got a query, that exports certain fields from a table to another table:
SELECT ALLQuotes.Ddate, ALLQuotes.Ttime, ALLQuotes.Price INTO 27012016_29012016
FROM ALLQuotes
ORDER BY ALLQuotes.Code;
The problem is, for some rows it mixes up and codes do not come ordered, like they are in the initial table. Here's a screenshot:
image1
And here how it looks in the original table:
image2
Could this be perhaps caused by the large ammount of raws that i have? (around 4.2 million)?
And how can I export sorted data so that I don't counter this problem?
Thanks!
Related
What is the best approach for bulk cleaning a database table that has a large amount of duplicated data loaded every day (snowflake db)
Thanks in advance for reading this, I hope I explain my problem. In one of our domains, we have a pipeline (Multiple) where data flows from S3 into a snowflake staging table using airflow. The data itself originates from a number of different applications but the process is always the same. The data is extracted from the application by the support teams (multiple support teams across multiple countries, using different technologies), then into AWS S3 and then bulk loaded into snowflake. Due to limitations on the data from source their often isn't any filter on the data itself and effectively the staging table is loaded with the raw CSV every single day, a file date column is added to the data itself. The result is that we have tables that have been loaded with the same data every single day since 2009. However the data does change, from day to day a column value will change and so the file date is very useful in tracking changed attributes and something that I want to exploit. Further if the data was cleansed we would need approximately 1% of the data. These tables are huge some contain around 16 trillion rows but can we be quite narrow. I would like to optimally loop through each days worth of data and then only load into the staging tables new data as apposed to just loading everything each day. I have tried the following A query that windows over the entire set and compares the hashed value of each row (minus the file date) and then only returns if it did not appear in the previous dates data set. This works but not for the larger tables as the warehouse starts to write to disk and then it takes hours. A day by day loop that looks at each file date data set and compares to the previous day and only loads the difference, this takes to long on the initial clean of the tables but is what I am doing once the data has been cleaned and will form the initial load procedure. The current solution is where I dynamically create multiple minus set statement where I look at each day minus the day before then batch these into blocks of 10-20 based of the average daily row size so as an example INSERT INTO TEMP TABLE (Select * FROM TABLE A WHERE FILE_DATE = 040123 MINUS Select * FROM TABLE A WHERE FILE_DATE = 030123) UNION ALL (Select * FROM TABLE A WHERE FILE_DATE = 030123 MINUS Select * FROM TABLE A WHERE FILE_DATE = 020123) etc... This is not pretty though does work however its taking me around 12 hours to process 70 odd tables. I would like advice on if their is another approach. Please bear in mind that I am limited to using snowflake due to resourcing issues and politics. Any guidance and ideas would be much appreciated. Regards
How to populate all possible combination of values in columns, using Spark/normal SQL
I have a scenario, where my original dataset looks like below Data: Country,Commodity,Year,Type,Amount US,Vegetable,2010,Harvested,2.44 US,Vegetable,2010,Yield,15.8 US,Vegetable,2010,Production,6.48 US,Vegetable,2011,Harvested,6 US,Vegetable,2011,Yield,18 US,Vegetable,2011,Production,3 Argentina,Vegetable,2010,Harvested,15.2 Argentina,Vegetable,2010,Yield,40.5 Argentina,Vegetable,2010,Production,2.66 Argentina,Vegetable,2011,Harvested,15.2 Argentina,Vegetable,2011,Yield,40.5 Argentina,Vegetable,2011,Production,2.66 Bhutan,Vegetable,2010,Harvested,7 Bhutan,Vegetable,2010,Yield,35 Bhutan,Vegetable,2010,Production,5 Bhutan,Vegetable,2011,Harvested,2 Bhutan,Vegetable,2011,Yield,6 Bhutan,Vegetable,2011,Production,3 Image of the above csv: Now there is a very small country lookup table which has all possible countries the source data can come with, listed. PFB: I want to have the output data's number of columns always fixed (this is to ensure the reporting/visualization tool doesn't get dynamic number columns with every day's new source data ingestions depending on the varying distinct number of countries present). So, I've to somehow join the source data with the country_lookup csv and populate all those columns with default value as F. Every country column would be binary with T or F being the possible values. The original dataset from the above has to be converted into below: Data (I've kept the Amount field unsolved for column Type having Derived Yield as is, rather than calculating them below for a better understanding and for you to match with the formulae): Country,Commodity,Year,Type,Amount,US,Argentina,Bhutan,India,Nepal,Bangladesh US,Vegetable,2010,Harvested,2.44,T,F,F,F,F,F US,Vegetable,2010,Yield,15.8,T,F,F,F,F,F US,Vegetable,2010,Production,6.48,T,F,F,F,F,F US,Vegetable,2010,Derived Yield,(2.44+15.2)/(6.48+2.66),T,T,F,F,F,F US,Vegetable,2010,Derived Yield,(2.44+7)/(6.48+5),T,F,T,F,F,F US,Vegetable,2010,Derived Yield,(2.44+15.2+7)/(6.48+2.66+5),T,T,T,F,F,F US,Vegetable,2011,Harvested,6,T,F,F,F,F,F US,Vegetable,2011,Yield,18,T,F,F,F,F,F US,Vegetable,2011,Production,3,T,F,F,F,F,F US,Vegetable,2011,Derived Yield,(6+10)/(3+9),T,T,F,F,F,F US,Vegetable,2011,Derived Yield,(6+2)/(3+3),T,F,T,F,F,F US,Vegetable,2011,Derived Yield,(6+10+2)/(3+9+3),T,T,T,F,F,F Argentina,Vegetable,2010,Harvested,15.2,F,T,F,F,F,F Argentina,Vegetable,2010,Yield,40.5,F,T,F,F,F,F Argentina,Vegetable,2010,Production,2.66,F,T,F,F,F,F Argentina,Vegetable,2010,Derived Yield,(2.44+15.2)/(6.48+2.66),T,T,F,F,F,F Argentina,Vegetable,2010,Derived Yield,(15.2+7)/(2.66+5),F,T,T,F,F,F Argentina,Vegetable,2010,Derived Yield,(2.44+15.2+7)/(6.48+2.66+5),T,T,T,F,F,F Argentina,Vegetable,2011,Harvested,10,F,T,F,F,F,F Argentina,Vegetable,2011,Yield,90,F,T,F,F,F,F Argentina,Vegetable,2011,Production,9,F,T,F,F,F,F Argentina,Vegetable,2011,Derived Yield,(6+10)/(3+9),T,T,F,F,F,F Argentina,Vegetable,2011,Derived Yield,(10+2)/(9+3),F,T,T,F,F,F Argentina,Vegetable,2011,Derived Yield,(6+10+2)/(3+9+3),T,T,T,F,F,F Bhutan,Vegetable,2010,Harvested,7,F,F,T,F,F,F Bhutan,Vegetable,2010,Yield,35,F,F,T,F,F,F Bhutan,Vegetable,2010,Production,5,F,F,T,F,F,F Bhutan,Vegetable,2010,Derived Yield,(2.44+7)/(6.48+5),T,F,T,F,F,F Bhutan,Vegetable,2010,Derived Yield,(15.2+7)/(2.66+5),F,T,T,F,F,F Bhutan,Vegetable,2010,Derived Yield,(2.44+15.2+7)/(6.48+2.66+5),T,T,T,F,F,F Bhutan,Vegetable,2011,Harvested,2,F,F,T,F,F,F Bhutan,Vegetable,2011,Yield,6,F,F,T,F,F,F Bhutan,Vegetable,2011,Production,3,F,F,T,F,F,F Bhutan,Vegetable,2011,Derived Yield,(2.44+7)/(6.48+5),T,F,T,F,F,F Bhutan,Vegetable,2011,Derived Yield,(10+2)/(9+3),F,T,T,F,F,F Bhutan,Vegetable,2011,Derived Yield,(6+10+2)/(3+9+3),T,T,T,F,F,F The image of the above expected output data for a structured look at it: Part 1 - Part 2 - Formulae for populating Amount Field for Derived Type: Derived Amount = Sum of Harvested of all countries with T (True) grouped by Year and Commodity columns divided by Sum of Production of all countries with T (True)grouped by Year and Commodity columns. So, the target is to have a combination of all the countries from source and calculate the sum of respective Harvested and Production values which then has to be divided. The commodity can be more than one in the actual scenario for any given country, but that should not bother as the summation of amount happens on grouped commodity and year. Note: The users in the frontend can select any combination of countries. The sole purpose of doing it in the backend rather than dynamically doing it in the frontend is because AWS QuickSight (our visualisation tool), even though can populate sum on selected column filters but doesn't yet support calculation on those derived summed fields. Hence, the entire calculation of all combination of countries has to be pre-populated (very naive approach) in order to make it available in report on dynamic users selection of countries. Also if you've any better approach (than the above naive approach mentioned in note) to solve this problem, you are most welcome to guide me. I've also posted a question on the same problem without writing my expected approach for experts to show me the path on how we can solve this kind of a problem better than this naive approach. If you want to help solve it with some other technique, you're most welcome, here is the link to that question. Any help shall be greatly acknowledged.
SQL Server Query: Daily Data Snapshot Comparison (Counting Delta Occurrences)
I am working towards counting customer subscription ("package") changes. To do this, I am selecting all data from my package table once, every day. I am calling the daily query results "snapshots" (approx 500k rows). I then load the snapshot data into a new table. After 10 days I have a total of 5 million rows in the snapshots table (500k rows * 10 days). The majority of customers do not changes packages (65%). I need to report which customers, of the remaining 35%, are switching packages, when they are switching packages, what package changes they are making (from "package X" to "package y") and which customers are changing packages most frequently.
The query I have written uses a self-join. I am identifying the changes but my results contain duplicate rows.
This is my query:
select *
from UserPackageDump UPD1, UserPackageDump UPD2
where UPD1.user_id = UPD2.user_id
and UPD1.package_id <> UPD2.package_id
How can I change this query to yield only distinct results?
SELECT DISTINCT * FROM UserPackageDump UPD1 JOIN UserPackageDump UPD2 ON UPD1.user_id = UPD2.user_id WHERE UPD1.package_id <> UPD2.package_id
You have many options for doing this, and I'm not sure your approach is the right one to take. Firstly to answer your specific question, you could perform a DISTINCT as per #sqlab's answer. Or you could include the date in the join, ensuring that UDP1 only matches a record in UDP2 that is one day different. However, to come back to the approach, there should be no need to take a full copy of all the data. You have lots of other options for more efficient data storage, some of which being: Put a "LastUpdated" datetime2 field in the database, to be populated each time the row is changed. Copy only those rows that have a LastUpdated more recent than the last time the copy was made. Assuming the only change possible to the table is to change the package_id then you will now only have rows in the table for users that have changed. Create a UserPackageHistory table into which rows are written each time a user subscribes to a package, at the same time that UserPackage is updated. This then leaves you with much the same result as the first bullet, but in advance of running the copy job. Then, with any one of these sets of data, to satisfy the reporting requirements you could populate a cube. Your source would be a set of rows containing user_id, old_package_id, new_package_id and date. You would create a measure group containing these measures: Distinct count of user_id Count of switches (basically just the row count of the source data) This measure group could then be related to the following dimensions: Date, so you can see when the switches are taking place User, so you can drill down on who is switching Switch Type, which is a dimension built from the selecting the old_package_id and new_package_id from your source data. This gives you the ability to see the popularity of particular shifts.
VBA, SQL, Queries
I have an access form with few controls on it, like start-date, end-date, move_type, mover_name etc, when user fills this field n clicks on query command button, a select query is run and it fetches records from various tables depending upon the criteria mentioned by the user. Later the same records are exported to excel file and a report is generated for user. Now I need to do more with this, my select query has a field "quoted-price", depending upon the price quoted the records need to be sorted out. EX : if quoted_price < 500 then it is a "domestic apartment1", if quoted-price is >500 And <1500 it is "domestic Apartment1" etc. Now I need to do more with this, my select query has a field "quoted-price", depending upon the price quoted the records need to be sorted out. EX : if quoted_price < 500 then it is a "domestic apartment1", if quoted-price is >500 And <1500 it is "domestic Apartment1" etc. and when all the records generated through select query are searched on this criteria, the records need to be grouped accordingly. Later the count for individual category is multiplied by a unit value and total charge is calculated. I tried building many queries to do this and tables to refer to value but of no use. Any help will be much appreciated.
are you looking to create a new field depending on the condition? That may be step one of what you are trying to do, If so that is pretty easy. See here: http://allenbrowne.com/casu-14.html
One to Many - Calculated Column
I am trying to teach myself the new Tabular model for SQL 2012 SSAS to handle some analytic reports that were previously handled in (slow) stored procedures. I've made decent progress on most of it, just figuring out how things work and how to add the calculations I need but I have been banging my head against the following: I have a table that has file information -- it has: ID FileName CurrentStatus UploadedBy And then a table that has statuses that the file went through (a many relationship to the file table): FileID StatusID TimeStamp What I'm trying to do is to add a calculated column to the File table that returns the TimeStamp information when a file was in a particular status. ie: StatusID=100 is uploaded. I want to add a calculated column called UploadedDate on the File table that has the associated TimeStamp information from the FileStatus table. It seems like this should be doable with DAX but I just can't seem to wrap my head around it. Any ideas out there? In advance, many thanks, Brent
Here's a formula that should work for what you want to do... =MAXX( CALCULATETABLE( 'FileStatus' ,'FileStatus'[StatusID] = 100 ) ,'FileStatus'[TimeStamp] )
I'm assuming each file can only be in each status once (there is only one row per FileID that has StatusID 100). I believe you can just use a lookupvalue formula. The formula for your UploadedDate calculated column would be something like =LOOKUPVALUE(FileStatus[Timestamp], File[FileID], FileStatus[FileID], FileStatus[StatusID], 100) Here's the MSDN description of LOOKUPVALUE. You provide the column containing the value you want returned, the column you want to search, and the value you are searching for. You can add multiple criteria to your lookup table. Here's a blog post that contains a good example.