So here's the scenario: I have a set of tables named job_listings_yyyyMMdd. Every day, a new table, using the aforementioned naming convention is created and populated with that day's job listings.
When that table is populated, a process is kicked off that transforms the data in the table so that a front-end app can use it.
So, as time goes on, I have a set of tables, something like
job_listings_20151226,
job_listings_20151227,
job_listings_20151228,
...
They all have the exact same table structure, but each table contains only that day's job listings.
What I'd like to do is reference a table, from the service that provides the front-end with this data, named job_listings. Ideally, my daily process would create the new day's table, and after all processing is done and that day's data is ready to be served, then have the process change the synonym/alias (i.e., job_listings) to point to the newly populated and processed table for that day.
The idea is that there is not any seam in between data refreshes. Oracle used to have a concept called Synonyms, but I'm having a hard time figuring out how to do this with PostgreSQL.
Different database systems have different methods for federation and aliasing. But I think what you want is something that is available in any SQL system -- a view.
CREATE VIEW MOST_RECENT_JOB_LISTINGS AS
SELECT * FROM job_listings_20151228
Just change this view's definition every day after the new table is created.
Related
I am using the following query to populate my fact table:
Select sh.isbn_l,sh.id_c,sh.id_s, sh.data,sh.quantity, b.price
from Book as b
inner join Sales as sh
on l.isbn=sh.isbn_l
The main thing is that I want to load the table from a specific time to a specific time. So if I load today, I will get all the records from today till the last time I loaded.
And if I load it the day after tomorrow, I will get the datas from today after load time, till the day after tomorrow.
What I mean is NO DUBLICATED ROWS or DATAS. What should I do ?
Any idea pleasee ?
Thank you in advance
Streams (and maybe Tasks) are your friend here.
A Snowflake Stream records the delta of change data capture (CDC) information for a table (such as a staging table), including inserts and other DML changes. A stream allows querying and consuming a set of changes to a table, at the row level, between two transactional points of time.
In a continuous data pipeline, table streams record when staging tables and any downstream tables are populated with data from business applications using continuous data loading and are ready for further processing using SQL statements.
Snowflake Tasks may optionally use table streams to provide a convenient way to continuously process new or changed data. A task can transform new or changed rows that a stream surfaces. Each time a task is scheduled to run, it can verify whether a stream contains change data for a table (using SYSTEM$STREAM_HAS_DATA) and either consume the change data or skip the current run if no change data exists.
Users can define a simple tree-like structure of tasks that executes consecutive SQL statements to process data and move it to various destination tables.
https://docs.snowflake.com/en/user-guide/data-pipelines-intro.html
I have a set of referential tables with different schema which we use as a reference data during integration of files. The reference data can be modified from the GUI.
And the requirement is, I need to create a snapshot of data if there are any changes. For eg., Users should be able to see which referential data has been used for particular date.
Option 1: Historize all the tables over night everyday with date. This way when users want to see the data used for particular date, we can easily query the corresponding history table. As users doesnt change the data everyday, this way we will make the database bigger day by day.
Option 2: Historize only the data(rows) which has been modified with modified date and use the view to fetch the data for particular days. But this way I need to write many views as the schema is different for different tables.
If you know of the best way I can use, I would appreciate it if you share your knowledge.
Thanks,
Not sure if possible but:
Option 3: Create/Edit triggers OnInsert/Update/Delete to write new values to an "historical table" and include a timestamp.
To get the Admin data used on day "X" just use the timestamp.
Another option (again not sure if possible) is to add "start_dt/end_dt" to the admin tables and have the processes lookup only the active data
Sérgio
I know you can run SELECT queries on top of SELECT queries in Access, but the application also provides the Make Table query type.
I'm wondering what the benefits/reasons for using Make Table might be?
You would usually use Make Table for performance reasons. If you have a fairly complex query that returns a subset of your table's data, and that you may need to retrieve multiple times, it can be expensive to re-run the query multiple times.
Using Make Table allows you to incur the cost of running the expensive query once, and make a copy of the query results into a table. Querying this copy would then be a lot less expensive than running your original expensive query.
This is usually a good option when you don't expect your original data to change frequently, or if you don't care that you are working of a copy of the data that may not be 100% up-to-date with the original data.
Notice what the following article on Create a make table query has to say:
Typically, you create make table queries when you need to copy or archive data. For example, suppose you have a table (or tables) of past sales data, and you use that data in reports. The sales figures cannot change because the transactions are at least one day old, and constantly running a query to retrieve the data can take time — especially if you run a complex query against a large data store. Loading the data into a separate table and using that table as a data source can reduce workload and provide a convenient data archive. As you proceed, remember that the data in your new table is strictly a snapshot; it has no relationship or connection to its source table or tables.
The main defense here is that a make table query creates a table. And when you done with the table then effort and time to delete that table and recover the VERY LARGE increase in the database file will have to occur. For general reports and a query of data make much more send. A comparison would be to build a NEW garage every time you want to park your car.
The database engine and query system can fetch and pull rows at a very high rate and those results are then able to be rendered into a report or form, and this occurs without having to create a temp table. It makes little sense to go through all of the trouble of having the system create a WHOLE NEW table for such results of data when they can with ease be sent to a report.
In other words creating a whole table just to display or use some data that the database engine already fetched and returned makes little sense. A table is a set of rows that holds data that can be updated and the results are permanent. A query is a “on the fly” results or sub set of data that only exists in memory and is discarded after you use the results.
So for general reporting and display of data, it makes no sense to create a temp table. MUCH WORSE of an issue is that if you have two users wanting to run a report, if they both need different results and you send the results to the SAME temp table, then you have a big mess and collision between the two users. So use of a temp table in Access for the most part makes little sense, and this is EVEN MORE so when working in a multi-user environment. And as noted, once the table is created, then after you are done you need to delete and remove the table. And with many users in a multi-user database this becomes even more of a problem and issue.
However in a multi-user environment as pointed out that if the resulting data needs additional processing, then sending the results to a temp table can be of use. This approach however suggests that EACH USER has their own front end and own copy of the application side. And better is that the temp table is created outside of the front end application that resides on each computer. Since the application part (front end) is placed on each computer, then creating of a temp table does not occur in the production database (back end) and as a result you can have multiple users function correctly without each individual user creating a temp table in the production back end database. So if one is to adopt a make table query, it likely should occur on each local workstation and not in the back end database when you have a multiple user database application.
Thus for the most part a make table and that of reports and query of data are VERY different goals and tasks. You don't want nor as a general rule create a whole brand new table for a simple query. In a multi user database system the users might run 100's of reports in a given day and FEW if any systems will send such data to a temp table in place of sending the query results directly to the report.
It creates a table - which is useful if you have a need for that table which you may have for temporary use where you have to modify the data for calculations or further processing while not disturbing the original data.
I'm currently working on my first application that uses a database so I'm very new to this. The database has multiple tables that are what you would expect to normally see.
However, I created one table which only has one row and one column used to keep a count of the total items processed by the program so it's available to access elsewhere. I can't just use
SELECT COUNT(*) FROM table_name
because these items that I am processing I do not want to actually keep in a table.
It seems like a waste to use a table to store one value so I am wondering if there a better way to keep track of this value.
What is your table storing? it's storing some kind of processing audit. So make it a little more useful - add a column storing the last datetime that the data was processed. Add a column for the time it took to process. Add another column which stores the username (or some identifier) of whoever ran the process. Now add a row for every table that is processed (there's only one now but there might be more in future). Try and envisage how your processing is going to grow in future
I'm wanting to create a temporary table, purely to make a simpler user interface feature, and it has to be visible only to the current user. I can't see how to achieve this in Access: apparently the "create" in Access is not like the standard SQL: there is a "Temporary" option however (according to http://msdn.microsoft.com/en-us/library/bb177893(v=office.12).aspx):
"When a TEMPORARY table is created it is visible only within the
session in which it was created. It is automatically deleted when the
session is terminated. Temporary tables can be accessed by more than
one user."
It's that last sentence that I have a problem with. I can see "normal databases" you would use SQL "create table #whatever"... so I want to imitate that with Access.
It's a bit long winded to explain the whole situation, apologies if i'm not being clear enough as I'm trying to avoid writing a stupid amount of unnecessary detail: essentially what I have is an "employee" record with a number of "tasks" they perform. My "EmployeeTasks" table has a "percentage" field for each task (i.e. in plain english "employee A(f.key) performs task B(f.key) X% of the day".
To maintain that information in the user interface, it's a bit "messy" to ask users to manually enter percentages... to my mind, people don't really think "well I work 7.6 hours a day, i do 10 tasks, i do this task 3.528% of the time, this task 9.813% of the time... " etc... What I want to present to the user, is their list of tasks, (in a continuous form) with their task effort expressed as hours and minutes per day.
So my theory is, create a temporary table including hours and minute extrapolation, display a form based on that table, the user can then edit those hours and minutes, and "update" function will take those figures and convert back to percentages based on sum. This way the user doesn't have to worry about being accurate in assuring all hours and minutes add to 7.6 hours, and the don't have to worry about all percentages adding to 1 etc... There's a large acceptable margin of error (because obviously most people don't perform tasks for a regimented amount of time, we're only gathering rough information)
It seems the easiest approach is to create a form based on a temp table [EDIT ADDITION]: but if more than one user edits a different employee, they would be overwriting each others temporary tables unless I can create a user-unique table somehow [/EDIT]. Another method I guess would be to dynamically create a list of controls for each task and read from them, but that would get messy quickly when employees have a large number of tasks. Thanks for your help, Simon
It is true that Access SQL does not support CREATE TABLE #TableName to create session-specific temporary tables like T-SQL does, but practically speaking it doesn't need to. Here's why:
For your Microsoft Access database application to support multiple concurrent users
you must split your database into a front-end database file (containing queries, forms, reports, code) linked to a back-end database file (containing just the data tables), and
each user must have their own (local) copy of the front-end database file.
No two users should ever directly open the same .mdb or .accdb file at the same time, e.g., by double-clicking it or doing File > Open in Access. (More details here).
Your VBA code in the front-end can create a temporary table in the front-end and your application can use it. Access allows us to build queries that JOIN local tables with linked tables, so the (local) temporary table can be used like a #Temporary table in T-SQL.
Since each user has their own copy of the front-end file (point #2, above), they each have their own copy of any temporary tables that your application might create.