Let's say we have a SQL Server database and a table Orders with columns id, date, total, type_id. This table is being updated very frequently (about every 5 seconds) and I would like to use an event listener, which is triggered on every INSERT.
Even more - I would like the event listener to send the newly inserted data to the listening function. Further more, I would like the trigger to work on a SQL WHERE condition (where as the last part with the WHERE cause is not mandatory for me) - so for example only trigger for specific data type: WHERE Orders.type_id=2.
Afaik one would use SQL Server notifications for this. If so, I would like to understand it more deeply. If it works, the way I understand it, the trigger would run every 5 sec - will the trigger wait for the script to end its work?
What does this mean for the performance - will my database be much slower? How can this be measured?
And what is the best way to implement this? I've read that there are different approaches - using C# or VB.
Related
I'm using Firebird 2.5.9. I have a table of information on a set of hardware impact devices that includes a running counter of the # of times the device has impacted. Each time a device is "fired", the hardware will impact 1 or more times; upon completion of the firing event, that device's row is updated with the timestamp and a result code, and I need to increment the running counter column with the number of impacts for that fire event.
I can do this as a separate query to get the field's current value, increment it and use that new value in the update statement, but that seems like a lot of extra overhead. This sort of scenario can't be that uncommon, so I assume that there's some straightforward way to do this within an update statement, but I don't know what it is. I also realize that I could do this as a stored procedure, but for now I want to just do it in the update statement if possible.
I've done this for now by expanding the existing before-insert trigger to a before-insert-or-update trigger:
CREATE TRIGGER TBIU_RPRS1 FOR RPRS ACTIVE BEFORE INSERT OR UPDATE
AS BEGIN
IF (INSERTING AND NEW.ID IS NULL) THEN NEW.ID = NEXT VALUE FOR SEQ_GLOBAL;
IF (UPDATING) THEN NEW.STRIKES = OLD.STRIKES + NEW.STRIKES;
END;
Running counters, sums, etc used to be called "stored aggregates". They are usually maintained by triggers on events tables. But before using them make sure that a simple view with plain count() is not fast enough for you.
We are performance tuning, both in terms of cost and speed, some of our queries and the results we get are a little bit weird. First, we had one query that did an overwrite on an existing table, we stopped that one after 4 hours. Running the same query to an entirely new table and it only took 5 minutes. I wonder if the 5 minute query maybe used a cached result from the first run, is that possible to check? Is it possible to force BigQuery to not use cache?
If you run query in UI - expand Options and make sure Use Cached Result properly set
Also, in UI, you can check Job Details to see if cached result was used
If you run your query programmatically - you should use respective attributes - configuration.query.useQueryCache and statistics.query.cacheHit
I would like to get the row count of job that was run using:
bigquery.startQuery(options)
The naive way of doing this, would be to stream the result (e.g. using):
job.getQueryResultsStream()
And count one by one. This obviously isn't very efficient, especially for large results. Another way I thought of is using the metadata of the job:
job.on('complete', function(metadata) {...}
Where I could kind of "reverse engineer" the response, to get the query plan, and see the number of written rows in the last step. I could find that in:
statistics.query.queryPlan[statistics.query.queryPlan.length - 1].recordsWritten
While a sample of different queries convinced me that this might work, it feels like a "hack", and it's difficult to say how robust it will be. Seems like I might need to handle different cases (failed queries, etc.)
EDIT: Another option suggested below is "SELECT COUNT"ing the temp table created by the original query (available in the job metadata). While this absolutely is a straightforward way to get the result I'm looking for, it has the disadvantage of requiring another roundtrip to query the BigQuery service, which costs several seconds. It is a 0 "bytes billed" query (counting a full table uses table metadata only), but it seems redundant when the job "knows" how many rows it has written to the output.
Is there a straightforward and "correct" way to get this count from the job object, without a roundtrip to BQ service? Perhaps a field I missed / misinterpreted, or a function in the job object that returns this?
Any job has destination table - even when you do not explicitly set it - result is still saved in so-called anonymous table that you can in turn query to get the count of output rows. So below simple extra query will work (note - names are just as an example)
SELECT COUNT(1)
FROM `yourProject._0511743a77ca76c1b55482d7cb1f8e91ac5c7b36.anon17286defe54b5c07ba6810a71abfdba6388ac4e0`
The actual destination table to use - can be retrieved from configuration.query.destinationTable property of job
job.on('complete', function(metadata) {
console.log(metadata.statistics.query.numDmlAffectedRows)
}
I have column "date" in my table.I need to call my function for this table every time when the current time is equal to time in my "date" column. I don't know if it's possible to do this in ms sql server?
It seems like you are trying to implement some kind of scheduling.
You could try implementing one using one of SQL Server services called SQL Server Agent. It may not be fit for all kinds of response to time events, though, but it should be able to manage certain tasks.
You would need to set up a SQL Server Agent job for it.
A job would need to consist of at least one job step and have at least one schedule to be runnable. Perhaps, it would be easiest for you at this point to use the Transact-SQL type of job step.
A Transact-SQL job step is just a Transact-SQL script, a multi-statement query. In your case it would probably first check if there are rows matching the current time. Then, either for every matching row separately or for the entire set of them, it would perform whatever kind of operation Transact-SQL allows you to perform.
I have an INSERT query in Oracle 10g that is getting stuck on a "SQL*Net message from dblink" event. It looks like:
INSERT INTO my_table (A, B, C, ...)
SELECT A, B, C, ... FROM link_table#other_system;
I do not see any locks on my_table besides the one from the INSERT I'm trying to do. The SELECT query on link_table#other_system completes without any trouble when run on its own. I only get this issue when I try to do the INSERT.
Does anyone know what could be going on here?
UPDATE
The SELECT returns 4857 rows in ~1.5 mins when run alone. The INSERT was running over an hour with this wait message before I decided to kill it.
UPDATE
I found an error in my methods. I was using a date range to limit the results. The date range I used when testing the SELECT only was before the last OraStats run on the link_table, but the date range that I used when testing the INSERT was after the last OraStats run on the link_table. So, that mislead me to believe there was a problem with the INSERT. Not very scientific of me to do this; my mistake.
SQL*Net message from dblink generally means that your local system is waiting on the network to transfer the data across the network. It's a very normal wait event for this sort of query.
How many rows does the SELECT statement return? How much data (in MB/ GB) does that represent?
When you say that it "completes without any trouble on its own", are you actually fetching all the data? If you're using something like TOAD or SQL Developer, the GUI will generally fetch the first N rows and return to you. That can be very quick but it doesn't imply that the database is done executing the query-- it may take much more time to finish producing all the rows your query is going to return. It's pretty common for people to measure the time required to fetch the first N rows rather than the time to fetch the last row-- your INSERT statement, obviously, can't return until all the rows have been fetched from the remote table.
Are you using a /*+ driving_site(link_table) */ hint to make Oracle perform the joins on the remote server?
If so, that hint will not work with DML, as explained by Jonathan Lewis on this page.
This may be a rare case where running the query just as a SELECT uses a very different plan than running the query as part of an INSERT. (You will definitely want to learn how to generate explain plans in your environment. Most tools have a button to do this.)
As Andras Gabor recommended in the link, you may want to use PL/SQL BULK COLLECT to improve performance. This may be a rare case where PL/SQL will work faster than SQL.