I am trying to fetch 50k records using SAP Query connector and send it to S3. but it is taking 7 minutes to process 50 k records in batch.It is taking time at the time at loading phase for instance. I have put S3 to aggregator. Still facing issue and also for next 50k records batch is taking 15 to 16 minutes and next records 30minutes. Can anyone help.
enter image description here
Related
I have this table in a database that holds the number of a countries credit card transactions broken out in 10 minute intervals. I have a separate process which generates Excel files for updated transaction counts every 10 minutes. I want to create a stored procedure that reads the data in from these network folders every 10 minutes, and I want to compare a countries most recent 10 minute period to its prior 10 minute period. If it increases by more than 10 percent, it should alert a handful of people with an email alert. I know that I will need a stored procedure to append this data to the current transaction table to the database, but I was wondering if SQL Server Job Agent is to be used when attempting to read in this file every 10 minutes, and then again when the potential alert is sent out?
I am building a web application and using postgreSQL as database. I need to fetch and update thousands of rows every 5-10 mins. Let's say i have 1M rows with this following schema in my table:
ServiceStatus {
id: string,
userid: string,
status: string,
}
I will be fetching all the rows based on service status (let's assume 100,000 rows every 5 mins) and based on the status i'll do some processing and update status in db. As i said i'll do this every 5-10 mins. What's the most efficient approach to this?
Fetching 100,000 row once every 5 minutes will not be strenuous. Updating them should not be either, but probably don't update one just to set it back to the same value as it already holds.
If this turns out to be a problem, it will come down to some detail you haven't described to us, and which we can't guess.
I am looking to pull data between two time periods at only 15 to 30 mins apart. I want to be able to rerun the code multiple times to constantly update the data I had already pulled. I know there is a function for current system time but I am unable to use it effectively in SQL developer.
I have tried using the function CURRENT_TIMESTAMP but could not get it to work effectively.
Currently i am using the following code and just pulling over a broad time frame, but i would like to shrink that down to 15 to 30 minute intervals that could be used to continue to pull updated data.
I expect to be able to pull current data within 15 to 30 minute segments of time.
I have been going through the book Google BigQuery Analytics. On page 354 it states that a table copy completes in less than a minute irrespective of table size. Is this correct? How is it possible?
Let's test that assessment.
I have a 2TB table with 55 billion rows:
I will ask BigQuery to make a copy of it:
The requested job was completed in 55 seconds - less than a minute:
So let me answer the 2 questions above:
On page 354 it states that a table copy completes in less than a minute irrespective of table size. Is this correct?
The book is old, but the answer is still "yes".
How is it possible?
BigQuery is powered by
I have exceeded the daily limit for the number of import to a specific table.
(Max=1000 imports according to the documentation here: https://developers.google.com/bigquery/quota-policy#import )
I would like to know when exactly does the quota reset back to 0? Is it 24hours after I exceeded the quota, or is it at a specific time?
As of this July 18 2014, all daily quotas are partially replenished every 10 minutes or so.
The first time you run a load job to a table (or if you haven't done so in a while) you'll get 1000 loads. Every few minutes, the quota will partially replenish, up to a maximum of 1000 available.
While this sounds complex, it means that you never get in a situation where you run out of daily quota and have to wait up to 24 hours for quota to reset. Instead, if you run out of quota you can start running jobs fairly soon thereafter (as long as you stay within the replenishment rate).
Hope that is helpful.