I own an amount about 80 000 rows on 3 tables of database, every time I execute my code, it can take 4-5 minutes to display the result. Do you know if it's possible to reduce this time by adding whatever ?
def findAirportAndRunwayByCountry(code: String)(implicit session : DBSession = AutoSession) : List[(Airports,Runway)] = {
val (c, a, r) = (Country.syntax("c"), Airports.syntax("a"), Runway.syntax("r"))
withSQL { select.from(Airports as a).innerJoin(Runway as r).on(r.ID, a.ID).innerJoin(Country as c).on(c.Code, a.Country).where.eq(c.Code, code).or.like(c.Name, code + "%") }.map(rs => (Airports(rs),Runway(rs))).list.apply()
}
This is the code taking the time described (~4-5 mins)
Related
I just want to compare 2 periods (e.g. i want to compare between 3w4d23h58m and 20h in order to track time). All i got is
I need to convert this PT2H to just 2 hours to count percentage between estimation and spent time.
https://www.jetbrains.com/help/youtrack/devportal/v1-PeriodProjectCustomField.html?q=getMinutes
Here is an excerpt from a workflow that calculates remaining time by subtracting spent time from estimation:
action: (ctx) => {
const issue = ctx.issue;
var periodestimate = issue.Estimation;
var minutesestimate = !periodestimate ? 0 : (periodestimate.getMinutes() + 60 * (periodestimate.getHours() + 8 * (periodestimate.getDays() + 5 * periodestimate.getWeeks())));
var periodspent = ctx.issue.fields.SpentTime;
var minutesspent = !periodspent ? 0 : (periodspent.getMinutes() + 60 * (periodspent.getHours() + 8 * (periodspent.getDays() + 5 * periodspent.getWeeks())));
var remain = minutesestimate - minutesspent;
ctx.issue.fields.Remaining = dateTime.toPeriod(remain + 'm');
},
I suppose that you can use it as an example to calculate the needed percentage.
I'm trying to run through a few hundred rows of data and delete rows that have 0 in both C and D. Deleting the ones that match that case seems to be working, but it really takes its time getting through. I'm a neophyte at scripting (I'm not a programmer, just trying to learn) and I can't figure how to make it faster. The main script came from someone else, I added the logic to do two columns and not just one. Any Ideas?
function deleteZeroes() {
var ss = SpreadsheetApp.getActiveSpreadsheet();
var s = ss.getSheetByName('Email_Help_Total_Input');
var range1 = s.getRange('C:C');
var cValues = range1.getValues();
var range2 = s.getRange('D:D');
var dValues = range2.getValues();
for(var i=cValues.length-1;i>=0;i--)
if(cValues[0,i]==0 && dValues[0,i]== 0)
s.deleteRow(i+1);
}
It works, but slowly. Just trying to make it a little quicker. Thanks!
Array rather than reads and writes!
data = s.getDataRange().getValues();
result = [];
for (i=0;i<data.length;i++) {
if (data[i][2]==0) continue; //checks C, if it finds it continues the loop
if (data[i][3]==0) continue; //checks D, if it finds it continues the loop
result.push(data[i]);//only those that fell through the 0 checks will be here
}
s.getDataRange().clear();
s.getRange(1,1,result.length,result[0].length).setValues(result);
So this reads all the data at once, checks all the data and copies the good rows to a new result array, deletes the old data and then pastes back the good data. Depending on the size you should see this function in maybe 30 seconds.
I'm trying to run parallel sql queries using GPars. But somehow it isn't working as I expected. Since I'm relatively new to groovy/java concurrency I'm not sure how to solve my issue.
I following code:
def rows = this.sql.rows(
"SELECT a_id, b_id FROM data_ids LIMIT 10 OFFSET 10"
)
With this code I get a List of ids. Now I want the load the data for each loaded id and this should happen parallel to improve my performance, because I have a large database.
To get the detail data I use the following code:
GParsPool.withPool() {
result = rows.collectParallel {
// 2. Get the data for each source and save it in an array.
def tmpData = [:]
def row = it
sql.withTransaction {
if (row.a_id != null) {
tmpData.a = sql.firstRow("SELECT * FROM data_a WHERE id = '" + row.a_id + "'")
}
if (row.b_id != null) {
tmpData.b = sql.firstRow("SELECT * FROM data_b WHERE id = '" + row.b_id + "'")
}
}
return tmpData
}
// 3. Return the loaded data.
return result
Now I run the code and everything works fine except that the code isn't executed parallel. Using the JProfiler I can see that I have blocked threads and waiting threads, but 0 runnable threads.
Thanks for any help. I you need more information, I will provide them :)
Daniel
Here's my table:
CREATE TABLE public.logs (
logid bigint NOT NULL DEFAULT nextval('applog_logid_seq'::regclass),
applicationname character varying(50),
loglevel character varying(10),
logmessage character varying(500),
stacktrace character varying(4096),
occurredon timestamp without time zone,
loggedon timestamp without time zone,
username character varying(50),
groupname character varying(50),
useragent character varying(512),
CONSTRAINT applog_pkey PRIMARY KEY (logid)
);
When I run SELECT *... on it, it takes 40 seconds to return 50000 rows on my local machine. I have the same table on a local install of SQL Server, and that takes less than a second to return the same amount of data.
I'm in the middle of an evaluation of PostgreSQL for our new stack and this is very concerning to me. Why am I doing wrong/why is PostgreSQL so slow?
Edit:
Here's what I get from EXPLAIN (BUFFERS, ANALYZE, TIMING OFF) SELECT * FROM public.logs:
So it looks like the server's going to execute this in about 6 ms. I guess that means all the overhead is in pgAdmin III, but how is SSMS able to do this so much faster?
As we suspected, and as we can see from the EXPLAIN output, almost none of the time is spent on the server, but for transfer (network latency) and rendering the output. We see 7 ms in the EXPLAIN output you added to the question.
Rendering large amounts of text in pgAdmin III is not very fast. Two hints:
Most of the time, you don't need to see all the content of huge columns. To speed things up, you can set a maximum number of display characters In pgAdmin III. But don't get confused if you only see the leading fragment of long strings:
Why are query results shortened in pgAdmin 1.18.1?
pgAdmin 4 beta 2 is out (2016-6-24). Not release yet, but since it's a complete rewrite performance may be substantially different. More on project's site.
Thanks so much for everyone's help in talking me down from the cliff:)
I composed a node console app app that puts my concerns to bed. In fact, my Postres instance beats SQL Server by about 50% (as #Guillaume F. pointed out). In the same client here are the results:
Postres (RDS) query duration: 7062ms
Postres (RDS) rows returned: 50000
Postgres (Local) query duration: 1919ms
Postgres (Local) rows returned: 46154
MSSQL (local) query duration: 4681ms
MSSQL (local) rows returned: 50000
Here's the sample app if anyone's interested in duplicating my results in their own environment:
'use strict';
let pgp = require('pg-promise')();
let db = pgp("postgres://username:password#server:5432/db");
let localdb = pgp("postgres://username:password#server:5432/db");
var mssql = require('mssql');
let start = new Date();
db.query('select * from logs').then((result) => {
console.log("Postres (RDS) query duration: " + (new Date() - start) + "ms");
console.log("Postres (RDS) rows returned: " + result.length);
console.log("");
let localstart = new Date();
localdb.query('select * from logs').then((localresult) => {
console.log("Postgres (Local) query duration: " + (new Date() - localstart) + "ms");
console.log("Postgres (Local) rows returned: " + localresult.length);
console.log("");
var config = {
user: 'username',
password: 'password',
server: 'server', // You can use 'localhost\\instance' to connect to named instance
database: 'db'
};
mssql.connect(config).then(function () {
// Query
let localMSSqlStart = new Date();
new mssql.Request().query('select TOP 50000 * from dbo.AppLog ORDER BY 1 DESC').then(function (recordset) {
console.log("MSSQL (local) query duration: " + (new Date() - localMSSqlStart) + "ms");
console.log("MSSQL (local) rows returned: " + result.length);
console.log("");
}).catch(function (err) {
// ... query error checks
console.log("Problem querying MSSQL: " + err);
});
}).catch(function (err) {
// ... connect error checks
console.log("Problem connecting to MSSQL: " + err);
});
});
});
EDIT: (by pg-promise author)
On a side note, this is just to show how to benchmark PostgreSQL in a more civilized way:
let pgp = require('pg-promise')();
let db = pgp("postgres://username:password#server:5432/db");
let localdb = pgp("postgres://username:password#server:5432/db");
db.result('select * from logs')
.then(r => {
console.log("Postres (RDS) rows returned:", r.rows.length);
console.log("Postres (RDS) query duration:", r.duration + 'ms\n');
return localdb.result('select * from logs')
.then(r => {
console.log("Postgres (Local) rows returned:", r.rows.length);
console.log("Postgres (Local) query duration:", r.duration + 'ms\n');
})
})
.catch(error=> {
console.log(error);
});
The advantage of using method result for benchmarks is that pg-promise automatically extends the Result with property duration.
I have a reporting application written in grails. it fires SQL at a production back office database, and simply lists the resultant rows back to the user.
The bit which executes the sql is this:
class ReportService {
static transactional = false
def dataSource_target
def runReport(sql) {
def rows = null
def start
def con
start = System.currentTimeMillis()
try {
con = new Sql(dataSource_target)
rows = con.rows(sql)
} finally {
con.close()
}
def time = System.currentTimeMillis() - start
def response = [rows: rows, time:time,]
response
}
When this runs, it takes say 60 seconds (60000 ms). If I run the exact same SQL using mysql work bench or similar, it comes back in 30 seconds. Thats a big difference!
Typicically on 30 or so rows are returned, so there is not much network overhead.
Any ideas why grails should be to run the query and red the results?
The time variable in your code measures the total of
time to create the connection
time to execute the query
time to destroy the connection
To more accurately measure just (2), change your code to
def runReport(sql) {
def rows = null
def time
def con
try {
con = new Sql(dataSource_target)
def start = System.currentTimeMillis()
rows = con.rows(sql)
time = System.currentTimeMillis() - start
} finally {
con.close()
}
def response = [rows: rows, time:time,]
response
}