We have a table called Purchases:
| PRSNumber | ... | ... | ProjectCode |
| PRJCD-00001 | | | PRJCD |
| PRJCD-00002 | | | PRJCD |
| PRJCD-00003 | | | PRJCD |
| PRJX2-00003 | | | PRJX2 |
| PRJX2-00003 | | | PRJX2 |
Note: ProjectCode is the prefix of PRSNumber.
Before, when there is no ProjectCode field in the table, our former developers use this query to search for purchases with specific supplier:
select * from Purchases where left(PRSNumber,5) = #ProjectCode
Yes, they concatenate the PRSNumber in order to obtain and compare the ProjectCode. Although, the code above works fine regardless of the table design.
But when I added a new field, the ProjectCode, and use this query:
select * from Purchases where ProjectCode = #ProjectCode
I receive this exception:
Timeout expired. The timeout period elapsed prior to completion
of the operation or the server is not responding.
I can't believe, that the first query, which needs concatenation before the compare, is faster than the second one which has to do nothing but compare. Can you please tell me why is this happening?
Some information which might be helpful:
PRSNumber is varchar(11) and is the primary key
ProjectCode is nvarchar(10)
Both query works fine in SQL Server Management Studio
First query works in ASP.NET website, but the second does not
ProjectCode is indexed
The table has 32k rows
Update
ProjectCode is now indexed, still no luck
First thing I would do is check the index on PRSNumber, I assume there is an index on that field and the table is very large.
Adding an index to your new field will likely fix the problem (if that is the case).
The code to add an index:
CREATE INDEX IX_Purchases_ProjectCode
ON dbo.Purchases (ProjectCode);
Update:
I would also try adding the field as a varchar to eliminate the datatype change from the equation.
I set the CommandTimeout property of my SqlCommand higher instead of making the query faster. It didn't solve the speed but solved the timeout issue.
Related
I'm running the following test on ignite 2.10.0
I have 2 tables created with a query_parallelism=1 and without affinity key.
When I join the 2 following tables I have the result as expected.
0: jdbc:ignite:thin://localhost:10800> SELECT "id" AS "_A_id", "source_id" AS "_A_source_id" FROM PUBLIC."source_ml_blue";
+--------------------------------------+--------------------------------------+
| _A_id | _A_source_id |
+--------------------------------------+--------------------------------------+
| 86c068cd-da89-11eb-a185-3da86c6c6bb3 | 86c068cc-da89-11eb-a185-3da86c6c6bb3 |
+--------------------------------------+--------------------------------------+
1 row selected (0.004 seconds)
0: jdbc:ignite:thin://localhost:10800> SELECT "id" AS "_B_id", "flx_src_ip_text" AS "_B_src_ip" FROM PUBLIC."source_nprobe_tcp_blue";
+--------------------------------------+-----------+
| _B_id | _B_src_ip |
+--------------------------------------+-----------+
| 86c068cc-da89-11eb-a185-3da86c6c6bb3 | 1.1.1.1 |
+--------------------------------------+-----------+
1 row selected (0.003 seconds)
0: jdbc:ignite:thin://localhost:10800> SELECT _A."id" AS "_A_id", _A."source_id" AS "_A_source_id", _B."id" AS "_B_id", _B."flx_src_ip_text" AS "_B_src_ip" FROM PUBLIC."source_ml_blue" AS "_A" INNER JOIN PUBLIC."source_nprobe_tcp_blue" AS "_B" ON "_A"."source_id"="_B"."id";
+--------------------------------------+--------------------------------------+--------------------------------------+-----------+
| _A_id | _A_source_id | _B_id | _B_src_ip |
+--------------------------------------+--------------------------------------+--------------------------------------+-----------+
| 86c068cd-da89-11eb-a185-3da86c6c6bb3 | 86c068cc-da89-11eb-a185-3da86c6c6bb3 | 86c068cc-da89-11eb-a185-3da86c6c6bb3 | 1.1.1.1 |
+--------------------------------------+--------------------------------------+--------------------------------------+-----------+
1 row selected (0.005 seconds)
If I delete and create the same tables with a query_parallelism = 8, I do not have a SQL error (the parallelism is equal on the 2 tables) BUT the result of the join is empty.
any idea why I get this behavior ?
You observe this behaviour because of optimisations for parallel query execution. Most likely your records landed to different partitions (handled by a different thread). If you increase the number of records in both tables you will see a subset of this join as a result.
The most elegant option here is to let "_A"."source_id" and "_B"."id" be affinity keys. Most likely ignite.jdbc.distributedJoins is going to affect performance for clustered installation. Affinity collocation will make items with matching "_A"."source_id" and "_B"."id" reside in the same partition to avoid cross-partitional interaction (for clustered environments it would lead to additional networks hops).
The problem comes from the SQL client : it has to be aware of the parallelism.
On DBeaver, I had to enable ignite.jdbc.distributedJoins in the connection properties to make the request works properly.
A query I am working on is showing a rather interesting behaviour that I couldn't debug so far.
This is the query before it gets buggy:
QryCount
SELECT EmpId, [Open/Close], Count([Open/Close]) AS Occurences, Attribute1, Market, Tier, Attribute2, MtSWeek
FROM qrySource
WHERE (Venue="NewYork") AND (Type="TypeA")
GROUP BY EmpId, [Open/Close], Attribute1, Market, Tier, Attribute2, MtSWeek;
The query gives precisely the results that I would expect it to:
#01542 | Open | 5 | Call | English | Tier1 | Complain | 01/01/2017
#01542 | Closed | 2 | Call | English | Tier2 | ProdInfo | 01/01/2017
#01542 | Open | 7 | Mail | English | Tier1 | ProdInfo | 08/01/2017
etc...
But as a matter of fact in doing so it provides more records than needed at a subsequent step thereby creating cartesians.
qrySource.[Open/Close] is a string type field with possible attributes (you guessed) "open", "Closed" and null and it is actually provided by a mapping table at the creation stage of qrySource (not sure, but maybe this helps).
Now, the error comes in when I try to limit qryCount only to records where Open/Close = "Open".
I tried both using WHERE and HAVING to no avail. The query would result in 0 records, which is not what I would like to see.
I thought that maybe it is because "open" is a reserved term, but even by changing it to "D_open" in the source table didn't fix the issue.
Also tried to filter for the desired records in a subsequent query
SELECT *
FROM QryCount
WHERE [Open/Close] ="D_Open"
But nothing, still 0 records found.
I am suspicious it might be somehow related to some inherent proprieties of the COUNT function but not sure. Any help would be appreciated.
Everyone who participated, thank you and apologies for providing you with insufficient/confusing information. I recon the question could have been drafted better.
Anyhow, I found that the problem was apparently caused by the "/" in the Open/Closed field name. As soon as I removed it from the field name in the original mapping table the query performed as expected.
it is possible to store a function IN the table to automatically sum a group of columns and store the result in a final column?
ie:
+----+------------+-----------+-------------+------------+
| id | appleCount | pearCount | bananaCount | totalFruit |
+----+------------+-----------+-------------+------------+
| 1 | 300 | 60 | 120 | 480 |
+----+------------+-----------+-------------+------------+
where the column totalFruit is automatically calculated from the previous three columns and updated as the other columns update. in this specific application, there is ONLY going to be the one row. it would be spanky-handy to be able to just push the updated counts and then pull the calculated total out. i seem to recall reading about this ability somewhere, but for the life of me, i can't recall where... :poop:
if there is not way to do this, that's cool. but if there is... :smile:
TIA!
WR!
Yes, it is possible. But is it worth it? It is simple enough to do
SELECT ...
appleCount + pearCount + bananaCount AS totalFruit
...
See MariaDB Generated Columns for how to generate the extra column -- either as a real extra column or "virtual". What version of MariaDB?--There are a number of changes over time.
(MySQL users: 5.7.6 has a similar MySQL Generated Columns.)
While trying to build a data warehousing application using Talend, we are faced with the following scenario.
We have two tables tables that look like
Table master
ID | CUST_NAME | CUST_EMAIL
------------------------------------
1 | FOO | FOO_BAR#EXAMPLE.COM
Events Table
ID | CUST_ID | EVENT_NAME | EVENT_DATE
---------------------------------------
1 | 1 | ACC_APPLIED | 2014-01-01
2 | 1 | ACC_OPENED | 2014-01-02
3 | 1 | ACC_CLOSED | 2014-01-02
There is a one-to-many relationship between master and the events table.Since, given a limited number of event names I proposing that we denormalize this structure into something that looks like
ID | CUST_NAME | CUST_EMAIL | ACC_APP_DATE_ID | ACC_OPEN_DATE_ID |ACC_CLOSE_DATE_ID
-----------------------------------------------------------------------------------------
1 | FOO | FOO_BAR#EXAMPLE.COM | 20140101 | 20140102 | 20140103
THE DATE_ID columns refer to entries inside the time dimension table.
First question : Is this a good idea ? What are the other alternatives to this scheme ?
Second question : How do I implement this using Talend Open Studio ? I figured out a way in which I moved the data for each event name into it's own temporary table along with cust_id using the tMap component and later linked them together using another tMap. Is there another way to do this in talend ?
To do this in Talend you'll need to first sort your data so that it is reliably in the order of applied, opened and closed for each account and then denormalize it to a single row with a single delimited field for the dates using the tDenormalizeRows component.
After this you'll want to use tExtractDelimitedFields to split the single dates field.
Yeah, this is a good idea, this is called a cumulative snapshot fact. http://www.kimballgroup.com/2012/05/design-tip-145-time-stamping-accumulating-snapshot-fact-tables/
Not sure how to do this in Talend (dont know the tool) but it would be quite easy to implement in SQL using a Case or Pivot statement
Regarding only your first question, it's certainly a good idea -- unless there is any possibility of the same persons applying-opening-closing their account more than once AND you want to keep all this information in their history (so UPDATE wouldn't help).
Snowflaking is definitely not a good option if you are going to design a data warehouse. So, denormalizing will certainly be a good choice in this case. Following article almost fits perfectly to clear the air over such scenarios,
http://www.kimballgroup.com/2008/09/design-tip-105-snowflakes-outriggers-and-bridges/
I was wondering if anyone can suggest a suitable design pattern for achieving the following:
I have a payslip, each slip shows my previous pay and my current pay. Each payslip should not need to duplicate fields, but rather point the current value to the previous value in the next slip.
On top of all this, I also need to be able to retrieve any given payslip at any point in time (preferably O(1)).
Here's a visual to help understand my problem.
[key:"1"] [key:"2"] [key:"3"]
+------+ +------+ +------+
| | | | | |
| Curr | <--- | Prev | | Curr |
| Null | | Curr | <--- | Prev |
| | | | | |
+------+ +------+ +------+
Any help would be greatly appreciated
Give your PaySlip class a PreviousPaySlipKey property, which will be null for the very first pay slip. In your database, this should be a foreign key to the PaySlip's Key property.
This way, if you have a PaySlip, you can find the key of the previous pay slip, and query the database for the payslip with that key.
Disclaimer: I am not a VB programmer, and the syntax below may not apply. Assuming VB has a Map interface, you could use that for your lookups O(1)
Map<Integer, CompositeValue> = some hash based map
and declare a class (assuming VB lets you create classes) CompositeValue as such:
class CompositeValue {
Integer previousKey;
Value value;
}
Now, once you've retrieved a CompositeValue from the Map, you have the means to get it's real value (value), and to retrieve the previous value using previousKey.
Just a thought.
If we are talking DBs here, every payslip should be given a sequential ID (also called auto-increment in some RDBMS). This will be the primary key (so it's automatically indexed). Assuming you retrieved the payslip you wanted, here is how to get previous one from DB:
SELECT TOP 1 * FROM PaySlip WHERE PaySlipID < #PaySlipID ORDER BY PaySlipID Desc
And the next one:
SELECT TOP 1 * FROM PaySlip WHERE PaySlipID > #PaySlipID ORDER BY PaySlipID
You will probably have payslips for multiple employees stored in a single table, so just add to the WHERE condition.
It is counter productive to store references to previous/next payslip on each payslip.