I am currently working on robust optimization and here I use the dual problem to make the optimizationn tractable. As I want to start to create larger problems, I don't want to have two optimization programs all the time. Therefore, I was wondering if it is possible to create the dual problem from the primal problem in pyomo, which I can then adjust (add uncertainty set) and optimize.
Until now, I have just found solutions which give you the dual values AFTER optimization. But I am looking for the whole dual problem before optimization.
Related
I am asking this question purely out of interest since I can't for the life of me explain why this is happening, especially to this degree. As far as I knew, combining result sets without checking for duplicates would always be as fast as / (usually) faster than with checking!
Basically, there are two views returning ~78.000 and ~63.000 records in ~7 seconds respectively. They have the same super long query, just reading from different tables (same indexes).
Combining the views
SELECT * FROM v1
UNION ALL
SELECT * FROM v2
This results in 141.167 rows after ~14 seconds. Twice the data, twice the time. Makes sense.
Now, purely out of interest I ran the same query but instead of a UNION ALL I just used a UNION on the result sets.
To my surprise, I got the same 141.167 rows in a whopping ~7 seconds!
The records returned by the views are already unique, so I would generally use UNION ALL, but this is twice as fast when I eliminate duplicates that don't exist, so what gives?
Execution plans
The execution plans are the same except for the last few steps since, obviously, they are different. I'm not that well versed in understanding them (especially being so long), but I don't want to leak any sensitive data here:
I was going to upload the execution plans to pastetheplan but they're both >2MB so it wouldn't let me. Instead, here are google drive links for the cleaned version (I only replaced table names):
UNION.sqlplan
UNION_ALL.sqlplan
I'm having more than one schemas inside a single database.
Say,
DBTEST (Database)
SCH001 (schema1)
SCH002 (schema2)
Similarly, I'm having a table called Tbl which is present in the both schemas given above.
Now i had retrieved datum from the tables SCH001.tbl and SCH002.tbl by using the union all key word like this below,
select * From SCH001.tbl union all select * From SCH002.tbl
The above query is working with out any issue, But my boss is asking me to use some other technique to achieve the same(with out union all / Union). So what i am trying to ask is,
Is there any remedy available for Union all to get the same result in my case ? If not then, kindly tell the reason that why Union all cannot be replaced in this case.?
If your boss thinks its ugly to use this union all (all) the time, why not make a view of it?
just reference all the fields in both union parts, and present this as combined view
(i say this cause its generally bad to use * in views)
i do not know if you could index such a view, however its properly worth checking out,
i believe you have to use one schema only when doing indexed views, but i'm unsure of this
(however if possible, this would be a better solution than "just" the union)
(new to that area)
:EDIT: you cannot (in Microsoft sql) make the materialized views on a union, and/or thereby on different schemas, (they are also schema bound)
i would personally use the view anyway, its just a nice way to give a programmatic interface to your data, that you can replace one day!.
:EDIT:
but fact is... its still a union under the hood
This question already has answers here:
UNION ALL vs OR condition in sql server query
(3 answers)
Closed 9 years ago.
I have a problem where I need to find records that either have a measurement that matches a value, or do not have that measurement at all. I solved that problem with three or four different approaches, using JOINs, using NOT IN and using NOT EXISTS. However, the query ended up being extremely slow every time. I then tried splitting the query in two, and they both run very fast (three seconds). But combining the queries using OR takes more than five minutes.
Reading on SO I tried UNION, which is very fast, but very inconvenient for the script I am using.
So two questions:
Why is UNION so much faster? (Or why is OR so slow)?
Is there any way I can force MSSQL to use a different approach for the OR statement that is fast?
The reason is that using OR in a query will often cause the Query Optimizer to abandon use of index seeks and revert to scans. If you look at the execution plans for your two queries, you'll most likely see scans where you are using the OR and seeks where you are using the UNION. Without seeing your query it's not really possible to give you any ideas on how you might be able to restructure the OR condition. But you may find that inserting the rows into a temporary table and joining on to it may yield a positive result.
Also, it is generally best to use UNION ALL rather than UNION if you want all results, as you remove the cost of row-matching.
There is currently no way in SQL Server to force a UNION execution plan if no UNION statement was used. If the only difference between the two parts is the WHERE clause, create a view with the complex query. The UNION query then becomes very simple:
SELECT * FROM dbo.MyView WHERE <cond1>
UNION ALL
SELECT * FROM dbo.MyView WHERE <cond2>
It is important to use UNION ALL in this context when ever possible. If you just use UNION SQL Server has to filter out duplicate rows, which requires an expensive sort operation in most cases.
In SQL 2008, I have a query like so:
QUERY A
UNION
QUERY B
UNION
QUERY C
Will it be slower/faster than putting the result of all 3 queries in say, a temporary table and then SELECTing them with DISTINCT?
It depends on the query -- without knowing the complexity of queries A, B or C it's not one that can be answered, so your best bet is to profile and then judge based on that.
However...
I'd probably go with a union regardless: a temporary table can be quite expensive, especially as it gets big. Remember with a temporary table, you're explicitly creating extra operations and thus more i/o to stress the disk sub-system out. If you can do a select without resorting to a temporary table, that's always (probably) going to be faster.
There's bound to be an exception (or seven) to this rule, hence you're better off profiling against a realistically large dataset to make sure you get some solid figures to make a suitable decision on.
DISTINCT and UNION stand for totally different tasks. The first one eliminates, while the second joins result sets. I don't know what you want to do, but it seems you want distinct rows from 3 different queries with joined results. In that case:
query A UNION query B......
that would be the fastest, depending of course on what you want to do.
I have two queries that I'm UNIONing together such that I already know there will be no duplicate elements between the two queries. Therefore, UNION and UNION ALL will produce the same results.
Which one should I use?
You should use the one that matches the intent of what you are looking for. If you want to ensure that there are no duplicates use UNION, otherwise use UNION ALL. Just because your data will produce the same results right now doesn't mean that it always will.
That said, UNION ALL will be faster on any sane database implementation, see the articles below for examples. But typically, they are the same except that UNION performs an extra step to remove identical rows (as one might expect), and it may tend to dominate execution time.
SQL Server article
Oracle article
MySQL article
DB2 documentation
I see that you've tagged this question PERFORMANCE, so I assume that's your primary consideration.
UNION ALL will absolutely outperform UNION since SQL doesn't have to check the two sets for dups.
Unless you need SQL to perform the duplicate checking for you, always use UNION ALL.
I would use UNION ALL anyway. Even though you know that there are not going to be duplicates, depending on your database server engine, it might not know that.
So, just to provide extra information to DB server, in order for its query planner a better choice (probably), use UNION ALL.
Having said that, if your DB server's query planner is smart enough to infer that information from the UNION clause and table indexes, then results (performance and semantic wise) should be the same.
Either case, it strongly depends on the DB server you are using.
According to http://blog.sqlauthority.com/2007/03/10/sql-server-union-vs-union-all-which-is-better-for-performance/ at least for performance it is better to use UNION ALL, since it does not actively distinct duplicates and as such is faster
Since there will be no duplicates from the two use UNION ALL. You don't need to check for duplicates and UNION ALL will preform the task more efficiently.