I'm having a hard time trying to figure out how to make this work :
I have a big SQL query used for a report, that is run using "execute(#sql)".
Withing #sql I have various select statements that I would like to filter using the values in the multi-value parameter passed from my report #filtervalues.
I've tested the following (this is an example query):
Set #Sql='Select * from my table where field in ('+#filtervalues+')'
When I select a single value in the parameter, the profiler reads it as follows:
Select * from mytable where field in(1234356-1234-1234-1234-123456)
So the selected Guid is without quotes, resulting in "error near 123456"
On the other hand, if I selected several values:
Select * from mytable where field in ('+N'1234456-1231-1234-1244566',N'2345333-3243-2342-2342-23423423'+)
So it adds extra ' and +
Can you help me with this ?
Note:
I cannot use the suggested solution on many websites to apply the filter on the table directly, because this would mess up the data.
Edit:
The multi-value parameter is filled using this query dataset:
select 'No Filter' as fullname,'00000000-0000-0000-0000-000000000000' as systemuserid
union all
select distinct su.fullname, convert(nvarchar(90),su.systemuserid)
from FilteredSystemUser su
I managed to fix this using the following :
1- In my Dataset, instead of passing the parameter as it is, I changed it to :
Join (Parameters!FilterValues.Value," ',' ")
to force the values to be sent as strings, the parameter value is as follows :
1234567','122345','12345
2- In SQL, in my procedure, I added additional ' at the beginning and the end of the parameter to make up for the missing ones. So my code shows the following:
Set #Sql='Select * from my table where field in ('''+#filtervalues+''')'
Related
I have created a linked server to query some CSV files. The query below works with no errors:
SELECT * FROM OPENQUERY(LINKEDSERVER, 'Select * FROM [import.csv]')
There is a field that could contain either an number or text and the query seems to want to treat it strictly as an integer and returns NULL as the values for any records that have text in that field. All of the records that contain numeric data in that field are displaying properly.
My first thought was maybe that I needed to cast or convert that field to a string to make sure that the query returned data regardless of whether it was numeric or text, but I haven't been able to get that to work. Something like:
SELECT * FROM OPENQUERY(LINKEDSERVER, 'Select cast(field1, varchar(50)) as newfield, field2 FROM [import.csv]')
Does anyone have any ideas?
I want to do something along the lines of:
SELECT some_things
FROM `myproject.mydataset.mytable_#suffix`
But this doesn't work because the parameter isn't expanded inside the table name.
This does work, using wildcard tables:
SELECT some_things
FROM `myproject.mydataset.mytable_*`
WHERE _TABLE_SUFFIX = #suffix
However, it has some problems:
If I mistype the parameter, this query silently returns zero rows, rather than yelling at me loudly.
Query caching stops working when querying with a wildcard.
If other tables exist with the mytable_ prefix, they must have the same schema, even if they don't match the suffix. Otherwise, weird stuff happens. It seems like BigQuery either computes the union of all columns, or takes the schema of an arbitrary table; it's not documented and I didn't look at it in detail.
Is there a better way to query a single table whose name depends on a query parameter?
Yes, you can, here's a working example:
DECLARE tablename STRING;
DECLARE tableQuery STRING;
##get list of tables
CREATE TEMP TABLE tableNames as select table_name from nomo_nausea.INFORMATION_SCHEMA.TABLES where table_name not in ('_sdc_primary_keys', '_sdc_rejected', 'fba_all_order_report_data');
WHILE (select count(*) from tableNames) >= 1 DO
SET tablename = (select table_name from tableNames LIMIT 1);
##build dataset + table name
SET tableQuery = CONCAT('nomo_nausea.' , tablename);
##use concat to build string and execute
EXECUTE IMMEDIATE CONCAT('SELECT * from `', tableQuery, '` where _sdc_deleted_at is not null');
DELETE FROM tableNames where table_name = tablename;
END WHILE;
In order to answer your stated problems:
Table scanning happens in FROM clause, in WHERE clause happens filtering [1] thus if WHERE condition is not match an empty result would be returned.
"Currently, Cached results are not supported when querying with wildcard" [2].
"BigQuery uses the schema for the most recently created table that matches the wildcard as the schema" [3]. What kind of weird stuff you have faced in your use case? "A wildcard table represents a union of all the tables that match the wildcard expression" [4].
In BigQuery parameterized queries can be run, But table names can not be parameterized [5]. Your wildcard solution seems to be the only way.
You can actually use tables as parameters if you use the Python API, but it's not documented yet. If you pass the tables as parameters through a formatted text string vs. a docstring, your query should work.
SQL example:
sql = "SELECT max(_last_updt) FROM `{0}.{1}.{2}` WHERE _last_updt >= TIMESTAMP(" +
"CURRENT_DATE('-06:00'))".format(project_id, dataset_name, table_name)
SQL in context of Python API:
bigquery_client = bigquery.Client() #setup the client
query_job = bigquery_client.query(sql) #run the query
results = query_job.result() # waits for job to complete
for row in results:
print row
I am using SSIS in VS 2013.
I need to get a list of IDs from 1 database, and with that list of IDs, I want to query another database, ie SELECT ... from MySecondDB WHERE ID IN ({list of IDs from MyFirstDB}).
There is 3 Methods to achieve this:
1st method - Using Lookup Transformation
First you have to add a Lookup Transformation like #TheEsisia answered but there are more requirements:
In the Lookup you Have to write the query that contains the ID list (ex: SELECT ID From MyFirstDB WHERE ...)
At least you have to select one column from the lookup table
These will not filter rows , but this will add values from the second table
To filter rows WHERE ID IN ({list of IDs from MyFirstDB}) you have to do some work in the look up error output Error case there are 2 ways:
set Error handling to Ignore Row so the added columns (from lookup) values will be null , so you have to add a Conditional split that filter rows having values equal NULL.
Assuming that you have chosen col1 as lookup column so you have to use a similar expression
ISNULL([col1]) == False
Or you can set Error handling to Redirect Row, so all rows will be sent to the error output row, which may not be used, so data will be filtered
The disadvantage of this method is that all data is loaded and filtered during execution.
Also if working on network filtering is done on local machine (2nd method on server) after all data is loaded is memory.
2nd method - Using Script Task
To avoid loading all data, you can do a workaround, You can achieve this using a Script Task: (answer writen in VB.NET)
Assuming that the connection manager name is TestAdo and "Select [ID] FROM dbo.MyTable" is the query to get the list of id's , and User::MyVariableList is the variable you want to store the list of id's
Note: This code will read the connection from the connection manager
Public Sub Main()
Dim lst As New Collections.Generic.List(Of String)
Dim myADONETConnection As SqlClient.SqlConnection
myADONETConnection = _
DirectCast(Dts.Connections("TestAdo").AcquireConnection(Dts.Transaction), _
SqlClient.SqlConnection)
If myADONETConnection.State = ConnectionState.Closed Then
myADONETConnection.Open()
End If
Dim myADONETCommand As New SqlClient.SqlCommand("Select [ID] FROM dbo.MyTable", myADONETConnection)
Dim dr As SqlClient.SqlDataReader
dr = myADONETCommand.ExecuteReader
While dr.Read
lst.Add(dr(0).ToString)
End While
Dts.Variables.Item("User::MyVariableList").Value = "SELECT ... FROM ... WHERE ID IN(" & String.Join(",", lst) & ")"
Dts.TaskResult = ScriptResults.Success
End Sub
And the User::MyVariableList should be used as source (Sql command in a variable)
3rd method - Using Execute Sql Task
Similar to the second method but this will build the IN clause using an Execute SQL Task then using the whole query as OLEDB Source,
Just add an Execute SQL Task before the DataFlow Task
Set ResultSet property to single
Select User::MyVariableList as Result Set
Use the following SQL command
DECLARE #str AS VARCHAR(4000)
SET #str = ''
SELECT #str = #str + CAST([ID] AS VARCHAR(255)) + ','
FROM dbo.MyTable
SET #str = 'SELECT * FROM MySecondDB WHERE ID IN (' + SUBSTRING(#str,1,LEN(#str) - 1) + ')'
SELECT #str
If the column has string data type you should add quotation before and after values as below:
SELECT #str = #str + '''' + CAST([ID] AS VARCHAR(255)) + ''','
FROM dbo.MyTable
Make sure that you have set the DataFlow Task Delay Validation property to True
This is a classic case for using LookUp Transformation. First, use a OLE DB Source to get data from the first database. Then, use a LookUp Transformation to filter this data-set based on the ID values from the second data-set. Here is the steps for using a LookUp Transformation:
In the General tab, select Full Cash, OLE DB Connection Manager and Redirect rows to no match output as shown in the following picture. Notice that using Full Cash provides great performance for your package.
General Setting
In the Connection tab, use OLE DB Connection Manager to connect to your second server. Then, you can either directly select the data-set with ID values or (as is shown in the picture below) you can use SQL code to select the IDs from the filtering data-set.
Connection:
Go to Columns tab and select ID columns from the both datasets. For each record from your first data-set, it will check to see if its ID is in the Available LookUp Column. If it is, it will go to the Matching output, else to No Matching output.
Match ID columns:
Click on OK to close the LookUp. Then you need to select the LookUp Match Output.
Match Output:
The "best" answer depends on data volumes and source systems involved.
Many of the other answers propose building out a list of values based on clever concatenation within SQL Server. That doesn't work so well if the referenced system is Oracle, MySQL, DB2, Informix, PostGres, etc. There may be an equivalent concept but there might not be.
For best performance, you need to filter against the second db before any of those rows ever hit the data flow. That means adding a filtering condition, as the others have suggested, to your source query. The challenge with this approach is that your query is going to be limited by some practical bounds that I don't remember. Ten, one hundred, a thousand values in your where clause is probably fine. A lakh, a million - probably not so much.
In the cases where you have large volumes of values to filter against the source table, it can make sense to create a table on that server and truncate and reload that table (execute sql task + data flow). This allows you to have all of the data local and then you can index the filter table and let the database engine do what it's really good at.
But, you say the source database is some custom solution that you can't make tables in. You can look at the above approach with temporary tables and within SSIS you just need to mark the connection as singleton/persisted (TODO: look this up). I don't much care for temporary tables with SSIS as debugging them is a nightmare I'd not wish upon my mortal enemy.
If you're still reading, we've identified why filtering in the source system might not be "doable", even if it will provide the best performance.
Now we're stuck with purely SSIS solutions. To get the best performance, do not select the table name in the drop down - unless you absolutely need every column. Also, pay attention to your data types. Pulling LOB (XML, text, image (n)varchar(max), varbinary(max)) into the dataflow is a recipe for bad performance.
The default suggestion is to use a Lookup Component to filter the data within the data flow. As long as your source system supports and OLE DB provider (or you can coerce the data into a Cache Connection Manager)
If you can't use a Lookup component for some reason, then you can explicitly sort your data in your source systems, mark your source components as such, and then use a Merge Join of type Inner Join in the data flow to only bring in matched data.
However, be aware that sorts in source systems are going to be sorted according to native rules. I ran into a situation where SQL Server was sorting based on the default ASCII sort and my DB2 instance, running on zOS, provided an EBCDIC sort. Which was great when my domain was only integers but went to hell in a handbasket when the keys became alphanumeric (AAA, A2B, and AZZ will sort differently based on this).
Finally, excluding the final paragraph, the above assumes you have integers. If you're performing string matching, you get an extra level of ugliness because different components may or may not perform a case sensitive match (sorting with case sensitive systems can also be a factor).
I would first create a String variable e.g. SQL_Select, at the Scope of the Package. Then I would assign that a value using an Execute SQL Task against the 1st database. The ResultSet property on the General page should be set to Single row. Add an entry to the Result Set tab to assign it to your Variable.
The SQL Statement used needs to be designed to return the required SELECT statement for your 2nd database, in a single row of text. An example is shown below:
SELECT
'SELECT * from MySecondDB WHERE ID IN ( '
+ STUFF ( (
SELECT TOP 5
' , ''' + [name] + ''''
FROM dbo.spt_values
FOR XML PATH(''), TYPE).value('(./text())[1]', 'VARCHAR(4000)'
) , 1 , 3, '' )
+ ' ) '
AS SQL_Select
Remove the TOP 5 and replace [name] and dbo.spt_values with your column and table names.
Then you can use the variable SQL_Select in a downstream task e.g. an OLE DB Source against database 2. OLE DB Sources and OLE DB Command Tasks both let you specify a Variable as the SQL Statement source.
You could add a LinkedServer between the two servers. The SQL command would be something like this:
EXEC sp_addlinkedserver #server='SRV' --or any name you want
EXEC sp_addlinkedsrvlogin 'SRV', 'false', null, 'username', 'password'
SELECT * FROM SRV.CatalogNameInSecondDB.dbo.SecondDBTableName s
INNER JOIN FirstDBTableName f on s.ID = f.ID
WHERE f.ID IN (list of values)
EXEC sp_dropserver 'SRV', 'droplogins'
I have a Crystal Report based on a SQL Command that, in part, consists of procedure names. What I'm trying to do is add a multi-value string parameter to the SQL Command such that the end users can enter partial procedure names and the report will return only those relevant procedures that string match.
For example, a user should be able to enter "%KNEE%" and "%HIP%" into the parameter and return all procedures that contain the words "KNEE" and "HIP". The problem is that I can't figure out how to manipulate the parameter value in the SQL to accomplish this. I've done this before with a report parameter (as opposed to a SQL Command parameter) by simply adding the line {table.procedure_name} like {?name match parameter} to the record selection formula, but taking the same approach in the SQL Command gets me an "ORA-00907: Missing right parenthesis" error.
Any suggestions on how I can manipulate the multi-value string parameter to accomplish this?
I dont like to post this as an answer because I don't care for the solution however it is the only way I have found to work around this.
I have had to instruct users to enter '%KNEE%','%HIP%','%ETC%' at the parameter prompt. Then the {table.procedure_name} like {?name match parameter} should work in your SQL. Not optimal, especially for your scenario with the %. I would love to hear someone provide a better solution because I have wrestled with this many times.
Here's an approach:
SELECT column0
FROM table0
INNER JOIN (
SELECT trim('%' || x.column_value.extract('e/text()') || '%') SEARCH
FROM ( SELECT 'arm,knee' options FROM dual ) t,
TABLE (xmlsequence(xmltype('<e><e>' || replace(t.options,',','</e><e>')|| '</e></e>').extract('e/e'))) x
) v ON column0 LIKE v.search
Use Oracle's XML functionality to convert a comma-delimited string to an equivalent number of rows, wrapping each clause with %%. Then join those rows to the desired table.
To use with CR, create a single-value, string parameter and add it to the code:
...
FROM ( SELECT '{?search_param}' options FROM dual ) t,
...
I have a report with an input parameter P_Region that allows the user to select 1-* options. I then need to pass the results of this parameter to a DataSet query. It works if i only select one option but not if i select two.
What i have:
The DataSet parameter for it is set to:
=join(Parameters!P_Region.Value, ",")
The query is:
WHERE ... AND D.City IN(#P_Region)
I have trieda few different things with no luck as well:
=Split(Parameters!Item.Value, ",")
=Split(Join(Parameters!#ReportParameter1.Value,","),",")
WHERE D.Region IN (SELECT * FROM dbo.split(#Department)) with no luck.
Any help is much appreciated!
Solution: (Sorry I feel like this is a pretty easy solution but I did not see it anywhere online and I'm super new with SSRS)
Right click on the dataset
Go to the Parameters tab and set the query parameter back to the default value (No Joins or Splits)
Go to the Filters tab and add the expression of the field you want to be in the parameter, choose the operator "In" and set the value as [#<'Parameter Name'>']
Note: Understand that the other records will still be returned by the query but they will just be filtered. If you are returning large sets of data and using very little of it, this may not be the best route to go.
I think you where on the right track with WHERE D.Region IN (SELECT * FROM dbo.split(#Department)) with no luck.
I use a tabled valued function to create a filter table and parse the values there.
WHERE D.Region IN (SELECT ID FROM dbo.CreateIntIDTable(#Department))
ALTER FUNCTION [dbo].[CreateIntIDTable](#IDList NVARCHAR(3000))
RETURNS #T TABLE (ID INT)
AS BEGIN
WHILE(CHARINDEX(',',#IDList)>0)BEGIN
INSERT INTO #T
SELECT LTRIM(RTRIM(SUBSTRING(#IDList,1,CHARINDEX(',',#IDList)-1)))
SET #IDList = SUBSTRING(#IDList,CHARINDEX(',',#IDList)+LEN(','),LEN(#IDList))
END
INSERT INTO #T SELECT LTRIM(RTRIM(#IDList))
RETURN
END
Try to Create separate dataset for your selection dropdown then you can use your where condition.
for ex.
1) Data set (SELECT Company FROM Companymaster ORDER BY Company)
2) where
company in (#CompanyName)
Did you set up the parameter properties to accept multiple values ?
you don't need to do any Join or Split function if properties ar set up properly.