So I've got a tables being imported daily from firebase and I'm exporting them from google cloud to power bi. The naming schema stays roughly the same, the only thing that changes daily is the date at the end of the table name. Been trying to create a query that makes the process fully automated, but I've been stuck. The concat is correct, but the string doesn't work to call from. Help is appreciated.
DECLARE Query STRING(100);
SET Query = CONCAT("report-viewer.analytics_278375497.events_", FORMAT_DATE("%Y%m%d", CURRENT_DATE()));
SELECT
//Various stuff
FROM Query LIMIT 1000
You should use EXECUTE IMMEDIATE in this scenario
Using your particular simplified example - it could look like below
DECLARE table_reference STRING(100);
SET table_reference = 'report-viewer.analytics_278375497.events_' || FORMAT_DATE('%Y%m%d', CURRENT_DATE());
EXECUTE IMMEDIATE (
'''SELECT //Various stuff
FROM `''' || table_reference || '''` LIMIT 1000
''');
Related
In Oracle's SQL Developer, I can execute a "dynamic" SQL select statement in the Script Output pane with something like:
script
var tabName = 'all_users';
sqlcl.setStmt('select * from ' + tabName);
sqlcl.run();
/
Now, I am wondering if it is possible to execute a dynamic select statement such that its result is displayed in the result grid.
SqlDev is implemented in Java which includes Nashorn scripting engine. Therefore, you can basically execute JDBC in your Nashorn script. Here is one way to do it.
Paste
select REGIONS,
COUNTRIES,
LOCATIONS,
DEPARTMENTS,
JOBS,
EMPLOYEES,
JOB_HISTORY,
TM_USER_INFO,
USER_ROLES,
PAYMENT_PRICE_SHOP,
SHOP_USER from dual
into your worksheet.
Open "Code Outline" panel. Type "arbori", or open the following "querySQL.arbori" file:
include "std.arbori"
prelude: runOnce -> {
var ConnectionResolver = Java.type('oracle.dbtools.db.ConnectionResolver');
var connectionList = ConnectionResolver.getConnectionNames();
var conn = ConnectionResolver.getConnection('IdeConnections%23local_hr');
}
queries: [node) column -> {
var node = tuple.get('node');
var table = target.input.substring(
target.src[node.from].begin,
target.src[node.to-1].end
);
var query = 'select * from ' + table;
var ps = conn.createStatement();
var rs = ps.executeQuery(query);
while( rs.next() )
print(rs.getString(1));
}
It outputs the first column of each table to the standard java output
, so it would require some reverse engineering to get a handle to the SqlDev script output panel and channel the result there.
Dynamic SQL can be displayed in an Oracle SQL Developer grid using a function returning a ref cursor (sort of), a polymorphic table function (18c+), or Oracle Data Cartridge (requires custom PL/SQL packages and types).
Function Returning Ref Cursor
As explained in this answer, the output from a function returning a ref cursor can be displayed in the "Output Variables" window. The example from the answer used static SQL, but it's pretty easy to make it dynamic:
create or replace function refcursor_function return sys_refcursor as
c sys_refcursor;
begin
open c for 'select * from all_objects';
return c;
end;
/
The downside is that getting the results takes a few more clicks than a normal query, and the Output Variables grid is not nearly as powerful as the regular results grid. If you just need a window for viewing and copying and pasting, Output Variables is good enough. But it doesn't allow any advanced GUI features.
Polymorphic Table Function
Since Oracle 18c, you can create a polymorphic table function that accepts input and has variable output. You have to program how to handle the tables and columns, but if you just need simple pass-through logic it's not that difficult. See my answer here for an example of a query that returns every column from a table excluding specific columns. The results are "regular" SQL as far as any program knows, and will work in any grid GUI.
Oracle Data Cartridge
My open source program Method4 can run dynamic SQL in SQL. After you download and install the packages and types, you can write a SQL statement that generates another SQL statement. If you need to use PL/SQL to generate the query, you may need to use a PL/SQL WITH function. Like the polymorphic table function, the results look like normal SQL and will work in any grid.
select * from table(method4.dynamic_query(
q'[
--Query that builds another query.
select replace(
q'!
select * from #TABLE_NAME#
!', '#TABLE_NAME#', table_name) sql_statement
from
(
--Enter your script here that returns a table name to select from.
select 'ALL_USERS' table_name
from dual
)
]'
));
You may want to add some details about what exactly you're trying to do and why; that might help narrow down the possible solutions.
I am creating a result set where I want the column name to be equal to a variable name that is et during run time. Is that possible ? How do I do that?
In the example below the user choses the date (myDate) before running the query (e.g 2015-06-11). The I want the column name to be that date (2015-06-11). How do I do that? FYI: I'm using Teradata.
SELECT
table_A.Cnt as ?myDate
/* I can't write ?myDate like that. I also tried to convert it to a string */
FROM
(
SELECT COUNT(*) AS Cnt FROM A
WHERE theDate=?myDate
) AS table_A
What you are trying to do is parameterize an object (or the name of an object) rather than parameterize a value, which seems straight forward when you think up the idea, but it's a bit more difficult to pull off.
First off, only an SP allows you to write and execute SQL dynamically, which is what you are doing here. Second, it's a little verbose. Third, it opens you up to SQL injection issues since you are slipping a parameter from a user into SQL then executing it, so proceed cautiously and do what you can to prevent a-holes from mucking up your system.
CREATE PROCEDURE paramMyField
(
IN myDate Date,
--This has to be less than 30 otherwise Teradata will be angry.
--I would set it low just to keep injection possibilities to minimum
IN fieldName VARCHAR(10)
)
--Tell it how many result sets this thing is going to return:
DYNAMIC RESULT SETS 1
--Set the security (using the security of the bloke that sets this thing off, if you don't trust them, neither do I)
SQL SECURITY INVOKER
BEGIN
--We'll need a variable to hold the dynamically generated sql statement
DECLARE dynSQL VARCHAR(5000);
--And we'll need a cursor and a statement
DECLARE dynCursor CURSOR WITH RETURN ONLY FOR dynStatement;
SET dynSQL = '
SELECT
table_A.Cnt as ' || fieldName || '
FROM
(
SELECT COUNT(*) AS Cnt FROM A
WHERE theDate = DATE ''' || myDate || '''
) AS table_A;';
--Now to prep the statement
PREPARE dynStatement FROM dynSQL;
--And open the cursor (we will open and not close it so it's sent back as a resultset
OPEN dynCursor;
END;
There's a lot happening there, but basically it's a stored procedure that takes in two parameters (the date and the name of the field) and spits back a record set that is the results of the SQL statement with a dynamically named field. It does this by using a dynamic SQL statement.
This is executed by running something like:
CALL paramMyField(DATE '2015-06-15', 'Whatever');
(1) Is there a good/reliable way to query the system catalogue in order
to find all stored procedures which create some temporary tables in their
source code bodies but which don't drop them at the end of their bodies?
(2) In general, can creating temp tables in a SP and not dropping
them in the same SP cause some problems and if so, what problems?
I am asking this question in the contexts of
SQL Server 2008 R2 and SQL Server 2012 mostly.
Many thanks in advance.
Not 100% sure if this is accurate as I don't have a good set of test data to work with. First you need a function to count occurrences of a string (shamelessly stolen from here):
CREATE FUNCTION dbo.CountOccurancesOfString
(
#searchString nvarchar(max),
#searchTerm nvarchar(max)
)
RETURNS INT
AS
BEGIN
return (LEN(#searchString)-LEN(REPLACE(#searchString,#searchTerm,'')))/LEN(#searchTerm)
END
Next make use of the function like this. It searches the procedure text for the strings and reports when the number of creates doesn't match the number of drops:
WITH CreatesAndDrops AS (
SELECT procedures.name,
dbo.CountOccurancesOfString(UPPER(syscomments.text), 'CREATE TABLE #') AS Creates,
dbo.CountOccurancesOfString(UPPER(syscomments.text), 'DROP TABLE #') AS Drops
FROM sys.procedures
JOIN sys.syscomments
ON procedures.object_id = syscomments.id
)
SELECT * FROM CreatesAndDrops
WHERE Creates <> Drops
1) probably no good / reliable way -- though you can extract the text of sp's using some arcane ways that you can find in other places.
2) In general - no this causes no problems -- temp tables (#tables) are scope limited and will be flagged for removal when their scope disappears.
and table variables likewise
an exception is for global temp tables (##tables) which are cleaned up when no scope holds a reference to them. Avoid those guys -- there are usually (read almost always) better ways to do something than with a global temp table.
Sigh -- if you want to go down the (1) path then be aware that there are lots of pitfalls in looking at code inside sql server -- many of the helper functions and information tables will truncate the actual code down to a NVARCHAR(4000)
If you look at the code of sp_helptext you'll see a really horrible cursor that pulls the actual text..
I wrote this a long time ago to look for strings in code - you could run it on your database -- look for 'CREATE TABLE #' and 'DROP TABLE #' and compare the outputs....
DECLARE #SearchString VARCHAR(255) = 'DELETE FROM'
SELECT
[ObjectName]
, [ObjectText]
FROM
(
SELECT
so.[name] AS [ObjectName]
, REPLACE(comments.[c], '#x0D;', '') AS [ObjectText]
FROM
sys.objects AS so
CROSS APPLY (
SELECT CAST([text] AS NVARCHAR(MAX))
FROM syscomments AS sc
WHERE sc.[id] = so.[object_id]
FOR XML PATH('')
)
AS comments ([c])
WHERE
so.[is_ms_shipped] = 0
AND so.[type] = 'P'
)
AS spText
WHERE
spText.[ObjectText] LIKE '%' + #SearchString + '%'
Or much better - use whatever tool of choice you like on your codebase - you've got all your sp's etc scripted out into source control somewhere, right.....?
I think SQL Search tool from red-gate would come handy in this case. You can download from here. This tool will find the sql text within stored procedures, functions, views etc...
Just install this plugin and you can find sql text easily from SSMS.
Summary
Is there an efficient way to run large numbers of Dynamic SQL (on SQL Server 2005)?
Details
Our system allows users to create "email alert" subscriptions - where new matches on the system are emailed to them on a daily basis.
The subscription allows for multiple options, including the use of search keywords. A parser written by myself outputs the appropriate SQL code, taking into account and, or and brackets (). The parser will not allow anything through that could be used for SQL Injection.
For example, the keywords might be entered by the user as this (that or other) and the resultant query would end up roughly as...
SELECT *
FROM [VW_EMAIL_ALERT]
WHERE ([SEARCH] LIKE '%this%' AND ([SEARCH] LIKE '%that%' OR [SEARCH] LIKE '%other%'))
Each night, all those subscriptions are processed individually, because each one is potentially unique. The result is that the batch processing has to run a cursor over every subscription and run the SQL through sp_executesql.
Obviously this is highly inefficient, and can cause serious overloading - leading in some cases to timeouts. The stored-procedure that runs this processing is coded to split the subscriptions into blocks, so they're not all being called at once.
Is there a better/more efficient way to do this?
Note: Unfortunately we are currently stuck supporting a minimum of SQL Server 2005, as some of our clients still use that technology
If you are looking for keywords that is the least efficient way you could do it
A like '%anything does not use an index
Use a FullText search to index the words
Or write you own parser to index the unique words
You would build up a keywords table
And index the keyword
This is a very efficient query
select id
from keywords
where keyword = 'this'
intersect
select id
from keywords
where keyword in ( 'that','other')
Even with wildcards in the keywords it is still much more efficient than searching the entire text
I hope this will help. At my work, we replaced cursor with this kind of implementation.
DECLARE
#strSQL NVARCHAR(MAX) = ''
CREATE TABLE #tmp
(
Result_Query VARCHAR(MAX)
)
INSERT INTO
#tmp
SELECT
'SELECT * FROM [VW_EMAIL_ALERT] WHERE ([SEARCH] = ''%this%'' AND ([SEARCH] = ''%that%'' OR [SEARCH] = ''%other%''))'
UNION
SELECT
'SELECT * FROM [VW_EMAIL_ALERT] WHERE ([SEARCH] = ''%this1%'' AND ([SEARCH] = ''%that1%'' OR [SEARCH] = ''%other1%''))'
SELECT
#strSQL = #strSQL + Result_Query + ';'
FROM
#tmp
SET
#strSQL = LEFT(#strSQL, LEN(#strSQL) - 1)
PRINT #strSQL
EXEC(#strSQL)
I have a MS SQL Query that is pulling data via from a remote server. The data that I'm pulling down needs to be filtered by a date that is determined at run time.. When I run the query like this:
SELECT * FROM SERVER.Database.dbo.RemoteView
WHERE EntryDate > '1/1/2009'
then the filter is applied remotely... However, I don't actually want to use '1/1/2009' as the date - I want the date to be supplied by a user-defined function, like this:
SELECT * FROM SERVER.Database.dbo.RemoteView
WHERE EntryDate > dbo.MyCustomCLRDateFunction()
where the function is a custom CLR scalar-valued function that returns a date time... (You may ask why I need to do this... the details are a bit complicated, so just trust me - I have to do it this way.)
When I run this query, the remote query is NOT filtered remotely - the filtering is done after all of the data is pulled down (400,000 rows vs 100,000 rows) and it makes a significant difference.
Is there a way that I can force the query to do the filtering remotely?
Thanks!
You could also construct a string and use an openquery ...
set #sqlString =
' select into myTable from openquery
(remoteServer,
"SELECT * FROM Database.dbo.RemoteView WHERE EntryDate > %DTSTART"
)
'
set #sqlString =
replace(#sqlString, '%DTSTART',
(select cast(dbo.MyCustomCLRDateFunction() as char(8))
)
EXECUTE sp_executesql #stmt=#sqlString
Can't you just send a query like this, or does the clr function have to actually be called inside the select statement?
Declare #datetime datetime
Set #datetime = dbo.MyCustomCLRDateFunction()
SELECT * FROM SERVER.Database.dbo.RemoteView
WHERE EntryDate > #datetime
You need to properly decorate your CLR function to mark it as Deterministic, Precise and Data Access/System Data Access as DataAccessKind.None.