In the custom SQL node of mosaic-decisions, I used the system parameter “current-time” . However, in the output, it is coming in as “Text” type – instead of “Date” type. What should be done to get it resolved as date in the output?
When you use any parameters in custom SQL node, input reader SQL, writer node pre/post SQL etc. (in short any SQL option), Mosaic Decisions replaces the parameter value inside the SQL as is.
So the below SQL:
SELECT COL1,
'$(currentTime)' as T1
FROM TABLE1..
will be converted as:
SELECT COL1,
'2020-06-17 21:44:55' as T1
FROM TABLE1..
before it is run on Spark. If you want to get this column as date/timestamp, you will have to cast it explicitly.
However, also note that, if you are using these parameters in any other nodes (say for example, custom-select node), that will appear as date type only.
Related
I am quite new to GBQ and any help is appreciated it.
I have a query below:
#Standard SQL
create or replace table `xxx.xxx.applications`
as select * from `yyy.yyy.applications`
What I need to do is to add today's date at the end of the table name so it is something like xxx.xxx.applications_<todays date>
basically create a filename with Application but add date at the end of the name applications.
I am writing a procedure to create a table every time it runs but need to add the date for audit purposes every time I create the table (as a backup).
I searched everywhere and can't get the exact answer, is this possible in Query Editor as I need to store this as a Proc.
Thanks in advance
BigQuery doesn't support dynamic SQL at the moment which means that this kind of construction is not possible.
Currently BigQuery supports Parameterized Queries but its not possible to use parameters to dynamically change the source table's name as you can see in the provided link.
BigQuery supports query parameters to help prevent SQL injection when
queries are constructed using user input. This feature is only
available with standard SQL syntax. Query parameters can be used as
substitutes for arbitrary expressions. Parameters cannot be used as
substitutes for identifiers, column names, table names, or other parts
of the query.
If you need to build a query based on some variable's value, I suggest that you use some script in SHELL, Python or any other programming language to create the SQL statement and then execute it using the bq command.
Another approach could be using the BigQuery client library in some of the supported languages instead of the bq command.
I have a table with a 'query' field containing a select sql and another 'parameters' field containing the sql parameters. I have merged these two fields into a new field containing a correct select sql statement. Now I need to execute this new field containing select sql, get the return from select (the output fields) and generate an excel file.
Use Table-Input if you are interested in a query result set. Table-Input supports SQL parameters, so no need to build the statement yourself using e.g. Replace-In-String, and tripping over escapes on your way. Also, there's variable substitution, just in case you can't live with a single template.
Update 21:14 GMT
I'm not very fond of the way you try to prepare the SELECT statement, but here we go, assuming it's a single statement we have:
Create a job with a Start entry and 2 Transformation entries (T1, T2). Let T1 produce the field containing your SELECT statement and use a Set-Variables step to make the statement available to T2 as variable SELECT. In T2 use a Table-Input step referencing ${SELECT} in the SQL statement text area. Don't forget to enable option "Replace variables in script".
From now on it's a matter of taste. I would prefer to create a CSV file using Text-File-Output. Using the right field separator Excel will open the file after double-clicking it. The advantage of Text-File-Output is that you don't have to specify the fields you don't know at design-time anyway. An empty field list will just handle all fields coming in. Comparable to the total projection in a Table-Input which will create the necessary fields from the retrieved columns downstream.
If you must produce an Excel workbook, you'll have to learn about metadata injection. That would be a separate project for a beginner, though. There are samples in your Kettle installation folder. And there is a very active community if you find yourself in trouble.
I am using pentaho for data migration testing. I have set a "table input" step where many parts of the query inside "table inputs" are variables. I have been looking for a way to capture that query after it gets executed during runtime.
I was wondering if there is any specific system log variables for sql or is it to do with metadata. need help! Thanks
Maybe the following approach will help:
We assume a transformation reading a CSV file to get the dynamic portion of the SELECT statement (e.g. the columns) and setting the variable columns with it.
The second transformation uses this variable to generate the SELECT statement and store it into the variable sql_statement.
In the main transformation we use ${sql_statement} as the SELECT statement of the table input and write the data to an output file (that's the business process so to say). From the same input we copy the output to another path. There we add the current time as a field (use element "Get system data") and we add the generated SQL statement, join them as a cartesian product and group the result by the sql_statement. That way we can compute the first time and the last time that the statement was used. These results are written to a text file.
The last thing we need is a job calling the three transformations sequentially.
This is a sample output:
sql_statement;min_time;max_time
SELECT my_column FROM test_table;2014/05/08 00:41:21.143;2014/05/08 00:41:21.144
Thank you Marcus! I did some thing similar.
It works. awesome.
I gathered parts of queries from table field where they were kept and formed a full query in javascript. After that full query will be sent as parameter to a transformation that will run and log the query.
I have a query like
SELECT *
FROM myTable
WHERE key LIKE 'XYZ'
The value 'XYZ' is entered by users (and may include % and _)
If I construct the query using string concatenation it runs in 10 seconds.
But this is unsafe, and I should use a parameterised query.
So I'm constructing the query using the odbc command object and it's execute method, and passing a parameter.
SELECT *
FROM myTable
WHERE key LIKE ?
Unfortunately the parameterised SQL execute method takes a full minute.
This query is one of many that are part of a drill-down / investigation package, and I've had similar slow downs with all the parameterised queries (compared to string concatenation).
How do I find out where the time is going (and fix it) ?
Here's my guess without further information.
I've had similar problems on SQL Server. In SQL Server when the column on your table is 'varchar' and the parameterised query parameter is 'nvarchar' (or vice versa), this causes SQL Server to ignore an available index because the parameter type doesn't match the index type, which in turn results in a table scan.
It's possible the same thing happens for Sybase. If you can see the generated query you can confirm if there's a type mismatch.
If this is the case, then two solutions would be
explicitly set the type of the parameter to match the column type
change the type of the column to match the parameter type being generated
Mitch had the right suggestion.
I had to change the connection string to use the OLEDB driver, then I could set the options:
Optimize Prepare=None
Select Method=Direct
We have a SQL Server table for user settings. Originally the settings were domain objects which had been serialized as XML into the table but we recently begun serializing them as binary.
However, as part of our deployment process we statically pre-populate the table with predefined settings for our users. Originally, this was as simple as copying the XML from a customized database and pasting it into an INSERT statement that was ran after the database was built. However, since we've moved to storing the settings as binary data we can't get this to work.
How can we extract binary data from a varbinary column in SQL Server and paste it into a static INSERT script? We only want to use SQL for this, we don't want to use any utilities.
Thanks in advance,
Jeremy
You may find it easier to store a template value in a config table somewhere, then read it into a variable and use that variable to fill your inserts:
DECLARE #v varbinary(1000)
SELECT #v = templatesettings from configtable
INSERT INTO usertable VALUES(name, #v, ....)
From SQL Server 2008 onwards you can use Tasks > Generate Scripts and choose to include data. That gives you INSERT statements for all rows in a table which you can modify as needed.
Here's the steps for SQL 2008. Note that the "Script Data" option in SQL 2008 R2 is called "Types of data to script" instead of "Script Data".
I presume you're OK with utilities like Query Analyzer/Mangement Studio?
You can just copy and paste the binary value returned by your select statement (make sure that you are returning sufficient data), and prefix it with "0x" in your script.
If I understand you correctly, you want to generate a static script from your data. If so, consider performing a query on the old data that concatenates strings to form the SQL statements you'll want in the script.
First, figure out what you want the scripted result to look like. Note that you'll need to think of the values you're inserting as constants. For example:
INSERT INTO NewTable VALUES 'value1', 'value2'
Now, create a query for the old data that just gets the values you'll want to move, like this:
SELECT value1, value2
FROM OldTable
Finally, update your query's SELECT statement to produce a single concatenated string in the form of the output you previous defined:
SELECT 'INSERT INTO NewTable VALUES ''' + value1 + ''', ''' + value2 + ''''
FROM OldTable
It's a convoluted way to do business, but it gets the job done. You'll need a close attention to detail. It will allow a small (but confusing) query to quickly output very large numbers of static DML statements.
David M's suggestion of using the 0x prefixing works but i had to add an extra 0 at the end of varbinary data that i was trying to insert.
See the stackoverflow entry below to see the issue with additional 0 that gets added when converting to varbinary or saving to varbinary column
Insert hex string value to sql server image field is appending extra 0