In there any way to declare a variable in CrateDB to be used than in queries?
I’ve tried with a function that echo a value, but since I will show the data on Grafana, some issue rise when I try to get the data to visualize (Grafana did not recognize the function, and this result in an error...)
Thanks!
Related
I am doing something that seems like it should be very easy yet I have yet to figure this out. I have read countless posts and tried everything I can think of and still no success.
Here goes:
I created a Pipeline Parameter pplLastWritten with a default value of 2022-08-20 12:19:08 (I have tried without the time for troubleshooting and still get errors)
Then I create a Data Flow Parameter ptblTableName
I have tried to convert to a Date, keeping as is and converting later...you name it still errors out.
In the expression builder I tried this and many more ways to build out to a sql statement:
"SELECT * FROM xxxxxx."+$ptblTableName+"where Lastwritten>='{$ptblLastWritten}'"
This is the post I got the idea from: ADF data flow concat expression with single quote
This is the error I got most of the time.
Operation on target df_DynamicSelect failed: {"StatusCode":"DF-Executor-StoreIsNotDefined","Message":"Job failed due to reason: at Source 'RptDBTEST'(Line 5/Col 0): The store configuration is not defined. This error is potentially caused by invalid parameter assignment in the pipeline.","Details":""}
I have tried so many things but in the end nothing has worked. I am new to Data Factory and come from the SSIS world which was so much easier. I would greatly appreciate someone helping. Please explain this like I'm a kindergartener because this tool is making me feel like it. :) Thank you in advanced.
I have tried various ways to format
Using different ideas in the expression builder
the ideas in this post: ADF data flow concat expression with single quote
You can use concat() function in the Data flow dynamic expression like below.
Here is the sample data in SQL.
I have created two dataflow parameters mytable and mydate.
Passed the values like below. Check the expression checkbox. For date you can also pass like this '2022-11-07T00:00:00.0000000'.
In the Query option use below Expression.
concat('select * from dbo.',$table_name,' where mydate >=','\'',$mydate,'\'')
Values inserted in Target table.
I'm a complete beginner with Azure Data Factory and I'm struggling to set a parameter using a lookup output, and then use the parameter within my query to import data.
I'm using an Azure SQL database as the source for the lookup and an Oracle database as the source for the copy data
My lookup query works ok and my copy data query works without a parameter so I know I'm part way there, but I'm struggling with the parameter part.
My lookup details are below with the output query
The copy data query is below with the lookup activity as the parameter added but as you can see it's not correct.
I'm not sure if I should be using the lookup as the parameter or if I should be using a parameter with the value generated from the lookup. In which case I have no idea how to do this.
I'm also pretty sure the parameter in the query needs some work as even hardcoding a parameter doesn't seem to work.
If anyone can help to get this working I'd be very grateful.
There doesn't seem to be much help with turning lookups into parameters on the internet.
Thanks in advance for any help anyone can give.
Try enclosing the lookup output as follows. Select column from table where rowstamp> '#{activity('Lookup1').output.firstrow.RowStamp}'
I am currently using a function in SQL Server to get the max-value of a certain column. I Need this value to generate a specific number of dummy files to insert flowfiles that are created later on.
Is there a way of calling this function via a nifi-processor?
By using ExecuteSQL I Always get error like unable to execute SQL select query or the column "ab" was not found, when using select ab.functionname() (ab is the loginname of the db)
In SQL Server I can just use select ab.functionname() and get the desired results.
If there is no possible way of calling this function, is there another way to create #flowfiles dummyfiles to reserve this place for them in the DB so that no one else could insert or use this ids (not autoincremt, because it is not possible) while the flowfiles are getting processed?
I tried using $flowfile.count and the Counterprocessor, but this did not solve the Problem.
It should look like: INSERT INTO table (id,nr) values (max(id)+1,anynumber) for every flowfiles, unfortunately the ExecuteSQL is not able to do this.
Think this conversation can help you:
https://community.hortonworks.com/questions/26170/does-executesql-processor-allow-to-execute-stored.html
Gist:
You can use ExecuteScript or ExecuteProcess to call appropriate script. For example for ExecuteProcess just call sqlplus command. Choose type of command "sqlplus". In command arguments set something like: user_id/password#dbname #"script_path/someScript.sql". In someScript.sql you put something like:
execute spname(param)
You can write your own processor :) Of course it's more difficulty and often unnecessary
I'm looking how to run a local query with date filters on a saved pass-through queries in Access. I'm trying to leverage the pass through query as basically a View in my Access database... the local query is constructed dynamically in VBA, and intended to be a clean way to filter my pass through query. I'd like to avoid creating another pass through query, or altering the original, every time i run my Sub Procedure.
My problem is that the normal access date filter format #m/d/yyyy# doesn't seem to work. I've tried both altering the date format as well in the pass through query with 1. Convert(varchar(12),p.startDate,101); 2. Convert(date,p.StartDate,101);
but neither will work when the pass through query is queried against locally.
Does anyone know how to do this?
UPDATE - I just checked and Access is reading the field as Text... does anyone know how it can read it as a date? As i mentioned the CONVERT functions don't seem to be working to do this
In a passthru you MUST use the backend's syntax. If the BE is SQL Server then I'd use a syntax like this:
WHERE DocDate = '2015-03-17'
On trying to use a select query statement. The input variable has 8 characters just as expected.
I dont know why this error comes for select query because for a select query it will query and if its available it will return else it will retun blank rows.
Hibernate is used. Even in mapping its correctly mapped as 8 only.
This is what i found in log file:
Cause = com.ibm.db2.jcc.a.SqlException: The value of a host variable in the EXECUTE or OPEN statement is too large for its corresponding use.
Has anybody come across this error before? Please suggest me some solutions on why this error occurs..
One possibility... This issue can come even in SELECT statement. When the parameter passed into the query is more than the size of its datatype's size, this error will pop-up.
Example:
Datatype - CHAR(12)
Search Param: "123456789012345"