I love that I can parameterize SQL queries in Pycharm. I an write a statement like so
select * from table where date >= ?
and will then be prompted for the value to be filled in for the ?.
Alas, I will be prompted afresh every time. Is there a way of specifying a set of default parameters? Bonus points for being able to pull these parameter values from a JSON/yaml file.
Related
I currently use the following to dynamically alter the date range in my queries:
'${MY_DATE}'::timestamp
Is there a way to dynamically populate this rather than have the user input the value in a pop-up? I'd like to be able to run the query and it would automatically select the maximum date that exists in the table, add one day to that value and use that in the dynamic parameter throughout the queries in my script.
I've tried, something like this but it still generates the pop-up:
insert into '${MY_DATE}'::timestamp select max(SOME_DATE_FIELD) from MY_TABLE;
Thank you!
Can I define inside a "select value step" a variable?
like ${column_name}
How can I make it work, I have a parameter that will change this column every time.
That step doesn't allow using variables to select the columns of the stream you want to keep.
In this case you could use Metadata Injection, read Pentaho documentation as well as study the example in the samples folder of your installation on how to use Metadata Injection.
You just need to check the box in the step select values
I am currently using a function in SQL Server to get the max-value of a certain column. I Need this value to generate a specific number of dummy files to insert flowfiles that are created later on.
Is there a way of calling this function via a nifi-processor?
By using ExecuteSQL I Always get error like unable to execute SQL select query or the column "ab" was not found, when using select ab.functionname() (ab is the loginname of the db)
In SQL Server I can just use select ab.functionname() and get the desired results.
If there is no possible way of calling this function, is there another way to create #flowfiles dummyfiles to reserve this place for them in the DB so that no one else could insert or use this ids (not autoincremt, because it is not possible) while the flowfiles are getting processed?
I tried using $flowfile.count and the Counterprocessor, but this did not solve the Problem.
It should look like: INSERT INTO table (id,nr) values (max(id)+1,anynumber) for every flowfiles, unfortunately the ExecuteSQL is not able to do this.
Think this conversation can help you:
https://community.hortonworks.com/questions/26170/does-executesql-processor-allow-to-execute-stored.html
Gist:
You can use ExecuteScript or ExecuteProcess to call appropriate script. For example for ExecuteProcess just call sqlplus command. Choose type of command "sqlplus". In command arguments set something like: user_id/password#dbname #"script_path/someScript.sql". In someScript.sql you put something like:
execute spname(param)
You can write your own processor :) Of course it's more difficulty and often unnecessary
I am quite a beginner with SSIS packages so bear with me. What I am trying to do:
I have daily files in the format of yyyy-mm-dd_filename_bla_bla.tsv
The date of the file need to be added in the table were I am trying to import it. Currently I am doing this manually with a derived coloumn with the expression: (DT_DATE)(DT_DBDATE)"yyyy-mm-dd"
Is there a possibility to automatically take the file name and only take the date part to import it into the table.
The things I find on the internet is getting the date into the file name, but this is exactly the opposite.
I hope I provided enough information, and anyone can help me out with this problem.
thanks in advance.
If you know the file name then keep that file name in a variable
example let file name be : 01/02/2015_kjh.bgd
then by using derived column use string functions as left(#variable,10)
10-> length of date
then map it to your oledb destination
To update/insert into a table you can use the SSIS SQL Exec task and pass in the variable value (which you know how to get) into the update/insert statement as a parameter. Its not hard just be careful on your settings of parameter names 0.
Here is a detailed description on how to do exactly what you are asking once you have a parameter set up:
How to pass variable as a parameter in Execute SQL Task SSIS?
I have a decent size SQL statement that I have connected to an Excel worksheet and it runs fine. The question I have is would it be possible to have the end user enter a list of values in an excel sheet or somewhere else and have those values added to a WHERE clause in my SQL to limit the results per the users needs without the user having to go into connection and alter the SQL etc.? Thank you.
Yes. I do it as part of a VB script. I'm not a VB expert and I didn't set it up- I just know enough to tweak some changes that the users require.
Conceptually- provide an area in th Excel template for users to enter the parameters, use VB to get the values of those parameters, then pass them to the SQL statement.
You can create a Data Connection in Excel that utilizes a parameter. Whether you are calling a Stored Procedure on the DB or sending raw SQL, you can replace part of the statement with a ?. Tie that parameter to a specific cell. Now all your user needs to do is enter/change the value in that cell and it will re-query the database using that value as the parameter.
(I don't have Excel in front of me, otherwise I'd walk you through the exact steps)