I'm trying to use Apache JMeter to perform a database load test using a list of pre-generated SQL statements from a file. The pre-generated SQL statements are various stored procedures that were captured from a trace so they have the needed values for the parameters as part of the execution statement.
I'm following the same design as a load test for an HTTP Request
from an external file by setting the variable from the CSV_Data_Set_Config as the PATH value of the HTTP Request but replacing the HTTP Request with the JDBC Request and putting the variable from the CSV_Data_Set_Config for the SQL statement. Each example I've seen only takes the contents of the file as variables into a predefined SQL statement but nothing takes each line from the file as a complete statement to execute.
In addition to using the CSV_Data_Set_Config I've tried going another route by trying to read lines from the file using the CSVRead function and putting the statement in the parameter values of the JDBC Request and using the ? to fill in the SQL Statement at runtime however it seems to cut off the line after the first period in the three part name. For example a line would be exec {database}.{owner}.{procedure} and the request would only send in exec {database}
Can you use JMeter in this manner with the JDBC Request controller?
Related
I have a source connection which is a ADO.NET MySql, I execute a stored procedure on MYSQL connection which gives me the data set, as we know SSIS tries to read the data from few rows and define the metadata when I run this procedure it defines the data based on data which it receives, however I run this procedure in loops for different sites (server and connection details are parameterized) columns and data types are same for all the sites, since SSIS reads only few columns while defining the metadata it doesn't get the data length and precision's correctly and they error out at the source connection itself, I am writing these errors to the flat file to process them later, when SSIS writes the error it replaces actual value with NULL.
For Example for column which is defined as Numeric(4,2) when it encounters the values such as 243.32 instead of writing the actual value to error file it writes a NULL value.
Is there anyway we can make SSIS write the actual value to the error file, Below is the Job Flow, Appreciate any inputs on how we can handle this,
Explained in the details of the problem
I need some advise on solving this requirement for auditing purpose . I am using airflow composer - dataflow java operator job which spits out json file after job completion with status and error message details (into airflow data folder ) . I want to extract the status and error message from json file via some operator and then pass the variable to next pipeline job Bigqueryinsertjoboperator which calls the stored proc and passes status and error message as input parameter and finally gets written into BQ dataset table.
Thanks
You need to do XCom and JINJA templating. When you return meta-data from the operator, the data is stored in XCom and you can retrieve it using JINJA templating or Python code in Python operator (or Python code in your custom operator).
Those are two very good articles from Marc Lamberti (who also has really nice courses on Airlfow) describing how templating and jinja can be leveraged in Airflow https://marclamberti.com/blog/templates-macros-apache-airflow/ and this one describes XCom: https://marclamberti.com/blog/airflow-xcom/
By combining the two you can get what you want.
Requirement: From an UI I am getting selected list of tables of a database. Data from these tables are to extracted and stored in file location. We are expected to use NiFi Rest APIs as there is a requirement for custom UI. So we are invoking NiFi processors using REST APIs.
Issue: With ExecuteSQL processor I can execute one SQL query at a time. Since the query is same for all of the tables (select * ...) I can pass in the table name as an attribute to the processor. But I am getting the table names from a REST API call so its delimited string.
the issue is how to invoke the ExecuteSQL in a loop for each table name received in a delimited String.
Kindly let me know if additional information is required.
ExecuteSQL can't loop on a single flow file. Instead you can use ReplaceText to put the delimited string into the body of the flow file, then use SplitText to split on the delimiter. You will get multiple flow files, each with the body containing one of the fields. Then you can use ReplaceText again to match the entire text, and replace with the SELECT statement around the group:
SELECT * from $1
Then you can send all flow files to ExecuteSQL, it will execute each SELECT one at a time (or concurrently if you set multiple concurrent tasks.
I have a question using CSV file as an input for Jmeter. Is it possible to use CSV file for storing data like select statements or update statements and then passing all that to Jmeter in JDBC request so it can execute them accordingly.
I'd appreciate any suggestion or pointing me to the right solution.
Thanks
You can create a variable from a csv file using the CSV Data Set Config http://jmeter.apache.org/usermanual/component_reference.html#CSV_Data_Set_Config
Then you just need to use them in your Jdbc Sampler.
The problem here is that you must have you sql on one line.
You can also create variable from a select JDBC Sample. (See variable Names)
http://jmeter.apache.org/usermanual/component_reference.html#JDBC_Request
Yes you can do that.your statements need to be in different column of the CSV file.
define 2 variable names in CSV file and call them in JDBC Request file ${variable name given in CSV file}.
I have a specific SQL file that may be "connected" to another, more generic SQL init file.
Is it possible to somehow include reference from one SQL file to another SQL file?
I am using Oracle and the DB is populated using Spring DataSourceInitializer class.
If you are using SQL*Plus to run your script, you can use the # (or ##) sign to include another SQL script.
See the manual for details:
http://download.oracle.com/docs/cd/B19306_01/server.102/b14357/ch12002.htm#i2696724
http://download.oracle.com/docs/cd/B19306_01/server.102/b14357/ch12003.htm#i2696759