I am pretty new with spark. I have a task to fetch 3M record from a sql server through denodo data platform and write into s3. In sql server side it is a view on join of two tables. The view is time consuming.
Now I am trying to run a spark command as:
val resultDf = sqlContext.read.format("jdbc").option("driver","com.denodo.vdp.jdbc.Driver").option("url", url).option("dbtable", "myview").option("user", user).option("password", password)
I can see that spark is sending query like:
SELECT * FROM myview WHERE 1=0
And this portion is taking more than an hour.
Can anyone please tell me why the where clause is appending here?
Thanks.
If I'm understanding your issue correctly, Spark is sending SELECT * FROM myview WHERE 1=0 to the Denodo Server.
If that is the case, that query should be detected by Denodo as a query with no results due to incompatible conditions in the WHERE clause and the execution should be instantaneous. You can try to execute the same query in Denodo's VQL Shell (available in version 6), Denodo's Administration Tool or any other ODBC/JDBC client to validate that the query is not even sent to the data source. Maybe Spark is executing that query in order to obtain the output schema first?
What version of Denodo are you using?
I see this is an old thread -- however we are experiencing the same issue -- however it does not occur all of the time nor on all connections/queries -
SQOOP command is sent -- the AND (1=0) context ('i18n' = 'us_est') is added somewhere --we are using Denodo 7 -- jdbc driver com.denodo.vdp.jdbc.Driver
select
BaseCurrencyCode,BaseCurrencyName,TermCurrencyCode,TermCurrencyName,
ExchangeAmount,AskRate,BidRate,MidMarketRate,ExchangeRateStartDate,
ExchangeRateEndDate,RecCreateDate ,LastChangeDate
from
CurrencyExchange
WHERE
LastChangeDate > '2020-01-21 23:20:15'
And LastChangeDate <= '2020-01-22 03:06:19'
And (1 = 0) context ('i18n' = 'us_est' )
Related
I would like to find out which version of DB2 we are running on our IBM i server using only SQL SELECT.
I am executing my queries via installed ODBC drivers for i Access. The places I am executing the queries are Excel-ODBC and Excel-Microsoft Query (simply because I am not a developer and therefore don't have/don't know of another place to run queries).
The following solutions do not work for me:
How to check db2 version
Get DB2 instance name using SQL
Basic reasons why I have failed to get the above solutions to work:
I do not have a SYSPROC table/have access to SYSPROC table
SYSIBMADM table does not contain a ENV_INST_INFO table.
I think these answers may be tailored to those using IBM z, but I use IBM i.
My end goal is to be able to execute a SQL SELECT and get the version of DB2 used on our server.
Try this:
SELECT RELEASE_LEVEL, TEXT_DESCRIPTION
FROM QSYS2.SOFTWARE_PRODUCT_INFO
WHERE PRODUCT_ID = '5770SS1'
AND PRODUCT_OPTION = '27'
--or this instead of the above line:
--AND LOAD_TYPE = 'CODE' AND PRODUCT_OPTION = '*BASE'
I am trying to run a very simple query that just select all the rows based upon some multiple criteria and all are inclusive i.e. I am using AND in select HiveQL statement. The table is an external table in Hive and the storage handler is phoenix, so I checked in phoenix also about that query and it is working fine, but in Hive, it is showing some java IO exception error which I am not able to get where I am wrong. The query I am using is:
SELECT * FROM msme_api.msme_jk WHERE gender='Male' AND socialcategory='General';
The complete error message is:
Error: java.io.IOException: org.apache.hadoop.hive.ql.exec.UDFArgumentLengthException: The operator 'AND' accepts at least 2 argument. (state=,code=0)
I am trying for external and for internal Hive tables, In both, the issues are still the same but when I give order by, OR statement then it's surprising that it works.
SELECT * FROM msme_api.msme_jk WHERE gender='Male' AND socialcategory='General' order by aid;
SELECT * FROM msme_api.msme_jk WHERE gender='Male' OR socialcategory='General';
Both works fine but with AND, I am getting the error.
I still confused about how hive is taking and processing the above queries and why I am not able to execute simple select statement. Any help will be appreciated.
I am doing local development for SAP CAP with Nodejs project in VSCODE. I am connected to XSA Server. I am trying to create hana database user in XSA environment. I want to first check if the list of users that my app will create is already existing or not. For that, I am trying to execute a select query with where clause. Its a prepared statement, which i am trying to execute with execBatch(array). Below is the code.
let arr = [["POC_ADMIN_DEMO_USER_7"],["POC_ADMIN_DEMO_USER_1"]]
const checkuserexiststatement = await xsaDbConn.preparePromisified("SELECT USER_NAME FROM USERS WHERE USER_NAME = ?")
let readuserresult = await checkuserexiststatement.execBatch(arr)
console.table(readuserresult)
The execution of the query fails with the following error -
Error
Error: SQL command would generate a result set at
C:\Users\Documents\XSA\SAC_POC\cap_njs\cap_njs\user_management.js.createUsers
(C:\Users\Documents\XSA\SAC_POC\cap_njs\cap_njs\user_management.js:59:60)
I want to know if select query/statement supports execBatch() functionality in Hana as the same select statement works without any placeholder that is when the value of the user_name is provided directly in the where clause and exec() used instead of execBatch(), Or is it that I am missing some point here?
execBatch() - returns no of rows affected as a result of the query executed. Select query returns a result set (if any) or else an empty array in Node.js.
Therefore execBatch() is not compatible with Select query because execBatch() is used for batch execution and selection is one shot instead of batch.
I'm getting the below error on Tableau Desktop when using the custom query, I'm successfully able to connect and see contents from the table when directly drag the table to query builder section on Tableau Desktop.
Datasource used: AWS Athena
Driver version: AthenaJDBC42_2.0.2
Tableau Desktop version: 10.4
com.tableausoftware.jdbc.TableauJDBCException: Error reading metadata for executed query: SELECT * FROM ( select * from tablename ) "TableauSQL" LIMIT 0 [Simba][AthenaJDBC](100071) An error has been thrown from the AWS Athena client. Only one sql statement is allowed. Got: SELECT * FROM ( select * from tablename ;) "TableauSQL" LIMIT 0
There was a Java error.
Unable to connect to the server "Athena.us-east-1.amazonaws.com". Check that the server is running and that you have access privileges to the requested database.
Try removing the ; from the subquery - That always throws errors for me
I was having the same issue.
To provide some clarity to the above - the outer query is being added by Tableau. Here is my simple query and the error being returned by Tableau.
I was able to resolve by changing my query to refer to my table in the form "database"."table" and remove the closing ;
This custom query then worked for me:
Hi i connected Hive using DB visualizer and fired a simple join query to fetch two columns according to the filter applied. But the query was running for more than an hour with the status "Executing". I fired the same query in Hive logging through Putty and got the result in less than 20 seconds.
Can anyone help me to understand why the query in DB visualizer was running for a long time without producing any output.
Query used:
SELECT
A.ORDER,
B.ORDER1
FROM
ORDER A
INNER JOIN DUORDER B ON A.ORDER=B.ORDER1 AND A.TYPE ='50'
(The result set contain only 400 records)
To analyze why, we need more info. Please please open Tools->Debug Window in DbVisualizer and enable debugging (just for DbVisualizer, not JDBC). Execute the query again, stopping it after some time (say a few minutes). Then submit a support request using Help->Contact Support, and make sure that Attach Logs is enabled. This will give us the info we need to see what may be wrong.
Best Regards,
Hans (DbVisualizer team)