Pyodbc and Access with query parameter that contains a period - sql

I recently found a bug with some Access SQL queries that I can't seem to track down. I have a fairly straightforward SQL query that I use to retrieve data from an access database that's "managed" in an older application (ie the data is already in the database and I have no real control over what's in there).
import pyodbc
MDB = '******.MDB'
DRV = '{Microsoft Access Driver (*.mdb)}'
PWD = ''
con = pyodbc.connect('DRIVER={};DBQ={};PWD={}'.format(DRV, MDB, PWD))
sql = ('SELECT Estim.PartNo, Estim.Descrip, Estim.CustCode, Estim.User_Text1, Estim.Revision, ' +
'Estim.Comments, Routing.PartNo AS RPartNo, Routing.StepNo, Routing.WorkCntr, Routing.VendCode, ' +
'Routing.Descrip AS StepDescrip, Routing.SetupTime, Routing.CycleTime, ' +
'Routing.WorkOrVend, ' +
'Materials.PartNo as MatPartNo, Materials.SubPartNo, Materials.Qty, ' +
'Materials.Unit, Materials.TotalQty, Materials.ItemNo, Materials.Vendor ' +
'FROM (( Estim ' +
'INNER JOIN Routing ON Estim.PartNo = Routing.PartNo ) ' +
'INNER JOIN Materials ON Estim.PartNo = Materials.PartNo )')
if 'PartNo' in kwargs:
key = kwargs['PartNo']
sql = sql + 'WHERE Estim.PartNo=?'
cursor = con.cursor().execute(sql, key)
# use this for debuging only
num = 0
for row in cursor.fetchall():
num += 1
return num
This works fine for all PartNo except when PartNo contains a decimal point. Curiously, when PartNo contains a decimal point AND a hyphen, I get the appropriate record(s).
kwargs['PartNo'] = "100.100-2" # returns 1 record
kwargs['PartNo'] = "200.100" # returns 0 records
Both PartNos exist when viewed in the other application, so I know there should be records returned for both queries.
My first thought was to ensure kwargs['PartNo'] is a string key = str(kwargs['PartNo']) with no change.
I also tried to places quotes around the 'PartNo' value with no success. key = '\'' + kwargs['PartNo'] + '\''
Finally, I tried to escape the . with no success (I realize this would break most queries, but I'm just trying to track down the issue with a single period) key = str(kwargs['partNo']).replace('.', '"."')
I know using query parameters should handle all the escaping for me, but at this point, I'm just trying to figure out what's going on. Any thoughts on this?

So the issue isn't with the query parameters - everything works as it should. The problem is with the SQL statement. I incorrectly assumed - and never checked - that there was a record in the Materials table that matched PartNo.
INNER JOIN Materials ON Estim.PartNo = Materials.PartNo
will only return a record if PartNo is found in both tables, which in this particular case it is not.
Changing it to
LEFT OUTER JOIN Materials ON Estim.PartNo = Materials.PartNo
produces the expected results. See this for info on JOINS. https://msdn.microsoft.com/en-us/library/bb243855(v=office.12).aspx
As for print (repr(key)) - flask handles the kwarg type upstream properly
api.add_resource(PartAPI, '/api/v1.0/part/<string:PartNo>'
so when I ran this in the browser, I got the "full length" strings. When run in the cmd line using python -c ....... I was not handling the argument type properly as Gord pointed out, so it was truncating the trailing zeros. I didn't think the flask portion was relevant, so I never added that in the original question.

Related

How to write an Open SQL statement with substring in the JOIN ON condition? [duplicate]

I have the following select statement in ABAP:
SELECT munic~mandt VREFER BIS AB ZZELECDATE ZZCERTDATE CONSYEAR ZDIMO ZZONE_M ZZONE_T USAGE_M USAGE_T M2MC M2MT M2RET EXEMPTMCMT EXEMPRET CHARGEMCMT
INTO corresponding fields of table GT_INSTMUNIC_F
FROM ZCI00_INSTMUNIC AS MUNIC
INNER JOIN EVER AS EV on
MUNIC~POD = EV~VREFER(9).
"where EV~BSTATUS = '14' or EV~BSTATUS = '32'.
My problem with the above statement is that does not recognize the substring/offset operation on the 'ON' clause. If i remove the '(9) then
it recognizes the field, otherwise it gives error:
Field ev~refer is unknown. It is neither in one of the specified tables
nor defined by a "DATA" statement. I have also tried doing something similar in the 'Where' clause, receiving a similar error:
LOOP AT gt_instmunic.
clear wa_gt_instmunic_f.
wa_gt_instmunic_f-mandt = gt_instmunic-mandt.
wa_gt_instmunic_f-bis = gt_instmunic-bis.
wa_gt_instmunic_f-ab = gt_instmunic-ab.
wa_gt_instmunic_f-zzelecdate = gt_instmunic-zzelecdate.
wa_gt_instmunic_f-ZZCERTDATE = gt_instmunic-ZZCERTDATE.
wa_gt_instmunic_f-CONSYEAR = gt_instmunic-CONSYEAR.
wa_gt_instmunic_f-ZDIMO = gt_instmunic-ZDIMO.
wa_gt_instmunic_f-ZZONE_M = gt_instmunic-ZZONE_M.
wa_gt_instmunic_f-ZZONE_T = gt_instmunic-ZZONE_T.
wa_gt_instmunic_f-USAGE_M = gt_instmunic-USAGE_M.
wa_gt_instmunic_f-USAGE_T = gt_instmunic-USAGE_T.
temp_pod = gt_instmunic-pod.
SELECT vrefer
FROM ever
INTO wa_gt_instmunic_f-vrefer
WHERE ( vrefer(9) LIKE temp_pod ). " PROBLEM WITH SUBSTRING
"AND ( BSTATUS = '14' OR BSTATUS = '32' ).
ENDSELECT.
WRITE: / sy-dbcnt.
WRITE: / 'wa is: ', wa_gt_instmunic_f.
WRITE: / 'wa-ever is: ', wa_gt_instmunic_f-vrefer.
APPEND wa_gt_instmunic_f TO gt_instmunic_f.
WRITE: / wa_gt_instmunic_f-vrefer.
ENDLOOP.
itab_size = lines( gt_instmunic_f ).
WRITE: / 'Internal table populated with', itab_size, ' lines'.
The basic task i want to implement is to modify a specific field on one table,
pulling values from another. They have a common field ( pod = vrefer(9) ). Thanks in advance for your time.
If you are on a late enough NetWeaver version, it works on 7.51, you can use the OpenSQL function LEFT or SUBSTRING. Your query would look something like:
SELECT munic~mandt VREFER BIS AB ZZELECDATE ZZCERTDATE CONSYEAR ZDIMO ZZONE_M ZZONE_T USAGE_M USAGE_T M2MC M2MT M2RET EXEMPTMCMT EXEMPRET CHARGEMCMT
FROM ZCI00_INSTMUNIC AS MUNIC
INNER JOIN ever AS ev
ON MUNIC~POD EQ LEFT( EV~VREFER, 9 )
INTO corresponding fields of table GT_INSTMUNIC_F.
Note that the INTO clause needs to move to the end of the command as well.
field(9) is a subset operation that is processed by the ABAP environment and can not be translated into a database-level SQL statement (at least not at the moment, but I'd be surprised if it ever will be). Your best bet is either to select the datasets separately and merge them manually (if both are approximately equally large) or pre-select one and use a FAE/IN clause.
They have a common field ( pod = vrefer(9) )
This is a wrong assumption, because they both are not fields, but a field an other thing.
If you really need to do that task through SQL, I'll suggest you to check native SQL sentences like SUBSTRING and check if you can manage to use them within an EXEC_SQL or (better) the CL_SQL* classes.

SQL query with "as" keyword

I have a below SQL query running in one of my project. I am struggling to understand the "as" concept here. In the result "user_key" and "user_all" are appearing as empty. Where as at the front end "user_all" is the combination of "rx.ord_by_userid" + "rx.ord_by_inst_id,"
SELECT rx.rx_id,
rx.pt_visit_id,
rx.pt_id,
pt_visit.date_time_sch,
' ' as print_dea_ind,
' ' as phys_rx_label,
rx.ord_by_userid,
rx.ord_by_inst_id,
' ' as user_key,
pt_visit.visit_inst_id,
' ' as user_all,
' ' as tp_agt_ind,
FROM rx LEFT OUTER JOIN tx_pln ON rx.tp_name = tx_pln.tp_name AND rx.tp_vers_no = tx_pln.tp_vers_no, pt_visit
WHERE ( pt_visit.pt_visit_id = rx.pt_visit_id ) and
( pt_visit.pt_id = rx.pt_id ) and
( ( rx.pt_id = :pt_id ) and
( rx.rx_id = :rx_id ) )
Thanks.
I think when they query database, they need two fields called "user_key" and "user_all" with empty value for some purpose. However, in the front end, they need to display column "user_all" with the combination of "rx.ord_by_userid" + "rx.ord_by_inst_id" because of business rule.
The meaning of "AS" is just setting the alias of any field which is needed to have a new name. In this situation, new columns "user_key" and "user_all" are set with empty value.
AS just provides the field in the data set a name, or in SQL terms, an alias. In PB, this is usually done so that the DataWindow gives it a consistent, easy name. That is all that AS does.
The other part of your mystery is how these get populated with non-blank values. You were assuming this was done in the SQL with AS, but we can assure you that is not the case. Most likely, this value is being set in a script that fires in the client after the Retrieve() (if I were to bet, I'd bet a script on the DataWindow control, maybe RetrieveRow or RetrieveEnd).

Gain more Results on enhancing the query on one specific Server

I wrote an Access add-in (VBA) which works perfectly fine on my and other test servers, but on one server I encountered a very strange problem. When I execute the query with the data condition, I get the correct results (which are none). When I add the date condition I get more results than before. These results do not even match with my other conditions. I use the exact same database on both servers.
Here is the Query:
select BE.* from dbo.KHKBuchungserfassung AS BE
left join KHKSachkonten AS SSK on BE.KtoSoll = SSK.SaKto and BE.Mandant = SSK.Mandant
left join KHKSachkonten AS HSK on BE.KtoHaben = HSK.SaKto and BE.Mandant = HSK.Mandant
where ((((BE.KtoSollTyp = 1) or (BE.KtoSollTyp = 2)) and (HSK.Kontenart = 'BF' or HSK.Kontenart = 'BG')) or
(((BE.KtoHabentyp = 1) or (BE.KtoHabentyp = 2)) and (SSK.Kontenart = 'BF' or SSK.Kontenart = 'BG')))
and BE.Mandant= 88 and BE.Buchungsdatum>={d '2016-01-01'} AND BE.Buchungsdatum<={d '2016-06-30'}
If I execute this on that specific server with from date < a specific date ('2016-02-24') it also works perfectly fine (e.g. '2016-01-01' & '2016-06-30'). Any given date higher than this one gives results that violate my first condition... (the one in the parenthesis). I already checked the number of parenthesis. I also rotated the places of the given conditions. In addition to that, I am pretty sure my DBMS would give me an error if there is a problem with my syntax (not sure though).
This is the Result Set:
Dim rs As ADODB.Recordset
sQry = ...
Set rs = goMandant.oData.rsOpenRecordset(sQry, adOpenStatic)
Do Until rs.EOF
...
I am sorry that I can not provide test data due to juridical reasons, but if you would also select SSK.Kontenart and HSK.Kontenart there would be 'NULL' on both sides which is not at all possible in my first condition...
I would also like to note that this is my first question and if I made any mistakes, I would appreciate it if you told me.
Regards
TK

Using the results of a select sub query as the columns to select in the main query. Injection?

I have a table that contains a column storing sql functions, column names and similar snippets such as below:
ID | Columsql
1 | c.clientname
2 | CONVERT(VARCHAR(10),c.DOB,103)
The reason for this is to use selected rows to dynamically create results from the main query that match spreadsheet templates. EG Template 1 requires the above client name and DOB.
My Subquery is:
select columnsql from CSVColumns cc
left join Templatecolumns ct on cc.id = ct.CSVColumnId
where ct.TemplateId = 1
order by ct.columnposition
The results of this query are 2 rows of text:
c.clientname
CONVERT(VARCHAR(10),c.DOB,103)
I would wish to pass these into my main statement so it would read initially
Select(
select columnsql from CSVColumns cc
left join Templatecolumns ct on cc.id = ct.CSVColumnId
where ct.TemplateId = 1
order by ct.columnposition
) from Clients c
but perform:
select c.clientname, CONVERT(VARCHAR(10),c.DOB,103) from clients c
to present a results set of client names and DOBs.
So far my attempts at 'injecting' are fruitless. Any suggestions?
You can't do this, at least not directly. What you have to do is, in a stored procedure, build up a varchar/string containing a complete SQL statement; you can execute that string.
declare #convCommand varchar(50);
-- some sql to get 'convert(varchar(10), c.DOB, 103) into #convCommand.
declare #fullSql varchar(1000);
#fullSql = 'select c.clientname, ' + #convCommand + ' from c,ients c;';
exec #fullSql
However, that's not the most efficient way to run it - and when you already know what fragment you need to put into it, why don't you just write the statement?
I think the reason you can't do that is that SQL Injection is a dangerous thing. (If you don't know why please do some research!) Having got a dangerous string into a table - e.g 'c.dob from clients c;drop table clients;'- using the column that contains the data to actually execute code would not be a good thing!
EDIT 1:
The original programmer is likely using a C# function:
string newSql = string.format("select c.clientname, {0} from clients c", "convert...");
Basic format is:
string.format("hhh {0} ggg{1}.....{n}, s0, s1,....sn);
{0} in the first string is replaced by the string at s0; {1} is replaces by tge string at s1, .... {n} by the string at sn.
This is probably a reasonable way to do it, though why is needs all the fragments is a bit opaque. You can't duplicate that in sql, save by doing what I suggest above. (SQL doesn't have anything like the same string.format function.)

Multiple parameter values

I have a problem with BIRT when I try to pass multiple values from report parameter.
I'm using BIRT 2.6.2 and eclipse.
I'm trying to put multiple values from cascading parameter group last parameter "JDSuser". The parameter is allowed to have multiple values and I'm using list box.
In order to be able to do that I'm writing my sql query with where-in statement where I replace text with javascript. Otherwise BIRT sql can't get multiple values from report parameter.
My sql query is
select jamacomment.createdDate, jamacomment.scopeId,
jamacomment.commentText, jamacomment.documentId,
jamacomment.highlightQuote, jamacomment.organizationId,
jamacomment.userId,
organization.id, organization.name,
userbase.id, userbase.firstName, userbase.lastName,
userbase.organization, userbase.userName,
document.id, document.name, document.description,
user_role.userId, user_role.roleId,
role.id, role.name
from jamacomment jamacomment left join
userbase on userbase.id=jamacomment.userId
left join organization on
organization.id=jamacomment.organizationId
left join document on
document.id=jamacomment.documentId
left join user_role on
user_role.userId=userbase.id
right join role on
role.id=user_role.roleId
where jamacomment.scopeId=11
and role.name in ( 'sample grupa' )
and userbase.userName in ( 'sample' )
and my javascript code for that dataset on beforeOpen state is:
if( params["JDSuser"].value[0] != "(All Users)" ){
this.queryText=this.queryText.replaceAll('sample grupa', params["JDSgroup"]);
var users = params["JDSuser"];
//var userquery = "'";
var userquery = userquery + users.join("', '");
//userquery = userquery + "'";
this.queryText=this.queryText.replaceAll('sample', userquery);
}
I tryed many different quote variations, with this one I get no error messages, but if I choose 1 value, I get no data from database, but if I choose at least 2 values, I get the last chosen value data.
If I uncomment one of those additional quote script lines, then I get syntax error like this:
The following items have errors:
Table (id = 597):
+ An exception occurred during processing. Please see the following message for details: Failed to prepare the query execution for the
data set: Organization Cannot get the result set metadata.
org.eclipse.birt.report.data.oda.jdbc.JDBCException: SQL statement does not return a ResultSet object. SQL error #1:You have an error in
your SQL syntax; check the manual that corresponds to your MySQL
server version for the right syntax to use near 'rudolfs.sviklis',
'sample' )' at line 25 ;
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to
your MySQL server version for the right syntax to use near
'rudolfs.sviklis', 'sample' )' at line 25
Also, I should tell you that i'm doing this by looking from working example. Everything is the same, the previous code resulted to the same syntax error, I changed it to this script which does the same.
The example is available here:
http://developer.actuate.com/community/forum/index.php?/files/file/593-default-value-all-with-multi-select-parsmeter/
If someone could give me at least a clue to what I should do that would be great.
You should always use the value property of a parameter, i.e.:
var users = params["JDSuser"].value;
It is not necessary to surround "userquery" with quotes because these quotes are already put in the SQL query arround 'sample'. Furthermore there is a mistake because userquery is not yet defined at line:
var userquery = userquery + users.join("', '");
This might introduce a string such "null" in your query. Therefore remove all references to userquery variable, just use this expression at the end:
this.queryText=this.queryText.replaceAll('sample', users.join("','"));
Notice i removed the blank space in the join expression. Finally once it works finely, you probably need to make your report input more robust by testing if the value is null:
if( params["JDSuser"].value!=null && params["JDSuser"].value[0] != "(All Users)" ){
//Do stuff...
}