QueryDsl column case sensitivity bug on identifier - hsqldb

I have seen many post on column name case sensitivity for in memory database like hsqldb and h2. We are using sql server camel case column names. However, I am testing using HyperSql which is column name case sensitive. I don't see any settings to handle column name sensitivity in hypersql except when creating the table quote the column names which will make them what ever case are inside the quotes, for example
Insert Into AddressType ("AddressTypeName", "CreateUser")
Values ('Mailing', 'User')
This will create tables in hsqldb with column name AddressTypeName and CreateUser
sql server is not willing to make theirs all upper case. As a result, when creating the columns in hsqldb I quote them which creates them in camel case. Works good except when using Querydsl their identiifers does not quote them which then results in the lookup with upper case column names and the database now has them as camel case.
The whole problem started because QueryDsl Q types which were generated off the sql server columns has the metadata with quote columns and since sql server had them as camel case it looks them up as camel case. But now when doing a Q type query it uses the identifier un quoted which results in a upper case lookup which fails. I don't see any work around except to let the database make them upper case and change all the Q types to be upper case for the meta data column name look ups.
I assume this is a bug in querydsl.
Example QAddressType addMetadata. Notice the ColumnMetadata.named("AddressTypeName") By these being quoted the database must have them in camel case.
public void addMetadata() {
addMetadata(addressTypeID, ColumnMetadata.named("AddressTypeID").withIndex(1).ofType(Types.BIGINT).withSize(19).notNull());
addMetadata(addressTypeName, ColumnMetadata.named("AddressTypeName").withIndex(2).ofType(Types.VARCHAR).withSize(50).notNull());
ColumnMetadata.named("CreateUser").withIndex(3).ofType(Types.VARCHAR).withSize(100).notNull());
}
But if I switch the database to be camel case then the queryDslTemplate call fails because it does not quote the identifier. So in the below code .where(qAddressType.addressTypeID.eq(id)); does this in sql a.addressTypeID = 1 where a is the alias for QAddressType. It needs to be a."addressTypeID" = 1 or else hsqldb looks it up as a.ADDDRESTYPEID = 1
private static QAddressType qAddressType = new QAddressType("a");
#Override
public AddressType getById(Long id) {
AddressType addressType = null;
try {
SQLQuery sqlQuery = queryDslJdbcTemplate.newSqlQuery()
.from(qAddressType)
.where(qAddressType.addressTypeID.eq(id));
addressType = queryDslJdbcTemplate.queryForObject(sqlQuery, new AddressTypeProjection(qAddressType));
//the API was not throwing the Exception so let's explicitly throw it
return addressType;
}

Related

Permanently store sqoop --map-column-hive mapping for DB2

I am importing several DB2 database tables with an Oozie workflow that uses Sqoop to import to Hive. Currently I have to map each column with an unsupported data type manually with "--map-column-hive".
Is there any way to permanently store mappings for specific data types? I am importing several tables that contain DB2-"Character" columns which all have to be mapped to HIVE-"STRING" manually.
For ~50 tables there are ~200 columns that use the datatype "Character" for FKs which have to be mapped manually.
I want to permanently save that DB2-"Character" is mapped to the datatype HIVE-"STRING".
Can this be done?
regards
As far as I can see, Sqoop does not provide ability to pass type-to-type mappings as parameters.
They are all hardcoded explicitly:
switch (sqlType) {
case Types.INTEGER:
case Types.SMALLINT:
return HIVE_TYPE_INT;
case Types.VARCHAR:
case Types.CHAR:
case Types.LONGVARCHAR:
case Types.NVARCHAR:
case Types.NCHAR:
case Types.LONGNVARCHAR:
case Types.DATE:
case Types.TIME:
case Types.TIMESTAMP:
case Types.CLOB:
return HIVE_TYPE_STRING;
case Types.NUMERIC:
case Types.DECIMAL:
case Types.FLOAT:
case Types.DOUBLE:
case Types.REAL:
return HIVE_TYPE_DOUBLE;
case Types.BIT:
case Types.BOOLEAN:
return HIVE_TYPE_BOOLEAN;
case Types.TINYINT:
return HIVE_TYPE_TINYINT;
case Types.BIGINT:
return HIVE_TYPE_BIGINT;
default:
// TODO(aaron): Support BINARY, VARBINARY, LONGVARBINARY, DISTINCT,
// BLOB, ARRAY, STRUCT, REF, JAVA_OBJECT.
return null;
}
Also there's a specific case for XML columns in DB2:
if (colTypeName.toUpperCase().startsWith("XML")) {
return XML_TO_JAVA_DATA_TYPE;
}
If your column type is not recognized by this mapping + user-defined mappings via
--map-column-hive parameter, you'll get an exception.
What I'd do in your case (if not considering manual column mapping):
Make sure once again that type mapping for DB2-"Character" does not work
Download the sources of your Sqoop version, add a new if-branch in Db2Manager.toDbSpecificHiveType, build and test with some your tables
Create a PR and wait for the next release OR use the customized version of Sqoop (might be painful when you want to upgrade Sqoop version)

GORM many2many preload error

Currently using GORM to connect to two databases: POSTGRES AND sqlite (using a code switch to choose which one to use). I have a 2 database tables defined in my schema that look like this:
type TableClient struct {
Model
Synchronised bool
FacilityID string `gorm:"primary_key"`
Age int
ClientSexID int
MaritalStatusID int
SpecificNeeds []TableOptionList`gorm:"many2many:options_specific_needs"`
}
type TableOptionList struct {
ID int `gorm:"primary_key"`
Name string
Value string
Text string
SortKey int
}
Previously, I would preload related table with code like this:
var dbClient TableClient
Db.Where("facility_id = ? AND client_id = ? AND id = ?;", URLFacilityID, URLClientID, URLIncidentID).
Preload("ClientSex").
Preload("MaritalStatus").
Preload("CareTakerRelationShip").
Preload("HighestLevelOfEducation").
Preload("Occupation").
Preload("SpecificNeeds").
First(&dbClient)
Now that lookup fails with a syntax error and when I look at the SQL generated, it shows the following SQL is generated:
SELECT * FROM "table_option_lists" INNER JOIN "options_specific_needs" ON "options_specific_needs"."table_option_list_id" = "table_option_lists"."id" WHERE (("options_specific_needs"."table_client_id","options_specific_needs"."table_client_facility_id") IN (('one','LS034')))
Pasting that into the sqlite console also fails with the same error: (near ",": syntax error)
The crux of your issue is in your WHERE clause, you need to use "OR", not ",".
This is likely an issue with GORM and you could open a GitHub issue with them for this. You can get around the issue using db.Raw() from GORM. In general, I prefer to avoid ORMs. Part of the reason is it is often easier to just build an SQL query by hand than try to figure out the strange way to do it in the ORM (for things beyond basic CRUD).

How to use a dynamic parameter in a IN clause of a JPA named query?

my problem is about this kind of query :
select * from SOMETABLE where SOMEFIELD in ('STRING1','STRING2');
the previous code works fine within Sql Developer.
The same static query also works fine and returns me a few results;
Query nativeQuery = em.createNativeQuery(thePreviousQuery,new someResultSet());
return nativeQuery.getResultList();
But when I try to parameterize this, I encounter a problem.
final String parameterizedQuery = "select * from SOMETABLE where SOMEFIELD in (?selectedValues)";
Query nativeQuery = em.createNativeQuery(parameterizedQuery ,new someResultSet());
nativeQuery.setParameter("selectedValues","'STRING1','STRING2'");
return nativeQuery.getResultList();
I got no result (but no error in console).
And when I look at the log, I see such a thing :
select * from SOMETABLE where SOMEFIELD in (?)
bind => [STRING1,STRING2]
I also tried to use no quotes (with similar result), or non ordered parameter (:selectedValues), which leads to such an error :
SQL Error: Missing IN or OUT parameter at index:: 1
I enventually tried to had the parentheses set directly in the parameter, instead of the query, but this didn't work either...
I could build my query at runtime, to match the first (working) case, but I'd rather do it the proper way; thus, if anyone has an idea, I'll read them with great interest!
FYI :
JPA version 1.0
Oracle 11G
JPA support the use of a collection as a list literal parameter only in JPQL queries, not in native queries. Some JPA providers support it as a proprietary feature, but it's not part of the JPA specification (see https://stackoverflow.com/a/3145275/1285097).
Named parameters in native queries also aren't part of the JPA specification. Their behavior depends on the persistence provider and/or the JDBC driver.
Hibernate with the JDBC driver for Oracle support both of these features.
List<String> selectedValues = Arrays.asList("STRING1", "STRING2");
final String parameterizedQuery = "select * from SOMETABLE where SOMEFIELD in (:selectedValues)";
return em.createNativeQuery(parameterizedQuery)
.setParameter("selectedValues", selectedValues)
.getResultList();
Instead of:
nativeQuery.setParameter("selectedValues", params);
I had to use:
nativeQuery.setParameterList("selectedValues", params);
This worked for me in derby. parameter without "()".
List<String> selectedValues = Arrays.asList("STRING1", "STRING2");
final String parameterizedQuery = "select * from SOMETABLE where SOMEFIELD in
:selectedValues";
return em.createNativeQuery(parameterizedQuery)
.setParameter("selectedValues", selectedValues)
.getResultList();
Replace this:
nativeQuery.setParameter("selectedValues","'STRING1','STRING2'");
with
List<String> params;
nativeQuery.setParameter("selectedValues",params);
I also faced the same issue.
This is what I did:
List<String> sample = new ArrayList<String>();
sample.add("sample1");
sample.add("sample2");
And now you, can set the sample in params.

How to change sql generated by linq-to-entities?

I am querying a MS SQL database using Linq and Entity Framework Code First. The requirement is to be able to run a WHERE SomeColumn LIKE '%sometext'clause against the table.
This, on the surface, is a simple requirement that could be accomplished using a simple Linq query like this:
var results = new List<MyTable>();
using(var context = new MyContext())
{
results = context.MyTableQueryable
.Where(x => x.SomeColumn.EndsWith("sometext"))
.ToList();
}
// use results
However, this was not effective in practice. The problem seems to be that the column SomeColumn is not varchar, rather it's a char(31). This means that if a string is saved in the column that is less than 31 characters then there will be spaces added on the end of the string to ensure a length of 31 characters, and that fouls up the .EndsWith() query.
I used SQL Profiler to lookup the exact sql that was generated from the .EndsWith() method. Here is what I found:
--previous query code removed for brevity
WHERE [Extent1].[SomeColumn] LIKE N'%sometext'
So that is interesting. I'm not sure what the N means before '%sometext'. (I'll Google it later.) But I do know that if I take the same query and run it in SSMS without the N like this:
--previous query code removed for brevity
WHERE [Extent1].[SomeColumn] LIKE '%sometext'
Then the query works fine. Is there a way to get Linq and Entity Framework to drop that N from the query?
Please try this...
.Where(x => x.SomeColumn.Trim().EndsWith("sometext"))
Just spoke to my colleague who had a similar issue, see if the following works for you:
[Column(TypeName = "varchar")]
public string SomeColumn
{
get;
set;
}
Apparently setting the type on the column mapping will force the query to recognise it as a VARCHAR, where a string is normally interpreted as an NVARCHAR.

SQL Injection: is this secure?

I have this site with the following parameters:
http://www.example.com.com/pagination.php?page=4&order=comment_time&sc=desc
I use the values of each of the parameters as a value in a SQL query.
I am trying to test my application and ultimately hack my own application for learning purposes.
I'm trying to inject this statement:
http://www.example.com.com/pagination.php?page=4&order=comment_time&sc=desc' or 1=1 --
But It fails, and MySQL says this:
Warning: mysql_fetch_assoc() expects parameter 1 to be resource,
boolean given in /home/dir/public_html/pagination.php on line 132
Is my application completely free from SQL injection, or is it still possible?
EDIT: Is it possible for me to find a valid sql injection statement to input into one of the parameters of the URL?
The application secured from sql injection never produces invalid queries.
So obviously you still have some issues.
Well-written application for any input produces valid and expected output.
That's completely vulnerable, and the fact that you can cause a syntax error proves it.
There is no function to escape column names or order by directions. Those functions do not exist because it is bad style to expose the DB logic directly in the URL, because it makes the URLs dependent on changes to your database logic.
I'd suggest something like an array mapping the "order" parameter values to column names:
$order_cols = array(
'time' => 'comment_time',
'popular' => 'comment_score',
... and so on ...
);
if (!isset($order_cols[$_GET['order'])) {
$_GET['order'] = 'time';
}
$order = $order_cols[$_GET['order']];
Restrict "sc" manually:
if ($_GET['sc'] == 'asc' || $_GET['sc'] == 'desc') {
$order .= ' ' . $_GET['sc'];
} else {
$order .= ' desc';
}
Then you're guaranteed safe to append that to the query, and the URL is not tied to the DB implementation.
I'm not 100% certain, but I'd say it still seems vulnerable to me -- the fact that it's accepting the single-quote (') as a delimiter and then generating an error off the subsequent injected code says to me that it's passing things it shouldn't on to MySQL.
Any data that could possibly be taken from somewhere other than your application itself should go through mysql_real_escape_string() first. This way the whole ' or 1=1 part gets passed as a value to MySQL... unless you're passing "sc" straight through for the sort order, such as
$sql = "SELECT * FROM foo WHERE page='{$_REQUEST['page']}' ORDER BY data {$_REQUEST['sc']}";
... which you also shouldn't be doing. Try something along these lines:
$page = mysql_real_escape_string($_REQUEST['page']);
if ($_REQUEST['sc'] == "desc")
$sortorder = "DESC";
else
$sortorder = "ASC";
$sql = "SELECT * FROM foo WHERE page='{$page}' ORDER BY data {$sortorder}";
I still couldn't say it's TOTALLY injection-proof, but it's definitely more robust.
I am assuming that your generated query does something like
select <some number of fields>
from <some table>
where sc=desc
order by comment_time
Now, if I were to attack the order by statement instead of the WHERE, I might be able to get some results... Imagine I added the following
comment_time; select top 5 * from sysobjects
the query being returned to your front end would be the top 5 rows from sysobjects, rather than the query you try to generated (depending a lot on the front end)...
It really depends on how PHP validates those arguments. If MySQL is giving you a warning, it means that a hacker already passes through your first line of defence, which is your PHP script.
Use if(!preg_match('/^regex_pattern$/', $your_input)) to filter all your inputs before passing them to MySQL.