My table consists of 253 columns so loaded data to pig bag without schema method, when I apply filter condition to any data getting this error how to avoid it? Any jars needs to be added ?
Error: IMPLICIT_CAST_TO_CHARARRAY
Since schema is not defined in the load statement, so default data type for all the fields are bytearray.
Pig tries to implicitly cast from one type to another while using in the script.
All castings are not possible, so in that case, warnings are thrown.
You can refer the Pig Cast operators for details
When the schema is not specified the fields are loaded as bytearray.When you apply filter you will have to cast the field you are using for the filter.If you are filtering using a string use (chararray) in front of the field.
Related
I have sqlite files that have incorrect datatypes defined to some fields hence the ExecuteSQL shows type error.
How do I provide the correct datatype to the processor for it to execute the select query correctly?
I am saving the output of my AnyLogic model into an SQL server database. For non-AnyLogic aficionados, AnyLogic is based on Java. However, I am not sure what data types I need to specify for my columns in the database.
So far I am using these:
double in AnyLogic : float in SQL
string in AnyLogic : varchar in SQL
int in AnyLogic : int in SQL
I also have parameters that are of type Option list, which is, if I understand correctly, a form of Java enum. I tried to save those parameters as varchar, but this (obviously) does not work. In addition, my model contains various boolean parameters. For my boolean parameters, I add columns of type bit in SQL by running:
ALTER TABLE myTable
ADD my_bool BIT NOT NULL DEFAULT 0;
However, running the model returns this error
SQLServerException: Invalid column name 'false'. Caused by: Invalid column name 'false'
So concretely, how can I export parameters of type Option list and boolean?
This addresses the original question which was tagged MySQL.
I don't know all the issues around "option list". Seems like a string (with a length such as varchar(255)) would work. You can also look into the built-in enum type, although I would not normally recommend using enums.
I would recommend using boolean instead of bit as the equivalent for boolean. Seems more mnemonic.
That said, MySQL understands false as a constant. You can check this by running:
select false
This also works:
select "false"
However, this returns the error that you specify:
select `false`
I suspect that the code being generated is using this construct. You will need to look at the code -- and you might need to figure out some other way of handling this. In MySQL you can use 0 for false and that might fix your problem.
The AnyLogic database is a standard HSQLDB database (not something proprietary) but they've added AnyLogic client functionality to define 'column types' as though they are Java types (with special types for option lists and compiled-on-the-fly-and-run Java code).
If you look at the db.script file (HSQLDB just stores the persistent DB data as an SQL script which creates the tables and INSERTs the values) you can see the underlying HSQLDB types which map closely to SQL Server types.
boolean --> BOOLEAN
double --> DOUBLE
int --> INT
String --> VARCHAR(16777216)
Date --> TIMESTAMP
Code --> VARCHAR(16777216)
Option List --> VARCHAR(255)
NB: The 'Java column types' are supposed to make it easier for a non-technical user to understand what they will get from a Java perspective when querying that column but, for example, they are confusing in that queries will return Java nulls for missing values, so a boolean column actually effectively returns a Boolean.
That should help.
I managed to address part of my problem. I am now able to store String variables from Java into my SQL database. The issue was due to incorrect use of quotations.
Java uses double quotations for String variables (e.g.: ""). SQL expects single quotations (e.g.: '') for string-like columns such as varchar() and char()
I had to amend my SQL query to this:
String insertTableSQL = "INSERT INTO test (my_string) VALUES(" +" '"+my_variable_string+"' )";
Note that my_variable_string is a derivative of a Java enum, which I obtained by executing String my_variable_string= my_enum.name();
I use postgreSQL/pentaho CDE. I need to transmit parameter as schema prefix in my query. I need to run this query: select * from peredelkino_public.protocol
I used custom parameter in Pentaho (named 'selector_par'), in properties named 'Javascript code' i wrote 'peredelkino_public.protocol'.
Then I changed sql query in component 'sql over sqlJndi' like this: select * FROM (${selector_par})
But this query doesn't work! I get message 'Error processing component'. How I can transmit schema prefix in my query?
Check the Replace variables in script.
Note: if the parameter selector_par does not exists or is misspelled, you may get a hard to understand error message.
You cannot.
CDE can only use JDBC parameters and those cannot be used in the from clause.
If you absolutely need to do that, you should use a Kettle datasource in CDE, as PDI allows any variable to be used in any part of a SQL query.
But make sure you sanitize your inputs. Variable replacements such as that one are a gold mine for hackers.
I have five or six OLE DB Sources with a String[DT_STR], with a length of 500 and 1252 (Latin) as Code Page.
The format of the column is like 0,08 or 0,10 etc etc. As you can see, it is separated with a comma.
All of them are equal except one of them. In this one source, I have a POINT as separation. On this it is working when I set the Data Type in the advanced editor of the OLE DB Source. On another (with comma separated) it is also working, if I set the Data Type in the advanced editor of the OLE DB Source. BUT the weird thing is, that it isn't working with the other sources although they are the same (sperated with comma).
I tested Numeric(18,2) and decimal(2).
Another try to solve the problem with the conversion task and/or the derived column task, failed.
I'm using SQL Server 2008 R2
Slowly, I think SSIS is fooling me :)
Has anyone an idea?
/// EDIT
Here a two screens:
Is working:
click
Isn't working:
click
I would not set the Data Type in the Advanced Editor of the OLE DB Source. I would convert the data in the SQL Code of the OLE DB Source, or in a Script Transformation e.g. using Decimal.TryParse , which would update a new column.
SSIS is unbeleivably fussy over datatypes and trying to mess with its internals is not productive.
Check that there are any spaces in between the commas, so that the SSIS is throwing an error trying to convert the blank space to a number. A blank space does not equal nothing in between spaces.
Redirect error rows and output the data to a file. Then you can examine the data that is being rejected by the SSIS and determine why it's causing error.
Reason for the error
1) Null’s are not properly handled either in the destination database or during SSIS package creation. It is quite possible that the source contains a null database but the destination is not accepting the NULL data leading to build generate above error.
2) Data types between source and destination does not match. For example, source column has varchar data and destination column have an int data type. This can easily generate above error. There are certain kind of datatypes which will automatically convert to another data type and will not generate the error but there are for incompatible datatypes which will generate The value could not be converted because of a potential loss of data. error.
The Issue arises when there is unhandled space or null. I have worked around using the Conditional (Ternary) Operator which checks the length:
LEN(TRIM([Column Name])) >= 1 ? (DT_NUMERIC,38,8)[Column Name] : 0
I am performing a DB2 load, and I am struggling to understand the impact of using GeneratedOverride over IdentityOverride. When I run the following command:
db2 load from tab123.ixf of ixf replace into application.table_abc
All rows are rejected, with the following error being the culprit:
SQL3550W The field value in row row-number and column column-number is not NULL, but the target column has been defined as GENERATED ALWAYS.
So to try and step around this, I executed
:
db2 load from tab123.ixf of ixf modified by identityoverride replace into application.table_abc
But this immediately returned this error:
SQL3526N The modifier clause "IDENTITY OVERRIDE" is inconsistent with the current load command. Reason code: "3".
From checking the reason code I see that the issue is "Generated or identity related file type modifiers have been specified but the target table contains no such columns." .. but the SQL3550W error seems to infer that the columns are generated always!
The only way I can get these rows to commit to the table is to run..
db2 load from tab123.ixf of ixf modified by generatedoverride replace into application.table_abc
Can anyone enlighten me to why I am recieving the SQL3526N error, or what the implications of running generatedoverride are?
Thanks for sticking with me..
Generated columns are not necessarily identity columns, apparently that's the case in your situation. Check the CREATE TABLE syntax to see what are other ways to generate column values.
By using the GENERATEDOVERRIDE option during the load you are obviously replacing (overriding) the generated values with those from the input file.