We have configured Spark SQL(1.3.2) to work on top of Hive and we use Beeline to create the tables.
I was trying to create a table with BIGINT datatype.However I see that the table is getting created with INT datatype when I use the below command
CREATE TEMPORARY TABLE cars (blank bigint)
USING com.databricks.spark.csv
OPTIONS (path "cars.csv", header "false")
However when I use the command below I am able to create a table with bigint datatype
CREATE TABLE cars(blank bigint)
Can you let me know how can I create a tableBIGINT datatype using first method
Is it because of this
"Integral literals are assumed to be INT by default, unless the number exceeds the range of INT in which case it is interpreted as a BIGINT, or if one of the following postfixes is present on the number."
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Types#LanguageManualTypes-IntegralTypes(TINYINT,SMALLINT,INT,BIGINT)
Related
I have problem inserting values in my SQL server database on Azure, I am getting the following error:
Failed to execute query. Error: String or binary data would be truncated in table 'dummy_app.dbo.user_info', column 'uid'. Truncated value: 'u'.
The statement has been terminated.
I don't understand where I am wrong, I just created the server, and I am trying to experiment but cant fix this.
if not exists (select * from sysobjects where name='user_info' and xtype='U')
create table user_info (
uid varchar unique,
name varchar,
email varchar
)
go;
INSERT INTO dbo.user_info(uid, name, email) VALUES('uids', 'name', 'email') go;
Creating the table works fine, the only thing that doesn't work is the second command INSERT
I suspect that the reason is that you haven't defined a lenght for varchar and it defaults to 1 as length. Therefore your value gets truncated.
Set a varchar length to something like varchar(200) and you should be good to go.
This looks like the fact that the CREATE portion of your procedure for the table doesn't include a length of varchar, so you'd have to specify a length such as varchar(50) since the default is 1. Refer to the official MS docs in the link, in the remarks.
docs.miscrosoft.com
Also, here is the syntax for the CREATE TABLE in Azure which might be helpful as well.
Syntax of Azure CREATE TABLE
I'm trying to filter some data - I have a column which looks like it is mainly smallint/int. Is there anyway I can run a where statement to say where not int or where not small int??
Microsoft SQL Server manager.
If you want a where clause that can tell you if the column contain information that can't be converted to int or smallint, you can use try_cast:
SELECT *
FROM <TableName>
WHERE TRY_CAST(<ColumnName> AS Int) IS NULL
You can change the int to smallint to get values that can't be converted to smallint but might be convertible to int.
Don't forget to replace <TableName> and <ColumnName> to the names of the relevant table and column.
The Try_Cast built in function will return null if the value in <ColumnName> is null or if it can't be converted to int (and since all smallint values can also be converted to int, it also can't be converted to smallint).
I have the following working DB2 SQL function
CREATE OR REPLACE FUNCTION selector (param VARCHAR(3))
RETURNS TABLE (id INT, CMD VARCHAR(1024), ATTR CHAR(10))
LANGUAGE SQL
DETERMINISTIC
NO EXTERNAL ACTION
RETURN
SELECT id, cmd, attr
FROM test.commandtbl c
WHERE c.attr=param;
Calling it like:
select * from table (selector('c'))!
The problem is that I want the return table to be dynamic in size and type.
I want to use the function with a lot of return fields and and while testing I don't want to always check the return table it everything still matches.
For example:
Test1 is with 5 return columns: INT, INT, INT, CHAR(10), VARCHAR(100)
Test2 is with 20 return columns: 10 VARCHAR(100) and 10 INT
and so on.
Is there a way to do that?
You can consider SQL a statically typed language in that it has little ability to discover its variable (e.g. column) and object (e.g. result set) data types at run time; you have to declare types at the statement compilation time. In other words, what you want to achieve is not possible.
There is a concept of a generic table function which allows you to define a Java-based UDF that returns some result set:
CREATE FUNCTION selector (param VARCHAR(3))
RETURNS GENERIC TABLE
EXTERNAL NAME...
However, you still need to declare the result set structure on the receiving end:
SELECT t.* FROM TABLE (selector('c')) AS t (foo INT, bar INT, baz VARCHAR(10)...)
I have a package that contains two execute sql tasks followed by numerous DFTs. The first has the statement:
select CAST(floor(rand()*1000)+1 AS INT) AS SeqVar
And has the ResultSet Single row - this works perfect. It gives me a random number between 1 and 1000 and passes that value on to a variable I have called SeqVar. (I have also verified that this works)
The problem I am having is in my second execute SQL task where I try and use the SeqVar variable outputted from the first Execute SQL teask as a parameter in the following statement:
IF OBJECT_ID('tempdb.dbo.[##temp1]') IS NOT NULL
DROP TABLE [##temp1]
CREATE TABLE [##temp1] (
[RecID] int IDENTITY(?,1),
[Name] VARCHAR(30),
[ABA] VARCHAR(10),
[Account] VARCHAR(20),
[Type] VARCHAR(1),
[Date] date,
[Amount] money
);
Under parameter mapping I have the SeqVar variable name, Direction is Input, Data Type numeric, Parameter name is 0, and Parameter size is 1000.
The value I get has to go where I have the "?" in the create tempdb statement. I am trying to have my code start at a random number and increment by 1.
I know this would probably be easier with a Script task but that tool is broken on my maching (weird dts pipeline errors). Thanks in advance and this is all in SSIS 2008.
Using identity in this way seems like a strange solution just to start a sequence at some random value. Parameters formatted as ? definitely don't work in that context in SQL.
However, another way to manage this, if the method is a given, is to set the entire code of your SQL task using an expression where you sub in the value as a simple string concatenation and then run the resulting string.
Better do create table as like below then use RESEED property to set it to your Random Variable Value'
IF OBJECT_ID('tempdb.dbo.[##temp1]') IS NOT NULL
DROP TABLE [##temp1]
CREATE TABLE [##temp1] (
[RecID] int IDENTITY(1,1),
[Name] VARCHAR(30),
[ABA] VARCHAR(10),
[Account] VARCHAR(20),
[Type] VARCHAR(1),
[Date] date,
[Amount] money
);
DECLARE #RANDOM_VALUE INT = 50
DBCC CHECKIDENT('##TEMP1',RESEED,#RANDOM_VALUE)
I have a database table of that I have used to store the data returned from a web spider. I have a column that contains ticket prices for different events all in the varchar type (as the scrapy spider has to scrape the data in unicode). I'm trying to return the min price of the column and since the min() function only works for data of type INT, I tried to convert the column to integers using a solution from this SO post:
ALTER TABLE vs_tickets ALTER COLUMN ticketprice TYPE integer USING (ticketprice::integer);
but I got the error: ERROR: invalid input syntax for integer:
I also tried: change_column :vs_tickets, :ticketprice, 'integer USING CAST(ticketprice AS integer)' but that didn't work either.
What is the proper way to convert the column to type INT?
Edit:
You have decimal places in the string, so a simple cast is not going to work. You can do a double conversion:
cast(cast(ticketprice as decimal(10, 2)) as int)
or:
(ticketprice::decimal(10, 2))::int
(The parens are not strictly necessary.)
EDIT:
Or, as Erwin points out, just use numeric:
(ticketprice::numeric)::int
Postgres is much smarter about numeric than most other databases . . . after all, it supports numbers that are egregiously large ;)
The final query is:
ALTER TABLE vs_tickets
ALTER COLUMN ticketprice TYPE integer USING (ticketprice::numeric::integer);
I'm going to bet on your column have wrong characters.
Also you may want use float or numeric because you will lose decimals if convert to integers.
You need create a function to check if a text is numeric like this isnumeric-with-postgresql
Then check each row like this
select ticketprice
from vs_tickets
where ISNUMERIC(ticketprice) = false;
As your comment you also should try
SELECT ticketprice::float
You will be best off adding an INT column, moving your data with a cast and then removing the old varchar column.
ALTER TABLE vs_tickets ADD COLUMN ticketprice_int TYPE int;
GO
update vs_tickets SET ticketprice_int = cast(ticketprice as int);
// if you fail to cast the varchar to int you can use Gordon's method
// update vs_tickets SET ticketprice_int = cast(cast(ticketprice as decimal(10, 2)) as int);
GO
ALTER TABLE vs_tickets DROP COLUMN ticketprice;
GO
ALTER TABLE vs_tickets RENAME COLUMN ticketprice_int to ticketprice;
GO
With this at minimum you will be able to tell if and where a cast/convert fails and be able to check and recheck at each step before you can't turn back.