When querying table using "select * from t2p", the reponse is as blow. I think I have missed some concepts, please help me out.
Failed with exception java.io.IOException:java.lang.ClassCastException: org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyMapObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector
Step1, create table
create table t2p(id int, name string, score map<string,double>)
partitioned by (class int)
row format delimited
fields terminated by ','
collection items terminated by '\\;'
map keys terminated by ':'
lines terminated by '\n'
stored as textfile;
Step2, insert data like
1,zs,math:90.0;english:92.0
2,ls,chinese:89.0;math:80.0
3,xm,geo:87.0;math:80.0
4,lh,chinese:89.0;english:81.0
5,xw,physics:91v;english:81.0
Step3, add another column
alter table t2p add columns (school string);
Step4, change column's order
alter table t2p change school school string after name;
Step5, do query and get error as mentioned above.
select * from t2p;
This is an obvious error.
Your command alter table t2p change school school string after name; changes metadata only. If you are moving columns, the data must already match the new schema or you must change it to match by some other means.
Which means, the map column has to be matching to the new column. In other words, if you want to move column around, make sure new column and existing data types are same.
I did a simple experiment with int data type. It worked because data type are not hugely different but you can see metadata changed but data stayed same.
create table t2p(id int, name string, score int)
partitioned by (class int)
stored as textfile;
insert into t2p partition(class=1) select 100,'dum', 199;
alter table t2p add columns (school string);
alter table t2p change school school string after name;
MSCK REPAIR TABLE t2p ;
select * from t2p;
You can see new column school is mapped to position 3( defined as INT).
Solution - You can do this but make sure new structure+data type is compatible to old structure.
Related
I am trying to alter the column size of a view with the same command that we use for table like :
alter table
STUDENT
modify (
ROLL_NO VARCHAR2(80)
);
But its throwing error
SQL Error: ORA-00942: table or view does not exist
So how we can alter the column size of a view?
A view is simply saved query and "inherits" column type from underlying base table. So if you need to change metadata you should alter view definition:
ALTER VIEW view_students
AS
SELECT CAST(roll_no AS VARCHAR2(80)) AS roll_no,
...
FROM tab_students;
If you want to change data type to store longer strings, then you need to locate base table and alter it instead:
ALTER VIEW tab_students
MODIFY (ROLL_NO VARCHAR2(80));
Here is the procedure that I followed :
1- First find the base table for that view by running the following query
SELECT * FROM DBA_DEPENDENCIES
WHERE OWNER = '<scheman_name>'
AND NAME = '<view_name>'
AND TYPE = 'VIEW';
2- Above query will you a table where you will find the base table under the column name 'REFERENCED_NAME'.
3- Now Change the column size of that base table.
NOTE: The view can be made up of 1 or more than 1 tables, so you need to change the column size of all those base tables.
How can I alter the type of a column in an existing table in MonetDB? According to the documentation the code should be something like
ALTER TABLE <tablename> ALTER COLUMN <columnname> SET ...
but then I am basically lost because I do not know which standard the SQL used by MonetDB follows here and I get a syntax error. If this statement is not possible I would be grateful for a workaround that is not too slow for large (order of 10^9 records) tables.
Note: I ran into this problem while doing some bulk data imports from csv files into a table in my database. One of the columns is of type INT but the values in the file at some point exceed the INT limit of 2^31-1 (yes, the table is big) and so the transaction aborts. After I found out the reason for this failure, I wanted to change it to BIGINT but all versions of SQL code I tried failed.
This is currently not supported. However, there is a workaround:
Example table for this example, say we want to change the type of column b from integer to double.
create table a(b integer);
insert into a values(42);
Create a temporary column alter table a add column b2 double;
Set data in temporary column to original data update a set b2=b;
Remove the original column alter table a drop column b;
Re-create the original column with the new type alter table a add column b double;
Move data from temporary column to new column update a set b=b2;
Drop the temporary column alter table a drop column b2;
Profit
Note that this will change the ordering of columns if there are more than one. However, this is only a cosmetic issue.
I have a HIVE table, partitioned on date field and gets loaded every day. We got a request to add a new column at the end and load the data into the same HIVE table. Are there any better ways to handle this column change requests in keeping the existing data.
Do I need to delete the data in the existing table and recreate the table using the new columns and load the data.
In which format do you save the data?
If you are using avro-format, just add the new fields in the .avsc-filed and set a default-value:
{
"name": "yourData",
"type": ["string", "null"],
"default": "null"
}
If you store the data as csv, then it seems to be a little bit more complicated.
Changing the table with alter table didn't worked in my case (I have no idea why).
So I deleted the table, recreated it with the new columns and added the partitions and it works.
Make shure that your table is an external Table, then you don't have to delete the data.
eg:
Old Data:
889,5CE1,2016-07-25
New Data:
900,5DCBA,2016-07-25,2012-03-22,152047
hive:
create table somData (
anid int
,astring String
,extractDate date
)
PARTITIONED BY(cusPart STRING)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ','
STORED AS TextFile location "/your/location";
what you have to do:
ALTER TABLE somData SET TBLPROPERTIES('EXTERNAL'='TRUE');
drop table somData;
create table somData (
anid int
,astring String
,extractDate date
,anotherDate date
,someInt int
)
PARTITIONED BY(cusPart STRING)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ','
STORED AS TextFile location "/your/location";
ALTER TABLE someData ADD IF NOT EXISTS PARTITION (cusPart='foo') LOCATION '/your/paritioned/data';
I have database and some text fields are CLOB type, I need to change most of these into VARCHAR2.
Have tried to do that using SQL Developer tool, by clicking edit on table, but get error like this one:
The following SQL statement failed:
ALTER TABLE TBL_PEOPLE MODIFY (PERSON VARCHAR2(150) )
Want to ask, how can this change be done
You can't, directly. The code you tried will have got an `ORA-22859, presumably. (It's helpful to show the actual errors you get, of course).
You'll need to add a new varchar2 column; copy the data across - or a substring of it if it might be larger than the new column you're creating; drop the clob column. You can rename the columns so it looks fairly transparent.
As in this SQL Fiddle:
alter table tbl_people rename column person to clob_person;
alter table tbl_people add (person varchar2(150));
update tbl_people set person = clob_person;
alter table tbl_people drop column clob_person;
Obviously don't drop the old column until you're sure the data has copied without errors. Also take into account any constraints, indexes, etc. that might exist in the old column; they will need to be recreated. And anything that references the old column will have been invalidated - generally procedures will recompile themselves on next use.
So be careful, test it first, and plan some down time.
ALTER TABLE tablename ADD (FIELD_LIST_TEMP VARCHAR2);
UPDATE tablename SET FIELD_LIST_TEMP = FIELD_LIST;
ALTER TABLE tablename DROP COLUMN FIELD_LIST;
ALTER TABLE tablename RENAME COLUMN FIELD_LIST_TEMP TO FIELD_LIST;
Here FIELD_LIST existing column which is defined it as CLOB. With above query it will change from CLOB to VARCHAR2.
Based on the following tutorial, Hive has a map type. However, there does not seem to be a documented way to insert a new key-value pair into a Hive map, via a SELECT with some UDF or built-in function. Is this possible?
As a clarification, suppose I have a table called foo with a single column, typed map, named column_containing_map.
Now I want to create a new table that also has one column, typed map, but I want each map (which is contained within a single column) to have an additional key-value pair.
A query might look like this:
CREATE TABLE IF NOT EXISTS bar AS
SELECT ADD_TO_MAP(column_containing_map, "NewKey", "NewValue")
FROM foo;
Then the table bar would contain the same maps as table foo except each map in bar would have an additional key-value pair.
Consider you have a student table which contains student marks in various subjects.
hive> desc student;
id string
name string
class string
marks map<string,string>
You can insert values directly to table as below.
INSERT INTO TABLE student
SELECT STACK(1,
'100','Sekar','Mathematics',map("Mathematics","78")
)
FROM empinfo
LIMIT 1;
Here 'empinfo' table can be any table in your database.
And Results are:
100 Sekar Mathematics {"Mathematics":"78"}
for key-value pairs, you can insert like following sql:
INSERT INTO TABLE student values( "id","name",'class',
map("key1","value1","key2","value2","key3","value3","key4","value4") )
please pay attention to sequence of the values in map.
I think the combine function from brickhouse will do what you need. Slightly modifying the query in your original question, it would look something like this
SELECT
combine(column_containing_map, str_to_map("NewKey:NewValue"))
FROM
foo;
The limitation with this example is that str_to_map creates a MAP< STRING,STRING >. If your hive map contains other primitive types for the keys or values, this won't work.
I'm sorry, I didn't quite get this. What do you mean by with some UDF or built-in function?If you wish to insert into a table which has a Map field it's similar to any other datatype. For example :
I have a table called complex1, created like this :
CREATE TABLE complex1(c1 array<string>, c2 map<int,string> ) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' COLLECTION ITEMS TERMINATED BY '-' MAP KEYS TERMINATED BY ':' LINES TERMINATED BY '\n';
I also have a file, called com.txt, which contains this :
Mohammad-Tariq,007:Bond
Now, i'll load this data into the above created table :
load data inpath '/inputs/com.txt' into table complex1;
So this table contains :
select * from complex1;
OK
["Mohammad","Tariq"] {7:"Bond"}
Time taken: 0.062 seconds
I have one more table, called complex2 :
CREATE TABLE complex2(c1 map<int,string>);
Now, to select data from complex1 and insert into complex2 i'll do this :
insert into table complex2 select c2 from complex1;
Scan the table to cross check :
select * from complex2;
OK
{7:"Bond"}
Time taken: 0.062 seconds
HTH