I'm trying to creat an external table in Hive with this
CREATE EXTERNAL TABLE IF NOT EXISTS 1987(
YEAR INT,
MONTH INT,
DAYOFMONTH INT,
DAYOFWEEK INT,
DEPTIME INT,
CRS INT,
ARRTIME TIME,
CARRIER STRING,
FLIGHTNUM INT,
TAILNUM STRING,
ACTUALELAPSED INT,
CRSELAPSED INT,
AIRTIME INT,
ARRDELAY INT,
DEPDELAY INT,
ORIGIN STRING,
DEST STRING,
DISTANCE INT,
TAXIIN INT,
TAXIOUT INT,
CANCELLED INT,
CANCELLATIONCODE STRING,
DIVERTED INT,
CARRIERDELAY INT,
WEATHERDELAY INT,
NASDELAY INT,
SECURITYDELAY INT,
LATEAIRCRAFT INT,
Origin CHAR(1))
COMMENT 'A??O 1987'
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ','
STORED AS TEXTFILE
location '/user/raj_ops/PROYECTO/'1987.csv';
But get the following error:
org.apache.hive.service.cli.HiveSQLException: Error while compiling
statement: FAILED: ParseException line 1:36 cannot recognize input
near '1987' '(' 'YEAR' in table name
Anyone knows why?
Thanks :)
location should be '/user/raj_ops/PROYECTO/' (without file itself). If you have other files in the same location then move them to separate locations, like /user/raj_ops/PROYECTO/1987/ for 1987. Because table can be built on top of location, not file.
And table name cannot start with digits. use backquotes 1987 or rename it like year_1987
I think you probably need to escape the table name with back-ticks if its numeric:
`1987`
There is an extra quote in the location value.Remove that.
location '/user/raj_ops/PROYECTO/'1987.csv';
Should be
location '/user/raj_ops/PROYECTO/1987.csv';
Related
The following code throws error:
X = "CREATE OR REPLACE TABLE Invoices (InvoiceID INT, CustomerID INT, BillToCustomerID INT, OrderID INT, DeliveryMethodID INT, ContactPersonID INT, AccountsPersonID INT, SalespersonPersonID INT, PackedByPersonID INT, InvoiceDate TIMESTAMP, CustomerPurchaseOrderNumber INT, IsCreditNote STRING, CreditNoteReason STRING, Comments STRING, DeliveryInstructions STRING, InternalComments STRING, TotalDryItems INT, TotalChillerItems STRING, DeliveryRun STRING, RunPosition STRING, ReturnedDeliveryData STRING, ConfirmedDeliveryTime TIMESTAMP, ConfirmedReceivedBy STRING, LastEditedBy INT, LastEditedWhen TIMESTAMP) LOCATION '/mnt/adls/DQD/udl/Invoices/'; ALTER TABLE Invoices ADD COLUMN DQ_Check_Op SMALLINT"
spark.sql(X)
But, with magic command, inside a cell, it runs fine:
%sql
CREATE OR REPLACE TABLE Invoices (InvoiceID INT, CustomerID INT, BillToCustomerID INT, OrderID INT, DeliveryMethodID INT, ContactPersonID INT, AccountsPersonID INT, SalespersonPersonID INT, PackedByPersonID INT, InvoiceDate TIMESTAMP, CustomerPurchaseOrderNumber INT, IsCreditNote STRING, CreditNoteReason STRING, Comments STRING, DeliveryInstructions STRING, InternalComments STRING, TotalDryItems INT, TotalChillerItems STRING, DeliveryRun STRING, RunPosition STRING, ReturnedDeliveryData STRING, ConfirmedDeliveryTime TIMESTAMP, ConfirmedReceivedBy STRING, LastEditedBy INT, LastEditedWhen TIMESTAMP) LOCATION '/mnt/adls/DQD/udl/Invoices/'; ALTER TABLE Invoices ADD COLUMN DQ_Check_Op SMALLINT
What am I doing wrong here?
The reason you are getting a ParseException in spark.sql is because of the usage of multiple SQL query statements (CREATE and ALTER) with the help of ; . Pyspark is not reading this query in the same way as Databricks SQL. Databricks SQL is able to parse and execute multiple queries separated by a semicolon. But Pyspark throws a ParseException while reading any statement written after semicolon.
So, using a single query with spark.sql is recommended (single line query or multiline query) instead of using a single spark.sql to execute multiple SQL queries.
The workaround using Pyspark SQL would be to use one spark.sql for each SQL query. You can modify your code as following:
X = "CREATE OR REPLACE TABLE Invoices (InvoiceID INT, CustomerID INT, BillToCustomerID INT, OrderID INT, DeliveryMethodID INT, ContactPersonID INT, AccountsPersonID INT, SalespersonPersonID INT, PackedByPersonID INT, InvoiceDate TIMESTAMP, CustomerPurchaseOrderNumber INT, IsCreditNote STRING, CreditNoteReason STRING, Comments STRING, DeliveryInstructions STRING, InternalComments STRING, TotalDryItems INT, TotalChillerItems STRING, DeliveryRun STRING, RunPosition STRING, ReturnedDeliveryData STRING, ConfirmedDeliveryTime TIMESTAMP, ConfirmedReceivedBy STRING, LastEditedBy INT, LastEditedWhen TIMESTAMP) LOCATION '/mnt/repro/';"
Y = "ALTER TABLE Invoices ADD COLUMN DQ_Check_Op SMALLINT"
z = "SELECT * FROM Invoices"
spark.sql(X)
spark.sql(Y)
spark.sql(z)
In my actual example, my table wasn't in the default database - I didn't think that was a relevant detail. After I wrapped my database and table name with ` (backtick), the error went off.
So, the following helped:
CREATE TABLE `sch`.`Invoices` (InvoiceID INT, CustomerID INT, BillToCustomerID INT, OrderID INT, DeliveryMethodID INT, ContactPersonID INT, AccountsPersonID INT, SalespersonPersonID INT, PackedByPersonID INT, InvoiceDate TIMESTAMP, CustomerPurchaseOrderNumber INT, IsCreditNote STRING, CreditNoteReason STRING, Comments STRING, DeliveryInstructions STRING, InternalComments STRING, TotalDryItems INT, TotalChillerItems STRING, DeliveryRun STRING, RunPosition STRING, ReturnedDeliveryData STRING, ConfirmedDeliveryTime TIMESTAMP, ConfirmedReceivedBy STRING, LastEditedBy INT, LastEditedWhen TIMESTAMP) LOCATION '/mnt/repro/';ALTER TABLE `sch`.`Invoices` ADD COLUMN DQ_Check_Op SMALLINT"
CREATE EXTERNAL TABLE schema_vtvs_ai_ext.fire(
fire_number VARCHAR(50),
fire_year DATE,
assessment_datetime INTEGER,
size_class CHAR,
fire_location_latitude REAL,
fire_location_longitude REAL,
fire_origin VARCHAR(50),
general_cause_desc VARCHAR(50),
activity_class VARCHAR(50),
true_cause VARCHAR(50),
fire_start_date DATE,
det_agent_type VARCHAR(50),
det_agent VARCHAR(50),
discovered_date DATE,
reported_date DATE,
start_for_fire_date DATE,
fire_fighting_start_date DATE,
initial_action_by VARCHAR(50),
fire_type VARCHAR(50),
fire_position_on_slope VARCHAR(50),
weather_conditions_over_fire VARCHAR(50),
fuel_type VARCHAR(50),
bh_fs_date DATE,
uc_fs_date DATE,
ex_fs_date DATE
);
This is the SQL code i have written to add an external table in Redhsift schema but the below error. i can't seem to see where the error is?
[Amazon](500310) Invalid operation: syntax error at end of input Position: 684;
If your data is in Amazon S3, then you need to specify the file format (via STORED AS) and the path to data files in S3 (via LOCATION).
This is the example query for csv files (with 1 line header):
create external table <external_schema>.<table_name> (...)
row format delimited
fields terminated by ','
stored as textfile
location 's3://mybucket/myfolder/'
table properties ('numRows'='100', 'skip.header.line.count'='1');
See official doc for details.
I receive error as below every time when i select external table that i have created.
ORA-29913: bład podczas wykonywania wywołania (callout) ODCIEXTTABLEOPEN
ORA-29400: bład kartrydza danych
KUP-00554: error encountered while parsing access parameters
KUP-01005: syntax error: found "minussign": expecting one of: "badfile, byteordermark, characterset, column, data, delimited, discardfile, dnfs_enable, dnfs_disable, disable_directory_link_check, field, fields, fixed, io_options, load, logfile, language, nodiscardfile, nobadfile, nologfile, date_cache, dnfs_readbuffers, preprocessor, readsize, string, skip, territory, variable, xmltag"
KUP-01007: at line 4 column 23
29913. 00000 - "error in executing %s callout"
The external table is created successfully. Here is the script which creates external table:
CREATE TABLE TB_CNEI_01C
(
NEW_OMC_ID VARCHAR(2),
NEW_OMC_NM VARCHAR(8),
NEW_BSS_ID VARCHAR(6),
NEW_BSS_NM VARCHAR(20),
OMC_ID VARCHAR(2),
OMC_NM VARCHAR(8),
OLD_BSS_ID VARCHAR(6),
OLD_BSS_NM VARCHAR(20),
DEPTH_NO INTEGER,
NE_TP_NO INTEGER,
OP_YN INTEGER,
FAC_ALIAS_NM VARCHAR(20),
FAC_GRP_ALIAS_NM VARCHAR(20),
SPC_VAL VARCHAR(4),
INMS_FAC_LCLS_CD VARCHAR(2),
INMS_FAC_MCLS_CD VARCHAR(3),
INMS_FAC_SCLS_CD VARCHAR(3),
INMS_FAC_SCLS_DTL_CD VARCHAR(2),
LDEPT_ID VARCHAR(3),
FAC_ID VARCHAR(15),
MME_IP_ADDR VARCHAR(20),
MDEPT_ID VARCHAR(4),
HW_TP_NM VARCHAR(20),
MME_POOL_NM VARCHAR(20),
BORD_CNT INTEGER,
FAC_DTL_CLSFN_NM VARCHAR(50),
INSTL_FLOOR_NM VARCHAR(20),
INSTL_LOC_NM VARCHAR(30)
)
ORGANIZATION EXTERNAL
(
TYPE oracle_loader
DEFAULT DIRECTORY EXTERNAL_DATA
ACCESS PARAMETERS
(
RECORDS DELIMITED BY NEWLINE
badfile EXTERNAL_DATA:'testTable.bad'
logfile EXTERNAL_DATA:'testTable.log'
CHARACTERSET x-IBM949
FIELDS TERMINATED BY ','
MISSING FIELD VALUES ARE NULL
(
NEW_OMC_ID VARCHAR(2),
NEW_OMC_NM VARCHAR(8),
NEW_BSS_ID VARCHAR(6),
NEW_BSS_NM VARCHAR(20),
OMC_ID VARCHAR(2),
OMC_NM VARCHAR(8),
OLD_BSS_ID VARCHAR(6),
OLD_BSS_NM VARCHAR(20),
DEPTH_NO INTEGER,
NE_TP_NO INTEGER,
OP_YN INTEGER,
FAC_ALIAS_NM VARCHAR(20),
FAC_GRP_ALIAS_NM VARCHAR(20),
SPC_VAL VARCHAR(4),
INMS_FAC_LCLS_CD VARCHAR(2),
INMS_FAC_MCLS_CD VARCHAR(3),
INMS_FAC_SCLS_CD VARCHAR(3),
INMS_FAC_SCLS_DTL_CD VARCHAR(2),
LDEPT_ID VARCHAR(3),
FAC_ID VARCHAR(15),
MME_IP_ADDR VARCHAR(20),
MDEPT_ID VARCHAR(4),
HW_TP_NM VARCHAR(20),
MME_POOL_NM VARCHAR(20),
BORD_CNT INTEGER,
FAC_DTL_CLSFN_NM VARCHAR(50),
INSTL_FLOOR_NM VARCHAR(20),
INSTL_LOC_NM VARCHAR(30)
)
)
LOCATION ('TB_CNEI_01C.csv')
);
I have checked all permisions for data directory and data files
I had a few commented lines in my 'CREATE TABLE ..' script. I removed those commented lines and the error disappeared.
I received the above suggestion from : http://www.orafaq.com/forum/t/182288/
It seems your CHARACTERSET(x-IBM949) containing - character is not valid
You may try the other alternatives without that sign,
such as
AL32UTF8, US7ASCII, WE8MSWIN1252 .. etc.
I'm trying yo create a table unfortunately i'm getting an error.
Here is the query that i'm running:
CREATE TABLE UpStreamParam (
macaddress CHAR(20),
datats BIGINT,
cmtstimestamp BIGINT,
modulation INT,
chnlidx INT,
severity BIGINT,
rxpower FLOAT,
sigqnoise FLOAT,
noisedeviation FLOAT,
prefecber FLOAT,
uncorrectables BIGINT)
STORED AS ORC tblproperties ("orc.compress"="SNAPPY", "orc.bloom.filter.columns"="macaddress")
LOCATION '/usr/hive/warehouse/UpStreamParam'
PARTITIONED BY(cmtsid CHAR(50), date int);
This is the error that i'm getting
ERROR: syntax error at or near "STORED"
LINE 21: STORED AS ORC tblproperties ("orc.compress"="SNAPPY", "orc.b...
Any idea how to solve it?
I have version 8.4.20
You must eliminate this part:
STORED AS ORC tblproperties ("orc.compress"="SNAPPY", "orc.bloom.filter.columns"="macaddress")
LOCATION '/usr/hive/warehouse/UpStreamParam'
PARTITIONED BY(cmtsid CHAR(50), date int);
since these kinds of physical storage specifications are not allowed in PostgreSQL. See the Create Table page.
Closed. This question is not reproducible or was caused by typos. It is not currently accepting answers.
This question was caused by a typo or a problem that can no longer be reproduced. While similar questions may be on-topic here, this one was resolved in a way less likely to help future readers.
Closed 8 years ago.
Improve this question
While doing SSIS job I got an error in my create table query
Incorrect Syntax near '-' .(Microsoft SQL Server Native Client 10.0)
SQL statement:
CREATE TABLE Staging_ACD_Precision_Queue
(
PrecisionQueueID int,
BucketIntervalID int,
EnterpriseName varchar(32),
AgentOrdering int,
CallOrdering int,
Description varchar(255),
ServiceLevelThreshold int,
ServiceLevelType smallint,
ForceExpandingQueue varchar(1),
Deleted varchar(1),
ChangeStamp int,
Partner varchar(4),
Center varchar(4),
Partner-Center varchar(9),
LOB varchar(4),
Circle varchar(4),
TypeOfBusiness varchar(4)
)
I tried the above query. I guess from the error message that, the issue is in Partner-Center varchar(9) but I failed to fix this error. Since I'm new to SSIS and SQL queries I don't know to fix the issue.
You need to put square brackets around Partner-Center, e.g. [Partner-Center]. Embedded spaces or special characters are not allowed in field names, but you can escape them by putting the field name within square brackets.
CREATE TABLE Staging_ACD_Precision_Queue (
PrecisionQueueID int,
BucketIntervalID int,
EnterpriseName varchar(32),
AgentOrdering int,
CallOrdering int,
Description varchar(255),
ServiceLevelThreshold int,
ServiceLevelType smallint,
ForceExpandingQueue varchar(1),
Deleted varchar(1),
ChangeStamp int,
Partner varchar(4),
Center varchar(4),
[Partner-Center] varchar(9),
LOB varchar(4),
Circle varchar(4),
TypeOfBusiness varchar(4)
)
As others have stated here, I too would like to recommend you avoid using a hyphen in the field name and go with either Partner_Center or PartnerCenter as the field name instead of Partner-Center.
SQL Server does not allow - in bare field names. However, you can "escape" it by surrounding the field name with square brackets ([]):
CREATE TABLE Staging_ACD_Precision_Queue (
PrecisionQueueID int,
BucketIntervalID int,
EnterpriseName varchar(32),
AgentOrdering int,
CallOrdering int,
Description varchar(255),
ServiceLevelThreshold int,
ServiceLevelType smallint,
ForceExpandingQueue varchar(1),
Deleted varchar(1),
ChangeStamp int,
Partner varchar(4),
Center varchar(4),
[Partner-Center] varchar(9),
LOB varchar(4),
Circle varchar(4),
TypeOfBusiness varchar(4)
)
You can't have "-" in column names. you can do underscores. but not minus signs.
Actually you can however you have to [ ] around column name to create with a dash and everytime you want to access you have to [var]
Change
Partner-Center varchar(9),
to be
Partner_Center varchar(9),
or
[Partner-Center] varchar(9)