I have created EXTERNAL TABLE and I was able to get data from it.
Now I get error
"External file access failed because the specified path name '...' does not exist. Enter a valid path and try again."
But file path is correct and file exists in container.
Below is scripts which I used to create external table:
CREATE DATABASE SCOPED CREDENTIAL AzureStorageCredential
WITH
IDENTITY = 'SHARED ACCESS SIGNATURE',
SECRET = 'XXXX' ;
CREATE EXTERNAL DATA SOURCE AzureStorageContainerName
WITH
( LOCATION = 'wasbs://BlobContainerName#StorageAccountName.blob.core.windows.net' ,
CREDENTIAL = [AzureStorageCredential],
TYPE = HADOOP
) ;
CREATE EXTERNAL FILE FORMAT FileFormat_csv
WITH
(
FORMAT_TYPE = DELIMITEDTEXT,
FORMAT_OPTIONS ( FIELD_TERMINATOR = ';',
FIRST_ROW = 1,
USE_TYPE_DEFAULT = FALSE,
ENCODING = 'UTF8')
);
CREATE EXTERNAL TABLE TableExternal
(
[code] [nvarchar](100) ,
[name] [nvarchar](100)
)
WITH (
LOCATION = '/subfolder/FileName.csv',
DATA_SOURCE = AzureStorageContainerName,
FILE_FORMAT = FileFormat_csv
)
Related
We have requirement to load the data from ADLS delta data into synapse table. actually, we are writing the delta format data into ADLS gen2 from databricks. now we want to load the data from ADLS gen2(with delta table) to synapse table delta table. below steps we followed to create table but we are getting issues.
CREATE EXTERNAL FILE FORMAT DeltaFileFormat
WITH (
FORMAT_TYPE = DELTA
);
CREATE EXTERNAL DATA SOURCE test_data_source
WITH
( LOCATION = 'abfss://container#storage.dfs.core.windows.net/table_metadata/testtable'
--,CREDENTIAL = <database scoped credential>
);
CREATE EXTERNAL TABLE testtable (
job_id int,
source_type varchar(10),
server_name varchar(10),
database_name varchar(15),
table_name varchar(20),
custom_query varchar(100),
source_location varchar(500),
job_timestamp datetime2,
job_user varchar(50)
) WITH (
LOCATION = 'abfss://targetcontainer#targetstorage.dfs.core.windows.net/table_metadata/testtable',
data_source = test_data_source,
FILE_FORMAT = DeltaFileFormat
);
select * from testtable;
while query select statement, below issues throwing exception.
Content of directory on path
'https://container#storage.dfs.core.windows.net_delta_log/.' cannot
be listed.
I also tried and getting similar error.
Content of directory on path 'https://container#storage.dfs.core.windows.net_delta_log/.' cannot be listed.
This error message typically occurs when there is no data or delta files present in the _delta_log directory of the specified ADLS Gen2 storage account. The _delta_log directory is created automatically when you create a Delta table, and it contains the transaction log files for the Delta table.
In below code, the delta table files are located at /demodata/ and the external data source for this example is test_data_source18 which contain file path information and credentials. LOCATION the delta table folder or the absolute file itself, which would be located at /demodata/_delta_log/00000000000000000000.json.
CREATE EXTERNAL FILE FORMAT DeltaFileFormat
WITH (
FORMAT_TYPE = DELTA
);
CREATE EXTERNAL DATA SOURCE test_data_source18
WITH
( LOCATION = 'abfss://demo#dlsg2p.dfs.core.windows.net');
CREATE EXTERNAL TABLE testtable24(
Id varchar(20),
Name varchar(20)
) WITH (
LOCATION = '/demodata/',
data_source = test_data_source18,
FILE_FORMAT = DeltaFileFormat
);
select * from testtable24;
Output:
Reference: delta table with PolyBase
I have a Spotify CSV file in my Azure Data Lake. I am trying to create external table you SQL serverless pool in Azure Synapse.
I am getting the below error message
Bulk load data conversion error (type mismatch or invalid character for the specified codepage) for row 1, column 4 (Track_popularity) in data file https://test.dfs.core.windows.net/data/folder/updated.csv.
I am using the below script
IF NOT EXISTS (SELECT * FROM sys.external_file_formats WHERE name = 'SynapseDelimitedTextFormat')
CREATE EXTERNAL FILE FORMAT [SynapseDelimitedTextFormat]
WITH ( FORMAT_TYPE = DELIMITEDTEXT ,
FORMAT_OPTIONS (
FIELD_TERMINATOR = ',',
USE_TYPE_DEFAULT = FALSE
))
GO
IF NOT EXISTS (SELECT * FROM sys.external_data_sources WHERE name = 'test.dfs.core.windows.net')
CREATE EXTERNAL DATA SOURCE [test.dfs.core.windows.net]
WITH (
LOCATION = 'abfss://data#test.dfs.core.windows.net'
)
GO
CREATE EXTERNAL TABLE updated (
[Artist] nvarchar(4000),
[Track] nvarchar(4000),
[Track_id] nvarchar(4000),
[Track_popularity] bigint,
[Artist_id] nvarchar(4000),
[Artist_Popularity] bigint,
[Genres] nvarchar(4000),
[Followers] bigint,
[danceability] float,
[energy] float,
[key] bigint,
[loudness] float,
[mode] bigint,
[speechiness] float,
[acousticness] float,
[instrumentalness] float,
[liveness] float,
[valence] float,
[tempo] float,
[duration_ms] bigint,
[time_signature] bigint
)
WITH (
LOCATION = 'data/updated.csv',
DATA_SOURCE = [data_test_dfs_core_windows_net],
FILE_FORMAT = [SynapseDelimitedTextFormat]
)
GO
SELECT TOP 100 * FROM dbo.updated
GO
Below is the data sample
My CSV is utf-8 encoding. Not sure what is the issue. The error shows column (Track_popularity). Please advise
I’m guessing you may have a header row that should be skipped. Drop your external table and then drop and recreate the external file format as follows:
CREATE EXTERNAL FILE FORMAT [SynapseDelimitedTextFormat]
WITH ( FORMAT_TYPE = DELIMITEDTEXT ,
FORMAT_OPTIONS (
FIELD_TERMINATOR = ',',
USE_TYPE_DEFAULT = FALSE,
FIRST_ROW = 2
))
I need help in following-
Old CSV file has 3 columns- id, fname, lname
New CSV file has 4 columns- id, fname, lname, dob
I have u updated the external schema as below-
CREATE EXTERNAL TABLE [dbo].[employee]
(
[id] [int] NOT NULL,
[fname] [nvarchar](50) NULL,
[lname] [nvarchar](50) NULL,
[dob] [datetime2](7) NULL
)
WITH (DATA_SOURCE = [EmpWarehouseStorage],LOCATION = N'Employee/New',FILE_FORMAT = [PipeFileFormat],REJECT_TYPE = VALUE,REJECT_VALUE = 10)
External Data source Query-
CREATE EXTERNAL DATA SOURCE [EmpWarehouseStorage]
WITH (TYPE = HADOOP, LOCATION = N'abfss://emp#****.dfs.core.windows.net',
CREDENTIAL = [EmpWarehouseStorageKey])
External File Format -
CREATE EXTERNAL FILE FORMAT [PipeFileFormat] WITH (FORMAT_TYPE = DELIMITEDTEXT, FORMAT_OPTIONS (FIELD_TERMINATOR = N'|', USE_TYPE_DEFAULT = False))
Querying external table throws the error -
1*** rows rejected from external table [Employee] in plan step 2 of
query execution: Location: 'Employee/New/Employee_2***.csv'
Column ordinal: 79, Expected data type: datetime2.
Just for debugging purposes, I have increased the REJECT_VALUE = 10000.
I get to know that external shows only the new files data and does not show old files data.
Just from the error message, it seams that the column data of dob can not be converted to data type datetime2(7).
We don't know what string your dob data look like, please check it follow this link:Supported string literal formats for datetime2.
So the solution could be that change the datetime2 to other date data type.
Update:
Glad to hear you found the cause:
It is not just because of datetime2Actually it is related with any data type. As previous files doesn't
have that column it is considered as error data.
I keep getting this error (Referenced external data source "MyAzureBlobStorage" not found.) when loading csv from blob to Azure SQL. I am following this example and I set my blob to be public but the following just does not work:
CREATE EXTERNAL DATA SOURCE MyAzureBlobStorage
WITH ( TYPE = BLOB_STORAGE,
LOCATION = 'https://test.blob.core.windows.net/test'
);
BULK INSERT SubscriberQueue
FROM 'inputs.csv'
WITH (DATA_SOURCE = 'MyAzureBlobStorage', FORMAT='CSV');
Any ideas what I am missing here?
If you want to bulk insert from Azure blob, please refer to following script
my csv file
1,Peter,Jackson,pjackson#hotmail.com
2,Jason,Smith,jsmith#gmail.com
3,Joe,Raasi,jraasi#hotmail.com
script
create table listcustomer
(id int,
firstname varchar(60),
lastname varchar(60),
email varchar(60))
Go
CREATE EXTERNAL DATA SOURCE MyAzureBlobStorage
WITH ( TYPE = BLOB_STORAGE,
LOCATION = 'https://****.blob.core.windows.net/test'
);
Go
BULK INSERT listcustomer
FROM 'mycustomers.csv'
WITH (DATA_SOURCE = 'MyAzureBlobStorage', FORMAT='CSV');
Go
select * from listcustomer;
I'm trying to create external table as it described in
https://msdn.microsoft.com/en-us/library/mt652315.aspx
using following structure:
CREATE EXTERNAL TABLE dbo.Blob2 (
ID int NOT NULL,
Description nvarchar(max) NULL,
Title nvarchar(max) NULL
)
WITH (LOCATION='/Blob/',
DATA_SOURCE = AzureStorage,
FILE_FORMAT = TextFileFormat
);
and getting error:
Msg 46518, Level 16, State 12, Line 49
The type 'nvarchar(max)' is not supported with external tables.
Am I'm missing something? I'm seeing that in documentation for external tables nvarchar/string/text in the list
https://msdn.microsoft.com/en-us/library/dn935021.aspx
Is there any chance that I can store text data in that container?