I'm trying to create an external table in SQL Server 2019 on premise ( Polybase has been installed and all services are up and running [Instance and 2 services for polybase] , Hadoop configuration = 7).
I want to feed the external table with multiple ".csv" or ".txt" files , for this example, Ill use 1 ".csv" file.
I've the lattest
OBCD Driver for "Driver=Microsoft Access Text Driver (*.txt, *.csv)"
"SQL Server® 2019 for Microsoft® Windows Latest Cumulative Update"
My first step was creating the EXTERNAL DATA SOURCE in my Database where Polybase is emabled using the code below:
CREATE EXTERNAL DATA SOURCE MyODBC
WITH
(
LOCATION = 'odbc://localhost',
CONNECTION_OPTIONS = 'Driver=Microsoft Access Text Driver (*.txt, *.csv);Dbq=C:\Historical\Event\',
PUSHDOWN = OFF
);
Second, I've created the script to CREATE the EXTERNAL TABLE using dynamic SQL based on variables ( #TableName,#FolderPath variable values can change at any time).
Please see my script below:
DECLARE
#TableName VARCHAR(15) = 'BRAO',
#FolderPath VARCHAR(500) = 'BRAO\BRAO_EVEN0000_20220101.csv',
#DataSourceName VARCHAR(20) ='MyODBC',
#SQLScript NVARCHAR(MAX) =''
SET #SQLScript =N'
IF EXISTS
(
SELECT
*
FROM sys.objects
WHERE
object_id = OBJECT_ID(N''[dbo].[Gold' + #TableName + ']'')
AND type in (N''U'')
)
DROP EXTERNAL TABLE [dbo].[Gold' + #TableName + ']
CREATE EXTERNAL TABLE [dbo].[Gold' + #TableName + ']
(
[Column1] [varchar](255) NULL,
[Column2] [varchar](255) NULL,
[Column3] [varchar](255) NULL
)
WITH (DATA_SOURCE =' + #DataSourceName + ',LOCATION = ''' + #FolderPath + ''')'
EXEC sp_executesql #SQLScript
When I run my script above, I get the following error:
Msg 105121, Level 16, State 1, Line 1
105121;The specified LOCATION string 'BRAO\BRAO_EVEN0000_20220101.csv' could not be parsed. A 1-part identifier was found. Expected 2.
Have you looked at the following page? It loads a .csv file but the location is only a single folder: https://learn.microsoft.com/en-us/answers/questions/118102/polybase-load-csv-file-that-contains-text-column-w.html
Related
I've finished a big query in SSMS with a lot of columns. The result will be inserted in another server.
Usually, I create a temporary table to quickly get the data types of each columns (CREATE TABLE from a SELECT).
But now, I'm working on a server where it is forbidden to create a table (even a temporary one).
How can I get the structure of my query easily, without re-typing each column manually?
EDIT 1 :
I query on an Azure SQL Server Database.
And my result will go to an On Premise SQL Server.
EDIT 2 :
SQL Server 2016
If you have a query that you can construct as dynamic SQL (or in an object), you can usually inspect the types involved using sys.dm_exec_describe_first_result_set:
DECLARE #sql nvarchar(max) = N'SELECT database_id, name FROM sys.databases;';
SELECT name,
system_type_name,
is_nullable
FROM sys.dm_exec_describe_first_result_set(#sql, NULL, 0) AS f
ORDER BY f.column_ordinal;
Results (db<>fiddle):
name
system_type_name
is_nullable
database_id
int
False
name
nvarchar(128)
True
So you could use that output to build a CREATE TABLE statement, e.g.:
DECLARE #sql nvarchar(max),
#create nvarchar(max) = N'CREATE TABLE dbo.DBs
(--';
SET #sql = N'SELECT database_id, name FROM sys.databases;';
SELECT #create += ',' + CHAR(13) + CHAR(10) + ' '
+ name + ' '
+ system_type_name
+ CASE is_nullable WHEN 0 THEN ' NOT' ELSE ' ' END
+ ' NULL'
FROM sys.dm_exec_describe_first_result_set(#sql, NULL, 0) AS f;
SELECT #create += CHAR(13) + CHAR(10) + N');';
PRINT #create;
-- EXEC sys.sp_executesql #create;
Output (db<>fiddle):
CREATE TABLE dbo.DBs
(--,
database_id int NOT NULL,
name nvarchar(128) NULL
);
Some background:
SQL Server v.Next (Denali) : Metadata enhancements
How can I get a list of column names and types from a resultset?
I want to programmatically set the permissions (i.e. GRANT) on a newly created table. I was hoping to get SQL Server to show me the script for that by going to another table in the database and doing a right-click and then "Script Table as" but I don't see the option for GRANT underneath that.
Is it possible to get SQL Server to show me the script for this?
Look around some more. The simple version is to use the dialog where you set permissions. E.g.,
A more general and advanced approach - with far more options - is to use the script feature at the database level. On your database, rt click and select , , and then run through the wizard to select your table and the scripting options you want. Be sure to click the Advanced button in the scripting options tab - where you will see "object-level permissions" (off by default).
Too long to comment.
GRANT <n> ON YourDatabase.YourSchema.YourTable TO YourSpecificUser
In this case, since it's a table <n> can be one of the following:
DELETE
INSERT
REFERENCES
SELECT
UPDATE
Note, if a user has a fixed database role, they could have more access than you explicitly grant them. Read more about GRANT Object Permissions.
Of note, to return a list of permissions on a table, you can use sp_table_privileges
sp_table_privileges #table_name = 'YourTable'
You can capture these results and then loop through them to build a dynamic sql query.
Replace the script below with your TableName and what ever your NewTableName is. When you are satisfied with the print out, you can uncomment the exec(#sql) to execute the code that's printed.
if object_id('tempdb..#priv') is not null
drop table #priv
create table #priv( ID int identity (1,1)
,TABLE_QUALIFIER varchar(64)
,TABLE_OWNER VARCHAR(64)
,TABLE_NAME VARCHAR(64)
,GRANTOR VARCHAR(64)
,GRANTEE VARCHAR(64)
,PRIVILEGE VARCHAR(64)
,IS_GRANTABLE VARCHAR(8))
insert into #priv
exec sp_table_privileges #table_name = 'YourTableName'
declare #i int = 1
declare #max int = (select max(id) from #priv)
declare #sql varchar(max) = ''
while (#i <= #max)
begin
set #sql = #sql + (select ' GRANT ' + stuff(PRIVILEGE,1,0,' ') + ' ON ' + stuff(TABLE_NAME,1,0,' ') + ' TO ' + stuff(GRANTEE,1,0,' ') + char(13) + ' GO ' + char(13) from #priv where ID = #i)
set #i = #i + 1
end
print(#sql)
set #sql = replace(#sql,'YourTableName','NewTableName')
print(#sql)
--exec(#sql)
Working on a DB project in VS2015 (Azure SQL V12 proj). I need to use an external table reference so I have to create a credential object to authenticate with the remote server. Code and error below, not sure what I am missing.
SQL code
CREATE DATABASE SCOPED CREDENTIAL [mycredential]
WITH
IDENTITY = 'SomeIdentity',
SECRET = 'SomeSecret';
Errors:
Incorrect syntax near '[mycredential]'. Expecting '='
Incorrect syntax near 'IDENTITY'. Expecting AW_CHANGE_TRACKING_CONTEXT, AW_ID,
AW_XMLNAMESPACES, or QUOTED_ID.
Ok, I also encountered this in VS2017 DB project, the way I did it is to use stored procs, so that intellisense will not trigger an error. As i find the code is working when run. Below is the stored proc I used:
you define your external reference table in "YOUR_EXTERN_TABLE" of "CREATE EXTERNAL TABLE" statement (which, in this example, is set to have schema of ID and Name columns):
CREATE PROCEDURE [dbo].[StoredProcWithExternalRefs]
#DatabaseName AS NVARCHAR(30),
#Passw AS NVARCHAR(100),
#SaID AS NVARCHAR(100),
#DataSource AS NVARCHAR(512),
#Catalog AS NVARCHAR(200)
AS
BEGIN
SET NOCOUNT ON;
SET IMPLICIT_TRANSACTIONS OFF;
DECLARE #SQLString nvarchar(200);
PRINT 'START'
PRINT 'CREATE DATABASE'
SET #SQLString = N'CREATE DATABASE [' + #DatabaseName + ']'
EXECUTE sp_executesql #SQLString
SET #SQLString = N'CREATE MASTER KEY ENCRYPTION BY PASSWORD = ''' + #Passw + ''';
CREATE DATABASE SCOPED CREDENTIAL credential_name
WITH IDENTITY = ''' + #SaID + '''
, SECRET = ''' + #Passw + ''';
CREATE EXTERNAL DATA SOURCE RemoteReferenceData
WITH
(
TYPE=RDBMS,
LOCATION=''' + #DataSource + ''',
DATABASE_NAME=''' + #Catalog + ''',
CREDENTIAL= credential_name
);
CREATE EXTERNAL TABLE YOUR_EXTERN_TABLE(
[Id] [int] NOT NULL,
[Name] [nvarchar](20) NOT NULL,
) WITH ( DATA_SOURCE = RemoteReferenceData );'
...
EXECUTE sp_executesql #SQLString
PRINT 'DONE.'
END
you can add additional external tables with the same pattern in the "CREATE EXTERNAL TABLE" statement and the schema.
here is a reference to guide you: https://sqldusty.com/2017/05/30/setting-up-cross-database-queries-in-azure-sql-database/
Currently, our local builds include a database build & deploy step that deploys databases to the developers' local machines based on a database model contained within a Sql Server 2008 Database Project. This is fine for deploying the database schema, but is unable to deploy any database diagrams that have been defined.
In order to deploy the diagrams, we are currently having to include CREATE scripts for the sysdiagrams table and the stored procedures upon which the designer is dependent, and a data insert script for the diagram definitions, as part of the post deployment stage.
Is there a better way to deploy the sysdiagrams table and stored procedures? These are created within sql server management studio when clicking yes in the dialog that asks whether I'd like to create the support objects for diagramming - does this invoke something that I can hook into? I can't find a relevant stored procedure.
There is a tool I've used to import and export SQL diagrams. The version for 2008 is Tool_ScriptDiagram2008 by Craig Dunn. Take a look at http://www.conceptdevelopment.net/Database/ScriptDiagram2008/. I use it to export the diagrams to a text format that can be saved in source control, then these scripts from souce control can be used to recreate the diagrams on any machine.
There isn't a means of doing this and you should bear in mind that Database diagram format differs between releases of SQL Server thus a SQL2008 diagram can't be directlt inserted into a SQL 2000 database. The diagram is storede as a binary file within the database.
You could carry on doing what you're currently doing or have a lokk at the tool on this link.
NOTE: I have no connection to the site and so can't vouch for it, but it may give you some ideas.
There is no such a procedure. If you want to keep your diagrams in safe place - script the table sysdiagrams and supporting objects if any with data and keep the generated script. Thats all.
My minor modifications to a well used stored procedure that can create the sysdiagrams table (original source):
/**
<summary>
Script Sql Server 2008 diagrams (inspired by usp_ScriptDatabaseDiagrams for Sql Server 2000 by Clay Beatty,
and Tool_ScriptDiagram2005 by yours truly)
</summary>
<example>
USE [YourDatabaseName]
EXEC ScriptDiagram2008 'DiagramName'
</example>
<author>Craig Dunn</author>
<remarks>
Helpful Articles
----------------
1) Upload / Download to Sql 2005
http://staceyw.spaces.live.com/blog/cns!F4A38E96E598161E!404.entry
2) MSDN: Using Large-Value Data Types
http://msdn2.microsoft.com/en-us/library/ms178158.aspx
3) "original" Script, Save, Export SQL 2000 Database Diagrams
http://www.thescripts.com/forum/thread81534.html
<![CDATA[ http://groups-beta.google.com/group/comp.databases.ms-sqlserver/browse_frm/thread/ca9a9229d06a56f9?dq=&hl=en&lr=&ie=UTF-8&oe=UTF-8&prev=/groups%3Fdq%3D%26num%3D25%26hl%3Den%26lr%3D%26ie%3DUTF-8%26oe%3DUTF-8%26group%3Dcomp.databases.ms-sqlserver%26start%3D25 ]]>
4) SQL2008 'undocumented' sys.fn_varbintohexstr
http://www.sqlservercentral.com/Forums/Topic664234-1496-1.aspx
</remarks>
<param name="name">Name of the diagram in the Sql Server database instance</param>
*/
CREATE PROCEDURE [dbo].[ScriptDiagram2008]
(
#DiagramName VARCHAR(128)
)
AS
BEGIN
DECLARE #diagram_id INT
DECLARE #index INT
DECLARE #size INT
DECLARE #chunk INT
DECLARE #line VARCHAR(max)
-- Set start index, and chunk 'constant' value
SET #index = 1 --
SET #chunk = 32 -- values that work: 2, 6
-- values that fail: 15,16, 64
-- Get PK diagram_id using the diagram's name (which is what the user is familiar with)
SELECT
#diagram_id = diagram_id,
#size = DATALENGTH(definition)
FROM sysdiagrams
WHERE [name] = #DiagramName
IF #diagram_id IS NULL
BEGIN
PRINT '/**<error>'
PRINT 'Diagram name [' + #DiagramName + '] could not be found.'
PRINT '</error>*/'
END
ELSE -- Diagram exists
BEGIN
-- Now with the diagram_id, do all the work
PRINT '/**'
PRINT '<summary>'
PRINT 'Restore diagram ''' + #DiagramName + ''''
PRINT '</summary>'
PRINT '<remarks>'
PRINT 'Generated by ScriptDiagram2008'
PRINT 'Will attempt to create [sysdiagrams] table if it doesn''t already exist'
PRINT '</remarks>'
PRINT '<generated>' + LEFT(CONVERT(VARCHAR(23), GETDATE(), 121), 16) + '</generated>'
PRINT '*/'
PRINT 'PRINT ''=== ScriptDiagram2008 restore diagram [' + #DiagramName + '] ==='''
PRINT ' -- If the sysdiagrams table has not been created in this database, create it!
IF NOT EXISTS (SELECT * FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME = ''sysdiagrams'')
BEGIN
-- Create table script generated by Sql Server Management Studio
-- _Assume_ this is roughly equivalent to what Sql Server/Management Studio
-- creates the first time you add a diagram to a 2008 database
CREATE TABLE [dbo].[sysdiagrams](
[name] [sysname] NOT NULL,
[principal_id] [int] NOT NULL,
[diagram_id] [int] IDENTITY(1,1) NOT NULL,
[version] [int] NULL,
[definition] [varbinary](max) NULL,
PRIMARY KEY CLUSTERED
(
[diagram_id] ASC
)WITH (PAD_INDEX = OFF, IGNORE_DUP_KEY = OFF) ,
CONSTRAINT [UK_principal_name] UNIQUE NONCLUSTERED
(
[principal_id] ASC,
[name] ASC
)WITH (PAD_INDEX = OFF, IGNORE_DUP_KEY = OFF)
)
EXEC sys.sp_addextendedproperty #name=N''microsoft_database_tools_support'', #value=1 , #level0type=N''SCHEMA'',#level0name=N''dbo'', #level1type=N''TABLE'',#level1name=N''sysdiagrams''
PRINT ''[sysdiagrams] table was created as it did not already exist''
END
-- Target table will now exist, if it didn''t before'
PRINT 'SET NOCOUNT ON -- Hide (1 row affected) messages'
PRINT 'DECLARE #newid INT'
PRINT ''
PRINT 'PRINT ''Create row for new diagram'''
-- Output the INSERT that _creates_ the diagram record, with a non-NULL [definition],
-- important because .WRITE *cannot* be called against a NULL value (in the WHILE loop)
-- so we insert 0x so that .WRITE has 'something' to append to...
PRINT 'BEGIN TRY'
PRINT ' PRINT ''Write diagram ' + #DiagramName + ' into new row (and get [diagram_id])'''
SELECT #line =
' INSERT INTO sysdiagrams ([name], [principal_id], [version], [definition])'
+ ' VALUES (''' + [name] + ''', '+ CAST (principal_id AS VARCHAR(100))+', '+CAST (version AS VARCHAR(100))+', 0x)'
FROM sysdiagrams WHERE diagram_id = #diagram_id
PRINT #line
PRINT ' SET #newid = SCOPE_IDENTITY()'
PRINT 'END TRY'
PRINT 'BEGIN CATCH'
PRINT ' PRINT ''XxXxX '' + Error_Message() + '' XxXxX'''
PRINT ' PRINT ''XxXxX END ScriptDiagram2008 - fix the error before running again XxXxX'''
PRINT ' RETURN'
PRINT 'END CATCH'
PRINT ''
PRINT 'PRINT ''Now add all the binary data...'''
PRINT 'BEGIN TRY'
WHILE #index < #size
BEGIN
-- Output as many UPDATE statements as required to append all the diagram binary
-- data, represented as hexadecimal strings
SELECT #line =
' UPDATE sysdiagrams SET [definition] .Write ('
+ ' ' + UPPER(sys.fn_varbintohexstr (SUBSTRING (definition, #index, #chunk)))
+ ', null, 0) WHERE diagram_id = #newid -- index:' + CAST(#index AS VARCHAR(100))
FROM sysdiagrams
WHERE diagram_id = #diagram_id
PRINT #line
SET #index = #index + #chunk
END
PRINT ''
PRINT ' PRINT ''=== Finished writing diagram id '' + CAST(#newid AS VARCHAR(100)) + '' ==='''
PRINT ' PRINT ''=== Refresh your Databases-[DbName]-Database Diagrams to see the new diagram ==='''
PRINT 'END TRY'
PRINT 'BEGIN CATCH'
PRINT ' -- If we got here, the [definition] updates didn''t complete, so delete the diagram row'
PRINT ' -- (and hope it doesn''t fail!)'
PRINT ' DELETE FROM sysdiagrams WHERE diagram_id = #newid'
PRINT ' PRINT ''XxXxX '' + Error_Message() + '' XxXxX'''
PRINT ' PRINT ''XxXxX END ScriptDiagram2008 - fix the error before running again XxXxX'''
PRINT ' RETURN'
PRINT 'END CATCH'
END
END
Usage for the the dbo.ScriptDiagram2008(#DiagramName) stored procedure:
exec dbo.ScriptDiagram2008('My diagram')
It will generate a SQL script that you can save as a "portable" version of your diagram. Executing the script inserts the diagram named "My diagram" into the sysdiagrams table. It even creates that sysdiagrams table if it does not already exist.
I tried executing this stored procedure that is meant to copy the format of an already created excel sheet into another excel sheet; the former serves as a template. The stored procedure is then meant to populate the new excel sheet with the resultset from a SQL query.
When executing, it gives the following error:
Insert ExcelSource...[ExcelTable$] ( A,B,C ) select convert(varchar(200),USER_ID), FIRST_NAME,
Convert (varchar(20),CREATEDTIME)
from SERV..AaUser
OLE DB provider "Microsoft.Jet.OLEDB.4.0" for linked server "ExcelSource" returned message "Cannot start your application. The workgroup information file is missing or opened exclusively by another user.".
Msg 7399, Level 16, State 1, Line 1
The OLE DB provider "Microsoft.Jet.OLEDB.4.0" for linked server "ExcelSource" reported an error. Authentication failed.
Msg 7303, Level 16, State 1, Line 1
Cannot initialize the data source object of OLE DB provider "Microsoft.Jet.OLEDB.4.0" for linked server "ExcelSource".
the syntax for the stored procedure is:
Create proc sp_write2Excel (#fileName varchar(100),
#NumOfColumns tinyint,
#query varchar(200))
as
begin
declare #dosStmt varchar(200)
declare #tsqlStmt varchar(500)
declare #colList varchar(200)
declare #charInd tinyint
set nocount on
-- construct the columnList A,B,C ...
-- until Num Of columns is reached.
set #charInd=0
set #colList = 'A'
while #charInd < #NumOfColumns - 1
begin
set #charInd = #charInd + 1
set #colList = #colList + ',' + char(65 + #charInd)
end
-- Create an Empty Excel file as the target file name by copying the template Empty excel File
set #dosStmt = ' copy C:\emp\empty.xls ' + #fileName
exec master..xp_cmdshell #dosStmt
-- Create a "temporary" linked server to that file in order to "Export" Data
EXEC sp_addlinkedserver 'ExcelSource',
'Jet 4.0',
'Microsoft.Jet.OLEDB.4.0',
#fileName,
NULL,
'Excel 5.0'
-- construct a T-SQL statement that will actually export the query results
-- to the Table in the target linked server
set #tsqlStmt = 'Insert ExcelSource...[ExcelTable$] ' + ' ( ' + #colList + ' ) '+ #query
print #tsqlStmt
-- execute dynamically the TSQL statement
exec (#tsqlStmt)
-- drop the linked server
EXEC sp_dropserver 'ExcelSource'
set nocount off
end
Many thanks for your audience and anticipated help.
Cheers,
Tunde
Is Excel installed on the same machine as your SQL Server instance? There's a possibility that Office of the JET driver is missing.
Edit:
I think I misread the post - it sounds like the file is open already. Excel files can only be opened by one user at a time, and SQL Server is requiring exclusive access to this file. Using LockHunter may help determine what's tying the file up.