Initial situation:
I'm stuck in a simple task (in my opinion it should be simple...)
I have a collection of data which should be exported weekly to Excel.
This export contains 104 columns, from which 57 are nvarchar(max) and contains item descriptions and other information in different languages for our sales guys.
The report will have something around 2000 to 8000 rows.
I use a SQL Server 2017 CU 16
My intention:
I intended to do an SSI Job with an Excel template where the columns are predefined (width, data type and so on).
This job would have something like those steps:
Delete existing Excel file
copy Excel template as a new Excel file
DataFlowTask using SQL Server as the source and Excel destination as the target
What I already tried:
If I use the excel template with only headers, I get the following error for each of the nvarchar(max) columns:
[Excel Destination [2]] Error: An error occurred while setting up a
binding for the "ColumnName" column. The binding status was
"DT_NTEXT".
When I prepare the template having it prefilled with one row. This row has a long text (more than 255 characters) for the columns where nvarchar(max) is in the source, everything runs fine but, this dummy line is still existing.
Another try I did was dropping the sheet using an "Execute SQL Task" to the Excel File Connection and recreating the sheet using a create table statement in another "Execute SQL Task" to the same Excel file connection, I get the same error as above. Although I'm using NTEXT as the datatype for the relevant columns.
Question:
How can I export data seamlessly into a preformatted excel file which contains NTEXT?
Thank you very much in advance for any assistance.
I have to export a table from a SQL Server, the table contains a column that has a large text content with the maximum length of the text going up to 100,000 characters.
When I use Excel as an export destination, I find out that the length of this text is capped and truncated to 32,765.
Is there an export format that preserves the length?
Note:
I will eventually be importing this data into another SQL Server
The destination SQL Server is in another network, so linked servers and other local options are not feasible
I don't have access to the actual server, so generating back up is difficult
As is documented in the Excel specifications and limits the maximum characters that can be stored in a single Excel cell is 32,767 characters; hence why your data is being truncated.
You might be better off exporting to a CSV, however, note that Quote Identified CSV files aren't supported within bcp/BULK INSERT until SQL Server 2019 (currently in preview). You can use a characters like || to denote a field delimited, however, if you have any line breaks you'll need to choose a different row delimitor too. SSIS, and other ETL tools, however, do support quote identified CSV files; so you can use something like that.
Otherwise, if you need to export such long values and want to use Excel as much as you can (which I actually personally don't recommend due to those awful ACE drivers), I would suggest exporting the (n)varchar(MAX) values to something else, like a text file, and naming each file with the value of your Primary Key included. Then, when you import the data back you can retrieve the (n)varchar(MAX) value again from each individual file.
The .sql is the best format for sql table. Is the native format for sql table, with that, you haven't to concert the export.
First, a grumble: MS builds SQL Server Studio AND Excel, but can't make one save in the standard format of the other?
OK, I'm a data analyst, but not allowed to change/mod either the data or structures directly. So full READ, but no WRITE.
I'm trying to do a dump so I can do some of this analysis offline, as I have no remote access either.
So one VARCHAR2 column in this table is for comments on the purchase of the asset being described/tracked. Of course, there are commas. The only export types built into SQL Server Studio are .csv and .txt, and .csv just turns into a mess when 'comma' is included as a delimiter.
So after an hour or so of screwing around with this, (including reading a thread on methods for excluding the one column from a SELECT while still exporting the other 221 columns in the table, without having to write them all out manually (fun reading, impressive, but means I'd have to figure out which of them actually works, and then still export the one column separately and insert it in the Excel separately)) I am throwing this problem on the pile at StackOverflow.
Someone else must have worked around this frustration of the .csv format as export VS the commas embedded in 'comment' text.
Any help would be appreciated.
Why don't you simply select all data in ssms result window, then copy and then paste in a blank excel file?
It should copy paste all data in correct format including comma valued fields in single column.
Try that.
So If you replace the ' to some special character you can export it.
Select
Replace(columnName,'''','`')
from Table
Other solution if you use the manager studio
https://learn.microsoft.com/en-us/sql/integration-services/import-export-data/start-the-sql-server-import-and-export-wizard
Okay, so I basically have a huge excel list with two columns, ID and Value. I'd like to update an Employee table in my SQL and set a column in that table equal to the Value from the excel file where the ID of the excel file is equal to the ID of the Employee table.
I've looked at other similar questions and they use OpenRowSet and other SQL excel functionality, but unfortunately it looks like our DBA hasn't installed any of that. What can I do as an alternative to update that table?
As for what I've tried, I created two XML lists and separated each column of the DB into a temporary table so now I have two temporary tables, one for each excel column, but now I'm reading that combining two tables without a common identifier is really bad practice so I'm back at square one.
You can do that easily using SSIS. You can use excel source, Lookup transformation or Merge Join and OLE DB destination.
I've developed a huge table in excel and now facing problem in transferring it into the postgresql database. I've downloaded the odbc software and I'm able to open table created in postgresql with excel. However, I'm not able to do it in a reverse manner which is creating a table in excel and open it in the postgresql. So I would like to know it is can be done in this way or is there any alternative ways that can create a large table with pgAdmin III cause inserting the data raw by raw is quite tedious.
The typical answer is this:
In Excel, File/Save As, select CSV, save your current sheet.
transfer to a holding directory on the Pg server the postgres user can access
in PostgreSQL:
COPY mytable FROM '/path/to/csv/file' WITH CSV HEADER; -- must be superuser
But there are other ways to do this too. PostgreSQL is an amazingly programmable database. These include:
Write a module in pl/javaU, pl/perlU, or other untrusted language to access file, parse it, and manage the structure.
Use CSV and the fdw_file to access it as a pseudo-table
Use DBILink and DBD::Excel
Write your own foreign data wrapper for reading Excel files.
The possibilities are literally endless....
For python you could use openpyxl for all 2010 and newer file formats (xlsx).
Al Sweigart has a full tutorial from automate the boring parts on working with excel spreadsheets its very indepth and the whole book and accompanying Udemy course are great resources.
From his example
>>> import openpyxl
>>> wb = openpyxl.load_workbook('example.xlsx')
>>> wb.get_sheet_names()
['Sheet1', 'Sheet2', 'Sheet3']
>>> sheet = wb.get_sheet_by_name('Sheet3')
>>> sheet
<Worksheet "Sheet3">
Understandably once you have this access you can now use psycopg to parse the data to postgres as you normally would do.
This is a link to a list of python resources at python-excel also xlwings provides a large array of features for using python in place of vba in excel.
You can also use psql console to execute \copy without need to send file to Postgresql server machine. The command is the same:
\copy mytable [ ( column_list ) ] FROM '/path/to/csv/file' WITH CSV HEADER
A method that I use is to load the table into R as a data.frame, then use dbWriteTable to push it to PostgreSQL. These two steps are shown below.
Load Excel data into R
R's data.frame objects are database-like, where named columns have explicit types, such as text or numbers. There are several ways to get a spreadsheet into R, such as XLConnect. However, a really simple method is to select the range of the Excel table (including the header), copy it (i.e. CTRL+C), then in R use this command to get it from the clipboard:
d <- read.table("clipboard", header=TRUE, sep="\t", quote="\"", na.strings="", as.is=TRUE)
If you have RStudio, you can easily view the d object to make sure it is as expected.
Push it to PostgreSQL
Ensure you have RPostgreSQL installed from CRAN, then make a connection and send the data.frame to the database:
library(RPostgreSQL)
conn <- dbConnect(PostgreSQL(), dbname="mydb")
dbWriteTable(conn, "some_table_name", d)
Now some_table_name should appear in the database.
Some common clean-up steps can be done from pgAdmin or psql:
ALTER TABLE some_table_name RENAME "row.names" TO id;
ALTER TABLE some_table_name ALTER COLUMN id TYPE integer USING id::integer;
ALTER TABLE some_table_name ADD PRIMARY KEY (id);
As explained here http://www.postgresonline.com/journal/categories/journal/archives/339-OGR-foreign-data-wrapper-on-Windows-first-taste.html
With ogr_fdw module, its possible to open the excel sheet as foreign table in pgsql and query it directly like any other regular tables in pgsql.
This is useful for reading data from the same regularly updated table
To do this, the table header in your spreadsheet must be clean, the current ogr_fdw driver can't deal with wide-width character or new lines etc. with these characters, you will probably not be able to reference the column in pgsql due to encoding issue. (Major reason I can't use this wonderful extension.)
The ogr_fdw pre-build binaries for windows are located here http://winnie.postgis.net/download/windows/pg96/buildbot/extras/
change the version number in link to download corresponding builds.
extract the file to pgsql folder to overwrite the same name sub-folders.
restart pgsql. Before the test drive, the module needs to be installed by executing:
CREATE EXTENSION ogr_fdw;
Usage in brief:
use ogr_fdw_info.exe to prob the excel file for sheet name list
ogr_fdw_info -s "C:/excel.xlsx"
use "ogr_fdw_info.exe -l" to prob a individual sheet and generate a table definition code.
ogr_fdw_info -s "C:/excel.xlsx" -l "sheetname"
Execute the generated definition code in pgsql, a foreign table is created and mapped to your excel file. it can be queried like regular tables.
This is especially useful, if you have many small files with the same table structure. Just change the path and name in definition, and update the definition will be enough.
This plugin supports both XLSX and XLS file.
According to the document it also possible to write data back into the spreadsheet file, but all the fancy formatting in your excel will be lost, the file is recreated on write.
If the excel file is huge. This will not work. which is another reason I didn't use this extension. It load data in one time.
But this extension also support ODBC interface, it should be possible to use windows' ODBC excel file driver to create a ODBC source for the excel file and use ogr_fdw or any other pgsql's ODBC foreign data wrapper to query this intermediate ODBC source. This should be fairly stable.
The downside is that you can't change file location or name easily within pgsql like in the previous approach.
A friendly reminder. The permission issue applies to this fdw extensions. since its loaded into pgsql service. pgsql must have access privileged to the excel files.
It is possible using ogr2ogr:
C:\Program Files\PostgreSQL\12\bin\ogr2ogr.exe -f "PostgreSQL" PG:"host=someip user=someuser dbname=somedb password=somepw" C:/folder/excelfile.xlsx -nln newtablenameinpostgres -oo AUTODETECT_TYPE=YES
(Not sure if ogr2ogr is included in postgres installation or if I got it with postgis extension.)
I have used Excel/PowerPivot to create the postgreSQL insert statement. Seems like overkill, except when you need to do it over and over again. Once the data is in the PowerPivot window, I add successive columns with concatenate statements to 'build' the insert statement. I create a flattened pivot table with that last and final column. Copy and paste the resulting insert statement into my EXISTING postgreSQL table with pgAdmin.
Example two column table (my table has 30 columns from which I import successive contents over and over with the same Excel/PowerPivot.)
Column1 {a,b,...} Column2 {1,2,...}
In PowerPivot I add calculated columns with the following commands:
Calculated Column 1 has "insert into table_name values ('"
Calculated Column 2 has CONCATENATE([Calculated Column 1],CONCATENATE([Column1],"','"))
...until you get to the last column and you need to terminate the insert statement:
Calculated Column 3 has CONCATENATE([Calculated Column 2],CONCATENATE([Column2],"');"
Then in PowerPivot I add a flattened pivot table and have all of the insert statement that I just copy and paste to pgAgent.
Resulting insert statements:
insert into table_name values ('a','1');
insert into table_name values ('b','2');
insert into table_name values ('c','3');
NOTE: If you are familiar with the power pivot CONCATENATE statement, you know that it can only handle 2 arguments (nuts). Would be nice if it allowed more.
You can handle loading the excel file content by writing Java code using Apache POI library (https://poi.apache.org/). The library is developed for working with MS office application data including Excel.
I have recently created the application based on the technology that will help you to load Excel files to the Postgres database. The application is available under http://www.abespalov.com/. The application is tested only for Windows, but should work for Linux as well.
The application automatically creates necessary tables with the same columns as in the Excel files and populate the tables with content. You can export several files in parallel. You can skip the step to convert the files into the CSV format. The application handles the xls and xlsx formats.
Overall application stages are :
Load the excel file content. Here is the code depending on file extension:
{
fileExtension = FilenameUtils.getExtension(inputSheetFile.getName());
if (fileExtension.equalsIgnoreCase("xlsx")) {
workbook = createWorkbook(openOPCPackage(inputSheetFile));
} else {
workbook =
createWorkbook(openNPOIFSFileSystemPackage(inputSheetFile));
}
sheet = workbook.getSheetAt(0);
}
Establish Postgres JDBC connection
Create a Postgres table
Iterate over the sheet and inset rows into the table. Here is a piece of Java code :
{
Iterator<Row> rowIterator = InitInputFilesImpl.sheet.rowIterator();
//skip a header
if (rowIterator.hasNext()) {
rowIterator.next();
}
while (rowIterator.hasNext()) {
Row row = (Row) rowIterator.next();
// inserting rows
}
}
Here you can find all Java code for the application created for exporting excel to Postgres (https://github.com/palych-piter/Excel2DB).
the simplest answer is to use the psql command:
it's free and is include////
psql -U postgres -p 5432 -f sql-command-file.sql
I recently discovered https://sqlizer.io, it creates insert statements from an Excel file, supports MySQL and PostgreSQL. Not sure about if it supports large files though.
You can do that easily by DataGrip .
First save your excel file as csv formate . Open the excel file then SaveAs as csv format
Go to datagrip then create the table structure according to the csv file . Suggested create the column name as the column name as Excel column
right click on the table name from the list of table name of your database then click of the import data from file . Then select the converted csv file .
.