So I am importing some images stored in SQL image columns, and I need to change them to Byte arrays since I store my images as varbinary(max) and recreate them. I would LOVE it if there was a program to do this, or a really easy way since I don't have a ton of time.
Any ideas out there?
The image data type in Sql Server is a varbinary field that is being discontinued in future versions.
I would bet that a tool like bcp handles the "conversion" automatically. I use quotes because its a type conversion and not a format conversion.
Have you looked into writing a quick script in PowerShell? It has access to the full .NET framework, so should be somewhat simple if you're using those technologies.
Of course it's not simple if you have to learn PowerShell in order to write the script, but learning's always good :)
Related
How can you decompress an sql server field with nothing but tsql queries?
Assumptions:
The field was compressed using GZIP from an application.
The field is a varbinary(max).
MSSQL version is less than 2016, meaning you can't use decompress function.
I don't think this is possible, I tried searching on google and found nothing. I'd like to have confirmation on this.
I think your best bet is to look into a CLR solution. CLR functions are written in C# and then can be executed in a standard SQL call. There are loads of examples on the internet of CLR functions.
My answer assumes you have purely SQL Server and not a solution like SSIS or alike installed.
Here's a link for basic example. You'll load a library that handles gzips. Lots of solution directions here.
https://www.skylinetechnologies.com/Blog/Skyline-Blog/March-2013/CLR-Functions-in-SQL-Server-A-Tutorial
We are designing a database that needs to store various versions of a file (pdf/image/reduced image) in a table. The powers that be have opted against using Filestream for whatever reason so this is not up for debate.
I can't seem to find anything online that indicates what the appropriate datatype is for storing pdf and image data. That or I'm being just a total idiot while searching for it.
I'm not trying to start a debate, so I'm not looking for opinionated responses. But rather I am trying to find out if one or the other was actually designed for what I'm trying to do. If either will work, that's all I need to know.
Given your binary choice of nvarchar vs varbinary there's no choice: it's varbinary. nvarchar is for storing unicode character based data. varbinary is going to store a bit-perfect copy of the data you put in there. PDFs and images are binary file types so varbinary it is.
As for the BLOB suggestion, no. That's not even supported with 2012. Oh, and perhaps you meant TEXT/NTEXT/IMAGE data type. Those are deprecated too so don't build anything new using them.
Finally, you said you can't use FileStream, but what about FileTable. I'm not sure if you're looking for just storage of data or you need it searchable in which case, FileTable's pretty slick.
It's my first post ever... and I really need help on this one so any one who has some knowlege on the subject - please help!
What I need to do is to read an xml file into sql server data tables. I was looking over and over for solutions to this one and have found a few actualy. The problem is the size of the xml which is being loaded. It weights 2GB (and there will 10GB ones). I have managed to do this but I saw one particular solution which seems to me to be a great one but I cannot figure it out.
Ok lets get to the point. Currently I do it this way:
I read an entire XML using the openrowset into a variable. (this takes the whole ram memory...)
next I use the .node() thing to pull out the data and fill the tables with them.
Thats a two-step process. I was wondering if I could do it in only one step. I saw that there are things like format files and there are numerus examples on how to use that to pull out data from flat files or even excel documents in a record-based maner (in stead of sucking the whole thing into a variable) but I CANNOT find any example which would show how to read that huge XML into a table parsing the data on the fly (based on the format file). Is it even possible? I would really appreciate some help, or guidence on where to find a good example.
Pardon my English - it's been a while since I had to write so much in that language :-)
Thanks in advance!
For very large files, you could use SSIS: Loading XML data into SQL Server 2008
It gives you the flexibility of transforming the XML data, as well as reducing your memory footprint for very large files. Of course, it might be slower compared to using OPENROWSET in BULK mode.
What do wikis/stackoverflow/etc. do when it comes to storing text? Is the text broken at newlines? Is it broken into fixed-length chunks? How do you best store arbitrarily long chunks of text?
nvarchar(max) ftw. because over complicating simple things is bad, mmkay?
I guess if you need to offer the ability to store large chunks of text and you don't mind not being able to look into their content too much when querying, you can use CLobs.
This all depends on the RDBMS that you are using as well as the types of text that you are going to store. If the text is formatted into sizable chunks of data that mean something in and of themselves, like, say header/body, then you might want to break the data up into columns of these types. It may take multiple tables to use this method depending on the content that you are dealing with.
I don't know how other RDBMS's handle it, but I know that that it's not a good idea to have more than one open ended column in each table (text or varchar(max)). So you will want to make sure that only one column has unlimited characters.
Regarding PostgreSQL - use type TEXT or BYTEA. If you need to read random chunks you may consider large objects.
If you need to worry about keeping things like formatting strings, quotes, and other "cruft" in the text, as code would likely have, then the special characters need to be completely escaped first - otherwise on submission the db, they might end up causing an invalid command to be issued.
Most scripting languages have tools to do this built-in natively.
I guess it depends on where you want to store the text, if you need things like transactions etc.
Databases like SQL Server have a type that can store long text fields. In SQL Server 2005 this would primarily be nvarchar(max) for long unicode text strings. By using a database you can benefit from transactions and easy backup/restore assuming you are using the database for other things like StackOverflow.com does.
The alternative is to store text in files on disk. This may be fairly simple to implement and can work in environments where a database is not available or overkill.
Regards the format of the text that is stored in a database or file, it is probably very close to the input. If it's HTML then you would just push it through a function that would correctly escape it.
Something to remember is that you probably want to be using unicode or UTF-8 from creation to storage and vice-versa. This will allow you to support additional languages. Any problem with this encoding mechanism will corrupt your text. Historically people may have defaulted to ASCII based on the assumption they were saving disk space etc.
For SQL Server:
Use a varchar(max) to store. I think the upper limit is 2 GB.
Don't try to escape the text yourself. Pass the text through a parameterizing structure that will do the escapes properly for you. In .Net you'd add a parameter to a SqlCommand, or just use LinqToSQL (which then manages the SqlCommand for you).
I suspect StackOverflow is storing text in markdown format in arbitrarily-sized 'text' column. Maybe as UTF8 (but it might be UTF16 or something. I'm guessing it's SQL Server, which I don't know much about).
As a general rule you want to store stuff in your database in the 'rawest' form possible. That is, do all your decoding, and possibly cleaning, but don't do anything else with it (for example, if it's Markdown, don't encode it to HTML, leave it in its original 'raw' format)
I currently have a relatively small (4 or 5 tables, 5000 rows) MySQL database that I would like to convert to an sqlite database. As I'd potentially have to do this more than once, I'd be grateful if anyone could recommend any useful tools, or at least any easily-replicated method.
(I have complete admin access to the database/machines involved.)
I've had to do similar things a few times. The easiest approach for me has been to write a script that pulls from one data source and produces an output for the new data source. Just do a SELECT * query for each table in your current database, and then dump all the rows into an INSERT INTO query for your new database. You can either dump this into a file or pipe it straight into the database frontend.
It's not pretty, but honestly, pretty hardly seems to be a major concern for things like this. This technique is quick to write, and it works. Those are my primary criteria for things like this.
You might want to check out this thread, too. It looks like a couple of people have already put together basically what you need. I didn't look that far into it, though, so no guarantees.
As long as a MySQL dump file doesn't exceed the SQLite query language, you should be able to migrate fairly easily:
tgl#moto~$ mysqldump old-database > old-database-dump.sql
tgl#moto~$ sqlite3 -init old-database-dump.sql new-database
I haven't tried this myself.
UPDATE:
Looks like you'll need to do a couple edits of the MySQL dump. I'd use sed, or Google for it.
Just the comment syntax, auto_increment & TYPE= declaration, and escape characters differ.
Here is a list of converters:
http://www.sqlite.org/cvstrac/wiki?p=ConverterTools
An alternative method that would work nicely but is rarely mentioned is: use a ORM class that abstracts the specific database differences away for you. e.g. you get these in PHP (RedBean), Python (Django's ORM layer, Storm, SqlAlchemy), Ruby on Rails ( ActiveRecord), Cocoa (CoreData)
i.e. you could do this:
Load data from source database using the ORM class.
Store data in memory or serialize to disk.
Store data into source database using the ORM class.
If it's just a few tables you could probably script this in your preferred scripting langauge and have it all done by the time it'd take to read all the replies or track down a suitable tool. I would any way. :)