Whats the advantage/disadvantage between storing files as a byte array in a SQL table and using something like Amazon S3 to store them? Whats the advantage of S3 that makes it so I should use that instead of SQL?
Pros for storing files in the database:
transactional consistency
security (assuming you need it and that your database isn't wide open anyway)
Cons for storing files in the database:
much larger database files + backups (which can be costly if you are hosting on someone else's storage)
much more difficult to debug (you can't say "SELECT doc FROM table" in Management Studio and have Word pop up)
more difficult to present the documents to users (and allow them to upload) - instead of just presenting a link to a file on the file system, you must build an app that takes the file and stores it in the database, and pulls the file from the database to present it to the user.
typically, database file storage and I/O are charged at a much higher premium that flat file storage
Related
I am currently in the process of building a SQL database in Microsoft Azure for handling pictures, documents, etc. What is the most efficient/best way of storing data? Uploading the files directly to the DB, or by sourcing the files from something like Azure BLOB? I have read numerous posts about people uploading it directly to the DB, but I am concerned about its efficiency.
Thank you in advance for any replies.
You can store in something like Azure SQL DB for example but I would not recommend it, you should definitely store in Azure Storage (BLOB) and then for reference store in a DB. Azure has multiple relational and NoSQL data stores which are offered as platform services.
I would do two things, use a NoSQL platform data store like Cosmos DB using SQL Core API to store the metadata for the images, here you can use the filename as the partition ID to do a point read (this is very fast read and it would be a very cheap option with blazing fast performance) and secondly I would use Azure CDN to make sure images are accessed via CDN so that they are faster.
Azure CDN has three options; Akamai, Verizon and Microsoft. You can test which CDN is faster from where you are from here: https://cloudharmony.com/speedtest-for-azure
Using the above URL you can also use to test which Azure region is closer to you so to use that region, or test for your end-users and choose the region closer tot them.
I would say storing in Azure BLOBs is a better idea. Imagine you have 100 GB files stored in DB.
It will slow down your query if your table is not designed properly.
Backup & Restore DB will be very slow.
Azure DB is more expensive than Azure BLOB for the same size.
If your total file size is small enough, it doesn't make much difference.
I am writing a new application at the moment and certain entities in the application have images (just used for display purposes on the website).
I want to host my application on azure later and I am trying to figure out whether it would be better to use Azure Blob storage to store all images or just store them in the DB?
What is better performance wise when loading the images on the website?
SQL: Controller -> DB -> VIEW
Azure Blob: Controller -> Webcall to Azure DB -> VIEW
Could someone please explain benefits of either solution to me so that I can make up my mind?
How you design your database storage scheme is subjective, but there are objective things to consider in your scenario. I'll address those, and leave the "which should I choose" to you...
Azure Storage blobs are designed for bulk "block" data (such as documents, images, etc). Something like SQL Database is designed for metadata (stuff you search/index/query).
Everything can be done via SQL Database, and you would only need to worry about SQL queries (and it sounds like that's something you're already familiar with). SQL Server (and SQL Database) have always had the ability to store binary content via its BLOB type.
While you can store your images in SQL Database, you will find that your database size increases considerably, vs just storing queryable metadata. And while SQL Database service allows you to scale your storage, you'll find larger scale in blob storage (up to 500TB) at a lower cost than SQL Database service. If you run SQL Server in a VM, then you'll still have storage cost (attached disks) equivalent to blobs, along with VM costs.
Storage blobs, by themselves, don't provide a query language - you will need to know the container and/or blob name. So, for optimum searching, you'll want a queryable database with your metadata (e.g. SQL Database).
If you store your images in blobs, and reference them via URI in your database, you will be able to query against your database, find the image's URI, and then read from blob storage appropriately.
Also note: With blobs, you'll be able to provide direct image URI access to, say, a browser or an app (even if the blob is marked as private), which allows you to then bypass your app tier when delivering binary (image) content to the end-user. Blobs may also be cached in the CDN, which you cannot do with SQL Database.
Which you choose is ultimately up to you; I simply provided the objective reasons to use each.
Much cheaper in BLOB.
You are also probably going the get faster transfer as BLOB. Now the initial lookup may be a little faster with SQL but for a large image I think BLOB would win. SQL is just plain not optimized for big stuff and BLOB is.
And you keep SQL free to serve up short stuff.
I've seen examples of web apps that has user accounts and stores information about the users, but does not use a database. I've been searching for a while and cannot figure out how this would be done without a database. Can someone point me in a direction?
Clearly the data has to be stored somewhere. It could stored on the client, using cookies or HTML5 Web Storage, but this would make it nearly impossible for the data to be shared across devices. (Technically, the data could be shared between devices provided that another device maintains an active connection to the web server at the same time; the web server would merely transfer the data between devices without storing it. However, this would be extremely impractical.) If the user data is stored in any centralized location, there would effectively be a database, even if specialized database software, such as MySQL, is not used, i.e. just using the filesystem of the server directly.
Some used database such as SQLite to store data, but for other application they simply saved it to a file somewhere in the directory (may be cookies, temporary file, settings/configuration, ....etc).
There are 2 main ways of storing data based on the requirement:
Storing data On client
1.HTML5 Webstorage is now available, which store data locally and it can support data upto 5MB and it is more secured and faster.
2.Cookies
Storing data On Server
Through flat files that is text file or through XML files.XML is standard in many companies as a way to store data. This is really quick.
Having said that, there are downsides to it as well.
I'm extremely confused, so I've created an SQL Database in Windows Azure, created a "video table" with a "video_file" column as "varbinary(max)" because I want to upload a video file into that field, however Azure offers no "Upload" option like say, PHPMyAdmin does where you can hit "browse" and upload a video directly into the field. Can anyone guide me as to how to actually upload a file into a Windows Azure SQL Database so it can be read as a varbinary type? Can it be done within the Azure management portal? Or does it require some sort of external program/service?
To answer your question, the functionality to upload files directly into SQL Azure Database does not exist. This is something you have to do on your own.
Can anyone guide me as to how to actually upload a file into a Windows
Azure SQL Database so it can be read as a varbinary type?
Do a search for uploading files in SQL Server and you will find plenty of examples on how to do that. Take a look at this link for example: http://www.codeproject.com/Articles/225446/Uploading-and-downloading-files-to-from-a-SQL-Serv
Can it be done within the Azure management portal? Or does it require some sort of external program/service?
No. This functionality does not exist in Azure Management Portal. As mentioned above, you would need to write some code to do so.
A little bit off-topic comment:
May I suggest that instead of saving the image files in the database you save them in Blob Storage and store the URL of the blob in your table. There're some advantages I could see in this:
Compared to SQL Database, Azure Blob Storage is much cheaper. If you store video files (or in other words large files) in the database, you will end up with large database and thus end up paying more money.
You will be choking the database when reading this large data from the database which will impact the performance of your application.
I am about to create an ASP.NET MVC app which will have over 2000 products and each products will approx 20 photos. The app will be asp.net mvc app and
I am using sql server 2008 r2 to manage my data. which way is the better approcah here;
Uploading pictures to a path and
storing their file names to database
in order to be able to make a
relation to each other.
Storing pictures inside the database
as byte as well and retreive them
from there when needded.
definitely in the filesystem (store path) is better, i have done both in the past.
Against SQL server to store images
A) betting data in and out can be more difficult as have to used blob type objects and some ORMs don't really cater for this
B) your data base is much bigger so effects your backup/restore policy. The more frequently you backup the better but space will be increased. Storing in file, yep you still need to backup but backing up filesystem is easy.
C) when you run out of storage space you just add another NAS drive / server and start storing images there, so scales horizontally
The common perception is not as good as data stored in two places but for me its better as the type of data in stored in the best storage medium for the data types ..
Definitely storing as a path rather than the byte array. This means you can easily change the actual image itself without having to alter any code or muck around in SQL (as long s the new file has the same name as the old one).
Hope this helps.
In the database using FILESTREAM which combines the 2 ideas (file and database)
FILESTREAM integrates the SQL Server Database Engine with an NTFS file system by storing varbinary(max) binary large object (BLOB) data as files on the file system. Transact-SQL statements can insert, update, query, search, and back up FILESTREAM data. Win32 file system interfaces provide streaming access to the data.
This changes the file vs database arguments
If you want to store paths only, then you'll have to accept the fact that images and database will get out of synch over time.