I am currently enhancing a product to support web delivery of large file-content.
I would like to store it in the database, and whether or not I choose to FILESTREAM by BLOB, the following question still holds.
My WCF method will return a stream, meaning that the file stream will remain open while the content is read by the client. If the connection is slow, then the stream could be open for some time.
Question: Connection pooling assumes that connections are exclusively held, only for a short period of time. Am I correct in assuming, that given I have a connection pool of finite size, there could be a contention problem, if slow network connections are used to download files?
Under this assumption, I really want to use FILESTREAM, and open the file directly from the file-system, rather than the SQL connection. However, if the database is remote, I will have no choice but to pull the content from the SQL connection (until I have a local cache of the file anyway).
I realise I have other options, such as to server-buffer the stream, however that will have implications as well. I wish at this time, to discuss only the issues relating to returning a stream obtained from a DB connection.
Related
System processes is shown here
I have a small problem.. okay huge problem.
I have a SQL database that has uses a filestream to store huge files (10GB+). When using the sqlfilestream API to copy large files into the database the table becomes unresponsive until the sql connection has been closed.
Now, normally this wouldn't be an issue since normally the connection would only be open long enough to send the query and get the response but when uploading a 10GB file the connection is open for several minutes and so the table does not response to queries until the other connection has been closed.
Does anyone have an idea how to resolve this? It isn't possible to copy part of the file and then close the connection and then restart because this creates multiple files and would take up a lot of extra space.
Basically I have a bunch of performance analysis that [given naive interpetation] claims 70% of the time is spent in synchronization on our web application under heavy load, and mostly in SNIReadSyncOverAsync which internally in the data reader calls. (SNIReadSyncOverAsync actually ends up sitting on a kernalbase.dll!WaitForSingleObjectEx) It would be informative to see if these waits are caller initiated or callee initiated.
Is there a way to see (interpret) this in a Visual Studio Contention or Concurrency Report? Or some other way?
More importantly for my understanding, is there a way to see the incoming buffer that holds data before the data get's consumed by the data reader?
It seems my question was ill-informed.
The datareader reads a record at a time, but it reads it from the
underlying database driver. The database driver reads data from the
database in blocks, typically using a buffer that is 8 kilobytes.
If your result records are small and you don't get very many, they
will all fit in the buffer, and the database driver will be able to
feed them all to the data reader without having to ask the database
for more data.
If you fetch a result that is larger than the buffer, you will only be
able to read the first part of it and when there will no data exist in
network buffer then datareader will inform sql server to send next
block of data.
How much data can be stored in network buffer when datareader is used
I have a service that has a method to send a file to the service from the client. I notice that when I run the client and the service in the same machine and the file that I want to send is also in the local machine, all works very fast.
However, I the client and the service are in the same machine but the file is in other computer, then speed is very slow.
If I copy the file from one computer to other, the speed is fast, so the problem does not seem to be the bandwidth.
I try to use tcp and basicHttp Binding, but the results are the same.
This problem also occurrs when I try to send if the client are in other computer.
Thanks.
EDIT: If I open the task manager, in the network tab of the computer taht run the client, I can see that the use of the network is about 0.5%. Why?
WCF for transmitting large file is not the optimal method because WCF has a lot of layers and overhead that adds up and causes delay in file transmission. Morever, you may not have written the WCF service to continuously read chunks of byte and write to the response. You might be doing a File.ReadAll and then just return the whole string, which would cause a large amount of sync read on the server, a lot of memory allocation and then writing the large string to WCF buffer, which in turn write to IIS buffer and so on.
The best way to transmit large files is by using HttpHandlers. You can just use Response.TransmitFile to transfer the file and IIS will transmit the file in the most optimal way. Otherwise you can always read 8k at a time and then write to the Response stream and call Flush after every 8k write.
If you cannot go for HttpHandler for any weird reason, can you show me the WCF code?
Another thing. You might be expecting performance that is simply not possible when IIS is in the picture. First you should measure how long it takes for IIS to transmit the file if you just host the file directly on a website and download the file by doing a WebClient.downloadString.
Another thing is, how are you downloading? Via Browser? or via client side code? Client side code can be suboptimal as well if you are trying to transmit the whole file in one shot and trying to hold it in a string. For ex, WebClient.DownloadString would be the worst approach.
This is a very general question. I am a bit confused with the term state. I would like to know what do people mean by "state of an application"? Why do they call webserver as "stateless" and database as "stateful"?
How is the state of an application (in a VM) transferred, when the VM memory is moved from one machine to another during live migration.
Is transferring the memory, caches and register values of a system enough to transfer the state of the running application?
You've definitely asked a mouthful -- it's unfortunate that the word state is used in so many different contexts, but each one is a valid use of the word.
State of an application
An application's state is roughly the entire contents of its memory. This can be a difficult concept to get behind until you've seen something like Erlang's server loops, which explicitly pass all the state of the application in a variable from one invocation of the function to the next. In more "normal" programming languages, the "state" of the program is all its global variables, static variables, objects allocated on the heap, objects allocated on the stack, registers, open file descriptors and file offsets, open network sockets and associated kernel buffers, and so forth.
You can actually save that state and resume execution of the process elsewhere. The BLCR checkpoint tools for Linux do exactly this. (Though it is an extremely uncommon task to perform.)
State of a protocol
The state of a protocol is a different sort of meaning -- the statelessness of HTTP requests means that every web browser communication with webservers essentially starts over, from scratch -- every cookie is re-transmitted in both directions to try to "fake" some amount of a "session" for the user's sake. The servers don't hold any resources open for any given client across requests -- each one starts from scratch.
Networked filesystems might also be stateless (earlier versions of NFS) or stateful (newer versions of NFS). The earlier versions assumed every individual packet of reading, writing, or metadata control would be committed as it arrived, and every time a specific byte was needed from a file, it would be re-requested. This allowed the servers to be very simple -- they would do what the client packets told them to do and no effort was required to bring servers and clients back to consistency if a server rebooted or routers disappeared. However, this was bad for performance -- every client requested static data hundreds or thousands of times each day. So newer versions of NFS allowed some amount of data caching on the clients, and persistent file handles between servers and clients, and the servers had to keep track of the state of the clients that were connected -- and vice versa: the clients also had to know what promises they had made to the servers.
A stateful firewall will keep track of active TCP sessions. It knows which sessions the system administrators want to allow through, so it looks for those initial packets specifically. Once the session is set up, it then tracks the established connections as entities in their own rights. (This was a real advancement upon previous stateless firewalls which considered packets in isolation -- the rulesets on previous firewalls were much more permissive to achieve the same levels of functionality, but allowed through far too many malicious packets that pretended a session was already active.)
An application state is simply the state at which an application resides with regards to where in a program is being executed and the memory that is stored for the application. The web is "stateless," meaning everytime you reload a page, no information remains from the previous version of the page. All information must be resent from the server in order to display the page.
Technically, browsers get around the statelessness of the web by utilizing techniques like caching and cookies.
Application state is a data repository available to all classes. Application state is stored in memory on the server and is faster than storing and retrieving information in a database. Unlike session state, which is specific to a single user session, application state applies to all users and sessions. Therefore, application state is a useful place to store small amounts of often-used data that does not change from one user to another.
Resource:http://msdn.microsoft.com/en-us/library/ms178594.aspx
Is transferring the memory, caches and register values of a system enough to transfer the state of the running application?
Does the application have a file open, positioned at byte 225? If so, that file is part of the application's state because the next byte written should go to position 226.
Has the application authenticated itself to a secure server with a time-based key? Then that connection is part of the application's state, because if the application were to be suspended for 24 hours after saving memory, cache, and register values, when it resumes it will no longer have a valid connection to the secure server because it will have timed out.
Things which make an application stateful are easy to overlook.
I am working on a VB.NET project that grabs data from an Access DB file. All the code snipeets I have come across open the DB, do stuff and close it for each operation. I currently have the DB open for the entire time the application is running and only close it when the application exits.
My question is: Is there a benefit to opening the connection to the DB file for each operation instead of keeping it open for the duration the application is running?
In many database systems it is good practice to keep connections open only when they are in use, since an open connection consumes resources in the database. It is also considered good practice for your code to have as little knowledge as possible about the concrete database in use (for instance by programming against interfaces such as IDbConnection rather than concrete types as OleDbConnection.
For this reason, it could be a good idea to follow the practice of keeping the connection open as little as possible regardless of whether it makes sense or not for the particular database that you use. It simply makes your code more portable, and it increases your chance of not getting it wrong, in case you in your next project happen to work against a system where keeping connections open is a bad thing to do.
So, your question should really be reversed: is there anything to gain by keeping the connection open?
There is no benefit with the Jet/ACE database engine. The cost of creating the LDB file (the record locking file) is very high. You could perhaps avoid that by opening the file exclusively (if it's a single user), but my sense is that opening exclusive is slower than opening multi-user.
The advice for opening and closing connections is based around an assumption of a database server being on the other end of the connection. If you consider how that works, the cost of opening and closing the connection is very slight, as the database deamon has the data files already open, and handles locking on the fly via structures in memory (and perhaps on disk -- I really don't know about how it's implemented in any particular server database) that already exist once the server is up and running.
With Jet/ACE, all users are contending for two files, the data file and the locking file, and setting that up is much more expensive than the incremental cost of creating a new connection to a server database.
Now, in situations where you're aiming for high concurrency with a Jet/ACE data store, there might have to be a trade-off among these factors, and you might get higher concurrency by being much more miserly with your connections. But I would say that if you're into that realm with Jet/ACE, you should probably be contemplating upsizing to a server-based back end in the first place, rather than wasting time on optimizing Jet/ACE for an environment it was not designed for.