I am working on an app which will store images in the file system. For example, if a user uploads an image named "green.jpg" it will get stored as /wwwroot/images/f/c/3/fc3b16ee-9254-11e9-bc42-526af7764f64.jpg
In the database, I will have a table that contains the image name and its original name:
UUID Name
fc3b16ee-9254-11e9-bc42-526af7764f64 green.jpg
I want the generated html to look like:
<img src="/whatever/green.jpg">
So the question is how to serve /wwwroot/images/f/c/3/fc3b16ee-9254-11e9-bc42-526af7764f64.jpg when the browser requests "green.jpg" "
Generating /wwwroot/whatever/green.jpg is not a good idea because of possible name collisions and supplementary disk io.
What I want is that filenames in the generated HTML to be human readable but store that in a way that I can avoid filename collisions and that there aren't too many files in a directory.
Generating a new directory for each letter of the UUID would solve both problems, but I don't know if it's a good idea.
Using UUID is not important, just the uniqueness of file path is. Even if I use an integer as an unique identifier, if there will be millions of images, having millions of directories might be too much.
If that matters, I am using PostgreSQL as the DB engine and will run the app on Linux. I don't know what filesystem I'll use, probably ext4.
Related
I need to upload an excel file in load runner HTTP/HTML script with unique filename each time. The file must present in the directory. Copying files and renaming them will be more manual task. Can anyone tell me is there any automated way to do this? or Load Runnner itself can perform such tasks? Thank you.
On each of your load generators make sure that you have a ram drive for the file i/o for the new files. You are going to have ten, perhaps hundreds or thousands on your load generator. You do not want contention for the read/write heads of a physical hard drive acting as a drag anchor on the performance of your entire load generator. It is for this same reason that logging is minimized during test execution.
Include the base file as part of your virtual user
Use appropriate language API for making a copy of the file from the virtual user directory to the ramdrive on your virtual user generator with a new name. It might recommend a name which includes virtual user number and iteration number at the end to ensure uniqueness across your virtual user population.
Upload your file from the ramdrive as the source
Delete your newly created file to return to the same initial condition as the beginning of the iteration.
As you will be engaging a large amount of file i/o for the virtual users it is highly recommended that you monitor the load generators just as you would monitor your application under test. If you are new to LoadRunner and performance testing then this is an excellent opportunity for your mentor/trainer to guide you on a monitoring strategy.
Assuming the upload is done using a html form..
Use web_submit_data() with the FilePath argument.
but first lets create some parameters to get a real unique filename (very importent)
create a parameter VUSERID which outputs the current vuser id.
get/save the current timestamp
web_save_timestamp_param("TIMESTAMP", LAST);
and here is the request:
web_submit_data("i1",
"Action=https://{LR_SERVER}/{LR_JUNCTION}/upload",
"Method=POST",
"EncType=multipart/form-data",
"Snapshot=t1.inf",
"Mode=HTML",
ITEMDATA,
"Name=FIELDNAME", "Value={VUSERID}{TIMESTAMP}_LOADTEST.xlsx", "File=yes", "FilePath=REALFILEPATH.xlsx", "ContentType=WHATEVERCONTENTTYPE", ENDITEM,
LAST);
The Value={VUSERID}{TIMESTAMP}_LOADTEST.xlsx will be the new (unique) filename. (It is unique for each user and iteration! very importent)
The FilePath points to the real file and its content will be uploaded.
I need to build a tables related to manage documents such as jpg,doc,msg,pdf using a sql server 2008 .
As i know sql server support .jpg images, so my question is if it's possible to upload other kind of files into a db.
This is an example of the table (could be redefined if it's needed).
Document : document_id int(10)
name varchar(10)
type image (doesnt know how it might works)
Those are the initial values for a table, but i dont know how to make it useful for any type.
pd: do i need to assign a directory to save this documents into the server?
You can store almost any file type in an sql server table...if you do, you will almost certainly regret it.
Store a meta-data / a pointer to the file in your database instead, and store the files themselves on a disk directly where they belong.
Your database size - and thus hardware required to run it - will grow very rapidly, so you will be incurring large costs that you do not need to incur.
Use Filestream
https://learn.microsoft.com/en-us/sql/relational-databases/blob/filestream-sql-server
I know that a link-only answer is not an answer but I can't believe no one has mentioned it yet
The proper database design pattern is not to save Files into DBMS. You should develop a kind of File Manager Subsystem to manage your files for all of your projects.
File Manager Subsystem
This subsystem should be Reusable, Extendable, Secure and etc. All your projects that want to save Files, can use this subsystem.
Files can be saved in every where such as Local Hard, Network Drive, External Drives, Clouds and etc. So this subsystem should be design to support all kind of requests.
(you can improve the mentioned subsystem by adding a lot of features to it. for example checking duplicate files,...)
This subsystem, should generate a Unique Key for each file. After uploading and saving the files, the subsystem should generate that key.
Now, you can use this Unique Key to save in database (instead of file). Every time if you want to get the file, you can get the Unique Key from database and request to get file from the subsystem by unique key.
We are in the process of building a system which allows users to upload multiple images and videos to our servers.
The team I'm working with have decided to save all the assets belonging to a user in a folder named using the user's unique identifier. This folder in turn will be a sub-folder of our main assets folder on the file server.
The file structure they have proposed is as follows:
[asset_root]/userid1/assets1
[asset_root]/userid1/assets2
[asset_root]/userid2/assets1
[asset_root]/userid2/assets2
etc.
We are expecting to have thousands or possibly a million+ users in the life time of this system.
I always thought that it wasn't a good idea to have many sub-folders in a single location and suggested a year/month/day approach as follows:
[asset_root]/2010/11/04/userid1/assets1
[asset_root]/2010/11/04/userid1/assets2
[asset_root]/2010/11/04/userid2/assets1
[asset_root]/2010/11/04/userid2/assets2
etc.
Does anyone know which of the above approaches would be better suited for this many assets? Is there a better method to organize images/videos on a server?
The system in question will be an Windows IIS 7.5 with a SAN.
Many thanks in advance.
In general you are correct, in that many file systems impose a limit on the number of files and folders which may be in one folder. If you hit that limit with the number of users you have, your in trouble.
In general, I would simply use a uuid for each image, with some dimension of partitioning. e.g. A hash of ABCDEFGH would end up as [asset_root]/ABC/DEFGH. Using a hash gives you a greater degree of assurance about the number of files which will end up in each folder and prevents you from having to worry about, for example, not knowing which month an image you need was stored in.
I'm presuming your file system is NTFS? IF so, you've got a limit of 4,294,967,295 files on the disk - the limit of files in a folder is the same. If you have on the order of millions of users you should be fine, though you might want to consider having only one folder per user instead of several as your example indicates.
For the past 5 years, my typical solution for storing uploaded files (images, videos, documents, etc) was to throw everything into an "upload" folder and give it a unique name.
I'm looking to refine my methods for storing uploaded content and I'm just wondering what other methods are used / preferred.
I've considered storing each item in their own folder (folder name is the Id in the db) so I can preserve the uploaded file name. I've also considered uploading all media to a locked folder, then using a file handler, which you pass the Id of the file you want to download in the querystring, it would then read the file and send the bytes to the user. This is handy for checking access, and restricting bandwidth for users.
I think the file handler method is a good way to handle files, as long as you know to how make good use of resources on your platform of choice. It is possible to do stupid things like read a 1GB file into memory if you don't know what you are doing.
In terms of storing the files on disk it is a question of how many, what are the access patterns, and what OS/platform you are using. For some people it can even be advantageous to store files in a database.
Creating a separate directory per upload seems like overkill unless you are doing some type of versioning. My personal preference is to rename files that are uploaded and store the original name. When a user downloads I attach the original name again.
Consider a virtual file system such as SolFS. Here's how it can solve your task:
If you have returning visitors, you can have a separate container for each visitors (and name it by visitor login, for example). One of the benefits of this approach is that you can encrypt the container using visitor's password.
If you have many probably one-time visitors, you can have one or several containers with files grouped by date of upload.
Virtual file system lets you keep original filenames either as actual filesnames, or as a metadata for the files being stored.
Next, you can compress the data being stored in the container.
I am attempting to add a document storage module to our AR software.
I will be prompting the user to attach a doc/image to thier account. I will then put a copy of this file into our folder so that we can reference it without having to rely on them keeping the file in its original place. This system is not using a database but instead its using multiple flat files.
I am looking for guidance on how to handle these files once they have attached them to our system.
How should I store these attached files?
I was thinking I could copy the file over to a sub directory then renaming it to a auto-generated number so that we do not have duplicates. The bad thing about this, is the contents of the folder can get rather large.
Anyone have a better way? Should I create directories and store them...?
This system is not using a database but instead its using multiple flat files.
This sounds like a multi-user system. How are you handing concurrent access issues? Your answer to that will greatly influence anything we tell you here.
Since you aren't doing anything special with your other files to handle concurrent access, what I would do is add a new folder under your main data folder specifically for document storage, and write your user files there. Additionally, you need to worry about name collisions. To handle that, I'd name each file there with by appending the date and username to the original file name and taking the md5 or sha1 hash of that string. Then add a file to your other data files to map the hash values to original file names for users.
Given your constraints (and assuming a limited number of total users) I'd also be inclined to go with a "documents" folder -- plus a subfolder for each user. Each file name should include the date to prevent collisions. Over time, you'll have to deal with getting rid of old or outdated files either administratively or with a UI for users. Consider setting a maximum number of files or maximum byte count for each user. You'll also want to handle the files of departed users.