Is there an optimization for writing images or PDFs faster to the database? - sql

I'm facing an issue that my upload time for an image or PDF of 40+MB is more than 2.5 minutes (20+ seconds of which are just routing the request from the frontend to the backend but I'm worried more about the sql query slowlyness). I pasted some code snippets below. I also don't get an upload percentage indicator and the fetched bytes don't open as an image either in html using <img src="data:image/jpeg;base64,#Convert.ToBase64String(Model.Image)" /> or using Win10 Photos app.
I'm looking first of all for an optimization on how can I write an image or pdf file faster into the database?
Index.cshtml
using (Html.BeginForm("Index", "File", FormMethod.Post, new { enctype = "multipart/form-data" }))
{
#Html.AntiForgeryToken()
#Html.TextBoxFor(m => m.File, new { #type = "file", #accept = "image/jpeg,image/gif,image/png,application/pdf" })
<input type="submit" value="Upload" />
#Html.ValidationSummary()
}
Controller
[HttpPost]
[ValidateAntiForgeryToken]
public async Task<ActionResult> Index(Model model)
{
var file = model.File;
if (file != null)
{
if (file.ContentLength == 0)
{
ModelState.AddModelError(String.Empty, "File cannot be empty.");
}
const int maxFileSizeMB = 50;
if (file.ContentLength > maxFileSizeMB * 1024 * 1024)
{
ModelState.AddModelError(String.Empty, $"File cannot be bigger than {maxFileSizeMB} megabytes.");
}
//Check for content-type, file extension, and first bytes if it's indeed a valid image.
if (!file.IsImageOrPdfFile())
{
ModelState.AddModelError(String.Empty, "File is not a valid image.");
}
if (ModelState.IsValid)
{
var fileName = file.FileName;
var fileBytes = new byte[file.ContentLength];
file.InputStream.Read(fileBytes, 0, file.ContentLength);
await _api.SaveFile(fileName, fileBytes);
//success!
return Redirect(Url.Action("Index"));
}
}
//View with errors
return View(model);
}
API method
public async Task SaveFile(string fileName, byte[] fileBytes)
{
var client = CreateRestClient();
client.Timeout = 300000;
var request = CreateJsonPostRequest("File", new SaveFileRequest
{
FileName = fileName,
FileBytes = fileBytes
});
var response = await client.ExecuteAsync(request);
if (response.StatusCode != HttpStatusCode.NoContent && response.StatusCode != HttpStatusCode.OK)
{
throw new Exception(response.Content);
}
}
Backend API
[RoutePrefix("api/File")]
public class FileController : ApiController
{
private readonly IFileActionsRepository _fileActions;
public FileController(IFileActionsRepository fileActions)
{
_fileActions = fileActions;
}
[Authorize]
[HttpPost, Route("")]
public async Task Post([FromBody] SaveFileRequest request)
{
if (request == null) throw new ArgumentNullException(nameof(request));
if (request.FileBytes.Length == 0) throw new ArgumentException("request.FileBytes");
_fileActions.SaveFile(request.FileName, request.FileBytes);
}
}
FileActionsRepository.cs
public class FileActionsRepository : IFileActionsRepository
{
private readonly IDataContext _dataContext;
public FileActionsRepository(IDataContext dataContext)
{
_dataContext = dataContext;
//Get the ObjectContext related to this DbContext
var objectContext = (_dataContext.Context as IObjectContextAdapter).ObjectContext;
//Sets the command timeout for all the commands (since it's too slow)
objectContext.CommandTimeout = 300;
}
public void SaveFile(string fileName, byte[] fileBytes)
{
//Using sql query (with update .write() block) since it's faster than entity framework linq-2-entities.
_dataContext.Context.Database.ExecuteSqlCommand(
"if exists (select FileName from Files with (updlock,serializable) where FileName = #FileName)"
+ " update Files set FileBytes .write(#FileBytes, 0, null), SysModified = GETDATE()"
+ " where FileName = #FileName"
+ " else"
+ " insert into Files (FileName, FileBytes, SysCreated)"
+ " values (#FileName, CONVERT(varbinary, '0x00'), GETDATE())"
+ " update Files set FileBytes .write(#FileBytes, 0, null)"
+ " where FileName = #FileName",
new SqlParameter("#FileName", fileName),
new SqlParameter("#FileBytes", fileBytes));
}
}

Related

Blazor Server: Creating email attachments from files uploaded via InputFile

I am trying to send emails with attachments attached to the email. I have a InputFile with a progress bar that I am able to upload files. I have attempted to use the memory stream to make attachments to the MailMessage class. The issue is that when the email is received, I am able to see the attachments but I can't read or view the contents of the attachments. I've posted my code below so you can replicate the issue that I am having (Make sure to install the Meziantou.Framework.ByteSize nuget package)
#using System.Net.Mail
#using System.Globalization
#using Meziantou.Framework
<InputFile OnChange="e => LoadFiles(e)" multiple></InputFile>
#foreach (var file in uploadedFiles)
{
<div>
#file.FileName
<progress value="#file.UploadedBytes" max="#file.Size"></progress>
#file.UploadedPercentage.ToString("F1")%
(#FormatBytes(file.UploadedBytes) / #FormatBytes(file.Size))
</div>
}
<button type="button" #onclick="#HandleNotifSubmit" class="btn btn-primary submit">Send Email</button>
#code {
private MemoryStream fileContents { get; set; }
List<FileUploadProgress> uploadedFiles = new();
MailMessage message = new MailMessage();
StreamWriter writer { get; set; }
private async ValueTask LoadFiles(InputFileChangeEventArgs e)
{
var files = e.GetMultipleFiles(maximumFileCount: 100);
var startIndex = uploadedFiles.Count;
// Add all files to the UI
foreach (var file in files)
{
var progress = new FileUploadProgress(file.Name, file.Size);
uploadedFiles.Add(progress);
}
// We don't want to refresh the UI too frequently,
// So, we use a timer to update the UI every few hundred milliseconds
await using var timer = new Timer(_ => InvokeAsync(() => StateHasChanged()));
timer.Change(TimeSpan.FromMilliseconds(500), TimeSpan.FromMilliseconds(500));
// Upload files
byte[] buffer = System.Buffers.ArrayPool<byte>.Shared.Rent(4096);
try
{
foreach (var file in files)
{
using var stream = file.OpenReadStream(maxAllowedSize: 10 * 1024 * 1024);
while (await stream.ReadAsync(buffer) is int read && read > 0)
{
uploadedFiles[startIndex].UploadedBytes += read;
var readData = buffer.AsMemory().Slice(0, read);
}
fileContents = new MemoryStream(buffer);
writer = new StreamWriter(fileContents);
fileContents.Position = 0;
message.Attachments.Add(new Attachment(fileContents, file.Name));
startIndex++;
}
}
finally
{
System.Buffers.ArrayPool<byte>.Shared.Return(buffer);
StateHasChanged();
}
}
string FormatBytes(long value)
=> ByteSize.FromBytes(value).ToString("fi2", CultureInfo.CurrentCulture);
record FileUploadProgress(string FileName, long Size)
{
public long UploadedBytes { get; set; }
public double UploadedPercentage => (double)UploadedBytes / (double)Size * 100d;
}
private async void HandleNotifSubmit()
{
try
{
var sClient = new SmtpClient("FAKECOMPANYCLIENT");
sClient.Port = 25;
sClient.UseDefaultCredentials = false;
message.Subject = "Hello World";
message.From = new MailAddress("test#gmail.com");
message.IsBodyHtml = true;
message.To.Add(new MailAddress("Fake#gmail.com"));
message.Body = "Please view attachments below.";
sClient.Send(message);
message.Dispose();
}
catch
{
Console.WriteLine("error");
}
}
}
I have also tried to use a stream writer with no success. I have also tried various ways to do a file.CopyToAsync(memoryStreamname). I am not sure what else I am missing or doing wrong here.
Thank you in advance.

Retreive the file name and size uploaded via postman in asp.net core

I have tried the following code ..
When I try out the following code I get 404 content not found. Also my controller doesn't get hit when I try to debug the code ..
public async Task<IActionResult> Download(string filename)
{
if (filename == null)
return Content("filename not present");
var path = Path.Combine(Directory.GetCurrentDirectory(),"wwwroot", filename);
var memory = new MemoryStream();
using (var stream = new FileStream(path, FileMode.Open))
{
await stream.CopyToAsync(memory);
}
memory.Position = 0;
return File(memory, GetContentType(path), Path.GetFileName(path));
}
Upload file with size and filename as response:
public async Task<IActionResult> OnPostUploadAsync(IFormFile file)
{
long size = file.Length;//in bytes
if (file.Length > 0)
{
var name = Path.GetRandomFileName();
//var fileOriginName = file.FileName;
var path = Path.Combine(System.IO.Directory.GetCurrentDirectory(), "wwwroot", name);
using (var stream = System.IO.File.Create(path))
{
await file.CopyToAsync(stream);
}
return Ok(new { size = size, filename = name });
}
else {
return Ok(new { size = 0, filename = ""});
}
}
===============================================
I have an api like below and I had a file test.txt in wwwroot folder.
public async Task<IActionResult> Download(string filename)
{
if (filename == null)
return Content("filename not present");
var path = Path.Combine(System.IO.Directory.GetCurrentDirectory(), "wwwroot", filename);
var memory = new MemoryStream();
using (var stream = new FileStream(path, FileMode.Open))
{
await stream.CopyToAsync(memory);
}
memory.Position = 0;
var contentType = "APPLICATION/octet-stream";
return File(memory, contentType, Path.GetFileName(path));
}
Then when I call localhost:port/home/download?filename=test then I'll get exception that file can't find. And when I call localhost:port/home/Download?filename=test.txt it succeeds.
My API is in an asp.net core MVC project and I think you need to check the contentType.

Net core api to upload 1GB size csv file

I have following code segment it works for small file. But if the file is larger then application is loading for long and recieves No 'Access-Control-Allow-Origin' header is present on the requested resource.
[HttpPost]
[ScopeAuthorize(Constants.ClaimScopeSGCanManageAll, Constants.ClaimScopeUserCanManage)]
[DisableRequestSizeLimit, RequestFormLimits(MultipartBodyLengthLimit = int.MaxValue, ValueLengthLimit = int.MaxValue)]
public async Task<IActionResult> UploadFile()
{
if (!Request.Form.Files.Any())
{
throw new Common.Exceptions.ValidationException("Empty file");
}
IFormFile formFile = Request.Form.Files[0];
var csvDatas = new List<PatientsCSVItem>();
using (var reader = new StreamReader(formFile.OpenReadStream()))
{
while (!reader.EndOfStream)
{
var line = reader.ReadLine();
var values = line.Split(';');
//process csv rows
}
}
PatientCsvLog executionLog = _patientCsvManager.AddOrUpdatePatientsByCsvData(csvDatas, _userManager.GetLoggedUserId(User));
if (executionLog == null)
{
throw new ArgumentNullException(nameof(executionLog));
}
var response = new
{
NumberRecordImported = executionLog.NumberRecordImported,
NumberRecordUpdated = executionLog.NumberRecordUpdated,
NumberRecordDiscarded = executionLog.NumberRecordDiscarded,
DiscardedRecordList = executionLog.DiscardedRecordList
};
return Ok(response);
}

ASP .Net Core file upload - getting form data when [DisableFormValueModelBinding] attribute is in place

I went ahead and implemented an ASP .Net Core file upload controller per the documentation and it requires using a [DisableFormValueModelBinding] attribute for streaming large files. I got that working fine. Unfortunately, when using that attribute it seems to block my JSON properties coming in from the form.
Is there any way to get both the file and the form data here? Here is my controller code (the request.form calls are where I am having issues):
[Route("{caseNbr:int}/Document")]
[ResponseType(typeof(CaseDocumentModel))]
[DisableFormValueModelBinding]
[HttpPost]
public async Task<IActionResult> PostDocument(int caseNbr)
{
string errorTrackingFileName = string.Empty;
try
{
UserSessionModel userSessionModel = SessionExtensions.CurrentUserSession;
if (!MultipartRequestHelper.IsMultipartContentType(Request.ContentType))
{
return BadRequest("Bad Request");
}
var boundary = MultipartRequestHelper.GetBoundary(MediaTypeHeaderValue.Parse(Request.ContentType), _defaultFormOptions.MultipartBoundaryLengthLimit);
var reader = new MultipartReader(boundary, HttpContext.Request.Body);
var section = await reader.ReadNextSectionAsync();
while (section != null)
{
var hasContentDispositionHeader = ContentDispositionHeaderValue.TryParse(section.ContentDisposition, out var contentDisposition);
if (hasContentDispositionHeader)
{
if (!MultipartRequestHelper.HasFileContentDisposition(contentDisposition))
{
return BadRequest("Bad Request");
}
var fileName = WebUtility.HtmlEncode(contentDisposition.FileName.Value);
errorTrackingFileName = fileName;
var trustedFileNameForFileStorage = fileName; //Path.GetRandomFileName();
var streamedFileContent = await FileHelpers.ProcessStreamedFile(section, contentDisposition, ModelState, _permittedExtensions, _fileSizeLimit);
if (!ModelState.IsValid)
{
return BadRequest("Bad Request");
}
using (var targetStream = System.IO.File.Create(Path.Combine(_tempFilePath, trustedFileNameForFileStorage)))
{
await targetStream.WriteAsync(streamedFileContent);
**//This is where I am having trouble:**
string descrip = HttpContext.Request.Form["Description"].ToString();
string docType = HttpContext.Request.Form["DocType"].ToString() ?? "Document";
bool isGeneralFileUpload = false;
if (string.IsNullOrWhiteSpace(Request.Form["GeneralFileUpload"]) == false && AppHelper.IsBool(Request.Form["GeneralFileUpload"]))
isGeneralFileUpload = bool.Parse(Request.Form["GeneralFileUpload"]);
int transcriptionJobId = 0;
if (string.IsNullOrWhiteSpace(Request.Form["TranscriptionJobId"]) == false && AppHelper.IsNumeric(Request.Form["TranscriptionJobId"]))
transcriptionJobId = int.Parse(Request.Form["TranscriptionJobId"]);
CaseDocumentModel createdCaseDocumentModel;
if (docType.Equals("Dictation"))
createdCaseDocumentModel = DictationRepository.ProcessDictationFile(userSessionModel.DBID, caseNbr, _tempFilePath, fileName, userSessionModel);
else if (isGeneralFileUpload)
createdCaseDocumentModel = DashboardAdjusterRepository.CreateGeneralFileUploadDocument(_tempFilePath, fileName, userSessionModel, docType, descrip);
else if (docType.Equals("Transcription"))
createdCaseDocumentModel = TranscriptionRepository.UploadTranscriptionFile(userSessionModel.DBID, _tempFilePath, fileName, userSessionModel, transcriptionJobId);
else
createdCaseDocumentModel = CaseRepository.CreateCaseDocumentRecord(userSessionModel.DBID, caseNbr, descrip, docType, _tempFilePath, fileName, userSessionModel);
return Ok(createdCaseDocumentModel);
}
}
// Drain any remaining section body that hasn't been consumed and
// read the headers for the next section.
section = await reader.ReadNextSectionAsync();
}
}
catch (Exception ex)
{
AppHelper.WriteErrorLog("CaseController PostDocument failed due to " + ex.Message + " case number was " + caseNbr + " file name was " + errorTrackingFileName);
return BadRequest("Bad Request");
}
return BadRequest("Bad Request");
}
Here is a sample call with Postman:
Screen shot of Postman

Error while uploading file method in Client Object Model Sharepoint 2010

Error while uploading file method in Client Object Model + Sharepoint 2010. Once the file got uploaded. After that though the code compiles with no error
I get the error while executing
"{"Value does not fall within the expected range."}
{System.Collections.Generic.SynchronizedReadOnlyCollection}
I have a method which takes care of functionality to upload files
///////////////////////////////////////////////////////////////////////////////////////////
public void Upload_Click(string documentPath, byte[] documentStream)
{
String sharePointSite = "http://cvgwinbasd003:28838/sites/test04";
String documentLibraryUrl = sharePointSite +"/"+ documentPath.Replace('\\','/');
////////////////////////////////////////////////////////////////////
//Get Document List
List documentsList = clientContext.Web.Lists.GetByTitle("Doc1");
var fileCreationInformation = new FileCreationInformation();
//Assign to content byte[] i.e. documentStream
fileCreationInformation.Content = documentStream;
//Allow owerwrite of document
fileCreationInformation.Overwrite = true;
//Upload URL
fileCreationInformation.Url = documentLibraryUrl;
Microsoft.SharePoint.Client.File uploadFile = documentsList.RootFolder.Files.Add(
fileCreationInformation);
//uploadFile.ListItemAllFields.Update();
clientContext.ExecuteQuery();
}
/////////////////////////////////////////////////////////////////////////////////////////////////
In the MVC 3.0 application in the controller I have defined the following method to invoke the upload method.
//////////////////////////////////////////////////////////////////////////////////////////////////
public ActionResult ProcessSubmit(IEnumerable<HttpPostedFileBase> attachments)
{
System.IO.Stream uploadFileStream=null;
byte[] uploadFileBytes;
int fileLength=0;
foreach (HttpPostedFileBase fileUpload in attachments)
{
uploadFileStream = fileUpload.InputStream;
fileLength=fileUpload.ContentLength;
}
uploadFileBytes= new byte[fileLength];
uploadFileStream.Read(uploadFileBytes, 0, fileLength);
using (DocManagementService.DocMgmtClient doc = new DocMgmtClient())
{
doc.Upload_Click("Doc1/Doc2/Doc2.1/", uploadFileBytes);
}
return RedirectToAction("SyncUploadResult");
}
//////////////////////////////////////////////////////////////////////////////////////////////////
Please help me to locate the error
I think your documentLibraryUrl needs to be relative. This is working for me with Sharepoint 2013
[HttpPost]
[ValidateAntiForgeryToken]
[SharePointContextFilter]
public ActionResult Upload()
{
if (Request.Files.Count > 0)
{
HttpPostedFileBase file = Request.Files[0];
if (file != null && file.ContentLength > 0)
{
var fileName = Path.GetFileName(file.FileName);
var spContext = SharePointContextProvider.Current.GetSharePointContext(HttpContext);
using (var clientContext = spContext.CreateUserClientContextForSPHost())
{
if (clientContext != null)
{
FileCreationInformation newFile = new FileCreationInformation();
using (MemoryStream ms = new MemoryStream())
{
file.InputStream.CopyTo(ms);
byte[] array = ms.GetBuffer();
newFile.Content = array;
}
List docs = clientContext.Web.Lists.GetByTitle("Documents");
Folder folder = docs.RootFolder;
clientContext.Load(folder);
clientContext.ExecuteQuery();
newFile.Url = docs.RootFolder.ServerRelativeUrl + "/" + fileName;
Microsoft.SharePoint.Client.File uploadFile = docs.RootFolder.Files.Add(newFile);
clientContext.Load(uploadFile);
clientContext.ExecuteQuery();
//Set the metadata
Microsoft.SharePoint.Client.ListItem item = uploadFile.ListItemAllFields;
string docTitle = string.Empty;
item["Title"] = docTitle;
item.Update();
clientContext.ExecuteQuery();
}
}
}
}
return RedirectToAction("Index", new { SPHostUrl = SharePointContext.GetSPHostUrl(HttpContext.Request).AbsoluteUri });
}