Saving Files to SQL server using knockout js - file-upload

I have a requirement on my web app to save multiple attachments to my SQL server. The attachments can be in any format. Is it possible to create a knockout viewmodel to encode the file type on client side to binary and transferring that binary info to my SQL server?
I currently have this method in my REST service:
[OperationContract, WebInvoke(ResponseFormat = WebMessageFormat.Json)]
public string PostAttachments (string FileName)
{
try
{
var MyAttachment = new Binary(File.ReadAllBytes(FileName));
return "Attachments Created";
}
catch (Exception ex)
{
return DefaultError + ex;
}
}
I know this is incorrect but I am really not sure how to do this file upload. I believe I am over complicating it. Any Assistance would be greatly appreciated.

You will have to transfer the buffer to the server not just the filename, since the file exists on the client.
I used the HTML5 FileReader in my last KO project
First a custom binding to read the file from the DOM element
(function () {
ko.bindingHandlers.file = {
init: function (element, valueAccessor, allBindingsAccessor) {
ko.utils.registerEventHandler(element, "change", function () {
writeValueToProperty(valueAccessor(), allBindingsAccessor, "file", element.files[0]);
});
},
update: function (element, valueAccessor) {
if (ko.utils.unwrapObservable(valueAccessor()) == null) {
element.value = "";
}
}
};
var writeValueToProperty = function (property, allBindingsAccessor, key, value, checkIfDifferent) {
if (!property || !ko.isObservable(property)) {
var propWriters = allBindingsAccessor()['_ko_property_writers'];
if (propWriters && propWriters[key])
propWriters[key](value);
} else if (ko.isWriteableObservable(property) && (!checkIfDifferent || property.peek() !== value)) {
property(value);
}
};
} ());
FileReader usage (code specific for how I used it in my app). Check the FileReader part only.
Note if you have large files you need to stream instead
processFiles: function (viewModel, callback) {
var files = [];
ko.utils.arrayForEach(viewModel.files, function (file) {
if (file.fileUpload() == null) return;
var count = Enumerable.From(viewModel.files)
.Where(function(f) { return f.fileUpload() != null; })
.Count();
var reader = new FileReader();
reader.onload = function (e) {
var base64 = e.target.result.substr(e.target.result.indexOf(",") + 1);
files.push({ Type: file.type, Data: base64, Filename: file.fileUpload().name });
if (files.length == count) {
callback(files);
}
};
reader.readAsDataURL(file.fileUpload());
});
}
Update
Like said this is code for my use case, you will need to change it to fit your purpose
<div data-bind="foreach: files">
<p><label data-bind="text: typeName"></label><input data-bind="file: fileUpload" type="file" /></p>
</div>
Controller (I would use WebApi if I did this today)
public JsonResult UploadFiles(IEnumerable<FileUploadViewModel> uploads)
{
fileRepository.SaveUploadedFiles(Mapper.Map<IEnumerable<FileUploadViewModel>, IEnumerable <FileUpload>> (uploads));
return string.Empty.AsJson();
}
FileUploadViewModel
public class FileUploadViewModel
{
public FileType Type { get; set; }
public string Filename { get; set; }
public string Data { get; set; }
}
Automapper config
Mapper.CreateMap<FileUploadViewModel, FileUpload>()
.ForMember(to => to.Buffer, opt => opt.MapFrom(from => Convert.FromBase64String(from.Data)));

Related

Getting Unsupported Media Type for .net core

I'm new in backend development and Gettig 415 Unsupported Media Type in multipart API written in .net core. I have attached the postman image for your reference. Thanks in advance.
[HttpPost]
[Route("uploadFiles")]
public async Task<ActionResult<IEnumerable<status>>> UploadFilesAsync([FromBody] UploadFile uploadFile)
{
using (var client = new AmazonS3Client("Access key", "Secret key", Region))
{
status s = new status();
try
{
if (uploadFile.Image != null && uploadFile.Image.Length > 0 && uploadFile.Name != null && uploadFile.Name != "")
{
using (var fileStream = new FileStream(uploadFile.Image.FileName, FileMode.Create))
{
uploadFile.Image.CopyTo(fileStream);
var uploadRequest = new TransferUtilityUploadRequest
{
InputStream = fileStream,
Key = uploadFile.Name,
BucketName = "needbucket",
CannedACL = S3CannedACL.PublicRead
};
var fileTransferUtility = new TransferUtility(client);
await fileTransferUtility.UploadAsync(uploadRequest);
}
s.Status = "1";
s.resone = "File uploded sucesefully.";
return Ok(s);
}
else
{
s.Status = "0";
s.resone = "Image and Image name canot be blank";
return Ok(s);
}
}
catch (Exception e)
{
s.Status = "0";
s.resone = "Somthing went wrong." + e.Message;
return Ok(s);
}
}
}
Getting in response on the postman.
1.Change your [FromBody] to [FromForm]
2.In the Body tab, select the form-data option. Then hover your mouse over the row so you can see a dropdown appear that says Text. Click this dropdown and set it to File.
Below is a work demo, you can refer to it.
public class UploadFile
{
public string Name { get; set; }
public IFormFile Image { get; set; }//this is my key name
}
Result:

How to download multiple files at once from S3 using C# AWS SDK

How to download multiple files from s3 buckets. I could not find any better option on SO.
Here is my code for single file download. Given list of Urls, I am looping to download multiple files.
public async Task Download(string url, Stream output)
{
var s3Uri = new AmazonS3Uri(url);
GetObjectRequest getObjectRequest = new GetObjectRequest
{
BucketName = s3Uri.Bucket,
Key = System.Net.WebUtility.UrlDecode(s3Uri.Key)
};
using (var s3Client = new AmazonS3Client(s3Uri.Region))
{
// dispose the underline stream when writing to stream is done
using (var getObjectResponse = await s3Client.GetObjectAsync(getObjectRequest).ConfigureAwait(false))
{
using (var responseStream = getObjectResponse.ResponseStream)
{
await responseStream.CopyToAsync(output);
}
}
}
output.Seek(0L, SeekOrigin.Begin);
}
Download files given s3 urls
var list = new List<Stream>();
foreach(var url in urls)
{
var stream = new MemoryStream();
await Download(url,ms);
list.Add(stream);
}
Is there any better option to download multiple files at once from S3?
I finally decided to implement my own version
public class StreamWrapper
{
public string Url { get; set; }
public Stream Content { get; set; }
public string FileName { get; set; }
}
public async Task Download(IList<StreamWrapper> inout, int maxConcurrentDownloads)
{
if (maxConcurrentDownloads <= 0)
{
maxConcurrentDownloads = 20;
}
if (!inout.HasAny())
return;
var tasks = new List<Task>();
for (int i = 0; i < inout.Count; i++)
{
StreamWrapper wrapper = inout[i];
AmazonS3Uri s3Uri = null;
if (AmazonS3Uri.TryParseAmazonS3Uri(wrapper.Url, out s3Uri))
{
tasks.Add(GetObject(s3Uri, wrapper.Content));
}
if (tasks.Count == maxConcurrentDownloads || i == inout.Count - 1)
{
await Task.WhenAll(tasks);
tasks.Clear();
}
}
}
private async Task GetObject(AmazonS3Uri s3Uri, Stream output)
{
GetObjectRequest getObjectRequest = new GetObjectRequest
{
BucketName = s3Uri.Bucket,
Key = System.Net.WebUtility.UrlDecode(s3Uri.Key)
};
using (var s3Client = new AmazonS3Client(s3Uri.Region))
{
// dispose the underline stream when writing to local file system is done
using (var getObjectResponse = await s3Client.GetObjectAsync(getObjectRequest).ConfigureAwait(false))
{
using (var responseStream = getObjectResponse.ResponseStream)
{
await responseStream.CopyToAsync(output);
}
}
}
output.Seek(0L, SeekOrigin.Begin);
}

Is there an optimization for writing images or PDFs faster to the database?

I'm facing an issue that my upload time for an image or PDF of 40+MB is more than 2.5 minutes (20+ seconds of which are just routing the request from the frontend to the backend but I'm worried more about the sql query slowlyness). I pasted some code snippets below. I also don't get an upload percentage indicator and the fetched bytes don't open as an image either in html using <img src="data:image/jpeg;base64,#Convert.ToBase64String(Model.Image)" /> or using Win10 Photos app.
I'm looking first of all for an optimization on how can I write an image or pdf file faster into the database?
Index.cshtml
using (Html.BeginForm("Index", "File", FormMethod.Post, new { enctype = "multipart/form-data" }))
{
#Html.AntiForgeryToken()
#Html.TextBoxFor(m => m.File, new { #type = "file", #accept = "image/jpeg,image/gif,image/png,application/pdf" })
<input type="submit" value="Upload" />
#Html.ValidationSummary()
}
Controller
[HttpPost]
[ValidateAntiForgeryToken]
public async Task<ActionResult> Index(Model model)
{
var file = model.File;
if (file != null)
{
if (file.ContentLength == 0)
{
ModelState.AddModelError(String.Empty, "File cannot be empty.");
}
const int maxFileSizeMB = 50;
if (file.ContentLength > maxFileSizeMB * 1024 * 1024)
{
ModelState.AddModelError(String.Empty, $"File cannot be bigger than {maxFileSizeMB} megabytes.");
}
//Check for content-type, file extension, and first bytes if it's indeed a valid image.
if (!file.IsImageOrPdfFile())
{
ModelState.AddModelError(String.Empty, "File is not a valid image.");
}
if (ModelState.IsValid)
{
var fileName = file.FileName;
var fileBytes = new byte[file.ContentLength];
file.InputStream.Read(fileBytes, 0, file.ContentLength);
await _api.SaveFile(fileName, fileBytes);
//success!
return Redirect(Url.Action("Index"));
}
}
//View with errors
return View(model);
}
API method
public async Task SaveFile(string fileName, byte[] fileBytes)
{
var client = CreateRestClient();
client.Timeout = 300000;
var request = CreateJsonPostRequest("File", new SaveFileRequest
{
FileName = fileName,
FileBytes = fileBytes
});
var response = await client.ExecuteAsync(request);
if (response.StatusCode != HttpStatusCode.NoContent && response.StatusCode != HttpStatusCode.OK)
{
throw new Exception(response.Content);
}
}
Backend API
[RoutePrefix("api/File")]
public class FileController : ApiController
{
private readonly IFileActionsRepository _fileActions;
public FileController(IFileActionsRepository fileActions)
{
_fileActions = fileActions;
}
[Authorize]
[HttpPost, Route("")]
public async Task Post([FromBody] SaveFileRequest request)
{
if (request == null) throw new ArgumentNullException(nameof(request));
if (request.FileBytes.Length == 0) throw new ArgumentException("request.FileBytes");
_fileActions.SaveFile(request.FileName, request.FileBytes);
}
}
FileActionsRepository.cs
public class FileActionsRepository : IFileActionsRepository
{
private readonly IDataContext _dataContext;
public FileActionsRepository(IDataContext dataContext)
{
_dataContext = dataContext;
//Get the ObjectContext related to this DbContext
var objectContext = (_dataContext.Context as IObjectContextAdapter).ObjectContext;
//Sets the command timeout for all the commands (since it's too slow)
objectContext.CommandTimeout = 300;
}
public void SaveFile(string fileName, byte[] fileBytes)
{
//Using sql query (with update .write() block) since it's faster than entity framework linq-2-entities.
_dataContext.Context.Database.ExecuteSqlCommand(
"if exists (select FileName from Files with (updlock,serializable) where FileName = #FileName)"
+ " update Files set FileBytes .write(#FileBytes, 0, null), SysModified = GETDATE()"
+ " where FileName = #FileName"
+ " else"
+ " insert into Files (FileName, FileBytes, SysCreated)"
+ " values (#FileName, CONVERT(varbinary, '0x00'), GETDATE())"
+ " update Files set FileBytes .write(#FileBytes, 0, null)"
+ " where FileName = #FileName",
new SqlParameter("#FileName", fileName),
new SqlParameter("#FileBytes", fileBytes));
}
}

How to send file using Chopper?

In my kotlin project, I use retrofit and it works well.
suspend fun createPlan(
context: Context?,
name: String,
file: File?
): ABC? {
val fileSignImage = file?.let {
MultipartBody.Part.createFormData(
"image",
it.getName(),
RequestBody.create("image/*".toMediaTypeOrNull(), it)
)
}
return RetrofitFactory.apiCall(context) {
RetrofitFactory.makeRetrofitService().createPlan(
name.toRequestBody("text/plain".toMediaTypeOrNull()),
fileSignImage
)
}}
RetrofitService
#Multipart
#POST("create_plan")
fun createPlan(
#Part("name") name: RequestBody,
#Part image: MultipartBody.Part?
): Deferred<Response<WebApiResponse.ABCs>>
If I want to use Chopper, what is the correct way?
This is what I have tried
Future<Response> createPlan(
BuildContext context, String name,String path) async {
Response response;
try {
response = await _service.createPlan(
name,path);
return response;
} catch (e) {
rethrow;
}
}
Service
#Post(path: "create_plan")
#multipart
Future<Response> createPlan(
#Field('name') String name,#PartFile('image') String imagePath);
How can I convert the imagePath to file so I can pass it as file to server using Chopper?
Anyone?
Looking at the documentation for Chopper, the PartFile annotation supports three data types:
List<int>
String (path of your file)
MultipartFile (from package:http)
You are currently using String, but for reasons unknown it is not working for you. The first option would probably be the most straightforward, but the third option would be the most similar to what you currently have in Retrofit, so we could try that.
import 'package:http/http.dart';
...
Future<Response> createPlan(BuildContext context, String name, String path) async {
Response response;
try {
final bytes = (await File(path).readAsBytes()).toList();
final file = MultipartFile.fromBytes('image', bytes);
response = await _service.createPlan(
name,
file,
);
return response;
} catch (e) {
rethrow;
}
}
Service
#Post(path: "create_plan")
#multipart
Future<Response> createPlan(
#Field('name') String name,
#PartFile('image') MultipartFile image,
);
I was managed to upload file using http instead of Chopper.
Future<http.Response> createPlan(String name, String path) async {
var request = http.MultipartRequest(
"POST",
Uri.parse(
"http://xxx"));
request.fields['name'] = name;
request.files.add(await http.MultipartFile.fromPath(
'image',
path,
));
try {
var streamedResponse = await request.send();
var response = http.Response.fromStream(streamedResponse);
return response;
} catch (e) {
rethrow;
}
}

How to use Scroll while passing raw json Query to ElasticSearch using NEST

var query = #"
{
""query"": {
""match_all"": { }
}
}";
Func<SearchRequestParameters, SearchRequestParameters> requestParameters = a =>
a.SearchType(SearchType.Scan).Scroll(TimeSpan.FromSeconds(60));
var searchResult = await client.LowLevel.SearchAsync<SearchResponse<T>>(indexName, mappingName, query , requestParameters)
if (searchResult.Body.IsValid)
{
var scrollNo = 0;
var results = await client.ScrollAsync<T>("10s", searchResult.Body.ScrollId);
while (results.Documents.Any())
{
documents.AddRange(results.Documents);
scrollNo++;
results = await client.ScrollAsync<T>("10s", results.ScrollId);
return new Customresponse<T>
{
Documents = documents,
total = result.Body.Total
};
}
Would like to pull all data using scroll while passing raw json query. but scroll is not working properly while passing json raw query. Can anyone help on this ?.
Your example is nearly there but not quite; you're missing a closing brace for the while loop to collect all documents before returning the custom response.
Here's an example I just ran on the Stackoverflow data set, to return all questions tagged with nest
private IElasticClient _client;
void Main()
{
var pool = new SingleNodeConnectionPool(new Uri("http://localhost:9200"));
var defaultIndex = "default-index";
var connectionSettings = new ConnectionSettings(pool);
_client = new ElasticClient(connectionSettings);
var query = #"
{
""query"": {
""term"": {
""tags"": {
""value"": ""nest""
}
}
}
}";
var result = RunScrollAsync(query).Result.Dump();
}
private async Task<Customresponse<Question>> RunScrollAsync(string query)
{
var scrollTime = "10s";
// omit the .SearchType(Scan) which is deprecated. Not
// specifying means the first response contains the first set
// of documents
var esResponse = await _client.LowLevel.SearchAsync<SearchResponse<Question>>(
"posts",
"question",
query, r => r.Scroll(TimeSpan.FromSeconds(10))).ConfigureAwait(false);
if (esResponse.Body.IsValid && esResponse.Body.Documents.Any())
{
// assume you have less than 2,147,483,647 documents to return?
var documents = new List<Question>((int)esResponse.Body.Total);
documents.AddRange(esResponse.Body.Documents);
var scrollNo = 0;
var response = await _client.ScrollAsync<Question>(scrollTime, esResponse.Body.ScrollId).ConfigureAwait(false);;
// keep scrolling until no more documents are returned
while (response.Documents.Any())
{
documents.AddRange(response.Documents);
scrollNo++;
response = await _client.ScrollAsync<Question>(scrollTime, response.ScrollId).ConfigureAwait(false);;
}
return new Customresponse<Question>
{
Documents = documents,
total = response.Total
};
}
// return an empty result.
// Or throw an exception, or log - whatever you need to do
return new Customresponse<Question>
{
Documents = Enumerable.Empty<Question>(),
total = 0
};
}
public class Customresponse<T>
{
public IEnumerable<T> Documents { get; set; }
public long total { get; set; }
}
This returns all 342 questions, with a total of 342 (Data set is from June 2016).