I am trying to create a function for all blobs in a container. I took the help
How to get hold of all the blobs in a Blob container which has sub directories levels(n levels)?, which seems to use an overload that doesn't exist any more. I had added default values into the additional fields prefix and operationContext :
static internal async Task<List<string>> ListBlobNames(string ContainerName)
{
BlobContinuationToken continuationToken = null;
bool useFlatBlobListing = true;
BlobListingDetails blobListingDetails = BlobListingDetails.None;
var blobOptions = new BlobRequestOptions();
CloudBlobContainer container = Container(ContainerName);
var operationContext = new OperationContext();
var verify = container.GetBlobReference("A_Valid_Name.jpg");
var verify2 = container.GetBlobReference("NotAName.jpg");
using (var a = await verify.OpenReadAsync()) ;
//using (var a = await verify2.OpenReadAsync()); // doesn't work since it doesn't exist
return (await container.ListBlobsSegmentedAsync("", useFlatBlobListing, blobListingDetails, null, continuationToken, blobOptions, operationContext))
.Results.Select(s => s.Uri.LocalPath.ToString()).ToList();
}
The last line gave me an exception:
StorageException: The requested URI does not represent any resource on the server.
I then created the verfiy and verify2 variables to test if my container is valid. verify references a valid blob and verify2 references an invalid blob name. Running the code with the second using statement uncommented gave me an error in the second using statement. This shows that the verify works and thus the container is valid.
I am trying to create a function for all blobs in a container.
You could leverage the Azure Storage Client Library and install the package WindowsAzure.Storage, then you could follow the tutorial List blobs in pages asynchronously to achieve your purpose. For test, I just created my .Net Core console application as follows:
static void Main(string[] args)
{
MainAsync(args).GetAwaiter().GetResult();
}
static async Task MainAsync(string[] args)
{
// Retrieve storage account from connection string.
CloudStorageAccount storageAccount = CloudStorageAccount.Parse("{your-storage-connection-string}");
// Create the blob client.
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
string containerName = "images";
// Retrieve reference to a previously created container.
CloudBlobContainer container = blobClient.GetContainerReference(containerName);
if (await container.ExistsAsync())
await ListBlobsSegmentedInFlatListing(container);
else
Console.WriteLine($"Your container with the name:{containerName} does not exist!!!");
Console.WriteLine("press any key to exit...");
Console.ReadLine();
}
ListBlobsSegmentedInFlatListing:
async public static Task ListBlobsSegmentedInFlatListing(CloudBlobContainer container)
{
//List blobs to the console window, with paging.
Console.WriteLine("List blobs in pages:");
int i = 0;
BlobContinuationToken continuationToken = null;
BlobResultSegment resultSegment = null;
//Call ListBlobsSegmentedAsync and enumerate the result segment returned, while the continuation token is non-null.
//When the continuation token is null, the last page has been returned and execution can exit the loop.
do
{
//This overload allows control of the page size. You can return all remaining results by passing null for the maxResults parameter,
//or by calling a different overload.
resultSegment = await container.ListBlobsSegmentedAsync("", true, BlobListingDetails.All, 10, continuationToken, null, null);
if (resultSegment.Results.Count<IListBlobItem>() > 0) { Console.WriteLine("Page {0}:", ++i); }
foreach (var blobItem in resultSegment.Results)
{
Console.WriteLine("\t{0}", blobItem.StorageUri.PrimaryUri);
}
Console.WriteLine();
//Get the continuation token.
continuationToken = resultSegment.ContinuationToken;
}
while (continuationToken != null);
}
Test:
This works for me...
String myContainerName = "images";
CloudBlobContainer blobContainer = blobClient.GetContainerReference(myContainerName);
CloudBlockBlob blockBlob;
blockBlob = blobContainer.GetBlockBlobReference("NotAName.jpg");
bool verify2 = await blockBlob.ExistsAsync();
if (!verify2)
{
// the blob image does not exist
// do something
}
Related
I am new to .NET and I have to perform this. Assuming we have the connection string and the Environment variable setup, could someone give me resources or code or guide on how to do it?
I just need to upload a pdf file in Azure Blob Storage using Minimal API
From the Minimal API document, we can see that the Minimal API does not support the binding the IFormFile.
No support for binding from forms. This includes binding IFormFile. We plan to add support for IFormFile in the future.
So, to upload file in the Minimal API, you can get the upload file from the HttpRequest Form. Refer to the following code:
app.MapPost("/upload", (HttpRequest request) =>
{
if (!request.Form.Files.Any())
return Results.BadRequest("At least one fie is need");
//Do something with the file
foreach(var item in request.Form.Files)
{
var file = item;
//insert the file into the Azure storage
}
return Results.Ok();
});
The screenshot as below:
Then, to upload the file to Azure Blob Storage, refer the following tutorial:
Upload images/files to blob azure, via web api ASP.NET framework Web application
Code like this:
CloudStorageAccount storageAccount;
Dictionary<string, object> dict = new Dictionary<string, object>();
string strorageconn = ConfigurationManager.AppSettings.Get("MyBlobStorageConnectionString");
if (CloudStorageAccount.TryParse(strorageconn, out storageAccount))
{
try
{
// Create the CloudBlobClient that represents the
// Blob storage endpoint for the storage account.
CloudBlobClient cloudBlobClient = storageAccount.CreateCloudBlobClient();
// Create a container called 'quickstartblobs' and
// append a GUID value to it to make the name unique.
CloudBlobContainer cloudBlobContainer = cloudBlobClient.GetContainerReference("images");
await cloudBlobContainer.CreateIfNotExistsAsync();
// Set the permissions so the blobs are public.
BlobContainerPermissions permissions = new BlobContainerPermissions
{
PublicAccess = BlobContainerPublicAccessType.Blob
};
await cloudBlobContainer.SetPermissionsAsync(permissions);
var httpRequest = HttpContext.Current.Request;
foreach (string file in httpRequest.Files)
{
HttpResponseMessage response = Request.CreateResponse(HttpStatusCode.Created);
var postedFile = httpRequest.Files[file];
string imageName = ("images" + serverTime.Year.ToString() + serverTime.Month.ToString() + serverTime.Day.ToString() +
serverTime.Hour.ToString() + serverTime.Minute.ToString() + serverTime.Second.ToString() + serverTime.Millisecond.ToString()
+ postedFile.FileName );
if (postedFile != null && postedFile.ContentLength > 0)
{
int MaxContentLength = 1024 * 1024 * 1; //Size = 1 MB
IList<string> AllowedFileExtensions = new List<string> { ".jpg", ".gif", ".png" };
var ext = postedFile.FileName.Substring(postedFile.FileName.LastIndexOf('.'));
var extension = ext.ToLower();
if (!AllowedFileExtensions.Contains(extension))
{
var message = string.Format("Please Upload image of type .jpg,.gif,.png.");
dict.Add("error", message);
return Request.CreateResponse(HttpStatusCode.BadRequest, dict);
}
else if (postedFile.ContentLength > MaxContentLength)
{
var message = string.Format("Please Upload a file upto 1 mb.");
dict.Add("error", message);
return Request.CreateResponse(HttpStatusCode.BadRequest, dict);
}
else
{
CloudBlockBlob cloudBlockBlob = cloudBlobContainer.GetBlockBlobReference(imageName);
cloudBlockBlob.Properties.ContentType = postedFile.ContentType;
await cloudBlockBlob.UploadFromStreamAsync(postedFile.InputStream);
}
}
var message1 = string.Format("Image Updated Successfully.");
return Request.CreateErrorResponse(HttpStatusCode.Created, message1);
}
var res3 = string.Format("Please Upload a image.");
dict.Add("error", res3);
return Request.CreateResponse(HttpStatusCode.NotFound, dict);
}
catch (Exception ex)
{
HttpResponseMessage response2 = Request.CreateResponse(HttpStatusCode.BadRequest, ex.InnerException.ToString());
return response2;
}
}
else
{
var res = string.Format("Did not connect successfull.");
dict.Add("error", res);
return Request.CreateResponse(HttpStatusCode.NotFound, dict);
}
I have a very newbie question.
I am following this docs "Azure Blob storage client library v12 for .NET" - https://learn.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-dotnet
When I tested on my console, and my Azure storage, it works.
But I was wondering if I can make a controller out of the suggested 'Main' method?
Because I want these getting and posting to the server actions initiated when the user input changes from the front end side.
This is what the Main method inside of the Program.cs looks like based on the docs
static async Task Main()
{
Console.WriteLine("Azure Blob storage v12 - .NET quickstart sample\n");
string connectionString = Environment.GetEnvironmentVariable("My_CONNECTION_STRING");
BlobServiceClient blobServiceClient = new BlobServiceClient(connectionString);
string containerName = "quickstartblobs" + Guid.NewGuid().ToString();
BlobContainerClient containerClient = await blobServiceClient.CreateBlobContainerAsync(containerName);
string localPath = "./data/";
string fileName = "quickstart" + Guid.NewGuid().ToString() + ".txt";
string localFilePath = Path.Combine(localPath, fileName);
// Write text to the file
await File.WriteAllTextAsync(localFilePath, "Hello, World!");
// Get a reference to a blob
BlobClient blobClient = containerClient.GetBlobClient(fileName);
Console.WriteLine("Uploading to Blob storage as blob:\n\t {0}\n", blobClient.Uri);
// Open the file and upload its data
using FileStream uploadFileStream = File.OpenRead(localFilePath);
await blobClient.UploadAsync(uploadFileStream, true);
uploadFileStream.Close();
Console.WriteLine("Listing blobs...");
// List all blobs in the container
await foreach (BlobItem blobItem in containerClient.GetBlobsAsync())
{
Console.WriteLine("\t" + blobItem.Name);
}
Console.Write("Press any key to begin clean up");
Console.ReadLine();
string downloadFilePath = localFilePath.Replace(".txt", "DOWNLOAD.txt");
Console.WriteLine("\nDownloading blob to\n\t{0}\n", downloadFilePath);
// Download the blob's contents and save it to a file
BlobDownloadInfo download = await blobClient.DownloadAsync();
using (FileStream downloadFileStream = File.OpenWrite(downloadFilePath))
{
await download.Content.CopyToAsync(downloadFileStream);
downloadFileStream.Close();
}
}
So for example, in my HomeController Can I use post related functions as
[HttpPost]
public void Post([FromBody]string value)
{
//Create a unique name for the container
string containerName = "filedata" + Guid.NewGuid().ToString();
// Create the container and return a container client object
BlobContainerClient containerClient = await blobServiceClient.CreateBlobContainerAsync(containerName);
// Create a local file in the ./data/ directory for uploading and downloading
string localPath = "./data/";
string fileName = "textfile" + Guid.NewGuid().ToString() + ".txt";
string localFilePath = Path.Combine(localPath, fileName);
BlobServiceClient blobServiceClient = new BlobServiceClient(connectionString);
// Get a reference to a blob
BlobClient blobClient = containerClient.GetBlobClient(fileName);
Console.WriteLine("Uploading to Blob storage as blob:\n\t {0}\n", blobClient.Uri);
// Open the file and upload its data
using FileStream uploadFileStream = File.OpenRead(localFilePath);
await blobClient.UploadAsync(uploadFileStream, true);
uploadFileStream.Close();
}
Or is this a no-go?
Thanks for helping this super newbie!
So for example, in my HomeController Can I use post related functions Or is this a no-go?
Yes, you can achieve it.
You can use postman to send post request in local to test it. Remember to remove SSL for webserver setting.
Also, change public void Post to public async Task Post and remove using in code:
FileStream uploadFileStream = File.OpenRead(localFilePath);
await blobClient.UploadAsync(uploadFileStream, true);
uploadFileStream.Close()
I am trying to copy a blob document from one container to another along with the metadata. I have tried the following code from azure function but getting error mentioned in the code.
HTTP Request:
{
"SourceUrl": "https://devstorage.blob.core.windows.net/test-docs/123.jpeg",
"DestinationUrl": "https://devstorage.blob.core.windows.net/test-docs-completed/123.jpeg"
}
Azure Function Code:
public static async Task<HttpResponseMessage> Run([HttpTrigger(AuthorizationLevel.Anonymous, "get", "post", Route = null)]HttpRequestMessage req, ILogger log)
{
string reqAsString = await req.Content.ReadAsStringAsync();
MoveProcessedDocumentRequest blobCopyRequest = JsonConvert.DeserializeObject<MoveProcessedDocumentRequest>(reqAsString);
CloudBlockBlob cloudBlockSource = new CloudBlockBlob(new Uri(blobCopyRequest.SourceUrl));
await cloudBlockSource.FetchAttributesAsync();
CloudBlobContainer cloudBlockDestinationContainer = new CloudBlockBlob(new Uri(blobCopyRequest.DestinationUrl)).Container;
string name = cloudBlockSource.Uri.Segments.Last();
CloudBlockBlob cloudBlockDestination;
cloudBlockDestination = cloudBlockDestinationContainer.GetBlockBlobReference(name);
// Copy metadata
foreach (var meta in cloudBlockSource.Metadata)
{
cloudBlockDestination.Metadata.Add(new KeyValuePair<string, string>(meta.Key, meta.Value));
}
await cloudBlockDestination.StartCopyAsync(cloudBlockSource);
// Exception: Microsoft.Azure.Storage.Common: The specified resource does not exist.
return req.CreateResponse(HttpStatusCode.OK);
}
You should modify your code with CloudBlobContainer instance.
Change:
CloudBlobContainer cloudBlockDestinationContainer = new CloudBlockBlob(new Uri(blobCopyRequest.DestinationUrl)).Container;
To:
var uri = new Uri("blobCopyRequest.DestinationUrl");
var storage = new StorageCredentials("your account name", "your storage key");
CloudBlobContainer cloudBlockDestinationContainer = new CloudBlobContainer(uri, storage);
And the DestinationUrl is destcontainer url.
HTTP Request:
{
"SourceUrl": "https://devstorage.blob.core.windows.net/test-docs/123.jpeg",
"DestinationUrl": "https://devstorage.blob.core.windows.net/test-docs-completed"
}
I am getting following error when accessing HttpContext.Session from static method placed in separate task:
Session has not been configured for this application or request.
I used this article to implement access to HttpContext outside the controller
From controller I invoke this static method that used to retrieve image data:
public static void CreateDummyGallery(Gallery gallery)
{
Logger.LogDebug(LogModule.Dummy, $"Starting gallery creation.");
Task.Factory.StartNew(() =>
{
try
{
List<DummyPicture> pictures;
using (var context = new MyzeumContext())
{
int top = 10;
pictures = context.DummyPictures.FromSql($"SELECT * FROM dummypictures ORDER BY RAND() LIMIT {top}").ToList();
}
Logger.LogDebug(LogModule.Dummy, $"Starting retrieving images.");
Parallel.ForEach(pictures, picture => {
using (WebClient client = new WebClient())
{
}
});
Logger.LogDebug(LogModule.Dummy, $"Done retrieving images.");
}
catch(Exception e)
{
Logger.LogError(LogModule.Server, e.Message, e);
}
});
}
The problem occurs in Logger.LogDebug() because this is where I access HttpContext:
public void LogDebug(LogModule module, string message, Exception stackTrace = null)
{
Log record = new Log();
record.Module = module;
record.ThreadId = Environment.CurrentManagedThreadId;
record.SessionId = HttpContextHelper.Current?.Session?.Id;
record.Message = message;
record.Logged = DateTime.UtcNow;
if(stackTrace != null)
{
record.Message += $" :{stackTrace.StackTrace}";
}
queue.Enqueue(record);
}
The problem 99% occurs in the first call inside task:
Logger.LogDebug(LogModule.Dummy, $"Starting retrieving images.");
BUT, right after application starts this whole task block works fine and does not throw any exception. Problem starts after following requests.
I'm trying to access a request's raw input body/stream in ASP.net 5. In the past, I was able to reset the position of the input stream to 0 and read it into a memory stream but when I attempt to do this from the context the input stream is either null or throws an error (System.NotSupportedException => "Specified method is not supported.").
In the first example below I can access the raw request in a controller if I declare the controller method's parameter object type as dynamic. For various reasons, this is not a solution and I need to access the raw request body in an authentication filter anyways.
This Example Works, But Is Not a Reasonable Solution:
[HttpPost("requestme")]
public string GetRequestBody([FromBody] dynamic body)
{
return body.ToString();
}
Throws Error:
[HttpPost("requestme")]
public string GetRequestBody()
{
var m = new MemoryStream();
Request.Body.CopyTo(m);
var contentLength = m.Length;
var b = System.Text.Encoding.UTF8.GetString(m.ToArray());
return b;
}
Throws Error:
[HttpPost("requestme")]
public string GetRequestBody()
{
Request.Body.Position = 0;
var input = new StreamReader(Request.Body).ReadToEnd();
return input;
}
Throws Error:
[HttpPost("requestme")]
public string GetRequestBody()
{
Request.Body.Position = 0;
var input = new MemoryStream();
Request.Body.CopyTo(input);
var inputString = System.Text.Encoding.UTF8.GetString(input.ToArray());
return inputString;
}
I need to access the raw request body of every request that comes in for an API that I am building.
Any help or direction would be greatly appreciated!
EDIT:
Here is the code that I would like to read the request body in.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using Microsoft.AspNet.Mvc;
using Microsoft.AspNet.Http;
namespace API.Filters
{
public class CustomAuthorizationAttribute : Attribute, IAuthorizationFilter
{
public CustomAuthorizationAttribute()
{ }
public void OnAuthorization(AuthorizationContext context)
{
if (context == null)
throw new ArgumentNullException("OnAuthorization AuthorizationContext context can not be null.");
else
{
if (this.AuthorizeCore(context.HttpContext) == false)
{
// Do Other Stuff To Check Auth
}
else
{
context.Result = new HttpUnauthorizedResult();
}
}
}
protected virtual bool AuthorizeCore(HttpContext httpContext)
{
var result = false;
using (System.IO.MemoryStream m = new System.IO.MemoryStream())
{
try
{
if (httpContext.Request.Body.CanSeek == true)
httpContext.Request.Body.Position = 0;
httpContext.Request.Body.CopyTo(m);
var bodyString = System.Text.Encoding.UTF8.GetString(m.ToArray());
return CheckBody(bodyString); // Initial Auth Check returns true/false <-- Not Shown In Code Here on Stack Overflow
}
catch (Exception ex)
{
Logger.WriteLine(ex.Message);
}
}
return false;
}
}
}
This code would be accessed when a call is made to a controller method marked with the CustomAuthorization attribute like so.
[Filters.CustomAuthorizationAuthorization]
[HttpPost]
public ActionResult Post([FromBody]UserModel Profile)
{
// Process Profile
}
Update
The information below is pretty outdated by now. Due to performance reasons this is not possible by default, but fortunately can be changed. The latest solution should be to enable request buffering with EnableBuffering:
Request.EnableBuffering();
See also this blog post for more information: https://devblogs.microsoft.com/aspnet/re-reading-asp-net-core-request-bodies-with-enablebuffering/.
Old, outdated answer for reference
The implementation of Request.Body depends on the controller action.
If the action contains parameters it's implemented by Microsoft.AspNet.WebUtilities.FileBufferingReadStream, which supports seeking (Request.Body.CanSeek == true). This type also supports setting the Request.Body.Position.
However, if your action contains no parameters it's implemented by Microsoft.AspNet.Loader.IIS.FeatureModel.RequestBody, which does not support seeking (Request.Body.CanSeek == false). This means you can not adjust the Position property and you can just start reading the stream.
This difference probably has to do with the fact that MVC needs to extract the parameters values from the request body, therefore it needs to read the request.
In your case, your action does not have any parameters. So the Microsoft.AspNet.Loader.IIS.FeatureModel.RequestBody is used, which throws an exception if you try to set the Position property.
**Solution**: either do not set the position or check if you actually _can_ set the position first:
if (Request.Body.CanSeek)
{
// Reset the position to zero to read from the beginning.
Request.Body.Position = 0;
}
var input = new StreamReader(Request.Body).ReadToEnd();
The exceptions you see in your three last snippets are the direct consequence of trying to read the request body multiple times - once by MVC 6 and once in your custom code - when using a streamed host like IIS or WebListener. You can see this SO question for more information: Read body twice in Asp.Net 5.
That said, I'd only expect this to happen when using application/x-www-form-urlencoded, since it wouldn't be safe for MVC to start reading the request stream with lengthy requests like file uploads. If that's not the case, then it's probably a MVC bug you should report on https://github.com/aspnet/Mvc.
For workarounds, you should take a look at this SO answer, that explains how you can use context.Request.ReadFormAsync or add manual buffering: Read body twice in Asp.Net 5
app.Use(next => async context => {
// Keep the original stream in a separate
// variable to restore it later if necessary.
var stream = context.Request.Body;
// Optimization: don't buffer the request if
// there was no stream or if it is rewindable.
if (stream == Stream.Null || stream.CanSeek) {
await next(context);
return;
}
try {
using (var buffer = new MemoryStream()) {
// Copy the request stream to the memory stream.
await stream.CopyToAsync(buffer);
// Rewind the memory stream.
buffer.Position = 0L;
// Replace the request stream by the memory stream.
context.Request.Body = buffer;
// Invoke the rest of the pipeline.
await next(context);
}
}
finally {
// Restore the original stream.
context.Request.Body = stream;
}
});
I just had this same issue. Remove the parameters from the method signature, and then read the Request.Body Stream how you want to.
You need to call Request.EnableRewind() to allow the stream to be rewound so you can read it.
string bodyAsString;
Request.EnableRewind();
using (var streamReader = new StreamReader(Request.Body, Encoding.UTF8))
{
bodyAsString = streamReader.ReadToEnd();
}
I Know this my be late but in my case its Just I had a problem in routing as bellow
At startup.cs file I was beginning the routing with /api
app.MapWhen(context => context.Request.Path.StartsWithSegments(new PathString("/api")),
a =>
{
//if (environment.IsDevelopment())
//{
// a.UseDeveloperExceptionPage();
//}
a.Use(async (context, next) =>
{
// API Call
context.Request.EnableBuffering();
await next();
});
}
//and I was putting in controller
[HttpPost]
[Route("/Register", Name = "Register")]
//Just Changed the route to start with /api like my startup.cs file
[HttpPost]
[Route("/api/Register", Name = "Register")]
//and now the params are not null and I can ready the body request multiple