Azure Web Jobs Redis (RedLock) & Blob Storage Access Issues - azure-storage

We switched to WebJobs with our background tasks that are starting to work when a new item lands on an Azure Queue. Now we have some weird issues that he seems to have problems accessing Redis RedLock and Storage that I can't explain.
Now the biggest issue we have is RedLock. We are using RedLock.Net for distributed locking. Now this works all fine in our production web application and it also worked on the background workers we had but as soon as we switched to WebJobs he basically failed to aquire the lock. To back this up with some code...we are locking like this:
using (var redisLock = await _redLockConnection.RedisLockFactory.CreateAsync(resource, UserLockExpiryTime, UserLockWaitTime, UserLockRetryTime))
{
// make sure we got the lock
if (redisLock.IsAcquired)
{
// execute code...
}
else
{
throw new CouldNotAcquireRedLockException();
}
}
The problem here is, IsAcquired is always false within a Webjob and I have no clue why!?
The second thing that maybe relates to this problem is deleting a blob file in azure storage that fails with a 404 only within a WebJob.
var file = _blobContainer.GetBlockBlobReference("file.txt");
file?.Delete();
This will fail with a 404 Not found exception within the WebJob.
Is there anything I missed setting up the webjob? Could it be an access problem for write operations? Would be glad for any help!

IsAcquired is always false within a Webjob
I do a test with the following code using RedLock.net in an Azure WebJob, I can acquire lock on resource if the lock is available.
public static void ProcessQueueMessage([QueueTrigger("mymes")] string message, TextWriter log)
{
var azureEndPoint = new RedisLockEndPoint
{
EndPoint = new DnsEndPoint("{YOUR_CACHE}.redis.cache.windows.net", 6380),
Password = "YOUR_ACCESS_KEY",
Ssl = true
};
var eps = new[] { azureEndPoint };
var rlf = new RedisLockFactory(eps);
var resource = "https://{storageaccount}.blob.core.windows.net/{containername}/test.txt";
var expiry = TimeSpan.FromSeconds(50);
var wait = TimeSpan.FromSeconds(10);
var retry = TimeSpan.FromSeconds(1);
using (var redisLock = rlf.Create(resource, expiry, wait, retry))
{
Console.WriteLine("Lock acquired: " + redisLock.IsAcquired);
}
log.WriteLine(message);
}
Result of test:
deleting a blob file in azure storage that fails with a 404
As I mentioned in comment, Please check if that Blob is existing via Azure portal or Azure storage explorer, or call Exists method to check existence of the blob before you delete it.

Related

Design Minimal API and use HttpClient to post a file to it

I have a legacy system interfacing issue that my team has elected to solve by standing up a .NET 7 Minimal API which needs to accept a file upload. It should work for small and large files (let's say at least 500 MiB). The API will be called from a legacy system using HttpClient in a .NET Framework 4.7.1 app.
I can't quite seem to figure out how to design the signature of the Minimal API and how to call it with HttpClient in a way that totally works. It's something I've been hacking at on and off for several days, and haven't documented all of my approaches, but suffice it to say there have been varying results involving, among other things:
4XX and 500 errors returned by the HTTP call
An assortment of exceptions on either side
Calls that throw and never hit a breakpoint on the API side
Calls that get through but the Stream on the API end is not what I expect
Errors being different depending on whether the file being uploaded is small or large
Text files being persisted on the server that contain some of the HTTP headers in addition to their original contents
On the Minimal API side, I've tried all sorts of things in the signature (IFormFile, Stream, PipeReader, HttpRequest). On the calling side, I've tried several approaches (messing with headers, using the Flurl library, various content encodings and MIME types, multipart, etc).
This seems like it should be dead simple, so I'm trying to wipe the slate clean here, start with an example of something that partially works, and hope someone might be able to illuminate the path forward for me.
Example of Minimal API:
// IDocumentStorageManager is an injected dependency that takes an int and a Stream and returns a string of the newly uploaded file's URI
app.MapPost(
"DocumentStorage/CreateDocument2/{documentId:int}",
async (PipeReader pipeReader, int documentId, IDocumentStorageManager documentStorageManager) =>
{
using var ms = new MemoryStream();
await pipeReader.CopyToAsync(ms);
ms.Position = 0;
return await documentStorageManager.CreateDocument(documentId, ms);
});
Call the Minimal API using HttpClient:
// filePath is the path on local disk, uri is the Minimal API's URI
private static async Task<string> UploadWithHttpClient2(string filePath, string uri)
{
var fileStream = File.Open(filePath, FileMode.Open);
var content = new StreamContent(fileStream);
var httpRequestMessage = new HttpRequestMessage(HttpMethod.Post, uri);
var httpClient = new HttpClient();
httpRequestMessage.Content = content;
httpClient.Timeout = TimeSpan.FromMinutes(5);
var result = await httpClient.SendAsync(httpRequestMessage);
return await result.Content.ReadAsStringAsync();
}
In the particular example above, a small (6 bytes) .txt file is uploaded without issue. However, a large (619 MiB) .tif file runs into problems on the call to httpClient.SendAsync which results in the following set of nested Exceptions:
System.Net.Http.HttpRequestException - "Error while copying content to a stream."
System.IO.IOException - "Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host.."
System.Net.Sockets.SocketException - "An existing connection was forcibly closed by the remote host."
What's a decent way of writing a Minimal API and calling it with HttpClient that will work for small and large files?
Kestrel allows uploading 30MB per default.
To upload larger files via kestrel you might need to increase the max size limit. This can be done by adding the "RequestSizeLimit" attribute. So for example for 1GB:
app.MapPost(
"DocumentStorage/CreateDocument2/{documentId:int}",
[RequestSizeLimit(1_000_000_000)] async (PipeReader pipeReader, int documentId) =>
{
using var ms = new MemoryStream();
await pipeReader.CopyToAsync(ms);
ms.Position = 0;
return "";
});
You can also remove the size limit globally by setting
builder.WebHost.UseKestrel(o => o.Limits.MaxRequestBodySize = null);
This answer is good but the RequestSizeLimit filter doesn't work for minimal APIs, it's an MVC filter. You can use the IHttpMaxRequestBodySizeFeature to limit the size (assuming you're not running on IIS). Also, I made a change to accept the body as a Stream. This avoids the memory stream copy before calling the CreateDocument API:
app.MapPost(
"DocumentStorage/CreateDocument2/{documentId:int}",
async (Stream stream, int documentId, IDocumentStorageManager documentStorageManager) =>
{
return await documentStorageManager.CreateDocument(documentId, stream);
})
.AddEndpointFilter((context, next) =>
{
const int MaxBytes = 1024 * 1024 * 1024;
var maxRequestBodySizeFeature = context.HttpContext.Features.Get<IHttpMaxRequestBodySizeFeature>();
if (maxRequestBodySizeFeature is not null and { IsReadOnly: true })
{
maxRequestBodySizeFeature.MaxRequestBodySize = MaxBytes;
}
return next(context);
});
If you're running on IIS then https://learn.microsoft.com/en-us/iis/configuration/system.webserver/security/requestfiltering/requestlimits/#configuration

Task Module call from Ms Teams in Bot Framework

I am looking to open a task module (Pop up - iframe with audio/video) in my bot that is connected to Teams channel. I am having issues following the sample code provided on the GitHub page.
I have tried to follow the sample and incorporate to my code by did not succeed.
In my bot.cs file I am creating card action of invoke type:
card.Buttons.Add(new CardAction("invoke", TaskModuleUIConstants.YouTube.ButtonTitle, null,null,null,
new Teams.Samples.TaskModule.Web.Models.BotFrameworkCardValue<string>()
{
Data = TaskModuleUIConstants.YouTube.Id
}));
In my BotController.cs that inherits from Controller
[HttpPost]
public async Task PostAsync()
{
// Delegate the processing of the HTTP POST to the adapter.
// The adapter will invoke the bot.
await _adapter.ProcessAsync(Request, Response, _bot);
}
public async Task<HttpResponseMessage> Post([FromBody] Activity activity)
{
if (activity.Type == ActivityTypes.Invoke)
{
return HandleInvokeMessages(activity);
}
return new HttpResponseMessage(HttpStatusCode.Accepted);
}
private HttpResponseMessage HandleInvokeMessages (Activity activity)
{
var activityValue = activity.Value.ToString();
if (activity.Name == "task/fetch")
{
var action = Newtonsoft.Json.JsonConvert.DeserializeObject<Teams.Samples.TaskModule.Web.Models.BotFrameworkCardValue<string>>(activityValue);
Teams.Samples.TaskModule.Web.Models.TaskInfo taskInfo = GetTaskInfo(action.Data);
Teams.Samples.TaskModule.Web.Models.TaskEnvelope taskEnvelope = new Teams.Samples.TaskModule.Web.Models.TaskEnvelope
{
Task = new Teams.Samples.TaskModule.Web.Models.Task()
{
Type = Teams.Samples.TaskModule.Web.Models.TaskType.Continue,
TaskInfo = taskInfo
}
};
return msg;
}
return new HttpResponseMessage(HttpStatusCode.Accepted);
}
There is more code as per the GitHub sample but I won't paste it here. Can someone point me into the correct direction ?
I have got to the stage that it is displaying a pop up window but the content and title comes from manifest file instead of creating actual iframe also no video is rendering. My goal is to render video within my teams using iframe container.
The important part from the sample:
This sample is deployed on Microsoft Azure and you can try it yourself by uploading Task Module CSharp.zip to one of your teams and/or as a personal app. (Sideloading must be enabled for your tenant; see step 6 here.) The app is running on the free Azure tier, so it may take a while to load if you haven't used it recently and it goes back to sleep quickly if it's not being used, but once it's loaded it's pretty snappy.
So,
Your Teams Admin MUST enable sideloading
Your bot MUST be sideloaded into Teams
The easiest way to do this would be download the sample manifest, open it in App Studio, then edit your bot information in. You then need to make sure Domains and permissions > Valid Domains are set for your bot. Also ensure you change the Tabs URLs to your own.
You also need to make sure that in your Tasks, the URLs they call ALL use https and not http. If anywhere in the chain is using http (like if you're using ngrok and http://localhost), it won't work.

Call Azure API from WebJob/ Shared code between WebJob and web api

I have a web api in an ASE and an associated web job. I am trying to call this web api from the web job but it always fails with winhttpexception: a security error has occurred. I have put in all the tls related settings but still getting the error.
Any suggestions on the error?
Also is there a way to share code between WebJob and web api?
I was able to resolve the issue by setting the below in my code.This resolved the Security Error.
using(var handler = new HttpClientHandler
{
ServerCertificateCustomValidationCallback = (sender,certificate,chain,sslPolicyErrors) => true
})
You could create a console app and publish it as Azure WebJobs. For username and password you could click Get Publish Profile in your Azure webapp overview to get them.
Then you could use the following code in Console App to call your Azure Webapi.
string userName = "$xxxxxx";
string userPassword = "xxxxxxxxxxxxx";
string webAppName = "xxxxxx";
var base64Auth = Convert.ToBase64String(Encoding.Default.GetBytes($"{userName}:{userPassword}"));
using (var client = new HttpClient())
{
client.DefaultRequestHeaders.Add("Authorization", "Basic " + base64Auth);
var baseUrl = new Uri($"https://{webAppName}.azurewebsites.net/api/values");
var result = client.GetAsync(baseUrl).Result;
if (result.IsSuccessStatusCode)
{
var readTask = result.Content.ReadAsStringAsync();
readTask.Wait();
var value = readTask.Result;
Console.WriteLine(value.ToString());
}
}
Console.WriteLine("run successfully");
Output as below:

Sense/net using content query in web application

I try to use content query in web application but it throw an exception " Lucene.Net.Store.AlreadyClosedException: this IndexReader is closed". Please give help me resolve that problem.
var startSettings = new RepositoryStartSettings
{
Console = Console.Out,
StartLuceneManager = true, // <-- this is necessary
IsWebContext = false,
PluginsPath = AppDomain.CurrentDomain.BaseDirectory,
};
using (Repository.Start(startSettings))
{
var resultQuery = ContentQuery.Query("+InTree:#0 + DisplayName:*#1*", null, folderPath, q);
}
The recommended way to connect to Sense/Net from a different application (app domain) is through the REST API. It is much easier to maintain and involves less configuration (the only exception is where you are working inside the Sense/Net application itself, or you only have a single application and you do not want to access Sense/Net from anywhere else, and you are willing to deal with a local index of Sense/Net and all the config values it needs, etc).
Connecting through the REST API does not mean you have to send HTTP requests manually (although that is also not complicated at all): there is a .Net client library which does that for you. You can access all content metadata or binaries through the client, you can upload files, query content, manage permissions, etc.
// loading a content
dynamic content = await Content.LoadAsync(id);
DateTime date = content.BirthDate;
// querying
var results = await Content.QueryAsync(queryText);
Install: https://www.nuget.org/packages/SenseNet.Client
Source and examples: https://github.com/SenseNet/sn-client-dotnet
To use it in a web application, you have to do the following:
initialize the client context once, at the beginning of the application life cycle (e.g. app start)
if you need to make requests to Sense/Net in the name of the currently logged in user (e.g. because you want to query for documents accessible by her), than you have to create a new ServerContext object for every user with the username/password of that user, and provide this object to any client call (e.g. load or save content methods).
var sc = new ServerContext
{
Url = "http://example.com",
Username = "user1",
Password = "asdf"
};
var content = await Content.LoadAsync(id, sc);

Self updating .net CF application

I need to make my CF app self-updating through the web service.
I found one article on MSDN from 2003 that explains it quite well. However, I would like to talk practice here. Anyone really done it before or does everyone rely on third party solutions?
I have been specifically asked to do it this way, so if you know of any tips/caveats, any info is appreciated.
Thanks!
This is relatively easy to do. Basically, your application calls a web service to compare its version with the version available on the server. If the server version is newer, your application downloads the new EXE as a byte[] array.
Next, because you can't delete or overwrite a running EXE file, your application renames its original EXE file to something like "MyApplication.old" (the OS allows this, fortunately). Your app then saves the downloaded byte[] array in the same folder as the original EXE file, and with the same original name (e.g. "MyApplication.exe"). You then display a message to the user (e.g. "new version detected, please restart") and close.
When the user restarts the app, it will be the new version they're starting. The new version deletes the old file ("MyApplication.old") and the update is complete.
Having an application update itself without requiring the user to restart is a huge pain in the butt (you have to kick off a separate process to do the updating, which means a separate updater application that cannot itself be auto-updated) and I've never been able to make it work 100% reliably. I've never had a customer complain about the required restart.
I asked this same question a while back:
How to Auto-Update Windows Mobile application
Basically you need two applications.
App1: Launches the actual application, but also checks for a CAB file (installer). If the cab file is there, it executes the CAB file.
App2: Actual application. It will call a web service, passing a version number to the service and retrieve a URL back if a new version exists (). Once downloaded, you can optionally install the cab file and shut down.
One potiencial issue: if you have files that one install puts on the file system, but can't overwrite (database file, log, etc), you will need two separate installs.
To install a cab: look up wceload.exe http://msdn.microsoft.com/en-us/library/bb158700.aspx
private static bool LaunchInstaller(string cabFile)
{
// Info on WceLoad.exe
//http://msdn.microsoft.com/en-us/library/bb158700.aspx
const string installerExe = "\\windows\\wceload.exe";
const string processOptions = "";
try
{
ProcessStartInfo processInfo = new ProcessStartInfo();
processInfo.FileName = installerExe;
processInfo.Arguments = processOptions + " \"" + cabFile + "\"";
var process = Process.Start(processInfo);
if (process != null)
{
process.WaitForExit();
}
return InstallationSuccessCheck(cabFile);
}
catch (Exception e)
{
MessageBox.Show("Sorry, for some reason this installation failed.\n" + e.Message);
Console.WriteLine(e);
throw;
}
}
private static bool InstallationSuccessCheck(string cabFile)
{
if (File.Exists(cabFile))
{
MessageBox.Show("Something in the install went wrong. Please contact support.");
return false;
}
return true;
}
To get the version number: Assembly.GetExecutingAssembly().GetName().Version.ToString()
To download a cab:
public void DownloadUpdatedVersion(string updateUrl)
{
var request = WebRequest.Create(updateUrl);
request.Credentials = CredentialCache.DefaultCredentials;
var response = request.GetResponse();
try
{
var dataStream = response.GetResponseStream();
string fileName = GetFileName();
var fileStream = new FileStream(fileName, FileMode.CreateNew);
ReadWriteStream(dataStream, fileStream);
}
finally
{
response.Close();
}
}
What exactly do you mean by "self-updating"? If you're referring to configuration or data, then webservices should work great. If you're talking about automatically downloading and installing a new version of itself, that's a different story.
Found this downloadable sample from Microsoft- looks like it should help.
If you want to use a third-party component, have a look at AppToDate developed by the guys at MoDaCo.