We have been running a service for over a year now that uploads videos to YouTube using the API v3 Upload (using the .NET library Google.Apis.YouTube.v3).
In the last two days we have suddenly started having all uploads fail part way through upload, with the error being returned from YouTube "A task was cancelled". The videos are partially uploaded on YouTube, they are getting through varied progress (some only a couple of MB, some as much as 17-20MB, though it does appear that the larger progress ones are in the earlier list of failures).
This happened first on Friday night and I was able to push all the failed videos on Saturday morning without incident, but then later on Saturday morning it started again, and since hen all videos on Saturday afternoon and evening failed. We're talking about a total of 25 or so videos, so not hundreds. The videos are all around 40MB in size.
Only 2-3 videos would usually be uploading at once. While I was successfully pushing them all up on Saturday morning I was actually pushing up about 4-5 at a time.
Can anyone else suggest a possible cause, or any troubleshooting tips? We haven't seen any server connectivity issues otherwise, and the website that runs on this server hasn't had any apparent issues. I can't see any other error coming back other than this "A task was cancelled" error from YouTube, and up until that error the progress from the uploader is success.
Edit: I've also been able to reproduce this issue running one of our uploads from my dev environment. I've just tried getting a whole new set of API credentials (which are actually for a different YouTube account), refreshed the oAuth token and still having the same issue.
You need to set the timeout before you use the service and, if the user changes it, you need to pause all uploads, null the service and initialise it again.
I'm pretty sure I can confirm setting the YouTubeService.HttpClient.Timeout does solve the question.
Apologies for my delay to respond.
internal YouTubeService YouTubeService { get; set; }
internal async Task InitialiseYouTubeServiceAsync(int timeoutSeconds)
{
var scope = new List<string> { YouTubeService.Scope.YoutubeUpload, YouTubeService.Scope.Youtube, };
var clientSecrets = new ClientSecrets()
{
ClientId = Constants.ClientId,
ClientSecret = Constants.ClientSecret
};
var fileDataStore = new FileDataStore(Constants.FileDataStore);
try
{
var credentialTask = GoogleWebAuthorizationBroker.AuthorizeAsync(
clientSecrets, scope, Environment.UserName, CancellationToken.None, fileDataStore);
//This is a hack to allow a progress bar timeout when the user closes the
//browser without authorising. It's not associated with the question at hand.
await Task.WhenAny(credentialTask, Task.Delay(TimeSpan.FromSeconds(timeoutSeconds)));
if (credentialTask.Status == TaskStatus.WaitingForActivation) return;
var initializer = new BaseClientService.Initializer()
{
HttpClientInitializer = credentialTask.Result,
ApplicationName = Constants.ApplicationName,
};
YouTubeService = new YouTubeService(initializer);
//This is the answer to the question at hand.
YouTubeService.HttpClient.Timeout = TimeSpan.FromMinutes(Settings.Default.TimeoutMinutes);
}
catch (AggregateException)
{
Exceptions.LogSilent(new Exception(ResourceExceptions.AuthorisationDenied), MethodBase.GetCurrentMethod(), EventLogEntryType.Information);
}
}
Related
I am looking to open a task module (Pop up - iframe with audio/video) in my bot that is connected to Teams channel. I am having issues following the sample code provided on the GitHub page.
I have tried to follow the sample and incorporate to my code by did not succeed.
In my bot.cs file I am creating card action of invoke type:
card.Buttons.Add(new CardAction("invoke", TaskModuleUIConstants.YouTube.ButtonTitle, null,null,null,
new Teams.Samples.TaskModule.Web.Models.BotFrameworkCardValue<string>()
{
Data = TaskModuleUIConstants.YouTube.Id
}));
In my BotController.cs that inherits from Controller
[HttpPost]
public async Task PostAsync()
{
// Delegate the processing of the HTTP POST to the adapter.
// The adapter will invoke the bot.
await _adapter.ProcessAsync(Request, Response, _bot);
}
public async Task<HttpResponseMessage> Post([FromBody] Activity activity)
{
if (activity.Type == ActivityTypes.Invoke)
{
return HandleInvokeMessages(activity);
}
return new HttpResponseMessage(HttpStatusCode.Accepted);
}
private HttpResponseMessage HandleInvokeMessages (Activity activity)
{
var activityValue = activity.Value.ToString();
if (activity.Name == "task/fetch")
{
var action = Newtonsoft.Json.JsonConvert.DeserializeObject<Teams.Samples.TaskModule.Web.Models.BotFrameworkCardValue<string>>(activityValue);
Teams.Samples.TaskModule.Web.Models.TaskInfo taskInfo = GetTaskInfo(action.Data);
Teams.Samples.TaskModule.Web.Models.TaskEnvelope taskEnvelope = new Teams.Samples.TaskModule.Web.Models.TaskEnvelope
{
Task = new Teams.Samples.TaskModule.Web.Models.Task()
{
Type = Teams.Samples.TaskModule.Web.Models.TaskType.Continue,
TaskInfo = taskInfo
}
};
return msg;
}
return new HttpResponseMessage(HttpStatusCode.Accepted);
}
There is more code as per the GitHub sample but I won't paste it here. Can someone point me into the correct direction ?
I have got to the stage that it is displaying a pop up window but the content and title comes from manifest file instead of creating actual iframe also no video is rendering. My goal is to render video within my teams using iframe container.
The important part from the sample:
This sample is deployed on Microsoft Azure and you can try it yourself by uploading Task Module CSharp.zip to one of your teams and/or as a personal app. (Sideloading must be enabled for your tenant; see step 6 here.) The app is running on the free Azure tier, so it may take a while to load if you haven't used it recently and it goes back to sleep quickly if it's not being used, but once it's loaded it's pretty snappy.
So,
Your Teams Admin MUST enable sideloading
Your bot MUST be sideloaded into Teams
The easiest way to do this would be download the sample manifest, open it in App Studio, then edit your bot information in. You then need to make sure Domains and permissions > Valid Domains are set for your bot. Also ensure you change the Tabs URLs to your own.
You also need to make sure that in your Tasks, the URLs they call ALL use https and not http. If anywhere in the chain is using http (like if you're using ngrok and http://localhost), it won't work.
I have a .NET Core 2.2 app which has a controller acting as a proxy to my APIs.
JS makes a fetch to the proxy, proxy forwards call onto API's and returns response.
I am experiencing intermittent lock ups on the proxy app when its awaiting the response from the HttpClient. When this happens it locks up the entire server. No more requests will be processed.
According to the logs of the API that is being proxied to it is returning fine.
To reproduce this is i have to make 100+ requests in a loop on the client through the proxy. Then i have to reload the page multiple times, reloading it whilst the 100 requests are in flight. It usually takes around 5 hits before things start slowing down.
The proxy will lock up waiting for an awaited request to resolve. Sometimes it comes back after a 4 - 5 second delay, other times after a minuet. Most of the time i haven't waited longer then 10 min before giving up and killing the proxy.
I've distilled the code down to the following block that will reproduce the issue.
I believe im following best practices, its async all the way down, im using IHttpClientFactory to enable sharing of HttpClient instances, im implementing using where i believe it is required.
The implementation was based on this: https://github.com/aspnet/AspLabs/tree/master/src/Proxy
I'm hoping im making a rather obvious mistake that others with more experience can pin point!
Any help would be greatly appreciated.
namespace Controllers
{
[Route("/proxy")]
public class ProxyController : Controller
{
private readonly IHttpClientFactory _factory;
public ProxyController(IHttpClientFactory factory)
{
_factory = factory ?? throw new ArgumentNullException(nameof(factory));
}
[HttpGet]
[Route("api")]
async public Task ProxyApi(CancellationToken requestAborted)
{
// Build API specific URI
var uri = new Uri("");
// Get headers frpm request
var headers = Request.Headers.ToDictionary(x => x.Key, y => y.Value);
headers.Add(HeaderNames.Authorization, $"Bearer {await HttpContext.GetTokenAsync("access_token")}");
// Build proxy request method. This is within a service
var message = new HttpRequestMessage();
foreach(var header in headers) {
message.Headers.Add(header.Key, header.Value.ToArray());
}
message.RequestUri = uri;
message.Headers.Host = uri.Authority;
message.Method = new HttpMethod(Request.Method);
requestAborted.ThrowIfCancellationRequested();
// Generate client and issue request
using(message)
using(var client = _factory.CreateClient())
// **Always hangs here when it does hang**
using(var result = await client.SendAsync(message, requestAborted).ConfigureAwait(false))
{
// Appy data from request onto response - Again this is within a service
Response.StatusCode = (int)result.StatusCode;
foreach (var header in result.Headers)
{
Response.Headers[header.Key] = header.Value.ToArray();
}
// SendAsync removes chunking from the response. This removes the header so it doesn't expect a chunked response.
Response.Headers.Remove("transfer-encoding");
requestAborted.ThrowIfCancellationRequested();
using (var responseStream = await result.Content.ReadAsStreamAsync())
{
await responseStream.CopyToAsync(responseStream, 81920);
}
}
}
}
}
EDIT
So modified the code to remove the usings and return the proxied response directly as a string instead of streaming and still getting the same issues.
When running netstat i do see a lot of logs for the url of the proxied API.
4 rows mention the IP of the API being proxied to, probably about another 20 rows mentions the IP of the proxy site. Those numbers dont seem odd to me but i don't have much experience using netstat (first time ive ever fired it up).
Also i have left the proxy running for about 20 min. Its it technically still alive. Responses are coming back. Just taking a very long time between the API being proxied to returning data and the HttpClient resolving. However it wont service any new requests, they just sit there hanging.
We switched to WebJobs with our background tasks that are starting to work when a new item lands on an Azure Queue. Now we have some weird issues that he seems to have problems accessing Redis RedLock and Storage that I can't explain.
Now the biggest issue we have is RedLock. We are using RedLock.Net for distributed locking. Now this works all fine in our production web application and it also worked on the background workers we had but as soon as we switched to WebJobs he basically failed to aquire the lock. To back this up with some code...we are locking like this:
using (var redisLock = await _redLockConnection.RedisLockFactory.CreateAsync(resource, UserLockExpiryTime, UserLockWaitTime, UserLockRetryTime))
{
// make sure we got the lock
if (redisLock.IsAcquired)
{
// execute code...
}
else
{
throw new CouldNotAcquireRedLockException();
}
}
The problem here is, IsAcquired is always false within a Webjob and I have no clue why!?
The second thing that maybe relates to this problem is deleting a blob file in azure storage that fails with a 404 only within a WebJob.
var file = _blobContainer.GetBlockBlobReference("file.txt");
file?.Delete();
This will fail with a 404 Not found exception within the WebJob.
Is there anything I missed setting up the webjob? Could it be an access problem for write operations? Would be glad for any help!
IsAcquired is always false within a Webjob
I do a test with the following code using RedLock.net in an Azure WebJob, I can acquire lock on resource if the lock is available.
public static void ProcessQueueMessage([QueueTrigger("mymes")] string message, TextWriter log)
{
var azureEndPoint = new RedisLockEndPoint
{
EndPoint = new DnsEndPoint("{YOUR_CACHE}.redis.cache.windows.net", 6380),
Password = "YOUR_ACCESS_KEY",
Ssl = true
};
var eps = new[] { azureEndPoint };
var rlf = new RedisLockFactory(eps);
var resource = "https://{storageaccount}.blob.core.windows.net/{containername}/test.txt";
var expiry = TimeSpan.FromSeconds(50);
var wait = TimeSpan.FromSeconds(10);
var retry = TimeSpan.FromSeconds(1);
using (var redisLock = rlf.Create(resource, expiry, wait, retry))
{
Console.WriteLine("Lock acquired: " + redisLock.IsAcquired);
}
log.WriteLine(message);
}
Result of test:
deleting a blob file in azure storage that fails with a 404
As I mentioned in comment, Please check if that Blob is existing via Azure portal or Azure storage explorer, or call Exists method to check existence of the blob before you delete it.
I'm building a W10 Universal app and I would like to know who is logged in to Windows so I can associate their data on my server with something that uniquely identifies the user w/o requiring a separate login.
OneDrive SDK is supposed to make this simple and easy.
So, I registered my app with OneDrive, used nuget to install the packages, downloaded the samples and wrote the following code.....
var scopes = new string[] { "wl.signin", "wl.offline_access", "onedrive.readonly" };
var client = OneDriveClientExtensions.GetUniversalClient(scopes);
try {
await client.AuthenticateAsync();
}
catch {
blahlblahblah;
}
This doesn't throw an exception, but, after AuthenticateAsync executes, the client's IsAuthenticated property is still false and the ServiceInfo's UserId is null.
So, I tried this next:
var client = OneDriveClient.GetMicrosoftAccountClient(
this.Resources["AppID"].ToString(),
this.Resources["ReturnUri"].ToString(),
scopes
);
where the AppID and ReturnUri match the Client ID and Redirect URL that are registered with the app.
This actually throws a OneDrive.Sdk.Error with a message of "Failed to retrieve a valid authentication token for the user."
So, I don't know what I'm doing wrong here. I'm at a total loss. I pulled up Fiddler to see what was being sent back & forth and nothing shows up. There's just not enough information for me to figure this out.
Anyone got any ideas?
So, ginach's workaround for the problem seems to be the solution until the bug is fixed. So, to sum it up....
Don't use the IsAuthenticated property of the UniversalClient. Instead, check the client's AuthenticationProvider's CurrentAccountSession to see if it has a value and an AccessToken.
var client = OneDriveClientExtensions.GetUniversalClient(scopes);
await client.AuthenticateAsync();
if (client.AuthenticationProvider.CurrentAccountSession != null && client.AuthenticationProvider.CurrentAccountSession.AccessToken != null) {
blahblahblahblahblah
}
This seems to do the trick.
I have a small console application doing some persistence with Raven which is working fine, but I just can't get the Raven Studio Web-App working.
I think I have read every article/blog post on the web which is around, but I haven't got it working.
The project is referencing the Raven.Client.Embedded, Raven.Client.Lightweight and Raven.Storage.Esent assemblies)
Here is the really simple code starting up my console app:
class Program
{
static void Main(string[] args)
{
EmbeddableDocumentStore store = new EmbeddableDocumentStore { DataDirectory = #"C:\temp\ravendata", UseEmbeddedHttpServer = true };
store.Initialize();
Console.WriteLine("Initialized");
while (true)
{
string line = Console.ReadLine();
if (line == "w")
{
Changeset cs = CreateChangeset();
using (var session = store.OpenSession())
{
session.Store(cs);
session.SaveChanges();
}
Console.WriteLine("Written.");
}
}
The question is: Where to put the Raven.Studio.xap in order to get it running in the browser (http://localhost:8080/Raven/studio.html)?
It's not working in the bin/debug output folder of my console app (which would be the most logical area where it should be), as well as it isn't if I put it in the root of my console application.
Sorry to ask this thing again, but it seems there is some point I am missing on this to get it up and running. ;)
Thanks for your help, R's, Rene
You are right, I've tried it using a new console application project and had the same issues, altough I copied the file Raven.Studio.xap into the \bin\debug AFTER I had seen the error message for the first time.
I found out, that the reason for this has to do with browser-caching. Even though the file would be available now, the embedded http-server returns 304 Not Modified, because it had sent the If-None-Match header into the request. Therefore, the cached "not-found" page in the browser cache will be used.
I fixed it and sent a patch to Ayende. However the solution now is:
1) make sure Raven.Studio.xap is under \bin\debug
2) clear the browsers cache