I'm using Onedrive's "upload item from URL" functionality.
https://dev.onedrive.com/items/upload_url.htm
Much of the time it works okay -- over the past years I've uploaded several thousand files this way. But over the past few days I've been trying to batch-upload about 4000 items and this "upload from URL" hasn't worked right...
What I see on about 9/10 of my uploads is it accepts the upload-from-URL request, and gives me back a "progress URL" to monitor, but that progress URL stays consistently at "status=notStarted" and "percentageComplete=0%" until I give up after five minutes.
Has anyone used the upload-from-URL functionality? Any experience with it being flakey?
I don't think it's down to rate-limiting. That's because I stopped my code for the day, restarted it the next day, and got the same behavior.
I wonder if it has something to do with this particular target filename to upload to being wrong? maybe OneDrive has got in a weird state where it doesn't feel able to upload to this particular filename via sideload?
(I guess my next attempt will be to do a regular upload and see what kind of error message that gives...)
' Initiate a sideload from the blobURL into OneDrive
Dim sideReqUrl = $"https://api.onedrive.com/v1.0/drive/root:{escapedFolder}:/children?access_token={accessToken}"
Dim sideReqContent = JsonConvert.SerializeObject(New JObject From {{"#content.sourceUrl", blobUrl}, {"#name.conflictBehavior", "fail"}, {"name", unescapedName}, {"file", New JObject}})
Dim sideReq As New HttpRequestMessage(HttpMethod.Post, sideReqUrl)
sideReq.Headers.Add("Prefer", "respond-async")
sideReq.Content = New StringContent(sideReqContent, Text.Encoding.UTF8, "application/json")
Dim sideResp = Await client.SendAsync(sideReq, cancel)
If sideResp.StatusCode <> Net.HttpStatusCode.Accepted Then
Console.WriteLine($"sideload rejected - {sideResp}")
Throw New Net.Http.HttpRequestException("sideload rejected")
End If
Dim progUrl = sideResp.Headers.Location ' for monitoring progress
Console.WriteLine($"sideload started - {progUrl}")
' Poll the progress of that sideload until it's complete
Do
Dim progResp = Await client.GetAsync(progUrl, cancel)
If progResp.StatusCode <> Net.HttpStatusCode.Accepted Then progResp.EnsureSuccessStatusCode()
Dim progRespText = Await progResp.Content.ReadAsStringAsync()
Dim progRespJson = JObject.Parse(progRespText)
If progResp.StatusCode = Net.HttpStatusCode.Accepted Then
Console.WriteLine($"sideload ping - {CStr(progRespJson("status"))} - {CStr(progRespJson("percentageComplete"))}%")
Continue Do
End If
Console.WriteLine("SIDELOAD COMPLETE!")
Return
Loop
Related
first time post here cause for the first time I couldn't find an answer on Stack.
The program I'm fiddling with goes to an API website - gets the passport key - and uses that to load the info in XML and then parses it. The code is very long so I'm only posting the bit that results in the error.
myURL = "https://api.smartbidnet.com/project?PassportKey=" & PassportKey & "&ResultType=xml" 'This is the API URL where we use our passport key to get our data
IE.Navigate myURL 'Go to Webpage
Dim XDoc As Object
Set XDoc = CreateObject("MSXML2.DOMDocument")
XDoc.async = False
XDoc.validateOnParse = False
XDoc.Load (ThisWorkbook.Path & myURL) 'Here we will load the XML info from the web page
Set lists = XDoc.DocumentElement 'lists will be used to access XML nodes
Set Projects = lists.SelectNodes("//root/projects")
Now heres the weird part:
If I open excel by selecting "Blank Workbook" the code works just fine from the editor.
If I save the blank workbook and then try to run it or If I simply right click the desktop and open a blank sheet rather than workbook and try to run it. the last line results in
Runtime 91 error
Any help is of course greatly appreciated.
The bit of code before the above is simply navigating to the site to grab the passport key for the API and build the myURL variable.
Quick question about downloading Blobs from azure storage. I'm a little lost on how this works exactly, here's my code:
Try
Dim accountname As String = "macroqc"
Dim accountkey As String = My.Settings.Storagekey1
Dim creds As StorageCredentials = New StorageCredentials(accountname, accountkey)
Dim account As CloudStorageAccount = New CloudStorageAccount(creds, useHttps:=True)
Dim client = account.CreateCloudBlobClient()
Dim container As CloudBlobContainer = client.GetContainerReference("smallequipment1certifications")
Dim blob As CloudBlockBlob = container.GetBlockBlobReference(filenamelbl.Text)
Using MemoryStream = New MemoryStream()
blob.DownloadToStream(MemoryStream)
My.Settings.downloadsource = System.Text.Encoding.UTF8.GetString(MemoryStream.ToArray())
equipmentpdf.src = filenamelbl.Text
End Using
Catch ex As Exception
MessageBox.Show("Sorry an error has occured while opening your file: " & Environment.NewLine & ex.ToString, "Download Error", MessageBoxButtons.OK, MessageBoxIcon.Error)
End Try
So I am getting an error when I run this which looks like:
So when it uploads the file to Azure storage it saves blob.Uri.AbsoluteUri & blob.Uri.AbsolutePath to the database. Am I going down the right path using this method to download the file? The Blobs are private do I need an SAS to download? I am confused and am having a really hard time finding useful documenation on this! Can someone help point me in the right direction please!!
Thanks everyone!
The reason you're getting this error is because GetBlockBlobReference expects the name of the blob (and not the full blob URL) and it creates a URL based on that. Since you're passing the complete URL, it still creates a URL but includes the blob URL. If you put a breakpoint and check the blob object's URL property, you will see something like:
https://accountname.blob.core.windows.net/containername/https://accountname.blob.core.windows.net/containername/filename
Since the blob by the name https://accountname.blob.core.windows.net/containername/filename does not exist in the container, you're getting a 404 (Not Found) error. Please provide just the file name in GetBlockBlobReference and things should work just fine.
I'd like to know the last-modifed date of a remote file (defined via url).
And only download it, if it's newer than my locally stored one.
I managed to do that for local files, but can't find a solution to do that for remote files (without downloading them)
working:
Dim infoReader As System.IO.FileInfo = My.Computer.FileSystem.GetFileInfo("C:/test.txt")
MsgBox("File was last modified on " & infoReader.LastWriteTime)
not working:
Dim infoReader As System.IO.FileInfo = My.Computer.FileSystem.GetFileInfo("http://google.com/robots.txt")
MsgBox("File was last modified on " & infoReader.LastWriteTime)
I'd love to have a solution which will only have to download the headers of a file
You can use the System.Net.Http.HttpClient class to fetch the last modified date from the server. Because it's sending a HEAD request, it will not fetch the file contents:
Dim client = New HttpClient()
Dim msg = New HttpRequestMessage(HttpMethod.Head, "http://google.com/robots.txt")
Dim resp = client.SendAsync(msg).Result
Dim lastMod = resp.Content.Headers.LastModified
You could also use the If-Modified-Since request header with a GET request. This way the response should be 304 - Not Modified if the file has not been changed (no file content sent), or 200 - OK if the file has been changed (and the contents of the file will be sent in the response), although the server is not required to honor this header.
Dim client = New HttpClient()
Dim msg = New HttpRequestMessage(HttpMethod.Get, "http://google.com/robots.txt")
msg.Headers.IfModifiedSince = DateTimeOffset.UtcNow.AddDays(-1) ' use the date of your copy of the file
Dim resp = client.SendAsync(msg).Result
Select Case resp.StatusCode
Case HttpStatusCode.NotModified
' Your copy is up-to-date
Case HttpStatusCode.OK
' Your copy is out of date, so save it
File.WriteAllBytes("C:\robots.txt", resp.Content.ReadAsByteArrayAsync.Result)
End Select
Note the use of .Result, since I was testing in a console application - you should probably await instead.
If the server offers it, you can get it through the HTTP header Last-Modified property. But your still stuck at downloading the full file.
You could get it through FTP.
See if the server allows you to see the list of files in a folder.
If the website offer the date somewhere that you could pull through screen scrapping.
I know this is a little bit old question but, there's still a better answer.
Dim req As WebRequest = HttpWebRequest.Create("someurl")
req.Method = "HEAD"
Dim resp As WebResponse = req.GetResponse()
Dim remoteFileLastModified As String = resp.Headers.Get("Last-Modified")
Dim remoteFileLastModifiedDateTime As DateTime
If DateTime.TryParse(remoteFileLastModified, remoteFileLastModifiedDateTime) Then
MsgBox("Date Last Modified:" + remoteFileLastModifiedDateTime.ToString("d MMMM yyyy dddd HH:mm:ss"))
Else
MsgBox("could not determine")
End If
I have wrote a web service, in a nutshell it uses openpop to get email messages does stuff with the content to insert into databases and saves attachments which are images. That works fine when i save images locally, it does exactley what it is suppose to. Now an added requirment was to save images to an FTP directory, so i can create my folders dynamically (they are created based upon timestamp) and that works well. My problem comes from when i try to save them to the ftp. Yes my user name and password are correct, otherwise i wouldn't be creating the directory.
Private Sub UploadFile(ByVal fileToSave As FileInfo, ByVal path As String)
Dim UploadRequest As FtpWebRequest = DirectCast(WebRequest.Create("ftp://UserName:Passowrd#999.99.999.9" & path), FtpWebRequest)
UploadRequest.Credentials = New NetworkCredential("PicService", "grean.matching18")
UploadRequest.Method = System.Net.WebRequestMethods.Ftp.UploadFile
UploadRequest.UseBinary = True
UploadRequest.UsePassive = True
' Const BufferSize As Integer = 2048
' Dim content(BufferSize - 1) As Byte, dataRead As Integer
Dim bFile() As Byte = System.IO.File.ReadAllBytes(fileToSave.ToString)
'UploadRequest.ContentLength = content.Length
Using FileStream1 As FileStream = fileToSave.OpenRead()
Try
'open request to send
Using RequestStream As Stream = UploadRequest.GetRequestStream
End Using
Catch ex As Exception
Finally
'ensure file closed
FileStream1.Close()
End Try
End Using
End Sub
I have tried using Passive False and Binary False as well, i did more research on my stack trace.
And found this article but no solution as of yet. Any input would be appreciated, i am also posting another question on windows services for different issue. If you would like to take a shot at it, the other question isnt about ftp but permissions for a service on windows server 2003
This may not be the solution but I've found that the URI string has to be 'just right' and that what is 'just right' varies by the ftp server.
So ftp://server/directory/file works on some servers but needs to be ftp://server//directory/file to work on others (note the double slash after the server name)
Aso, your URI has 'password' spelled incorrectly: ftp://UserName:Passowrd#999.99.999.9 and you are supplying the credentials in a separate code line as well.
I've hit a wall on POST'ing data to a webform :(
Below is the code I've adapted from many places (here most recently: http://p2p.wrox.com/asp-net-1-0-1-1-professional/34517-webrequest-webresponse-form-submit-problem-help.html )
Dim url As String = "https://student.ashford.edu/student/"
Dim data As String = "username=myusername&password=mypassword"
Dim buffer As Byte() = Encoding.UTF8.GetBytes(data)
Dim result As String = ""
Dim req As HttpWebRequest = DirectCast(WebRequest.Create(url), HttpWebRequest)
req.Method = "POST"
req.ContentType = "application/x-www-form-urlencoded"
req.ContentLength = buffer.Length
req.CookieContainer = New CookieContainer()
' enable cookies
Dim reqst As Stream = req.GetRequestStream()
' add form data to request stream
reqst.Write(buffer, 0, buffer.Length)
reqst.Flush()
reqst.Close()
Console.WriteLine(vbLf & "Posting data to " & url)
Dim res As HttpWebResponse = DirectCast(req.GetResponse(), HttpWebResponse)
' send request,get response
Console.WriteLine(vbLf & "Response stream is: " & vbLf)
Dim resst As Stream = res.GetResponseStream()
' display HTTP response
Dim sr2 As New StreamReader(resst)
result = sr2.ReadToEnd()
Using writer As System.IO.StreamWriter = New StreamWriter("C:\Temp\checkcheck.html")
writer.Write(result)
End Using
When I check the checkcheck.html file, it doesn't show the page I normally see when I've successfully logged into my university, but instead shows the login page again prompting for a user name and password. Since it's prompting for a user/pass in the output response, does that mean nothing is posted? Or is the login button not being hit?
As far as the field names in the source, the name of the username field is "username", and the name of the password field is "password". From what I can tell, the name of the login button itself is "submit". Below is the source of the page, am I missing something?
view-source:https://student.ashford.edu/student/
https://student.ashford.edu/student/
What I want to have happen is that in my VB.NET backend, I click a button and my username and password are automatically entered and I am automagically logged in. Once the login is done, I want to retrieve the HTML of the logged in page so I can parse it for items relating to my studies. This may be unrelated, but do I need cookies to keep my application "logged in" if I navigate to other pages?
EDIT:
I tried plugging different URL's, to no avail. I have also tried adding submit=login and login=submit (alternately of course) to the POST data. I'm still getting the original login screen HTML from the result variable at the end, instead of my university homepage that I'm expecting after I normally login by hand as a human would. Maybe there is a follow up step that I am missing?
The actual form action is at https://student.ashford.edu/login/commands/login.php. You might plug that into url instead. Also, try including the submit button - submit=login - in the values.
I found out what I needed to do. First I have to make sure that I have a cookie container. Next, I used Fiddler to find out EXACTLY what is being sent back and forth between my browser and where I want to login. I post all the login info, and I get the response back. This seems to fill up my cookie container variable with the required authentication cookies? Next I make another POST request to the page I want to go to, and voila - it works.