I'm new to WebRequest authentication and have been researching how to authenticate w/ a couple websites to pull some excel data from them. Couple things I'm confused about is
a.) how to properly read a log from Fiddler (using this to pick up get/post data from the website authentication)
b.) how do use the data from Fiddler to program the VB.NET WebRequest properly.
I've been able to authenticate w/ websites that use simple authentication HTTPS, but any site that does any redirects/REST/cookie auth I'm lost...
Let me know if I can provide anymore detail.
Dim req As Net.HttpWebRequest = Net.WebRequest.Create(Url)
If Not Login = Nothing AndAlso Not Password = Nothing Then
Dim myCache As New System.Net.CredentialCache()
myCache.Add(New Uri(Url), "Basic", New System.Net.NetworkCredential(Login, Password))
req.Credentials = myCache
End If
Dim sr As New StreamReader(req.GetResponse().GetResponseStream())
Dim ss as string = sr.ReadToEnd
'Save it as excel & close stream
sr.Close()
Related
I created a console program on visual basic asp.net that sends a generated email through a post API call. I was successful in sending an email to any recipient email using a godaddy domain through sendgrid. On this domain godaddy, I set up a proofpoint encryption attachment that should create encrypted emails through a trigger of putting "[encrypt]" in the subject line. However, it did not trigger because for the encryption to be automatically attached to the email it has to pass through the office 365 server. Since I am using sendgrid according to the workflow it doesn't go through outlook whatsoever:
https://sendgrid.com/docs/ui/sending-email/email-flow/
I have been doing research and I see send sendgrid supports end-to-end encryption with TLS. However, I cannot find anywhere how to establish this encryption with sendgrid and visual basic API calls. This program is for a medical diagnostic company, who will be sending patient information via email, to maintain HIPPA compliance the emails must be encrypted in some way. Here is the code:
Private Sub SendEmail(PARAMETERS_FOR_AUTOMATING_EMAIL)
Try
System.Net.ServicePointManager.SecurityProtocol = System.Net.SecurityProtocolType.Tls12
Dim uri As String = "https://api.sendgrid.com/v3/mail/send"
Dim Request As WebRequest = WebRequest.CreateHttp(uri)
Request.Method = "POST"
Request.PreAuthenticate = True
Request.Headers.Add("Authorization", "Bearer API_KEY")
Dim json_data As String = "{""personalizations"": [{""To"": [{""email"": ""TO_EMAIL""}]}],""from"": {""email"": ""FROM_EMAIL""},""subject"":""[Encrypt]This is a automated report"",""content"": [{""type"": ""text/html"",""value"": ""Hello!,<br>Please find attachment.""}], ""attachments"": [{""content"": ""File_translated_to_64_encoded"", ""type"": ""EXCEL_MIME"", ""filename"": ""FILE_NAME_FOR_ATTACHMENT""}]}}"
Request.ContentType = "application/json"
Dim json_bytes() As Byte = Encoding.UTF8.GetBytes(json_data)
Request.ContentLength = json_bytes.Length
Using requeststream = Request.GetRequestStream
requeststream.Write(json_bytes, 0, json_bytes.Length)
End Using
Dim responsecontent As String = Nothing
Using Response = DirectCast(Request.GetResponse, HttpWebResponse),
responseStream = Response.GetResponseStream()
Using reader = New StreamReader(responseStream)
responsecontent = reader.ReadToEnd()
End Using
End Using
Catch ex As Exception
ErrLog(ex, currentLocID)
End Try
End Sub
If you guys have any ideas to give me a solution please share! thank you in advance.
I'm trying to send a request to a website but it keeps telling me my cookies are not enabled. (But i did used a cookiecontainer in my code)
Here is the code :
Private Function does_it_asks_for_cookies() As Boolean
Dim url As String = "https://www.myhabit.com/?hash="
Dim req As Net.HttpWebRequest = Net.HttpWebRequest.Create(URL)
req.CookieContainer = New CookieContainer
req.ContentType = "application/x-www-form-urlencoded"
Dim res As Net.HttpWebResponse = req.GetResponse()
Dim strm As New IO.StreamReader(res.GetResponseStream())
Dim html As String = strm.ReadToEnd()
strm.Dispose()
res.Dispose()
If html.Contains("To continue shopping at Amazon.com, please enable cookies in") Then
Return True
Else
Return False
End If
End Function
Usage : MsgBox(does_it_asks_for_cookies)
While you have a CookieContainer the code in your example doesn't actually put anything in it. HttpWebRequest doesn't know anything about previous requests and certainly doesn't magically store the contents of any cookies in your container, you will have to do that yourself. Having a CookieContainer isn't enough. Most (especially e-commerce) websites are very specific in what they expect there to be present in the form of a cookie, often reporting that cookies aren't turned on when the cookie doesn't contain the proper values. You will have to inspect the web site in a browser to figure out what cookies are set and what values they contain. Google "inspect cookies" to learn how to view cookies in the browser of your choice.
I'm trying to upload a file from an FTP site to Basecamp using the Basecamp API. I'm using a simple console application. Here's my code:
Try
Dim accountID As String = ConfigurationManager.AppSettings("BaseCampID")
Dim projectID As Integer = 9999999
Dim folderName As String = "XXXXX/XXXXX"
Dim fileName As String = "XXX.zip"
'The URL to access the attachment method of the API
Dim apiURL = String.Format("https://basecamp.com/{0}/api/v1/projects/{1}/attachments.json", accountID, projectID)
'Get the file from the FTP server as a byte array
Dim fileBytes As Byte() = GetFileBytes(String.Format("{0}\\{1}", folderName, fileName))
'Initialize the WebClient object
Dim client As New WebClient()
client.Headers.Add("Content-Type", "application/zip")
'Need to provide a user-agent with a URL or email address
client.Headers.Add("User-Agent", "Basecamp Upload (email#email.com)")
'Keep the connection alive so it doesn't close
client.Headers.Add("Keep-Alive", "true")
'Provide the Basecamp credentials
client.Credentials = New NetworkCredential("username", "password")
'Upload the file as a byte array to the API, and get the response
Dim responseStr As Byte() = client.UploadData(apiURL, "POST", fileBytes)
'Convert the JSON response to a BaseCampAttachment object
Dim attachment As BaseCampAttachment
attachment = JSonHelper.FromJSon(Of BaseCampAttachment)(Encoding.Default.GetString(responseStr))
Catch ex As Exception
Console.WriteLine(ex.Message)
Finally
Console.ReadLine()
End Try
But whenever it calls client.UploadData, I get the error message "The underlying connection was closed: The connection was closed unexpectedly." I ran into this issue earlier and thought I solved it by adding the "Keep-Alive" header, but it's not working anymore. The API works if I upload a local file with client.UploadFile, but I'd like to just upload the file from they byte array from the FTP rather than downloading the file locally then uploading it to Basecamp.
Any thoughts would be greatly appreciated. Thanks!
I never figured out what was wrong with the WebClient call, but I ended up using a Basecamp API wrapper from https://basecampwrapper.codeplex.com. That wrapper uses HTTPRequest and HTTPResponse instead of WebClient.UploadData. It's also much easier to just use that wrapper than to try writing my own code from scratch.
I am making a Windows Phone app and I am trying to get JSON data from a URL. The user needs to be logged into the website (which hosts the JSON data) in order to get JSON data and I cannot use the Web Browser control to display the data and then extract the string since the browser doesn't recognize it (for some weird reason) and asks to search for an app on Store which can handle that JSON file type. (If I open the URL in desktop browser on my Windows PC, I can see the raw JSON data). I can't use normal HTTPWebRequest or WebClient as to get JSON data the login cookies needs to be set (the JSON data is user-specific) and I can't extract the cookies from Web browser control and use it with WebClient or HTTPWebRequest. So the best thing I can do is use a special internal instance of IWebRequestCreate that is used internally by the WebBrowser. By opening background HTTP requests with that class, the cookies get automatically set as if they were created/sent by the WebBrowser control. But my code is not returning the JSON data, I get blank response, as in the string resp is empty.
Below is the code:
Dim browser = New WebBrowser()
Dim brwhttp = GetType(WebRequestCreator).GetProperty("BrowserHttp")
Dim requestFactory = TryCast(brwhttp.GetValue(Browser, Nothing), IWebRequestCreate)
Dim uri = New Uri("http://api.quora.com/api/logged_in_user?fields=inbox,notifs,following,followers")
Dim req = requestFactory.Create(uri)
req.Method = "GET"
req.BeginGetResponse(New AsyncCallback(AddressOf request_Callback), req)
Private Sub request_Callback(asyncResult As IAsyncResult)
Dim webRequest As HttpWebRequest = DirectCast(asyncResult.AsyncState, HttpWebRequest)
Dim webResponse As HttpWebResponse = DirectCast(webRequest.EndGetResponse(asyncResult), HttpWebResponse)
Dim tempStream As New MemoryStream()
webResponse.GetResponseStream().CopyTo(tempStream)
Dim sr As New StreamReader(tempStream)
Dim resp As String = sr.ReadToEnd
End Sub
What's wrong?
I found that CopyTo can leave the Stream's pointer at the end of the buffer, you probably need to reset tempStream's pointer to the beginning before attempting to read it with the StreamReader, here's the code...
webResponse.GetResponseStream().CopyTo(tempStream);
tempStream.Seek(0, SeekOrigin.Begin);
Dim sr As New StreamReader(tempStream);
I've hit a wall on POST'ing data to a webform :(
Below is the code I've adapted from many places (here most recently: http://p2p.wrox.com/asp-net-1-0-1-1-professional/34517-webrequest-webresponse-form-submit-problem-help.html )
Dim url As String = "https://student.ashford.edu/student/"
Dim data As String = "username=myusername&password=mypassword"
Dim buffer As Byte() = Encoding.UTF8.GetBytes(data)
Dim result As String = ""
Dim req As HttpWebRequest = DirectCast(WebRequest.Create(url), HttpWebRequest)
req.Method = "POST"
req.ContentType = "application/x-www-form-urlencoded"
req.ContentLength = buffer.Length
req.CookieContainer = New CookieContainer()
' enable cookies
Dim reqst As Stream = req.GetRequestStream()
' add form data to request stream
reqst.Write(buffer, 0, buffer.Length)
reqst.Flush()
reqst.Close()
Console.WriteLine(vbLf & "Posting data to " & url)
Dim res As HttpWebResponse = DirectCast(req.GetResponse(), HttpWebResponse)
' send request,get response
Console.WriteLine(vbLf & "Response stream is: " & vbLf)
Dim resst As Stream = res.GetResponseStream()
' display HTTP response
Dim sr2 As New StreamReader(resst)
result = sr2.ReadToEnd()
Using writer As System.IO.StreamWriter = New StreamWriter("C:\Temp\checkcheck.html")
writer.Write(result)
End Using
When I check the checkcheck.html file, it doesn't show the page I normally see when I've successfully logged into my university, but instead shows the login page again prompting for a user name and password. Since it's prompting for a user/pass in the output response, does that mean nothing is posted? Or is the login button not being hit?
As far as the field names in the source, the name of the username field is "username", and the name of the password field is "password". From what I can tell, the name of the login button itself is "submit". Below is the source of the page, am I missing something?
view-source:https://student.ashford.edu/student/
https://student.ashford.edu/student/
What I want to have happen is that in my VB.NET backend, I click a button and my username and password are automatically entered and I am automagically logged in. Once the login is done, I want to retrieve the HTML of the logged in page so I can parse it for items relating to my studies. This may be unrelated, but do I need cookies to keep my application "logged in" if I navigate to other pages?
EDIT:
I tried plugging different URL's, to no avail. I have also tried adding submit=login and login=submit (alternately of course) to the POST data. I'm still getting the original login screen HTML from the result variable at the end, instead of my university homepage that I'm expecting after I normally login by hand as a human would. Maybe there is a follow up step that I am missing?
The actual form action is at https://student.ashford.edu/login/commands/login.php. You might plug that into url instead. Also, try including the submit button - submit=login - in the values.
I found out what I needed to do. First I have to make sure that I have a cookie container. Next, I used Fiddler to find out EXACTLY what is being sent back and forth between my browser and where I want to login. I post all the login info, and I get the response back. This seems to fill up my cookie container variable with the required authentication cookies? Next I make another POST request to the page I want to go to, and voila - it works.