So, I'm trying to scrape the following public site using vba and the WinHttp library:
https://auctions.seecao.com/DAILY_AUCTION_LIST
Having examined the network traffic when the "Show Data" button is clicked, I came up with my code:
Sub test()
Dim border As String
Dim req As New WinHttpRequest
Dim url As String
Dim reqBodyObj As Object, respObj As Object, auction As Object
Dim reqBodyStr As String
Dim deliveryDay As Date
url = "https://auctions.seecao.com/api/DailyAuction/GetDailyAuctionList"
deliveryDay = Date
border = "ALME"
Set reqBodyObj = JsonConverter.ParseJson("{""parameters"":{""dayFrom"":""2021-04-01"",""dayTill"":""2021-04-01"",""auctionState"":[0,3,4,5,6,7,9]}}")
reqBodyObj("parameters")("dayFrom") = Format(deliveryDay + 1, "yyyy-mm-dd")
reqBodyObj("parameters")("dayTill") = Format(deliveryDay + 1, "yyyy-mm-dd")
reqBodyStr = JsonConverter.ConvertToJson(reqBodyObj)
With req
.Open "POST", url, False
.setRequestHeader "Content-Type", "application/json"
.Option(WinHttpRequestOption_SslErrorIgnoreFlags) = 256 '=0x0100 =ignore "Unknown certification authority (CA) or untrusted root" error refer to: https://learn.microsoft.com/en-us/windows/win32/winhttp/winhttprequestoption
.send reqBodyStr
Debug.Print .responseText
Set respObj = JsonConverter.ParseJson(.responseText)
End With
For Each auction In respObj("dailyAuctionListData")("rows")
If auction("columns")("auctionName") Like border & "*" Then
Debug.Print auction("columns")("id")
End If
Next auction
End Sub
It's probably worth noting that the first time you visit the site via a browser you'll get a warning saying that the server's certificate is not trusted and you'll have to add an exception to visit it:
To overcome this I used this .Option(WinHttpRequestOption_SslErrorIgnoreFlags) = 256 to ignore the errors.
Everything worked fine for about a month. Today however I started getting the following error:
Run-time error '-2147012711 (80072f99)': No credentials were available
in the client certificate.
So I suppose that the website started requesting a client certificate?
Requesting the data through a browser works normally without the need of any credentials and the POST request that is being sent under the hood seems to be the same as it used to be. Editing and resending the request via the browser's developer tools also works without any problems.
So my question is, what is going on here? Is the site asking for a client certificate?
And if so, which client certificate does Firefox use under the hood?
Does the browser use some kind of a default certificate for these cases?
Why am I asked for credentials when I run the code when clearly they are not needed when I visit the page?
Finally, when I try to specify a random certificate like so :
.SetClientCertificate "NameOfCertificate"
the request is being sent and I get the following response:
<html>
<head><title>400 The SSL certificate error</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<center>The SSL certificate error</center>
<hr><center>nginx/1.18.0</center>
</body>
</html>
Keep in mind that it's a public site accessible to everyone. No subscription needed.
EDIT
The same request works just fine with Postman after disabling SSL certificate verification. Is there a way to imitate this behavior in VBA?
Related
I'm trying to connect to server with certificate based Autherification. Certificate is installed on Win10.
My code is:
Dim request As Object
request = CreateObject("WinHttp.WinHttpRequest.5.1")
request.SetClientCertificate("Local Computer\Personal\Certificates\friently_cert_name")
Dim rest_request As String = "SOAP"
request.Open("GET", "<URL>")
request.SetRequestHeader("Accept", "*/*")
request.SetRequestHeader("Connection", "Keep-Alive")
request.Send(rest_request)
It lead to error: Value doesn't fall with expected range
After I tried to use:
request.SetClientCertificate("friently_cert_name")
It was led to issue: A certificate is required to complete client authentication
Could advise how to resolve an issue?
I am trying to connect to a REST endpoint via the GetHTTP Processor in NiFi 1.5.0.
The problem that I am faceing is, that the SSL certificate is issued to the domain but I only have direct access to the IP:Port address (company firewall).
With that I run into the problem that host name and certificate owners don't match up and the IP is not added as subject alternative name.
When I try to connect, I get this error message:
javax.net.ssl.SSLPeerUnverifiedException: Certificate for
<[IP-ADDRESS]> doesn't match any of the subject alternative names: []
Is there a way to bypass the host name verification?
I have found this NiFi Jira ticket but it doesn't seem to be addressed yet. Is there a workaround I could use?
You could try using InvokeHttp and use the "Trusted Hostname" property.
As the "Trusted Hostname" property is deprecated in recent versions of NiFi you can use the ExecuteScript processor with Ruby. The example is below. The body of the POST request must be in FlowFile contents. The body of the response will be in FlowFile contents after the processor.
require "uri"
require "net/http"
require "openssl"
java_import org.apache.commons.io.IOUtils
java_import java.nio.charset.StandardCharsets
java_import org.apache.nifi.processor.io.StreamCallback
# Define a subclass of StreamCallback for use in session.read()
class JRubyStreamCallback
include StreamCallback
def process(inputStream, outputStream)
text = IOUtils.toString(inputStream, 'utf-8')
url = URI("https://...")
https = Net::HTTP.new(url.host, url.port)
https.use_ssl = true
https.verify_mode = OpenSSL::SSL::VERIFY_NONE
request = Net::HTTP::Post.new(url)
request["Authorization"] = "Basic ..."
request["Content-Type"] = "application/json"
request.body = text
response = https.request(request)
outputStream.write((response.read_body).to_java.getBytes(StandardCharsets::UTF_8))
end
end
jrubyStreamCallback = JRubyStreamCallback.new
flowFile = session.get()
if flowFile != nil
flowFile = session.write(flowFile, jrubyStreamCallback)
session.transfer(flowFile, REL_SUCCESS)
end
I am connecting to a server via a proxy that requires a client certificate to authenticate. I can easily launch a browser from my script, utilizing an ssh tunnel already established in a browser. When I attempt to open a connection using Microsoft.XMLHTTP, I receive an error
Security certificate required to access this resource is invalid.
I am looking for one of two solutions:
utilize a session that is already (manually) established in a browser
send the client certificate (and PIN) via the script.
The logic that I'm currently using for this function is very simple:
Dim xHttp: Set xHttp = CreateObject("Microsoft.XMLHTTP")
Dim bStrm: Set bStrm = CreateObject("Adodb.Stream")
xHttp.Open "GET", "https://localhost:63619/ap_detail.xml?id=3502", False
xHttp.Send
With bStrm
.Type = 1 '//binary
.Open
.Write xHttp.responseBody
.SaveToFile "C:\xml\3502.xml", 2 '//overwrite
End With
I have the below code to get data from my https website having godaddy certificate.but i unable to get the result from the url.
port = CreateObject("roMessagePort")
req = createObject("roUrlTransfer")
req.SetMessagePort(port)
req.setCertificatesFile("pkg:/source/domain.crt")
req.AddHeader("X-Roku-Reserved-Dev-Id", "")
req.InitClientCertificates()
req.setUrl("https://domain.com/index.php/roku/getdata")
response = req.getToString()
print "response: "; response
Note: i edited /etc/apache2/sites-available/default-ssl.conf file to add godaddy certificate.in my website,https is working.i alwaise getting error code -77.i exported the crt file from mozilla firefox.is this a correct way?
No idea about the correct format of certificate export. But have you tried:
req.SetCertificatesFile("common:/certs/ca-bundle.crt")
I have to do a proof of concept and thus far I'm finding primarily old articles that reference IIS6 which isn't helping.
In short I have the following requirements.
I need to secure one file/page and this one file/page only using a client certificate. The rest of the site does need to continue operating under SSL but doesn't require client certificate, just this one file. User mapping is forbidden as mapping will be done programatically via C#/VB.NET.
Now I know this shouldn't be hard. I mean I should have access to the Request.ClientCertificate property but my problem is that in my testing I can't get a client certificate to travelling along the wire.
I've set IIS on one folder ( just to make my life simple ) require SSL and accept client certs as well as require client certs but all i get from iis once visiting the page is
HTTP/1.1 403 Forbidden. I never get asked to choose a client certificate to send to the server it just spews all over my request and drops it.
It gets even weirder when I'm using some code to test this. In this client code the CertPolicy class just returns true from a method to ignore cert errors and test.cer is a self signed cert made from using MakeCert. Just to make it clear though, only the client cert if self signed, the main cert is properly signed, but i play with fiddler alot and I haven't trusted that cert so that's why I have the hacky callback.
Dim Cert As X509Certificate = X509Certificate.CreateFromCertFile("Cert\test.cer")
' Handle any certificate errors on the certificate from the server.
ServicePointManager.CertificatePolicy = New CertPolicy()
' You must change the URL to point to your Web server.
Dim Request As HttpWebRequest = DirectCast(WebRequest.Create("https://local.domain.com/Cert/Server/"), HttpWebRequest)
Request.ClientCertificates.Add(Cert)
Request.UserAgent = "Client Cert Sample"
Request.Method = "GET"
Dim sr As StreamReader
Using Response As HttpWebResponse = DirectCast(Request.GetResponse, HttpWebResponse)
' Print the repsonse headers.
output.AppendFormat("{0}\r\n", Response.Headers)
output.AppendLine()
' Get the certificate data.
sr = New StreamReader(Response.GetResponseStream, Encoding.Default)
Dim count As Integer
Dim ReadBuf() As Char = New Char((1024) - 1) {}
Do
count = sr.Read(ReadBuf, 0, 1024)
If Not 0 = count Then
output.AppendLine(New String(ReadBuf))
End If
Loop While (count > 0)
End Using
The target page just returns the number of certs attached, which always returns if i set IIS to accept or ignore client certs but not required the.
Protected Overrides Sub OnLoad(ByVal e As System.EventArgs)
MyBase.OnLoad(e)
Dim cs As HttpClientCertificate = Request.ClientCertificate
Response.Write(cs.Count)
Response.End()
End Sub
If anyone can help me find out how to configure IIS7.5 to allow client certs to be attached to a request and just passed through that would be great.
This is an old question but I found it while searching for my own answers and figured it should be answered. In the web.config for the web site, to enable client certificates, you must first make sure that authentication module is installed, then enable the feature:
<location path="yourpath">
<system.webServer>
<security>
<access sslFlags="Ssl, SslNegotiateCert"/> <!-- or SslRequireCert -->
<authentication>
<iisClientCertificateMappingAuthentication enabled="true"
oneToOneCertificateMappingsEnabled="true">
<!-- or manyToOneCertificateMappingsEnabled="true" -->
</iisClientCertificateMappingAuthentication>
</authentication>
</security>
</system.webServer>
</location>
Then you add the one-to-one or many-to-one mappings inside of the iisClientCertificateMappingAuthentication element.
When the server asks the browser for the client certificate, it sends a list of certificate authorities it trusts. The browser then filters the available certificates based upon this information in order to display only relevant certificates (those issued by CAs the server trusts) in the certificate choice dialog.
(At least this is how Internet Explorer works; I don't know if other browsers perform such filtering.)
Therefore the client cert should not be self signed, but 1) should be issued by a certificate authority, 2) the certificate of that certificate authority should be installed on the server (in the Trusted Root Certificate Authorities store of the Local Machine account).
For testing purposes, you may set up your own CA, just make sure its certificate is installed on the server.