Why is it not possible to access a public website? - vba

Question

This url can not be opened with URL-copy&paste it shows only an error. I have tried in vain to get access through MSXML2.XMLHTTP, same result!
This is my code:
Sub GetDataWebsite()
Const URL = "http://Zvg-port.de/index.php"
Dim HTML As New HTMLDocument
Dim elmt As Object
Dim x As Long
With CreateObject("MSXML2.XMLHTTP")
.Open "POST", URL, False
.setRequestHeader "Content-Type", "application/x-www-form-urlencoded"
.send
HTML.body.innerHTML = .responseText
End With
Set elmt = HTML.querySelectorAll("tr") 'or any class or tag or name
For x = 0 To elmt.Length - 1
ActiveSheet.Cells(x + 2, 2) = elmt.Item(x).innerText
Next x
End Sub
Nothing happens! What could be the problem? Thanks!

Answer 1

For your first question, you need to add additional parameters to the initial request body.
Weirdly, one has to keep working off html, rather than setting anything into derived variables, if one wants to use querySelectorAll().
For your second question, the result urls are expected to be navigated to after coming from the search page. A little testing indicates a referer header is needed. I know an request, with/without html session, with referer header will work because I tested with Python, but I haven't worked out what the missing bit is for VBA; my current attempts are returning odd encoding that also looks truncated.
Currently, the easiest way I see, if sticking with VBA, to ensure following links, would be to automate a browser, gather the results and the result links, then navigate to each link.
Current code (which answers your first question):
Option Explicit
Public Sub GetDataZvgPort()
Const URL = "https://www.zvg-portal.de/index.php?button=Suchen"
Dim html As MSHTML.HTMLDocument, xhr As Object
Set html = New MSHTML.HTMLDocument
Set xhr = CreateObject("MSXML2.ServerXMLHTTP.6.0")
Dim headers As Variant
With xhr
.Open "POST", URL, False
.setRequestHeader "Content-Type", "application/x-www-form-urlencoded"
.send "land_abk=sh&ger_name=Norderstedt&order_by=2&ger_id=X1526"
headers = .getAllResponseHeaders
html.body.innerHTML = .responseText
End With
Dim x As Long, link As String, gatheredLinks()
ReDim gatheredLinks(html.querySelectorAll("td a").Length - 1)
For x = 0 To html.querySelectorAll("table a nobr").Length - 1
ActiveSheet.Cells(x + 2, 2) = html.querySelectorAll("table a nobr").Item(x).innerText
link = Replace$(html.querySelectorAll("td a").Item(x).href, "about:", "https://www.zvg-portal.de/")
ActiveSheet.Cells(x + 2, 3) = link
Dim j As Long
For j = 0 To html.querySelectorAll("tr").Length - 1
If InStr(html.querySelectorAll("tr").Item(j).innerHTML, "Amtsgericht") > 0 Then
ActiveSheet.Cells(x + 2, 4) = html.querySelectorAll("tr").Item(j).getElementsByTagName("b")(0).innerText
Exit For
End If
Next
gatheredLinks(x) = link
Next x
' With xhr
' For x = LBound(gatheredLinks) To UBound(gatheredLinks)
' .Open "GET", gatheredLinks(x), False
' .setRequestHeader "Referer", "https://www.zvg-portal.de/index.php?button=Suchen"
' .setRequestHeader "Content-Type", "text/html; charset=ISO-8859-1"
' .setRequestHeader "User-Agent", "python-requests/2.24.0"
' .setRequestHeader "Accept-Encoding", "gzip, deflate"
' .setRequestHeader "Connection", "keep-alive"
' .setRequestHeader "Accept", "text/html,application/xhtml+xml,application/xml;"
' .send
' ActiveSheet.Cells(x + 2, 5) = .Status
' html.body.innerHTML = .responseText 'test response
' Dim s As String
' s = .responseText
' ActiveSheet.Cells(x + 2, 6) = s
' Stop
'
' 'do something else
' Next
' End With
Stop
End Sub
Python (using session) which can successfully retrieve content from result links:
import requests
from bs4 import BeautifulSoup as bs
data = {'ger_name': 'Norderstedt','order_by': '2','land_abk': 'sh','ger_id': 'X1526'}
headers = {'Referer': 'https://www.zvg-portal.de/index.php?button=Suchen'}
with requests.Session() as s:
r = s.post('https://www.zvg-portal.de/index.php?button=Suchen', data=data)
print(r.status_code)
soup = bs(r.content, 'lxml')
links = ['https://www.zvg-portal.de/' + i['href'] for i in soup.select('td a')]
s.headers = headers
for link in links:
r = s.get(link)
# print(r.status_code)
soup = bs(r.content, 'lxml')
print(soup.select_one('td p').text)
Session is NOT required. It is used just for efficiency.
Without a session, which still works, the headers sent are:
{'User-Agent': 'python-requests/2.24.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Referer': 'https://www.zvg-portal.de/index.php?button=Suchen'}