I am trying to use browsermob-proxy to monitor all requests and responses for selenium tests. In my case, I am running the selenium tests with the py.test framework inside a docker image on jenkins. I am using the following versions:
selenium==3.8.0
pyvirtualdisplay==0.2.1
pytest==3.4.0
browsermob-proxy==0.8.0
also java openjdk-8-jdk.
Inside the py.test code I have the following line to create and start the server:
proxyserver = Server(
path="/root/tests/bsp_usecase_tests/bin/browsermob-proxy",
options={'port': 8090}
)
proxyserver.start()
driverproxy = proxyserver.create_proxy() # line 127
but in the last line I get the following error:
conftest.py:127: in basedriver
driverproxy = proxyserver.create_proxy()
/usr/local/lib/python2.7/dist-packages/browsermobproxy/server.py:40: in create_proxy
client = Client(self.url[7:], params)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <browsermobproxy.client.Client object at 0x7f1a5f73f1d0>
url = 'localhost:8090', params = {}, options = {}
def __init__(self, url, params=None, options=None):
"""
Initialises a new Client object
:param url: This is where the BrowserMob Proxy lives
:param params: URL query (for example httpProxy and httpsProxy vars)
:param options: Dictionary that can contain the port of an existing
proxy to use (for example 'existing_proxy_port_to_use')
"""
params = params if params is not None else {}
options = options if options is not None else {}
self.host = "http://" + url
if params:
urlparams = "?" + unquote(urlencode(params))
else:
urlparams = ""
if 'existing_proxy_port_to_use' in options:
self.port = options['existing_proxy_port_to_use']
else:
resp = requests.post('%s/proxy' % self.host + urlparams)
content = resp.content.decode('utf-8')
try:
jcontent = json.loads(content)
except Exception as e:
raise Exception("Could not read Browsermob-Proxy json\n"
> "Another server running on this port?\n%s..." % content[:512])
E Exception: Could not read Browsermob-Proxy json
E Another server running on this port?
E <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
E <html><head>
E <meta type="copyright" content="Copyright (C) 1996-2016 The Squid Software Foundation and contributors">
E <meta http-equiv="Content-Type" CONTENT="text/html; charset=utf-8">
E <title>ERROR: The requested URL could not be retrieved</title>
E <style type="text/css"><!--
E /*
E * Copyright (C) 1996-2016 The Squid Software Foundation and contributors
E *
E * Squid software is distributed under GPLv2+ license and incl...
/usr/local/lib/python2.7/dist-packages/browsermobproxy/client.py:37: Exception
Maybe I forgot to copy important files into the docker image?
I have the following setup for the browsermob-proxy (besides the pip install):
bsp_usecase_tests/bin:
browsermob-proxy
bsp_usecase_tests/lib:
browsermob-dist-2.1.4.jar
i.e. I have copied the file browsermob-proxy into bin and the file browsermob-dist-2.1.4.jar into the lib folder. Also, the server itself seem to be starting and running. But I get this weird error when I try to get the proxy information I can use for the selenium driver...
Any ideas what I have done wrong, or what is missing?
Seems to me that the port is already in use. You can check if the port is being used using lsof and then kill it, if it is in use. You can check using lsof -i:8090
EDIT :
And you have already done that in this thread -
How to fix 'Address already in use' error with browsermob-proxy?
I would initialize server with
proxyserver = Server(
path="/root/tests/bsp_usecase_tests/bin/browsermob-proxy",
options={'existing_proxy_port_to_use': 8090})
This should do it.
The internal code in Browsermob-Proxy is looking for key in 'options' dictionary named 'existing_proxy_port_to_use', else it uses default i.e. empty dictionary.
Related
I'm using a project in VB6.0 and I need to implement WebSocket for VB6.0 but unfortunately, I could not find any resources over the internet, so please assist me on this, I've been searching Winsock to use TCP/IP socket servers, I need proper guidance on this.
I found the below code from somewhere but it seems not useful.
Private Sub Winsock1_ConnectionRequest(Index As Integer, ByVal requestID As Long)
Winsock1(Index).Close
Winsock1(Index).Accept requestID
Dim Packet As String
With Winsock1(Index)
Packet = "HTTP/1.1 101 Web Socket Protocol Handshake" & vbNewLine
Packet = Packet & "Upgrade: WebSocket" & vbNewLine
Packet = Packet & "Connection: Upgrade" & vbNewLine
Packet = Packet & "Sec-WebSocket-Accept: s3pPLMBiTxaQ9kYGzzhZRbK+xOo=" & vbNewLine
Data = Empty
.SendData Packet
End With
End Sub
I have developed websocket solution for vb6 but comments / varables are in Slovenian language so... fast answer:
you have to calculate key in 'Sec-WebSocket-Accept'. It changes on every connection so you cannot use fixed response...
"Sec-WebSocket-Accept: " + Key + vbCrLf + _
I had a file (Theme file) that is stored on Sharepoint and first needed to be downloaded into the temp directory before being loaded applied to word. This worked for a while, but recently I am getting an "access denied error".
I looked around and tested other libraries CreateObject("MSXML2.ServerXMLHTTP.6.0") instead of CreateObject("Microsoft.XMLHTTP").
Interestingly, I don't get the access error message with CreateObject("MSXML2.ServerXMLHTTP.6.0"), but instead it dowloads a page with this error:
[![Screenshot of error message][1]][1]
We can't sign you in
Your browser is currently set to block cookies. You need to allow cookies to use this service.
Cookies are small text files stored on your computer that tell us when you're signed in. To learn how to allow cookies, check the online help in your web browser.
I hope someone has an idea about why this error occurs and how to solve it
Here is the code I use.
Public Sub Download(ByVal URL As String, ByVal FilePath As String, Optional ByVal Overwrite As Boolean = True)
Dim iOverwrite, oStrm
If (IsNull(Overwrite) Or Overwrite) Then
iOverwrite = 2
Else
iOverwrite = 1
End If
Dim HttpReq As Object
'NOTE: There are some issues downloading if not properly logged in! May need to loggin sharepoint again
' https://www.codeproject.com/Questions/1101499/Download-files-from-API-using-vbscript-cmd-prompt
' Based on https://stackoverflow.com/questions/22938194/xmlhttp-request-is-raising-an-access-denied-error
'Set HttpReq = CreateObject("MSXML2.ServerXMLHTTP.6.0")
Set HttpReq = CreateObject("Microsoft.XMLHTTP")
'Set HttpReq = CreateObject("MSXML2.ServerXMLHTTP.3.0")
HttpReq.Open "GET", URL, False, "username", "password"
On Error GoTo ErrorHandler
HttpReq.send
On Error GoTo 0
If HttpReq.Status = 200 Then
Set oStrm = CreateObject("ADODB.Stream")
oStrm.Open
oStrm.Type = 1
oStrm.Write HttpReq.responseBody
oStrm.SaveToFile FilePath, iOverwrite ' 1 = no overwrite, 2 = overwrite
oStrm.Close
End If
Exit Sub
ErrorHandler:
MsgBox "The file could not be downloaded. Verify that you are logged in SharePoint with word and browser.", vbCritical, "Download error"
Debug.Print "Download - Error Downloading file will not be downloaded - Error #: '" & Err.Number & "'. Error description: " & Err.description
End Sub```
[1]: https://i.stack.imgur.com/pdH6v.png
I use import function specifically designed for this. I use Sharepoint Teams site (no user/password can be sent for auth).
Private Declare Function URLDownloadToFile Lib "urlmon" Alias "URLDownloadToFileA" (ByVal pCaller As Long, ByVal szURL As String, ByVal szFileName As String, ByVal dwReserved As Long, ByVal lpfnCB As Long) As Long
Function downloadSP(ByVal url As String, ByVal nm As String) As Long
DownloadFileFromWeb = URLDownloadToFile(0, url, nm, 0, 0) ' nm includes filename
End Function
In addition. I have to first use an ADO query to the sharepoint library directly before. This ADO connection handles authentication and allows subsequent downloads to location. There probably is another method for sending Teams authentication, but this works just fine. (it's also a great way to get data from SP List/Libraries or even within Excel files)
If testConnected Then downloadSP url, nm
Function testConnected() As Boolean
Dim cn As Object
Dim rs As Object
Set cn = CreateObject("ADODB.Connection")
Set rs = CreateObject("ADODB.RecordSet")
listGUID = "B3657D15-5F5C-468E-B1C2-784B930FE2E6"
siteURL = "https://azuresite.sharepoint.com/sites/test/"
spSql = "Select * from ['https://azuresite.sharepoint.com/sites/test/SL%20Template/Forms/AllItems.aspx']"
cnStr = "Provider=Microsoft.ACE.OLEDB.16.0;WSS;IMEX=2;RetrieveIds=No;DATABASE=" & siteURL & "; LIST=" & listGUID & ";"
cn.ConnectionString = cnStr
On Error GoTo NotConnected
cn.Open
rs.Open spSql, cn, 1, 2
testConnected = True
cn.Close
Exit Function
NotConnected:
testConnected = False
Exit Function
End Function
So I managed to solve this issue after:
enabling accepting cookies from: https://login.microsoftonline.com/ and our SharePoint sites (also added them in trusted websites)
Clearing the cookies from the history
Use the "Microsoft.XMLHTTP" library which works (other libraries do not seem to work properly still)
I need to download a really large file in msaccess via a vba application.
Using the objects MSXML2.ServerXMLHTTP.6.0 and WinHttp.WinHttpRequest.5.1 result in an error stating that there is not enough storage available to complete this operation. Therefore i resorted in using the DoFileDownload method from shdocvw.dll.
What i want to do is pass an extra header (an API key) to the request sent by the function.
Here is roughly what i want to do.
Private Declare Function DoFileDownload Lib "shdocvw.dll" _
(ByVal lpszFile As String) As Long
Public Sub Download()
sDownloadFile = StrConv(<link_to_download>, vbUnicode)
'set a header before calling DoFileDownload
Call DoFileDownload(sDownloadFile)
End Sub
How do i approach this problem?
A WebRequest downloading a whole file at once stores the whole data in response.
Although there are options to chunk response, using Wget is less coding, but more options.
Private Sub DownloadFileWget()
Const PathToWget As String = "" 'if wget is not in path use "Path\To\Wget"
Dim LinkToFile As String
Dim SavePath As String
With CreateObject("WScript.Shell")
LinkToFile = "http://download.windowsupdate.com/microsoftupdate/v6/wsusscan/wsusscn2.cab" 'huge file > 500MB
SavePath = "C:\doc" 'folder to save download
.CurrentDirectory = SavePath
.Run Chr(34) & PathToWget & "wget.exe" & Chr(34) & " --header='name: value' " & Chr(34) & LinkToFile & Chr(34) & " -N", 1, True
' -N: Continue download only if the local version is outdated.
End With
End Sub
All of my settings work perfectly fine and have no issue as such. Its just that CircleCI is not getting the most simplest possible test work fine. I've created a home page and wrote a test for it. The test at local ubuntu machine runs fine with python manage.py runserver and with pytest as well. Also installation and configuration works just fine in logs at CircleCI.
I would appreciate if someone can point me to right direction.
My Django Test:
from django.test import TestCase
from django.urls import reverse
class TestPage(TestCase):
def test_home_page_works(self):
response = self.client.get(reverse("home"))
self.assertEqual(response.status_code, 200)
self.assertTemplateUsed(response, "base.html")
self.assertContains(response, "APTA")
My CircleCI config.yml:
version: 2
jobs:
build:
docker:
# specify the version you desire here
# use `-browsers` prefix for selenium tests, e.g. `3.6.1-browsers`
- image: circleci/python:3.6.2
# Specify service dependencies here if necessary
# CircleCI maintains a library of pre-built images
# documented at https://circleci.com/docs/2.0/circleci-images/
# - image: circleci/postgres:9.4
- image: postgres:9.6
environment:
- POSTGRES_USER=root
- POSTGRES_DB=lims
working_directory: ~/repo
steps:
- checkout
- run:
name: Wait for db
command: dockerize -wait tcp://localhost:5432 -timeout 1m
- run:
name: Install
command: |
python3 -m venv venv
. envs/test
pip install .[test]
- run:
name: Install dependencies
command: |
. venv/bin/activate
python --version
pip install -r requirements.txt
- run:
name: Collect static files
command: |
. venv/bin/activate
python apta/./manage.py collectstatic --noinput
- run:
name: Test
command: |
. venv/bin/activate
python apta/./manage.py migrate --noinput
pytest
- run:
name: Lint
command: |
. envs/test
flake8
- store_artifacts:
path: test-reports
destination: test-reports
My pytest.ini:
[pytest]
DJANGO_SETTINGS_MODULE=apta.settings
python_files = tests.py test_*.py
Error I've got at CircleCI:
apta/tests/test_views.py F [100%]
=================================== FAILURES ===================================
________________________ TestPage.test_home_page_works _________________________
self = <tests.test_views.TestPage testMethod=test_home_page_works>
def test_home_page_works(self):
> response = self.client.get(reverse("home"))
apta/tests/test_views.py:8:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
venv/lib/python3.6/site-packages/django/test/client.py:535: in get
response = super().get(path, data=data, secure=secure, **extra)
venv/lib/python3.6/site-packages/django/test/client.py:347: in get
**extra,
venv/lib/python3.6/site-packages/django/test/client.py:422: in generic
return self.request(**r)
venv/lib/python3.6/site-packages/django/test/client.py:503: in request
raise exc_value
venv/lib/python3.6/site-packages/django/core/handlers/exception.py:34: in inner
response = get_response(request)
venv/lib/python3.6/site-packages/django/core/handlers/base.py:145: in _get_response
response = self.process_exception_by_middleware(e, request)
venv/lib/python3.6/site-packages/django/core/handlers/base.py:143: in _get_response
response = response.render()
venv/lib/python3.6/site-packages/django/template/response.py:106: in render
self.content = self.rendered_content
venv/lib/python3.6/site-packages/django/template/response.py:81: in rendered_content
template = self.resolve_template(self.template_name)
venv/lib/python3.6/site-packages/django/template/response.py:63: in resolve_template
return select_template(template, using=self.using)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
template_name_list = ['base.html'], using = None
def select_template(template_name_list, using=None):
"""
Load and return a template for one of the given names.
Try names in order and return the first template found.
Raise TemplateDoesNotExist if no such template exists.
"""
if isinstance(template_name_list, str):
raise TypeError(
'select_template() takes an iterable of template names but got a '
'string: %r. Use get_template() if you want to load a single '
'template by name.' % template_name_list
)
chain = []
engines = _engine_list(using)
for template_name in template_name_list:
for engine in engines:
try:
return engine.get_template(template_name)
except TemplateDoesNotExist as e:
chain.append(e)
if template_name_list:
> raise TemplateDoesNotExist(', '.join(template_name_list), chain=chain)
E django.template.exceptions.TemplateDoesNotExist: base.html
venv/lib/python3.6/site-packages/django/template/loader.py:47: TemplateDoesNotExist
------------------------------ Captured log call -------------------------------
log.py 228 ERROR Internal Server Error: /
Traceback (most recent call last):
File "/home/circleci/repo/venv/lib/python3.6/site-packages/django/core/handlers/exception.py", line 34, in inner
response = get_response(request)
File "/home/circleci/repo/venv/lib/python3.6/site-packages/django/core/handlers/base.py", line 145, in _get_response
response = self.process_exception_by_middleware(e, request)
File "/home/circleci/repo/venv/lib/python3.6/site-packages/django/core/handlers/base.py", line 143, in _get_response
response = response.render()
File "/home/circleci/repo/venv/lib/python3.6/site-packages/django/template/response.py", line 106, in render
self.content = self.rendered_content
File "/home/circleci/repo/venv/lib/python3.6/site-packages/django/template/response.py", line 81, in rendered_content
template = self.resolve_template(self.template_name)
File "/home/circleci/repo/venv/lib/python3.6/site-packages/django/template/response.py", line 63, in resolve_template
return select_template(template, using=self.using)
File "/home/circleci/repo/venv/lib/python3.6/site-packages/django/template/loader.py", line 47, in select_template
raise TemplateDoesNotExist(', '.join(template_name_list), chain=chain)
django.template.exceptions.TemplateDoesNotExist: base.html
=========================== 1 failed in 0.85 seconds ===========================
Exited with code 1
VB.NET 4.5 here.
In order to dynamically load a class type from an external library, please take a look at my snippet of code from a thread context:
While (True)
Dim MyClassName = "MyNamespace.MyClass,MyNamespace"
Dim MyLibPath = "C:\Libs\MyLib.dll"
Sleep(3000)
Try
Dim MyClass As Type = _
Type.GetType(MyClassName, _
Function(AssemResolv) Reflection.Assembly.LoadFrom(MyLibPath), _
Nothing, _
False, _
True)
Catch ex As Exception
MsgBox(ex.Message)
End Try
End While
that is, every 3 secs load my class from my lib at that path.
Please refrain from making me notice that is useless to load the type at every loop, this snippet is just an example.
At this point my problem is that everything works fine as long as the libraries are already present at the path when the program launches, but if I try to erase the dll from C:\Libs while program is running, the dll continues being found and the type being loaded.
Same happens if the dll is not present at launch: if I copy it at the given path, the program continues not finding it. But, again if I stop and relaunch the dll is found.
Why does this happen?
How can I make the program to notice when a dll is copied/erased to/from the path?
Thank you in advance.
Edit:
I solved this way:
While (True)
Dim MyClassName = "MyNamespace.MyClass,MyNamespace"
Dim MyLibName = "MyLib"
Dim MyLibPath = "C:\Libs\MyLib.dll"
Sleep(3000)
Try
Dim MyClass As Type = _
Type.GetType(TypeStr, _
Function(AssemResolv) If(AssemResolv.Name = MyLibName, _
Reflection.Assembly.LoadFile(MyLibPath), _
Nothing), _
Function(assem, name, ignore) If(assem Is Nothing, _
Type.GetType(name, False, ignore), _
assem.GetType(name, False, ignore)), _
False, _
True)
Catch ex As Exception
MsgBox(ex.Message)
End Try
End While
In other words I used the Assembly.LoadFile (that is expecting a real FS path) in place of the Assembly.LoadFrom (which is expecting a URI) and implemented better assembly and type resolvers.
Now libraries are loaded when copied into the directory and not loaded when for example you change name to the dll file, and all of this while the program is running.
I solved this way:
While (True)
Dim MyClassName = "MyNamespace.MyClass,MyNamespace"
Dim MyLibName = "MyLib"
Dim MyLibPath = "C:\Libs\MyLib.dll"
Sleep(3000)
Try
Dim MyClass As Type = _
Type.GetType(TypeStr, _
Function(AssemResolv) If(AssemResolv.Name = MyLibName, _
Reflection.Assembly.LoadFile(MyLibPath), _
Nothing), _
Function(assem, name, ignore) If(assem Is Nothing, _
Type.GetType(name, False, ignore), _
assem.GetType(name, False, ignore)), _
False, _
True)
Catch ex As Exception
MsgBox(ex.Message)
End Try
End While
In other words I used the Assembly.LoadFile (that is expecting a real FS path) in place of the Assembly.LoadFrom (which is expecting a URI) and implemented better assembly and type resolvers.
Now libraries are found and loaded when copied into the directory and not found and loaded when for example you change name to the dll file, and all of this while the program is running.