The app that I am trying to test has an SFTP server that can be queried via API. Swagger shows the following sample cURL request to get domain files information, and I am having a hard time making this call with Karate:
curl -X GET -H 'Accept: application/json' 'https://{host}:{port}/api/{clientId}/'
I do the following where baseUrl is defined as https://sftp.mydomain.com:22 where 22 is the port number that I can successfully use to connect to the SFTP server via Cyberduck:
Feature:
Background:
* url baseUrl
* def moduleBase = '/api/12345/'
Scenario:
* path moduleBase
When method get
Then status 200
The error that I get is this:
ERROR com.intuit.karate - src/test/java/mytest.feature:9
When method get
http call failed after 815 milliseconds for url: https://sftp.mydomain.com:22/api/12345/
What am I doing wrong?
Clearly it may not be HTTP so I don't think Swagger and all is legit.
Maybe you can just delegate to the OS. Refer: https://stackoverflow.com/a/64352676/143475
I have a test using RabbitMq in Testcontainers. The test is working using HTTP
#Container private static final RabbitMQContainer RABBITMQ_CONTAINER =
new RabbitMQContainer()
.withLogConsumer(new Slf4jLogConsumer(LOG))
.withStartupTimeout(Duration.of(5, ChronoUnit.MINUTES))
.waitingFor(Wait.forHttp("/api/vhosts")
.forPort(15672)
.withBasicCredentials("guest", "guest"));
and fails when I switch to HTTPS
#Container private static final RabbitMQContainer RABBITMQ_CONTAINER =
new RabbitMQContainer()
.withLogConsumer(new Slf4jLogConsumer(LOG))
.withStartupTimeout(Duration.of(5, ChronoUnit.MINUTES))
.waitingFor(Wait.forHttp("/api/vhosts")
.usingTls()
.forPort(15671)
.withBasicCredentials("guest", "guest"))
.withSSL(forClasspathResource("/certs/server_key.pem", 0644),
forClasspathResource("/certs/server_certificate.pem", 0644),
forClasspathResource("/certs/ca_certificate.pem", 0644),
VERIFY_NONE,
false);
In logs I see that container can not start:
...
18:53:21.274 [main] INFO - /brave_swirles: Waiting for 60 seconds for URL: https://localhost:50062/api/vhosts (where port 50062 maps to container port 15671)
...
18:54:21.302 [main] ERROR - Could not start container
org.testcontainers.containers.ContainerLaunchException: Timed out waiting for URL to be accessible (https://localhost:50062/api/vhosts should return HTTP 200)
What do I miss? I'd want at least Testcontainers' waiting strategy works.
Using RabbitMQContainer with custom SSL certificates makes it hard to also use HttpWaitStrategy. In this case, you probably need to configure the whole JVM to trust those SSL certificates.
Alternatively, just continue to use the default wait strategy (which will be a LogMessageWaitStrategy).
For further examples, just look at the RabbitMQContainer tests in Testcontainers itself:
https://github.com/testcontainers/testcontainers-java/blob/c3f53b3a63e6b0bc800a7f0fbce91ce95a8986b3/modules/rabbitmq/src/test/java/org/testcontainers/containers/RabbitMQContainerTest.java#L237-L264
I wanted to use NiFi's posthttp/invokeHttp processor to post a PDF to an API.
But considering the following cURL request to replicate in NiFi:
curl -X POST "http://ipaddress:port/api/" -H "accept: application/json" -H
"Content-Type: multipart/form-data" -F "pdf_file=#sample.pdf;
type=application/pdf"
Which property takes the -F information in nifi attributes?
Configuration for invokehttp right now:
error:
"400 Bad Request: The browser (or proxy) sent a request that this server could not understand."
Configration for posthttp right now:
error:
server logs: readv() failed (104: Connection reset by peer) while reading upstream
In older version of nifi you will have to use your own script to build a multipart request and then use invoke to create post request. You can refer to this post for a ExecuteGroovyScript example.
https://stackoverflow.com/a/57204862
Since Nifi 1.12 you can directly use invokeHTTP by setting content-type
https://stackoverflow.com/a/69284300
When you use PostHttp/InvokeHttp you wouldn't be referencing an external file, you would be sending the content of the flow file. So you would first need to bring sample.pdf into NiFi by using GetFile or ListFile/FetchFile and then flow file coming out of those processors represents the PDF, and you would route that flow file to InvokeHttp which would POST the content of the flow file (the pdf).
Please note that my question is not about testing ssl/tls secured http links and not about making Webdriver accept certain certificates.
My question is about how to make the embedded Jetty of selenium standalone server provide a secured https connection.
In the sourcecode of 3.4.0 I can see this section:
HttpConfiguration httpConfig = new HttpConfiguration();
httpConfig.setSecureScheme("https");
httpConfig.setSecurePort(config.port);
log.info("Will listen on " + config.port);
ServerConnector http = new ServerConnector(server, new HttpConnectionFactory(httpConfig));
http.setPort(config.port);
server.addConnector(http);
From the logs I can see that this code is reached but the connection is not secured (how should it be, there isn't even a certificate involved):
10:57:00.023 INFO - Selenium build info: version: '3.4.0', revision: 'unknown'
10:57:00.024 INFO - Launching Selenium Grid hub
2017-05-09 10:57:01.707:INFO::main: Logging initialized #2044ms to org.seleniumhq.jetty9.util.log.StdErrLog
10:57:01.721 INFO - Will listen on 4444
2017-05-09 10:57:01.800:INFO:osjs.Server:main: jetty-9.4.3.v20170317
2017-05-09 10:57:01.851:INFO:osjs.session:main: DefaultSessionIdManager workerName=node0
Because of company security governance we are forced to provide all services secured. This means I need to secure at least the hub of selenium grid, nodes would be perfect too. I know that I could do some tunneling, proxying or ipsec but I want to avoid this complexity if possible.
I even tried if Jetty somehow "automagically" knows to respond accordingly if ssl is used but as expected this fails:
I have a Se 2 grid hub running. Where can I find documentation for the effects and schema of the -hubConfig parameter? Currently my grid hub shows: "updated with grid2 config : No hub config file specified. To specify one, use -hubConfig XXX.json where XXX.json is a hub config file". I can use the hub without this configured.
I have found examples in blogs and issues, but no clear documentation.
blogs: http://opensourcetester.co.uk/2011/07/06/selenium-grid-2/
issues: http://code.google.com/p/selenium/issues/detail?id=2399
I've found a good starting point is looking at examples in the Selenium Grid 2 code base.
Here is a sample hub file (assume we name the json file hub.json):
Use: java -jar selenium-server-standalone-2.6.0.jar -role hub -hubConfig hub.json
{
"host": null,
"port": 4444,
"newSessionWaitTimeout": -1,
"servlets" : [],
"prioritizer": null,
"capabilityMatcher": "org.openqa.grid.internal.utils.DefaultCapabilityMatcher",
"throwOnCapabilityNotPresent": true,
"nodePolling": 5000,
"cleanUpCycle": 5000,
"timeout": 300000,
"maxSession": 5
}
here is a sample node file (assume the file is named rc.json):
Use: java -jar selenium-server-standalone-2.6.0.jar -role rc -nodeConfig rc.json
{
"capabilities":
[
{
"browserName": "firefox",
"maxInstances": 5
},
{
"browserName": "chrome",
"maxInstances": 5
},
{
"browserName": "internet explorer",
"maxInstances": 1
}
],
"configuration":
{
"nodeTimeout":120,
"port":5555,
"hubPort":4444,
"hubHost":"localhost",
"nodePolling":2000,
"registerCycle":10000,
"register":true,
"cleanUpCycle":2000,
"timeout":30000,
"maxSession":5,
}
}
You can create use the similar format for role "wd" if it's required to have a different configuration.
The javadocs and various source code files are helpful to a certain extent but the wiki is not very helpful and the various configuration examples seem inconsistent until you dig into the code. For example, DefaultHub.json specifies "timeout" : 300000 which leads one to assume timeout is in ms but if you look at command-line examples you'll see that timeout is specified in seconds. If you look at the code you'll see that, indeed, the JSON configuration scheme does take timeout (and all other time values) in ms but all other configuration schemes take the timeout (and some other time values) in seconds.
The most readable and concise documentation I've found for the hub and node options is in an old copy of GridParameters.properties (which no longer seems to exist), but keep in mind that all time values in a JSON configuration file are in ms:
role = <hub|node> (default is no grid, just run an RC/webdriver server). When launching a node, the parameters will be forwarded to the server on the node, so you can use something like -role node -trustAllSSLCertificates. In that case, the SeleniumServer will be launch with the trustallSSLCertificates option.
# hub config
host = (hub & node) <IP | hostname> : usually not needed and determined automatically. For exotic network configuration, network with VPN, specifying the host might be necessary.
port = (hub & node) <xxxx> : the port the remote/hub will listen on. Default to 4444.
throwOnCapabilityNotPresent = (hub) <true | false> default to true. If true, the hub will reject test requests right away if no proxy is currently registered that can host that capability.Set it to false to have the request queued until a node supporting the capability is added to the grid.
newSessionWaitTimeout = (hub) <XXXX>. Default to no timeout ( -1 ) the time in ms after which a new test waiting for a node to become available will time out.When that happens, the test will throw an exception before starting a browser.
capabilityMatcher = (hub) a class implementing the CapabilityMatcher interface. Defaults to org.openqa.grid.internal.utils.DefaultCapabilityMatcher. Specify the logic the hub will follow to define if a request can be assigned to a node.Change this class if you want to have the matching process use regular expression instead of exact match for the version of the browser for instance. All the nodes of a grid instance will use the same matcher, defined by the registry.
prioritizer = (hub) a class implementing the Prioritizer interface. Default to null ( no priority = FIFO ).Specify a custom prioritizer if you need the grid to process the tests from the CI, or the IE tests first for instance.
servlets = (hub & node) <com.mycompany.MyServlet,com.mycompany.MyServlet2> to register a new servlet on the hub/node. The servlet will accessible under the path /grid/admin/MyServlet /grid/admin/MyServlet2
grid1Yml = (hub) a YML file following grid1 format.
hubConfig = (hub) a JSON file following grid2 format that defines the hub properties.
nodeConfig = (node) a JSON file following grid2 format that defines the node properties.
# config that will be inherited by the proxy and used for the node management.
cleanupCycle = (node) <XXXX> in ms. How often a proxy will check for timed out thread.
timeout = (node) <XXXX> the timeout in seconds before the hub automatically ends a test that hasn't had any activity in the last X seconds. The browser will be released for another test to use. This typically takes care of the client crashes.
browserTimeout= (hub/node) The timeout in seconds a browser can hang
hub = (node) <http://localhost:4444/grid/register> : the url that will be used to post the registration request. This option takes precedence over -hubHost and -hubPort options.
hubHost = (node) <IP | hostname> : the host address of a hub the registration request should be sent to. Default to localhost. Option -hub takes precedence over this option.
hubPort = (node) <xxxx> : the port listened by a hub the registration request should be sent to. Default to 4444. Option -hub takes precedence over this option.
proxy = (node) the class that will be used to represent the node. By default org.openqa.grid.selenium.proxy.DefaultRemoteProxy.
maxSession = (node) max number of tests that can run at the same time on the node, independently of the browser used.
registerCycle = (node) how often in ms the node will try to register itself again.Allow to restart the hub without having to restart the nodes.
nodePolling = (node) in ms. Interval between alive checks of node how often the hub checks if the node is still alive.
unregisterIfStillDownAfter = (node) in ms. If the node remains down for more than unregisterIfStillDownAfter millisec, it will disappear from the hub.Default is 1min.
downPollingLimit = (node) node is marked as down after downPollingLimit alive checks.
nodeStatusCheckTimeout = (node) in ms. Connection and socket timeout which is used for node alive check.
The wiki docs are sparse, but there are lots of helpful comments in the source. GridHubConfiguration.java is where you should start. Here's a sample:
/**
* how often in ms each proxy will detect that a session has timed out. All new proxy registering
* will have that value if they don't specifically mention the parameter.
*/
private int cleanupCycle;
/**
* how long a new session request can stay in the queue without being assigned before being
* rejected. -1 = forever.
*/
private int newSessionWaitTimeout;
I was looking for what the timeout config meant, found here:
/**
* how many ms can a session be idle before being considered timed out. Working together with
* cleanup cycle. Worst case scenario, a session can be idle for timeout + cleanup cycle before the
* timeout is detected.
*/
public static final JsonKey CLIENT_TIMEOUT = JsonKey.key("timeout");