Debug "about:blank" in CasperJs - phantomjs

I have the following simple casperjs script (I cannot name the actual URL - sorry) on a Windows 7 machine:
var casper = require('casper').create({verbose:true,logLevel: "debug"});
casper.start('https://[XXX].de', function() {
console.log(this.getCurrentUrl());
});
casper.run();
The Output states that it failed - and the current url is : "about:blank"
[info] [phantom] Starting...
[info] [phantom] Running suite: 2 steps
[debug] [phantom] opening url: https://[XXX].de, HTTP GET
[debug] [phantom] Navigation requested: url=https://[XXX].de, type=Other, lock=true, isMainFrame=true
[warning] [phantom] Loading resource failed with status=fail: https://[XXX].de
[debug] [phantom] Successfully injected Casper client-side utilities
about:blank
[info] [phantom] Step 2/2: done in 39205ms.
[info] [phantom] Done 2 steps in 39309ms
When send a GET request with the Firefox RESTCLient Plugin - I get:
Status Code: 200 OK
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Connection: close
Content-Type: text/html
Date: Tue, 11 Dec 2012 11:09:37 GMT
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Pragma: no-cache
Server: unknown
Transfer-Encoding: chunked
My question is:
How can I debug this? Is this a known issue ?

Seems like the command line arguments passed to casperjs are not being passed to PhantomJS. I also ran into this issue and to fix this I ran my script as follows:
PhantomJS.exe --ignore-ssl-errors=true myscript.js
When I tried passing the parameters the other way around
PhantomJS.exe myscript.js --ignore-ssl-errors=true
it does not work and gives the same error as you have been facing.

To pass phantomjs parameters to casperjs, you can directly update your casper binary(casperjs.py on linux or casperjs.bat on windows).
On Linux, Open the casperjs.py, Update the CASPER_COMMAND array which is forming the phantomjs command and execute.
To add "--ignore-ssl-errors=yes", extend the CASPER_COMMAND by the following,
CASPER_COMMAND.extend(['--ignore-ssl-errors=yes']);
For the same, on windows or For more info can be found here

Related

Use wget to download pdf with no direct link

Some websites provide pdf files for viewing but I can't download such pdf files with wget.
Calling the website in question from my browser views the pdf:
https://www.lokalmatador.de/epaper/ausgabe/gemeinderundschau-muehlhausen-14-2021/
But using the following code I only get a pdf file with 0 lenght.
wget --content-disposition -nd https://www.lokalmatador.de/epaper/ausgabe/gemeinderundschau-muehlhausen-14-2021/
I tried some combinations with saving and loading cookies and referer but nothing works.
At this point I'm just curious what is happening and why wget is not fetching anything except maybe an empty index.html.
When I was looking at server response, it was saying the content length was 0.
--2021-04-17 14:59:35-- https://www.lokalmatador.de/epaper/ausgabe/gemeinderundschau-muehlhausen-14-2021/
Resolving www.lokalmatador.de (www.lokalmatador.de)... 37.202.6.70
Connecting to www.lokalmatador.de (www.lokalmatador.de)|37.202.6.70|:443... connected.
HTTP request sent, awaiting response...
HTTP/1.1 200 OK
Date: Sat, 17 Apr 2021 13:59:36 GMT
Server: Apache
Set-Cookie: fe_typo_user=477e8a1d2b3dd74bc5b6b408a6d74edd; expires=Mon, 17-May-2021 13:59:36 GMT; Max-Age=2592000; path=/; domain=.lokalmatador.de; httponly; samesite=lax
Upgrade: h2,h2c
Connection: Upgrade, Keep-Alive
Content-Length: Array
Cache-Control: max-age=2592000
Expires: Mon, 17 May 2021 13:59:36 GMT
X-UA-Compatible: IE=edge
X-Content-Type-Options: nosniff
Keep-Alive: timeout=5, max=100
Content-Type: application/pdf
Length: 0 [application/pdf]
Remote file exists but does not contain any link -- not retrieving.
So looked at the manual:
https://www.gnu.org/software/wget/manual/html_node/HTTP-Options.html
And there is a command just exactly for this:
‘--ignore-length’
Unfortunately, some HTTP servers (CGI programs, to be more precise) send out bogus Content-Length headers, which makes Wget go wild, as it thinks not all the document was retrieved. You can spot this syndrome if Wget retries getting the same document again and again, each time claiming that the (otherwise normal) connection has closed on the very same byte.
With this option, Wget will ignore the Content-Length header—as if it never existed.
Then the wget command started working as expected:
wget --ignore-length -O epaper.pdf https://www.lokalmatador.de/epaper/ausgabe/gemeinderundschau-muehlhausen-14-2021
Here is output which I'm seeing with the ignore length:
--2021-04-17 14:56:19-- https://www.lokalmatador.de/epaper/ausgabe/gemeinderundschau-muehlhausen-14-2021
Resolving www.lokalmatador.de (www.lokalmatador.de)... 37.202.6.70
Connecting to www.lokalmatador.de (www.lokalmatador.de)|37.202.6.70|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: ignored [application/pdf]
Saving to: ‘epaper.pdf’
epaper.pdf [ <=> ] 4.39M 1.23MB/s in 3.6s
2021-04-17 14:56:23 (1.21 MB/s) - ‘epaper.pdf’ saved [4601842]

Gatling: WebSocketHandshakeException: Invalid handshake response getStatus: 400 Bad Request

I'm trying to open ws connection, but get the error:
val openConnection = exec( ws("Connect -> WS").wsName("user").connect("wss://socket.develop.test.com?access_token=eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9"))
Then, I get:
09:49:53.343 [DEBUG] i.g.h.c.i.DefaultHttpClient - Installing SslHandler for wss://socket.develop.test.com?access_token=eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9
09:49:53.381 [DEBUG] i.n.h.s.u.InsecureTrustManagerFactory - Accepting a server certificate: CN=Kubernetes Ingress Controller Fake Certificate, O=Acme Co
09:49:53.418 [DEBUG] i.n.h.c.h.w.WebSocketClientHandshaker13 - WebSocket version 13 client handshake key: YndmXwIGgZseWyRlmXBoyw==, expected response: H2YGNn6p+DyLyudnc1JCluHjj7E=
09:49:53.419 [DEBUG] i.g.h.c.i.WebSocketHandler - ctx.write msg=DefaultFullHttpRequest(decodeResult: success, version: HTTP/1.1, content: EmptyByteBufBE)
GET ?access_token=eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9
HTTP/1.1
accept: */*
upgrade: websocket
connection: upgrade
sec-websocket-key: YndmXwIGgZseWyRlmXBoyw==
host: socket.develop.test.com
origin: https://socket.develop.test.com
sec-websocket-version: 13
09:49:53.423 [DEBUG] i.n.h.s.SslHandler - [id: 0xfa53644a, L:/192.168.150.134:54082 - R:socket.develop.test.com/78.47.16.48:443] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
09:49:53.450 [DEBUG] i.g.h.c.i.WebSocketHandler - Read msg=HttpObjectAggregator$AggregatedFullHttpResponse(decodeResult: success, version: HTTP/1.1, content: CompositeByteBuf(ridx: 0, widx: 163, cap: 163, components=1))
HTTP/1.1 400 Bad Request
Server: openresty/1.15.8.1
Date: Tue, 10 Mar 2020 06:49:53 GMT
Content-Type: text/html
Content-Length: 163
Connection: close
09:49:53.452 [DEBUG] i.g.h.c.i.WebSocketHandler - Crash
io.netty.handler.codec.http.websocketx.WebSocketHandshakeException: Invalid handshake response getStatus: 400 Bad Request
at io.netty.handler.codec.http.websocketx.WebSocketClientHandshaker13.verify(WebSocketClientHandshaker13.java:267)
In gatling.conf, I set also:
enableSni = false
useOpenSsl = false
enableHostnameVerification = false
If I try to open connection with any WS clients like SmartWebsocketClient - everything is ok, but by means of gatling I can't to open connection. Any ideas to fix it?
wss://domain?access_token... is not a valid URL. The path component must start with a / and not a ?. This means that the request is invalid which can explain the response of 400 Bad Request.
The URL must be at least wss://domain/?access_token... (i.e. a / before the ?) but maybe the rest of the URL is unexpected by the server too. Please check the actual requirements of the server.
This is a bug in Netty that doesn't properly compute WebSocket handshake request url when path is empty.
I've just contributed a patch: https://github.com/netty/netty/pull/10095.
Sibling issue in Gatling: https://github.com/gatling/gatling/issues/3876

S3 backend: can't overwrite AWS_S3_ENDPOINT

I want to use an S3 backend to remotely store my tfstate.
Since I want to use a bucket on Wasabi, and not AWS, I set the endpoint to s3.wasabisys.com.
However, terraform still tries to use AWS.
I tried to use the TF_AWS_S3_ENDPOINT environment variable, but even hardcoded values don't work:
terraform {
backend "s3" {
bucket = "my-bucket"
key = "my-key"
region = "us-east-1"
endpoint = "s3.wasabisys.com"
access_key = "xxxx"
secret_key = "xxxx"
}
}
Output:
stanislas#mbp ~> terraform init
2018/10/25 08:53:35 [INFO] Terraform version: 0.11.10
2018/10/25 08:53:35 [INFO] Go runtime version: go1.11.1
2018/10/25 08:53:35 [INFO] CLI args: []string{"/usr/local/Cellar/terraform/0.11.10/bin/terraform", "init"}
2018/10/25 08:53:35 [DEBUG] Attempting to open CLI config file: /Users/stanislas/.terraformrc
2018/10/25 08:53:35 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2018/10/25 08:53:35 [INFO] CLI command args: []string{"init"}
2018/10/25 08:53:35 [DEBUG] command: loading backend config file: /Users/stanislas/git/xxx
2018/10/25 08:53:35 [DEBUG] command: no data state file found for backend config
Initializing the backend...
2018/10/25 08:53:35 [DEBUG] New state was assigned lineage "be941477-7111-2a13-ceed-55e4fba0bcbd"
2018/10/25 08:53:35 [INFO] Building AWS region structure
2018/10/25 08:53:35 [INFO] Building AWS auth structure
2018/10/25 08:53:35 [INFO] Setting AWS metadata API timeout to 100ms
2018/10/25 08:53:36 [INFO] Ignoring AWS metadata API endpoint at default location as it doesn't return any instance-id
2018/10/25 08:53:36 [INFO] AWS Auth provider used: "StaticProvider"
2018/10/25 08:53:36 [INFO] Initializing DeviceFarm SDK connection
2018/10/25 08:53:36 [DEBUG] [aws-sdk-go] DEBUG: Request sts/GetCallerIdentity Details:
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: sts.amazonaws.com
User-Agent: aws-sdk-go/1.14.31 (go1.11.1; darwin; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.10
Content-Length: 43
Authorization: AWS4-HMAC-SHA256 Credential=xxx, SignedHeaders=content-length;content-type;host;x-amz-date, Signature=xxx
Content-Type: application/x-www-form-urlencoded; charset=utf-8
X-Amz-Date: 20181025T065336Z
Accept-Encoding: gzip
Action=GetCallerIdentity&Version=2011-06-15
-----------------------------------------------------
2018/10/25 08:53:36 [DEBUG] [aws-sdk-go] DEBUG: Response sts/GetCallerIdentity Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 403 Forbidden
Connection: close
Content-Length: 306
Content-Type: text/xml
Date: Thu, 25 Oct 2018 06:53:36 GMT
X-Amzn-Requestid: xxx
-----------------------------------------------------
2018/10/25 08:53:36 [DEBUG] [aws-sdk-go] <ErrorResponse xmlns="https://sts.amazonaws.com/doc/2011-06-15/">
<Error>
<Type>Sender</Type>
<Code>InvalidClientTokenId</Code>
<Message>The security token included in the request is invalid.</Message>
</Error>
<RequestId>xxx</RequestId>
</ErrorResponse>
2018/10/25 08:53:36 [DEBUG] [aws-sdk-go] DEBUG: Validate Response sts/GetCallerIdentity failed, not retrying, error InvalidClientTokenId: The security token included in the request is invalid.
status code: 403, request id: xxx
2018/10/25 08:53:36 [DEBUG] plugin: waiting for all plugin processes to complete...
Error configuring the backend "s3": InvalidClientTokenId: The security token included in the request is invalid.
status code: 403, request id: xxx
Please update the configuration in your Terraform files to fix this error
then run this command again.
As you can see, Terraform sends a request to sts.amazonaws.com.
Am I missing something?
Have you tried to use the set the following options to true ?
skip_requesting_account_id
skip_credentials_validation
skip_get_ec2_platforms
skip_metadata_api_check
As those seems to be needed when not using AWS (https://github.com/hashicorp/terraform/pull/15553#issuecomment-383294678).

Kong responds 404 when accessed vi Postman

I have set up an API and Kong. I have successfully installed Kong. In fact, i can access my APIs via curl on cli. However, when I use Postman, which I use all of the time, I get the response
{
"request_path": "/api/v1/",
"message": "API not found with these values",
"request_host": [
"192.168.33.13"
]
}
Ps. Im new to kong so please bear with me.
$ kong start
[INFO] Kong 0.7.0
[INFO] Using configuration: /etc/kong/kong.yml
[INFO] Setting working directory to /usr/local/kong
[INFO] database...........cassandra keyspace=kong ssl=verify=false enabled=false replication_factor=1 contact_points=127.0.0.1:9042 replication_strategy=SimpleStrategy timeout=5000 data_centers=
[INFO] dnsmasq............address=127.0.0.1:8053 dnsmasq=true port=8053
[INFO] serf ..............-profile=wan -rpc-addr=127.0.0.1:7373 -event-handler=member-join,member-leave,member-failed,member-update,member-reap,user:kong=/usr/local/kong/serf_event.sh -bind=0.0.0.0:7946 -node=precise64_0.0.0.0:7946 -log-level=err
[INFO] Trying to auto-join Kong nodes, please wait..
[WARN] Cannot auto-join the cluster because no nodes were found
[WARN] ulimit is currently set to "1024". For better performance set it to at least "4096" using "ulimit -n"
[INFO] nginx .............admin_api_listen=0.0.0.0:8001 proxy_listen=0.0.0.0:8000 proxy_listen_ssl=0.0.0.0:8443
[OK] Started
$ curl -i -X POST --url http://localhost:8001/apis/ --data 'name=geospatial' --data 'upstream_url=http://192.168.33.10/' --data 'request_host=192.168.33.10'
HTTP/1.1 201 Created
Date: Fri, 08 Apr 2016 14:38:22 GMT
Content-Type: application/json; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Access-Control-Allow-Origin: *
Server: kong/0.7.0
{"upstream_url":"http:\/\/192.168.33.10\/","id":"240e6cc6-626f-4629-9551-0d341a57adba","name":"geospatial","created_at":1460126302000,"request_host":"192.168.33.10"}
When I curl the api...
$ curl -i -X GET -H "Host: 192.168.33.10" "http://192.168.33.13:8000/api/v1/"
HTTP/1.1 200 OK
Date: Fri, 08 Apr 2016 14:56:04 GMT
Content-Type: application/json
Content-Length: 70
Connection: keep-alive
Server: Werkzeug/0.11.4 Python/2.7.6
X-Kong-Upstream-Latency: 4
X-Kong-Proxy-Latency: 0
Via: kong/0.7.0
{"status": 200, "message": 200, "data": "Hello World!", "error": null}
How ever when I try to use Postman, I get a 404 status response.
with this body
{
"request_path": "/api/v1/",
"message": "API not found with these values",
"request_host": [
"192.168.33.13"
]
}
Am I missing something?
Disclaimer:
I am using Postman because I am lazy and because my clients will mostly be using Postman as well. So dont tell me to to use curl instead :)
Just encountered the same problem.
In order to send to Kong you need to add Host header, but Postman has some restricted headers that are blocked and Host is one of them.
You need to download Chrome Inspector to send the Host header.
For more details -
https://www.getpostman.com/docs/requests
You need to setup the header parameter named "Host" in order to Kong find your API.
You can see that your curl command have this parameter
-H "Host: 192.168.33.10"

Cannot load url and then, cannot send login parameters

I'm trying to login automatically in tumblr, running the script proposed here but phantomjs only shows failed attempts:
[info] [phantom] Starting...
[info] [phantom] Running suite: 3 steps
[debug] [phantom] opening url: https://tumblr.com/login, HTTP GET
[debug] [phantom] Navigation requested: url=https://tumblr.com/login, type=Other,willNavigate=true, isMainFrame=true
[warning] [phantom] Loading resource failed with status=fail: https://tumblr.com/login
[debug] [phantom] Successfully injected Casper client-side utilities
[debug] [phantom] Logging in
[info] [remote] attempting to fetch form element from selector: '#signup_form'
[info] [phantom] Step anonymous 2/3: done in 2899ms.
[info] [phantom] Step _step 3/3: done in 2912ms.
[warning] [phantom] Casper.waitFor() timeout
[error] [phantom] Wait timeout of 100000ms expired, exiting.
Wait timeout of 100000ms expired, exiting.
I increased the original timeout script without success, my workstation uses:
- Windows 7 64bits
- Casperjs 1.1.0
- Phantomjs 1.9.1
thanks.
I finally found a solution for this issue.
Tumblr is using the TLSv1.2 encryption for their site while PhantomJS defaults to SSLv3.
To fix this, you can either call CasperJS with either:
casperjs tumblr.js --ssl-protocol=tlsv1
or
casperjs tumblr.js --ssl-protocol=any
With my version of casperjs and phantomjs, the cli ssl protocol parameter was being ignored, I had to put it in a config file instead:
casperjs browse.js --config=<(echo '{"sslProtocol": "any"}') --ignore-ssl-errors=true