I am running a mock service using the karate standalone jar version 1.3.0.
Mocking is working fine.
However, when I make a GET request to the /__admin/stop endpoint, the process registers that the endpoint has been called, closes the listen port, but does not stop.
Execution output and scripts...
on startup:
java -jar /opt/karate/karate.jar -m src/test/java/mocks/fs/john.feature -p 80
16:17:16.671 [main] INFO com.intuit.karate - Karate version: 1.3.0
16:17:17.315 [main] INFO com.intuit.karate - mock server initialized: src/test/java/mocks/fs/john.feature
16:17:17.425 [main] DEBUG com.intuit.karate.http.HttpServer - server started: aad-9mpcfg3:65080
netstat on listen socket:
netstat -na | grep 65080
tcp6 0 0 :::65080 :::* LISTEN
submission of curl commands:
curl -v http://localhost:65080/john/transactionservice/ping
* Trying 127.0.0.1:65080...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 65080 (#0)
> GET /john/transactionservice/ping HTTP/1.1
> Host: localhost:65080
> User-Agent: curl/7.68.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: application/json
< content-length: 49
< server: Armeria/1.18.0
< date: Mon, 28 Nov 2022 16:19:37 GMT
<
* Connection #0 to host localhost left intact
{"message":"this is the JOHN TransactionService"}
curl -v http://localhost:65080/__admin/stop
* Trying 127.0.0.1:65080...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 65080 (#0)
> GET /__admin/stop HTTP/1.1
> Host: localhost:65080
> User-Agent: curl/7.68.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 202 Accepted
< content-type: text/plain; charset=utf-8
< content-length: 12
< server: Armeria/1.18.0
< date: Mon, 28 Nov 2022 16:19:54 GMT
<
* Connection #0 to host localhost left intact
output after curl commands:
java -jar /opt/karate/karate.jar -m src/test/java/mocks/fs/john.feature -p 80
16:17:16.671 [main] INFO com.intuit.karate - Karate version: 1.3.0
16:17:17.315 [main] INFO com.intuit.karate - mock server initialized: src/test/java/mocks/fs/john.feature
16:17:17.425 [main] DEBUG com.intuit.karate.http.HttpServer - server started: aad-9mpcfg3:65080
16:19:37.312 [armeria-common-worker-epoll-2-1] DEBUG com.intuit.karate - scenario matched at line 3: pathMatches('john/transactionservice/ping') && methodIs('get')
16:19:54.603 [armeria-common-worker-epoll-2-2] DEBUG com.intuit.karate.http.HttpServer - received command to stop server: /__admin/stop
At this point, netstat on listen port shows socket is closed, but the process continues to run:
ps -ef | grep karate
matt 15070 15069 1 16:17 pts/3 00:00:03 java -jar karate-1.3.0.jar -m src/test/java/mocks/fs/john.feature -p 65080
This is the test mock that I am using (src/test/java/mocks/fs/john.feature):
Feature: JOHN TransactionService mock
Scenario: pathMatches('john/transactionservice/ping') && methodIs('get')
* def response = {}
* set response.message = 'this is the JOHN TransactionService'
* def responseStatus = 200
Question:
Is there something else I should be doing to make the mock process stop? I think I'm following the guidance at https://github.com/karatelabs/karate/tree/master/karate-netty#stopping
Thank you in anticipation of responses.
I´m trying to implement the TLS server feature available with pgBackRest to use a secure connection between the DB server and the repo server, replacing the previous SSH passwordless setup (that was working fine).
After following the online documentation, I´m having the following error when issuing the stanza-create command :
pgbackrest#pgb-repo$ pgbackrest --stanza=training --log-level-console=info stanza-create
2022-06-13 12:56:55.677 P00 INFO: stanza-create command begin 2.39: --buffer-size=16MB --exec-id=8994-62e5ecac --log-level-console=info --log-level-file=info --pg1-host=pg1-primary --pg1-host-ca-file=/etc/pgbackrest/cert/ca.crt --pg1-host-cert-file=/etc/pgbackrest/cert/pg1-primary.crt --pg1-host-key-file=/etc/pgbackrest/cert/pg1-primary.key --pg1-host-type=tls --pg1-host-user=postgres --pg1-path=/data/postgres/13/pg_data --repo1-path=/backup/pgbackrest --stanza=training
WARN: unable to check pg1: [HostConnectError] unable to connect to 'pg1-primary:8432': [113] No route to host
ERROR: [056]: unable to find primary cluster - cannot proceed
HINT: are all available clusters in recovery?
2022-06-13 12:58:55.835 P00 INFO: stanza-create command end: aborted with exception [056]
The PostgreSQL server is up and running on the the DB host:
[postgres#pg1-primary ~]$ psql -c "SELECT pg_is_in_recovery();"
pg_is_in_recovery
-------------------
f
(1 row)
Question
Why am I having this [113] No route to host error ?
Configuration for each server :
pg1-primary
[postgres#pg1-primary ~]$ cat /etc/pgbackrest/pgbackrest.conf
[global]
repo1-path=/backup/pgbackrest
repo1-host-ca-file=/etc/pgbackrest/cert/ca.crt
repo1-host-cert-file=/etc/pgbackrest/cert/pgb-repo.crt
repo1-host-key-file=/etc/pgbackrest/cert/pgb-repo.key
repo1-host-type=tls
tls-server-address=*
tls-server-auth=pgb-repo=training
tls-server-ca-file=/etc/pgbackrest/cert/ca.crt
tls-server-cert-file=/etc/pgbackrest/cert/pg1-primary.crt
tls-server-key-file=/etc/pgbackrest/cert/pg1-primary.key
[postgres#pg1-primary ~]$ cat /etc/pgbackrest/conf.d/training.conf
[training]
pg1-path=/data/postgres/13/pg_data
pg1-socket-path=/tmp
repo1-host=pgb-repo
repo1-host-user=pgbackrest
[postgres#pg1-primary ~]$ ll /etc/pgbackrest/cert/
total 20
-rw-------. 1 postgres postgres 1090 Jun 13 12:12 ca.crt
-rw-------. 1 postgres postgres 977 Jun 13 12:12 pg1-primary.crt
-rw-------. 1 postgres postgres 1708 Jun 13 12:12 pg1-primary.key
-rw-------. 1 postgres postgres 977 Jun 13 12:23 pgb-repo.crt
-rw-------. 1 postgres postgres 1704 Jun 13 12:23 pgb-repo.key
pgb-repo
pgbackrest#pgb-repo$ cat /etc/pgbackrest/pgbackrest.conf
[global]
repo1-path=/backup/pgbackrest
tls-server-address=*
tls-server-auth=pg1-primary=training
tls-server-ca-file=/etc/pgbackrest/cert/ca.crt
tls-server-cert-file=/etc/pgbackrest/cert/pgb-repo.crt
tls-server-key-file=/etc/pgbackrest/cert/pgb-repo.key
pgbackrest#pgb-repo$ cat /etc/pgbackrest/conf.d/training.conf
[training]
pg1-host=pg1-primary
pg1-host-user=postgres
pg1-path=/data/postgres/13/pg_data
pg1-host-ca-file=/etc/pgbackrest/cert/ca.crt
pg1-host-cert-file=/etc/pgbackrest/cert/pg1-primary.crt
pg1-host-key-file=/etc/pgbackrest/cert/pg1-primary.key
pg1-host-type=tls
pgbackrest#pgb-repo$ ll /etc/pgbackrest/cert/
total 20
-rw-------. 1 pgbackrest pgbackrest 1090 Jun 13 12:27 ca.crt
-rw-------. 1 pgbackrest pgbackrest 977 Jun 13 12:27 pg1-primary.crt
-rw-------. 1 pgbackrest pgbackrest 1708 Jun 13 12:27 pg1-primary.key
-rw-------. 1 pgbackrest pgbackrest 977 Jun 13 12:27 pgb-repo.crt
-rw-------. 1 pgbackrest pgbackrest 1704 Jun 13 12:27 pgb-repo.key
The servers are reachable from one another:
[postgres#pg1-primary ~]$ ping pgb-repo
PING pgb-repo.xxxx.com (XXX.XX.XXX.117) 56(84) bytes of data.
64 bytes from pgb-repo.xxxx.com (XXX.XX.XXX.117): icmp_seq=1 ttl=64 time=0.365 ms
64 bytes from pgb-repo.xxxx.com (XXX.XX.XXX.117): icmp_seq=2 ttl=64 time=0.421 ms
pgbackrest#pgb-repo$ ping pg1-primary
PING pg1-primary.xxxx.com (XXX.XX.XXX.116) 56(84) bytes of data.
64 bytes from pg1-primary.xxxx.com (XXX.XX.XXX.116): icmp_seq=1 ttl=64 time=0.325 ms
64 bytes from pg1-primary.xxxx.com (XXX.XX.XXX.116): icmp_seq=2 ttl=64 time=0.298 ms
So actually the issue had to do with the firewall preventing access to the default TLS port (8432) used by pgBackRest.
[root#pgb-server ~]# firewall-cmd --zone=public --add-port=8432/tcp --permanent
[root#pgb-server ~]# firewall-cmd --reload
Once the port was accessible through the firewall I could issue a telnet command successfully (for testing access) - and of course run my pgBackRest commands too.
[pgbackrest#pgb-server]$ telnet pg1-server 8432
Trying 172.XX.XXX.XXX...
Connected to pg1-server.
Escape character is '^]'.
Currently I'm programmatically trying to upload files to a server using the WinSCP API.
However looking at the log files every time I attempt to do this I get an error stating that the remote file or it's directory doesn't exist.
Code:
Dim objftp As New WinSCP.SessionOptions
objftp.HostName = Server
objftp.PortNumber = Port
objftp.Protocol = Protocol.Sftp
objftp.UserName = User
objftp.Password = Password
objftp.PrivateKeyPassphrase = "passPhrase"
objftp.SshPrivateKeyPath = "\\PathToPrivatekey.ppk"
objftp.GiveUpSecurityAndAcceptAnySshHostKey = True
Dim session As New Session
session.Timeout = System.TimeSpan.FromSeconds(30)
session.SessionLogPath = "C:\Temp\ftpLog\winscp.log"
session.Open(objFTP)
Dim transferOptions As New TransferOptions
transferOptions.TransferMode = TransferMode.Binary
transferOptions.ResumeSupport.State = TransferResumeSupportState.Off
Dim transferResult As TransferOperationResult
strLocalFile = FilePath & FileName
strRemoteFile = FileName
transferResult = session.PutFiles(strLocalFile, "/In/", False, transferOptions)
transferResult.Check()
session.Close()
Now I have manually using WinSCP connected to the server and I can verify that the /In/ directory does exist. I am using a Windows OS to do this. I don't have full details on the server I'm connecting to as it's a third party I'm sending information to.
Is there something I'm missing in my code?
EDIT: As requested this is my logs:
Log from programmatically trying it:
. 2016-12-23 10:52:07.089 --------------------------------------------------------------------------
. 2016-12-23 10:52:07.089 WinSCP Version 5.9.3 (Build 7136) (OS 6.1.7601 Service Pack 1 - Windows 7 Professional)
. 2016-12-23 10:52:07.089 Configuration: nul
. 2016-12-23 10:52:07.089 Log level: Normal
. 2016-12-23 10:52:07.089 Local account: user\Andrew
. 2016-12-23 10:52:07.089 Working directory: \\WorkingDIr\
. 2016-12-23 10:52:07.089 Process ID: 12648
. 2016-12-23 10:52:07.090 Command-line: "\\WorkingDir\winscp.exe" /xmllog="C:\Users\andrew\AppData\Local\Temp\wscp46E9.02B7AFA0.tmp" /xmlgroups /xmllogrequired /nointeractiveinput /dotnet=593 /ini=nul /log="C:\Temp\ftpLog\winscp.log" /console /consoleinstance=_10192_57352375_875
. 2016-12-23 10:52:07.090 Time zone: Current: GMT+0, Standard: GMT+0 (GMT Standard Time), DST: GMT+1 (GMT Daylight Time), DST Start: 27/03/2016, DST End: 30/10/2016
. 2016-12-23 10:52:07.090 Login time: 23 December 2016 10:52:07
. 2016-12-23 10:52:07.090 --------------------------------------------------------------------------
. 2016-12-23 10:52:07.094 Script: Retrospectively logging previous script records:
> 2016-12-23 10:52:07.094 Script: option batch on
< 2016-12-23 10:52:07.094 Script: batch on
< 2016-12-23 10:52:07.094 Script: reconnecttime 120
> 2016-12-23 10:52:07.094 Script: option confirm off
< 2016-12-23 10:52:07.094 Script: confirm off
> 2016-12-23 10:52:07.094 Script: option reconnecttime 120
< 2016-12-23 10:52:07.094 Script: reconnecttime 120
> 2016-12-23 10:52:07.094 Script: open sftp://user:***#server.com:22 -hostkey="*" -privatekey="\\containingDir\PrivateKey.ppk" -passphrase=*** -timeout=15
. 2016-12-23 10:52:07.094 --------------------------------------------------------------------------
. 2016-12-23 10:52:07.094 Session name: user#server.com (Ad-Hoc site)
. 2016-12-23 10:52:07.094 Host name: server.com (Port: 22)
. 2016-12-23 10:52:07.095 User name: user(Password: Yes, Key file: Yes, Passphrase: Yes)
. 2016-12-23 10:52:07.095 Tunnel: No
. 2016-12-23 10:52:07.095 Transfer Protocol: SFTP
. 2016-12-23 10:52:07.095 Ping type: Off, Ping interval: 30 sec; Timeout: 15 sec
. 2016-12-23 10:52:07.095 Disable Nagle: No
. 2016-12-23 10:52:07.095 Proxy: None
. 2016-12-23 10:52:07.095 Send buffer: 262144
. 2016-12-23 10:52:07.095 SSH protocol version: 2; Compression: No
. 2016-12-23 10:52:07.095 Bypass authentication: No
. 2016-12-23 10:52:07.095 Try agent: Yes; Agent forwarding: No; TIS/CryptoCard: No; KI: Yes; GSSAPI: No
. 2016-12-23 10:52:07.095 Ciphers: aes,chacha20,blowfish,3des,WARN,arcfour,des; Ssh2DES: No
. 2016-12-23 10:52:07.095 KEX: ecdh,dh-gex-sha1,dh-group14-sha1,rsa,WARN,dh-group1-sha1
. 2016-12-23 10:52:07.095 SSH Bugs: Auto,Auto,Auto,Auto,Auto,Auto,Auto,Auto,Auto,Auto,Auto,Auto
. 2016-12-23 10:52:07.095 Simple channel: Yes
. 2016-12-23 10:52:07.095 Return code variable: Autodetect; Lookup user groups: Auto
. 2016-12-23 10:52:07.095 Shell: default
. 2016-12-23 10:52:07.095 EOL: LF, UTF: Auto
. 2016-12-23 10:52:07.095 Clear aliases: Yes, Unset nat.vars: Yes, Resolve symlinks: Yes; Follow directory symlinks: No
. 2016-12-23 10:52:07.095 LS: ls -la, Ign LS warn: Yes, Scp1 Comp: No
. 2016-12-23 10:52:07.095 SFTP Bugs: Auto,Auto
. 2016-12-23 10:52:07.095 SFTP Server: default
. 2016-12-23 10:52:07.095 Local directory: default, Remote directory: home, Update: Yes, Cache: Yes
. 2016-12-23 10:52:07.095 Cache directory changes: Yes, Permanent: Yes
. 2016-12-23 10:52:07.095 Recycle bin: Delete to: No, Overwritten to: No, Bin path:
. 2016-12-23 10:52:07.095 DST mode: Unix
. 2016-12-23 10:52:07.095 --------------------------------------------------------------------------
. 2016-12-23 10:52:07.106 Looking up host "server.com" for SSH connection
. 2016-12-23 10:52:07.126 Connecting to 11.11.11.11 port 22
. 2016-12-23 10:52:07.307 We claim version: SSH-2.0-WinSCP_release_5.9.3
. 2016-12-23 10:52:07.496 Server version: SSH-2.0-9.99 sshlib: 8.1.0.0
. 2016-12-23 10:52:07.496 We believe remote version has SSH-2 channel request bug
. 2016-12-23 10:52:07.496 Using SSH protocol version 2
. 2016-12-23 10:52:07.504 Doing Diffie-Hellman group exchange
. 2016-12-23 10:52:07.928 Doing Diffie-Hellman key exchange with hash SHA-1
. 2016-12-23 10:52:08.307 Host key fingerprint is:
. 2016-12-23 10:52:08.307 ssh-rsa fingerprint
. 2016-12-23 10:52:08.308 Verifying host key host key
* 2016-12-23 10:52:08.308 WARNING! Giving up security and accepting any host key as configured!
. 2016-12-23 10:52:08.310 Initialised AES-256 SDCTR client->server encryption
. 2016-12-23 10:52:08.310 Initialised HMAC-SHA1 client->server MAC algorithm
. 2016-12-23 10:52:08.310 Initialised AES-256 SDCTR server->client encryption
. 2016-12-23 10:52:08.310 Initialised HMAC-SHA1 server->client MAC algorithm
. 2016-12-23 10:52:08.741 Reading key file "\\PrivateKseyDir\PrivateKey.ppk"
. 2016-12-23 10:52:08.764 Pageant is running. Requesting keys.
. 2016-12-23 10:52:08.766 Pageant has 0 SSH-2 keys
. 2016-12-23 10:52:08.766 Configured key file not in Pageant
! 2016-12-23 10:52:08.766 Using username "user".
. 2016-12-23 10:52:08.958 Server offered these authentication methods: publickey,password
. 2016-12-23 10:52:08.958 Offered public key
. 2016-12-23 10:52:09.193 Offer of public key accepted
! 2016-12-23 10:52:09.193 Authenticating with public key "rsa-key-20161222"
. 2016-12-23 10:52:09.194 Prompt (passphrase, "SSH key passphrase", <no instructions>, "Passphrase for key "rsa-key-20161222": ")
. 2016-12-23 10:52:09.194 Using configured passphrase.
. 2016-12-23 10:52:09.316 Sent public key signature
! 2016-12-23 10:52:09.504 Server refused public-key signature despite accepting key!
. 2016-12-23 10:52:09.504 Server refused public-key signature despite accepting key!
. 2016-12-23 10:52:09.504 Server offered these authentication methods: password
. 2016-12-23 10:52:09.504 Prompt (password, "SSH password", <no instructions>, "&Password: ")
. 2016-12-23 10:52:09.504 Using stored password.
. 2016-12-23 10:52:09.505 Sent password
. 2016-12-23 10:52:09.831 Access granted
. 2016-12-23 10:52:09.831 Opening session as main channel
. 2016-12-23 10:52:10.017 Opened main channel
. 2016-12-23 10:52:10.426 Started a shell/command
. 2016-12-23 10:52:10.426 --------------------------------------------------------------------------
. 2016-12-23 10:52:10.432 Using SFTP protocol.
. 2016-12-23 10:52:10.432 Doing startup conversation with host.
> 2016-12-23 10:52:10.441 Type: SSH_FXP_INIT, Size: 5, Number: -1
< 2016-12-23 10:52:10.631 Type: SSH_FXP_VERSION, Size: 5, Number: -1
. 2016-12-23 10:52:10.631 SFTP version 3 negotiated.
. 2016-12-23 10:52:10.631 We believe the server has signed timestamps bug
. 2016-12-23 10:52:10.631 We will use UTF-8 strings until server sends an invalid UTF-8 string as with SFTP version 3 and older UTF-8 strings are not mandatory
. 2016-12-23 10:52:10.635 Getting current directory name.
. 2016-12-23 10:52:10.639 Getting real path for '.'
> 2016-12-23 10:52:10.639 Type: SSH_FXP_REALPATH, Size: 10, Number: 16
< 2016-12-23 10:52:10.825 Type: SSH_FXP_NAME, Size: 43, Number: 16
. 2016-12-23 10:52:10.825 Real path is '/Home/user'
. 2016-12-23 10:52:10.825 Startup conversation with host finished.
< 2016-12-23 10:52:10.826 Script: Active session: [1] server.com
> 2016-12-23 10:52:11.141 Script: pwd
< 2016-12-23 10:52:11.141 Script: /Home/user
> 2016-12-23 10:52:26.491 Script: put -nopermissions -preservetime -transfer="binary" -resumesupport="off" -- "\\fileDir\testFile.dat" "/In/testFile.dat"
. 2016-12-23 10:52:26.497 Copying 1 files/directories to remote directory "/In/"
. 2016-12-23 10:52:26.497 PrTime: Yes; PrRO: No; Rght: rw-r--r--; PrR: No (No); FnCs: N; RIC: 0100; Resume: N (102400); CalcS: No; Mask: testFile.dat
. 2016-12-23 10:52:26.497 TM: B; ClAr: No; RemEOF: No; RemBOM: No; CPS: 0; NewerOnly: No; InclM: ; ResumeL: 0
. 2016-12-23 10:52:26.497 AscM: *.*html; *.htm; *.txt; *.php; *.php3; *.cgi; *.c; *.cpp; *.h; *.pas; *.bas; *.tex; *.pl; *.js; .htaccess; *.xtml; *.css; *.cfg; *.ini; *.sh; *.xml
. 2016-12-23 10:52:26.502 File: '\\fileDir\testFile.dat' [2016-12-23T06:31:02.639Z] [212952]
. 2016-12-23 10:52:26.504 Copying "\\fileDir\testFile.dat" to remote directory started.
. 2016-12-23 10:52:26.504 Binary transfer mode selected.
. 2016-12-23 10:52:26.505 Opening remote file.
> 2016-12-23 10:52:26.509 Type: SSH_FXP_OPEN, Size: 50, Number: 259
< 2016-12-23 10:52:26.681 Type: SSH_FXP_STATUS, Size: 31, Number: 259
< 2016-12-23 10:52:26.681 Status code: 2, Message: 259, Server: No such file, Language: en
> 2016-12-23 10:52:26.681 Type: SSH_FXP_LSTAT, Size: 34, Number: 519
< 2016-12-23 10:52:26.849 Type: SSH_FXP_STATUS, Size: 31, Number: 519
< 2016-12-23 10:52:26.849 Status code: 2, Message: 519, Server: No such file, Language: en
* 2016-12-23 10:52:26.849 (ETerminal) No such file or directory.
* 2016-12-23 10:52:26.849 Error code: 2
* 2016-12-23 10:52:26.849 Error message from server (en): No such file
. 2016-12-23 10:52:26.850 Asking user:
. 2016-12-23 10:52:26.850 Cannot create remote file '/In/testFile.dat'. ("No such file or directory.
. 2016-12-23 10:52:26.850 Error code: 2
. 2016-12-23 10:52:26.850 Error message from server (en): No such file")
< 2016-12-23 10:52:26.850 Script: Cannot create remote file '/In/testFile.dat'.
< 2016-12-23 10:52:26.850 Script: No such file or directory.
< 2016-12-23 10:52:26.850 Error code: 2
< 2016-12-23 10:52:26.850 Error message from server (en): No such file
* 2016-12-23 10:52:26.862 (EScpSkipFile) Cannot create remote file '/In/testFile.dat'.
* 2016-12-23 10:52:26.862 No such file or directory.
* 2016-12-23 10:52:26.862 Error code: 2
* 2016-12-23 10:52:26.862 Error message from server (en): No such file
. 2016-12-23 10:52:26.862 Script: Failed
And this is the log from me doing it successfully with the GUI:
> 2016-12-23 10:59:12.037 Type: SSH_FXP_SETSTAT, Size: 54, Number: 2057
< 2016-12-23 10:59:12.221 Type: SSH_FXP_STATUS, Size: 40, Number: 2057
< 2016-12-23 10:59:12.221 Status code: 8, Message: 2057, Server: Operation unsupported, Language: en
* 2016-12-23 10:59:12.222 (ETerminal) The server does not support the operation.
* 2016-12-23 10:59:12.222 Error code: 8
* 2016-12-23 10:59:12.222 Error message from server (en): Operation unsupported
. 2016-12-23 10:59:12.226 Asking user:
. 2016-12-23 10:59:12.226 **Upload of file 'sftpTest.dat' was successful, but error occurred while setting the permissions and/or timestamp.**
. 2016-12-23 10:59:12.226
. 2016-12-23 10:59:12.226 If the problem persists, turn off setting permissions or preserving timestamp. Alternatively you can turn on 'Ignore permission errors' option. ("The server does not support the operation.
. 2016-12-23 10:59:12.226 Error code: 8
. 2016-12-23 10:59:12.226 Error message from server (en): Operation unsupported")
* 2016-12-23 10:59:14.385 (EScpSkipFile) **Upload of file 'sftpTest.dat' was successful, but error occurred while setting the permissions and/or timestamp.**
* 2016-12-23 10:59:14.385
* 2016-12-23 10:59:14.385 If the problem persists, turn off setting permissions or preserving timestamp. Alternatively you can turn on 'Ignore permission errors' option.
* 2016-12-23 10:59:14.385 The server does not support the operation.
* 2016-12-23 10:59:14.385 Error code: 8
* 2016-12-23 10:59:14.385 Error message from server (en): Operation unsupported
. 2016-12-23 10:59:14.386 Listing directory "/Home/user".
> 2016-12-23 10:59:14.386 Type: SSH_FXP_OPENDIR, Size: 20, Number: 4875
< 2016-12-23 10:59:14.602 Type: SSH_FXP_HANDLE, Size: 13, Number: 4875
> 2016-12-23 10:59:14.603 Type: SSH_FXP_READDIR, Size: 13, Number: 5132
< 2016-12-23 10:59:14.817 Type: SSH_FXP_NAME, Size: 245, Number: 5132
> 2016-12-23 10:59:14.817 Type: SSH_FXP_READDIR, Size: 13, Number: 5388
< 2016-12-23 10:59:15.022 Type: SSH_FXP_STATUS, Size: 30, Number: 5388
< 2016-12-23 10:59:15.022 Status code: 1
> 2016-12-23 10:59:15.022 Type: SSH_FXP_CLOSE, Size: 13, Number: 5636
. 2016-12-23 10:59:15.022 In;d;0;2016-12-20T15:48:00.000Z;"0" [0];"0" [0];rwxrwxrwx;1
. 2016-12-23 10:59:15.022 sftpTest.dat;-;212952;2016-12-23T10:59:00.000Z;"0" [0];"0" [0];rw-rw-rw-;1
While I cannot tell for sure, as you didn't provide enough information, my guess is that your real target path should be like /Home/user/In/, not just /In/.
From the GUI log, it seems that your account is not chrooted, as the home directory is /Home/user/, not /.
. 2016-12-23 10:59:14.386 Listing directory "/Home/user".
Documentation talks about the full path to the remote file, not the directory with the file. You can try to specify full path:
transferResult = session.PutFiles(strLocalFile, "/In/remote_file.ext", False, transferOptions)
I've created a custom implementation of the PasswdAuthenticationProvider interface, based on OAuth2. I think the code is irrelevant for the problem I'm experiencing, nevertheless, it can be found here.
I've configured hive-site.xml with the following properties:
<property>
<name>hive.server2.authentication</name>
<value>CUSTOM</value>
</property>
<property>
<name>hive.server2.custom.authentication.class</name>
<value>com.telefonica.iot.cosmos.hive.authprovider.OAuth2AuthenticationProviderImpl</value>
</property>
Then I've restarted the Hive service and I've connected a JDBC based remote client with success. This is an example of a successful run found in /var/log/hive/hiveserver2.log:
2016-02-01 11:52:44,515 INFO [pool-5-thread-5]: authprovider.HttpClientFactory (HttpClientFactory.java:<init>(66)) - Setting max total connections (500)
2016-02-01 11:52:44,515 INFO [pool-5-thread-5]: authprovider.HttpClientFactory (HttpClientFactory.java:<init>(67)) - Setting default max connections per route (100)
2016-02-01 11:52:44,799 INFO [pool-5-thread-5]: authprovider.HttpClientFactory (OAuth2AuthenticationProviderImpl.java:Authenticate(65)) - Doing request: GET https://account.lab.fiware.org/user?access_token=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx HTTP/1.1
2016-02-01 11:52:44,800 INFO [pool-5-thread-5]: authprovider.HttpClientFactory (OAuth2AuthenticationProviderImpl.java:Authenticate(76)) - Response received: {"organizations": [], "displayName": "frb", "roles": [{"name": "provider", "id": "106"}], "app_id": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", "email": "frb#tid.es", "id": "frb"}
2016-02-01 11:52:44,801 INFO [pool-5-thread-5]: authprovider.HttpClientFactory (OAuth2AuthenticationProviderImpl.java:Authenticate(104)) - User frb authenticated
2016-02-01 11:52:44,868 INFO [pool-5-thread-5]: thrift.ThriftCLIService (ThriftCLIService.java:OpenSession(188)) - Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V6
2016-02-01 11:52:44,871 INFO [pool-5-thread-5]: session.SessionState (SessionState.java:start(358)) - No Tez session required at this point. hive.execution.engine=mr.
2016-02-01 11:52:44,873 INFO [pool-5-thread-5]: session.SessionState (SessionState.java:start(358)) - No Tez session required at this point. hive.execution.engine=mr.
The problem is after that the following error appears in a recurrent manner:
2016-02-01 11:52:48,227 ERROR [pool-5-thread-4]: server.TThreadPoolServer (TThreadPoolServer.java:run(215)) - Error occurred during processing of message.
java.lang.RuntimeException: org.apache.thrift.transport.TTransportException
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:189)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.thrift.transport.TTransportException
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:182)
at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
... 4 more
2016-02-01 11:53:18,323 ERROR [pool-5-thread-5]: server.TThreadPoolServer (TThreadPoolServer.java:run(215)) - Error occurred during processing of message.
java.lang.RuntimeException: org.apache.thrift.transport.TTransportException
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:189)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.thrift.transport.TTransportException
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:182)
at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
... 4 more
Why? I've seen in several other questions this occurs when using the default value of hive.server2.authentication, i.e. SASL, and the client is not doing the handshake. But in my case, the value of such a property is CUSTOM. I cannot understand it, and any help would be really appreciated.
EDIT 1
I've found there are periodical requests to the HiveServer2... from the HiveServer2 itself! These are the requests that are resulting in Thrift SASL errors:
$ sudo tcpdump -i lo port 10000
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lo, link-type EN10MB (Ethernet), capture size 65535 bytes
...
...
10:18:48.183469 IP dev-fiwr-bignode-11.hi.inet.ndmp > dev-fiwr-bignode-11.hi.inet.55758: Flags [.], ack 7, win 512, options [nop,nop,TS val 1034162147 ecr 1034162107], length 0
^C
21 packets captured
42 packets received by filter
0 packets dropped by kernel
[fiware-portal#dev-fiwr-bignode-11 ~]$ sudo netstat -nap | grep 55758
tcp 0 0 10.95.76.91:10000 10.95.76.91:55758 CLOSE_WAIT 7190/java
tcp 0 0 10.95.76.91:55758 10.95.76.91:10000 FIN_WAIT2 -
[fiware-portal#dev-fiwr-bignode-11 ~]$ ps -ef | grep 7190
hive 7190 1 1 10:10 ? 00:00:10 /usr/java/jdk1.7.0_71//bin/java -Xmx1024m -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/var/log/hadoop/hive -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/lib/hadoop -Dhadoop.id.str=hive -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/lib/hadoop/lib/native/Linux-amd64-64:/usr/lib/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx1024m -Xmx4096m -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /usr/lib/hive/lib/hive-service-0.13.0.2.1.7.0-784.jar org.apache.hive.service.server.HiveServer2 -hiveconf hive.metastore.uris=" " -hiveconf hive.log.file=hiveserver2.log -hiveconf hive.log.dir=/var/log/hive
1011 14158 12305 0 10:19 pts/1 00:00:00 grep 7190
Any idea?
EDIT 2
More research about the connections sent from HiveServer2 to HiveServer2. Data packets always sent 5 bytes, the following ones (hexadecimal): 22 41 30 30 31
Any idea about these connections?
I finally "fixed" this. Since the message was sent by the Ambari agent running in the HiveServer2 machine (some king of weird ping), I simply added an iptables rule blocking all the connections to TCP/10000 port on the loopback interface:
iptables -A INPUT -i lo -p tcp --dport 10000 -j DROP
Of course, now Ambari warns the HiveServer2 is not alive (the pings are droped). And the above rule must be removed if I want to restart the server from Ambari (there is another alive check in the starting script); then after the restart I can enable the rule again. Well, I can live with that.
Im using Apache FtpServer and built my own ftp server. I`m using this server on a VM (VPS). Im able to login and list the files however, Im couldn't download or upload files. Every time I try I get the error
220 Service ready for new user.
USER testuser
331 User name okay, need password for testuser.
PASS ********
230 User logged in, proceed.
FEAT
211-Extensions supported
SIZE
MDTM
REST STREAM
LANG en;zh-tw;ja;is
MLST Size;Modify;Type;Perm
AUTH SSL
AUTH TLS
MODE Z
UTF8
TVFS
MD5
MMD5
MFMT
211 End
OPTS UTF8 ON
200 Command OPTS okay.
SYST
215 UNIX Type: Apache FtpServer
STAT /
212-
drwx------ 3 user group 0 Oct 25 02:46 lost+found
drwx------ 3 user group 0 Oct 25 06:27 v
-rw------- 1 user group 4496 Oct 25 06:30 working.txt
212 End of status.
TYPE I
200 Command TYPE okay.
PORT 10,9,226,222,220,45
200 Command PORT okay.
STOR /Records.xls
150 File status okay; about to open data connection.
220 Service ready for new user.
When I run the code on localhost, I can download, modify and upload files. Why this isn't working on a VM?