I am using com.jcraft.jsch version 0.1.55 and connecting to a number of Linux servers, one at a time. Some server connections are successful, while others are not.
Ostensibly the code in the servers is identical. I need to find out why the different behaviour.
Client code. Set up
JSch jsch = new JSch();
Session session = jsch.getSession("username", ip_address, 22);
session.setPassword("password");
session.setConfig("StrictHostKeyChecking","no");
session.connect()
Hangs on calling UserAuthGSSAPIWithMIC.start()
Working flow:
Session.start()
line 22 super.start(session) sets buf[5] = 51
line 25 this.buf.putByte((byte)50) sets buf[5] = 50
flow continues to line 39 this.buf = session.read(this.buf) which returns
and has set buf[5] to 51 forcing the auth to move on the next auth method.
Eventually succeeds using the password method
Broken flow:
Session.start()
super.start() sets buf[5] = 51
this.buf.putByte((byte)50) sets buf[5] = 50
flow continues to line 39 this.buf = session.read(this.buf) which returns
and has set buf[5] to 60 which sends the code to create a new context,
part of which calls oid.equals() which hangs on line 160 -
return this.oid.equals.((Object) ((Oid) other).oid);
There are no log messages in this part of the code, so no information about what the code is doing.
I can make the problem go away by setting session.setConfig("PreferredAuthentications", "password");
I want to know why the code is hanging on the UserAuthGSSAPIWithMIC on some servers but not others. Also how I can get some informative logging information.
Related
My application(Javascript + html) posts a CSV file with ajaxForm and does something on the server side, then the server returns a json data which indicates whether it succeeds or not.
When the CSV file is small, the server processed it within a few minutes. In this case, the browser returns a response as expected; however, if the CSV file is large and it takes 5 or 6 seconds to process it. In this case, the application does not receive anything.
I checked the message on the console, but couldn't see anything wrong.
I set the timeout to 0 as follows:
48 var options5 = {
49 dataType: 'json',
50 success: processRegister,
51 error: processError,
52 timeout: 0
53 };
54 $('#form_user_register').ajaxForm(options5);
Could someone please give me some advice to tackle this issue?
Also, what is the best way to handle a request which would take a long time (but want to inform its result to a client side)?
I have created a Java-based LSP client, but none of the method calls are completed & it waits indefinitely.
Socket socket = new Socket("localhost", 6008);
Launcher<LanguageServer> createClientLauncher = LSPLauncher.createClientLauncher (languageClient,
socket.getInputStream(), socket.getOutputStream());
LanguageServer server = createClientLauncher.getRemoteProxy();
createClientLauncher.startListening();
InitializeResult result = server.initialize(new InitializeParams()).get();
System.out.println("end");
initialize method never returns. The Language Server is working fine when tested with the VSCode instance.
Seems like requests are not reaching the server as nothing is printed in trace logs of server.
I'm maintaining an antedeluvian Notes application which connects to a SAP back-end via a manually done 'Webservice'
The server is running Domino Release 7.0.4FP2 HF97.
The Webservice is not the more recently Webservice Consumer, but a large Java agent which is using Apache soap.jar (org.apache.soap). Below an example of the calling code.
private Call setupSOAPCall() {
Call call = new Call();
SOAPHTTPConnection conn = new SOAPHTTPConnection();
call.setSOAPTransport(conn);
call.setEncodingStyleURI(Constants.NS_URI_SOAP_ENC);
There has been a change in the SAP system which is now taking 8 minutes to complete (verified by SAP Team).
I'm getting an error message as follows:
[SOAPException: faultCode=SOAP-ENV:Client; msg=For input string: "906 "; targetException=java.lang.NumberFormatException: For input string: "906 "]
I found a blog article describing the error message quite closely:
https://thejavablog.wordpress.com/category/jmeter/
and I've come to the hypothesis that it is a timeout message that is returning to my Call object and that this timeout message is being incorrectly parsed, hence the NumberFormat Exception.
Looking at my logs I can see that there is a time difference of 62 seconds between my call and the response.
I recommended that the server setting in the server document, tab Internet Protocols/HTTP/Timeouts/Request timeouts be changed from 60 seconds to 600 seconds, and the http task restarted with
tell http restart
I've re-run the tests and I am getting the same error, and the time difference is still slightly more than 60 seconds, which is not what I was expecting.
I read Michael Rulnau's blog entry
http://www.mruhnau.net/2014/06/how-to-overcome-domino-webservice.html
which points to this APR
http://www-01.ibm.com/support/docview.wss?uid=swg1LO48272
but I'm not convinced that this would apply in this case, since there is no way that IBM would know that my Java agent is in fact making a Soap call.
My current hypothesis is that I have to use either the setTimeout() method on
org.apache.axis.client.Call
https://axis.apache.org/axis/java/apiDocs/org/apache/axis/client/Call.html
or on the org.apache.soap.transport.http.SOAPHTTPConnection
https://docs.oracle.com/cd/B13789_01/appdev.101/b12024/org/apache/soap/transport/http/SOAPHTTPConnection.html
and that the timeout value is an apache default, not something that is controlled by the Domino server.
I'd be grateful for any help.
I understand your approach, and I hope this is the correct one to solve your problem.
Add a debug (console write would be fine) that display the default Timeout then try to increase it to 10 min.
SOAPHTTPConnection conn = new SOAPHTTPConnection();
System.out.println("time out is :" + conn.getTimeout());
conn.setTimeout(600000);//10 min in ms
System.out.println("after setting it, time out is :" + conn.getTimeout());
call.setSOAPTransport(conn);
Now keep in mind that Dommino has also a Max LotusScript/Java execution time, check this value and (at least for a try) change it: http://www.ibm.com/support/knowledgecenter/SSKTMJ_9.0.1/admin/othr_servertasksagentmanagertab_r.html (it's version 9 help but this part should be identical)
I've since discovered that it wasn't my code generating the error; the default timeout for the apache axis SOAPHTTPConnetion is 0, i.e. no timeout.
I have an application that I have started developing that monitors websocket messages from all clients connected to the websocket server by relaying all messages received from the server to this application.
Problem
When I run my program (In visual studio I hit Start), it builds and starts up perfectly, and does most of the functionality the same everytime. However, I have a common occurance of a portion of code that will not run the same. Below is the small snippet of that code.
msg = "set name monitor"
SendMessage2(socket, msg, msg.Length)
msg = "set monitor 1"
SendMessage2(socket, msg, msg.Length)
Console.WriteLine("We are after our second SendMessage2 function")
I know that the two calls to SendMessage2 are always executed because visual studio's debug console will output the following
We are at the end of the SendMessage2 Sub
We are at the end of the SendMessage2 Sub
We are after our second SendMessage2 function
I also know when it executes correctly because my websocket server will either output one of the two blocks
Output when app runs correctly
Client 4 connected
New thread created
Connection received. Parsing headers.
Message from socket #4: "set name monitor"
Message from socket #4: "set monitor 1"
Output when app runs incorrectly
Client 4 connected
New thread created
Connection received. Parsing headers.
Message from socket #4: "set name monitor"
Notice how the second output is missing the second message from the monitor application.
What have I tried
Using a string variable to call the functions
Calling the functions using static string arguments (not using the variable msg)
SyncLocking the functions separately
SyncLocking inside the SendMessage2 function
Reordering the functions (swapping the strings to change behavior)
TL;DR
Why is it that even when I do not change my code, my program will execute two separate ways? Am I doing something incorrectly when calling my SendMessage2 Sub?
I am all out of ideas. I am willing to try any recommendation to fix this problem.
All code can be found on GitHub here
So I figured it out.
It is actually not the VB application that is messing up. Nor was my server. While debugging I was looking at the number of bytes received by my server and I noticed the following:
Client 4 connected
New thread created
Connection received. Parsing headers.
bytes read: 25
Message from socket #4: "set name monitor"
bytes read: 22
Message from socket #4: "set monitor 1"
Ok great we have 25 bytes from set name monitor and 22 bytes from set monitor 1
Client 4 connected
New thread created
Connection received. Parsing headers.
bytes read: 47
Message from socket #4: "set name monitor"
And boom. Both programs were doing their jobs, sending the correct number of bytes every time and reading the correct number. However, the VB application is sending them so quickly back to back that my server was reading all 47 bytes at a time instead of the separate 25 and 22 bytes.
Solution
I solved this problem by implementing a secondary buffer in my server to store off all bytes after the first message should multiple messages by group like this. Now I check if my secondaryBuffer is empty before reading in new bytes.
Here is a portion of the code used to solve the problem
/*Byte Check*/
for (j=0; j < bytes; j++) {
if (j == 0)
continue;
if (readBuffer[j] == '\x81' && readBuffer[j-1] == '\x00' && readBuffer[j-2] == '\x00') {
secondaryBytes = bytes - j;
printf("Potential second message attached to this message\nCopying it to the secondary buffer.\n");
memcpy(secondaryBuffer, readBuffer + j, secondaryBytes);
break;
}//END IF
}//END FOR LOOP
/**/
I am calling 5 external servers to retrieve XML-based data for each request for a particular webpage on my IIS 6 server. Present volume is between 3-5 incoming requests per second, meaning 15-20 outgoing requests per second.
99% of the outgoing requests from my server (the client) to the external servers (the server) work OK but about 100-200 per day end up with a "The operation has timed out" exception.
This suggests I have a resource problem on my server - some shortage of sockets, ports etc or a thread lock but the problem with this theory is that the failures are entirely random - there are not a number of requests in a row that all fail - and two of the external servers account for the majority of the failures.
My question is how can I further diagnose these exceptions to determine if the problem is on my end (the client) or on the other end (the servers)?
The volume of requests precludes putting an analyzer on the wire - it would be very difficult to capture these few exceptions. I have reset CONNECTIONS and THREADS in my machine.config and the basic code looks like:
Dim hRequest As HttpWebRequest
Dim responseTime As String
Dim objWatch As New Stopwatch
Try
' calculate time it takes to process transaction
objWatch.Start()
hRequest = System.Net.WebRequest.Create(url)
' set some defaults
hRequest.Timeout = 5000
hRequest.ReadWriteTimeout = 10000
hRequest.KeepAlive = False ' to prevent open HTTP connection leak
hRequest.SendChunked = False
hRequest.AllowAutoRedirect = True
hRequest.MaximumAutomaticRedirections = 3
hRequest.Accept = "text/xml"
hRequest.Proxy = Nothing 'do not waste time searching for a proxy
hRequest.ServicePoint.Expect100Continue = False
Dim feed As New XDocument()
' use *Using* to auto close connections
Using hResponse As HttpWebResponse = DirectCast(hRequest.GetResponse(), HttpWebResponse)
Using reader As XmlReader = XmlReader.Create(hResponse.GetResponseStream())
feed = XDocument.Load(reader)
reader.Close()
End Using
hResponse.Close()
End Using
objWatch.Stop()
' Work here with returned contents in "feed" document
Return XXX' some results here
Catch ex As Exception
objWatch.Stop()
hRequest.Abort()
Return Nothing
End Try
Any suggestions?
By default, HttpWebRequest limits you to 2 connections per HTTP/1.1 server. So, if your requests take time to complete, and you have incoming requests queuing up on the server, you will run out of connection and thus get timeouts.
You should change the max outgoing connections on ServicePointManager.
ServicePointManager.DefaultConnectionLimit = 20 // or some big value.
You said that you are doing 5 outgoing request for each incoming request to the ASP page. Is that 5 different servers, or the same server?
DO you wait for the previous request to complete, before issuing the next one? Is the timeout happening while it is waiting for a connection, or during the request/response?
If the timeout is happening during the request/response then it means that the target server is under stress. The only way to find out if this is the case, is to run wireshark/netmon on one of the machines, and look at the network trace to see if the request from the app is even making it through to the server, and if it is, whether the target server is responding within the given timeout.
If this is a thread starvation issue, then one of the ways to diagnose it is to attach windbg.exe debugger to w3wp.exe process, when you start getting timeout. Then load the sos.dll debugging extension. And run the !threads command, followed by !threadpool command. It will show you how many Worker threads and completion port threads are utilized/remaining. If the #completionport threads or worker threads are low, then that will contribute to the timeout.
Alternatively, you can monitor ASP.NET and System.net perf counters. See if the ASP.NET request queue is increasing monotonically - this might indicate that your outgoing requests are not completing fast enough.
Sorry, there are no easy answers here. THere is a lot of avenues you will need to explore. If I were you, I would start off by attaching windbg.exe to w3wp when you start getting timeouts and do what I described earlier.