Kestrel/IIS extremely slow on .NET 4.6.2 - asp.net-core

I have a NET 4.6.2 MVC (Frontend) which talks to a API (backend) via Rest and all of a sudden has become extremely slow on our test server.
Other applications on the server appear fine, and running this application locally it's blinding fast.
I've tested the API at it's end-points and the response time is <10ms (Caching), using NLog, I've got some results from Kestrel and it appears to be stalling at an incoming request.
Looking at the logs, there is a clear 3 seconds where the request comes in and nothing happens.
I'm pretty much lots of ideas here - Could it be a server issue? As it works fine locally?
2016-12-23 16:34:53.5251|1|Microsoft.AspNetCore.Hosting.Internal.WebHost|INFO|Request starting HTTP/1.1 GET http://myApplication.myTestServer.com/Request/Create
2016-12-23 16:34:57.3845|1|Microsoft.AspNetCore.Routing.RouteConstraintMatcher|DEBUG|Route value 'Request' with key 'pageNumber' did not match the constraint 'Microsoft.AspNetCore.Routing.Constraints.IntRouteConstraint'.
2016-12-23 16:34:57.3845|1|Microsoft.AspNetCore.Routing.RouteBase|DEBUG|Request successfully matched the route with name 'default' and template '{controller=Home}/{action=Index}/{id?}'.
2016-12-23 16:34:57.3845|2|Microsoft.AspNetCore.Mvc.Internal.ActionSelector|DEBUG|Action 'myApplication.Controllers.RequestController.Create (myApplication)' with id 'de2f360e-a000-4ff0-94ea-2cff3afb5af1' did not match the constraint 'Microsoft.AspNetCore.Mvc.Internal.HttpMethodActionConstraint'
2016-12-23 16:34:57.3845|1|Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker|DEBUG|Executing action myApplication.Controllers.RequestController.Create (myApplication)
2016-12-23 16:34:57.3845|1|Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker|INFO|Executing action method myApplication.Controllers.RequestController.Create (myApplication) with arguments ((null)) - ModelState is Valid
2016-12-23 16:34:57.4001|2|Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker|DEBUG|Executed action method myApplication.Controllers.RequestController.Create (myApplication), returned result Microsoft.AspNetCore.Mvc.ViewResult.
2016-12-23 16:34:57.4001|2|Microsoft.AspNetCore.Mvc.Razor.RazorViewEngine|DEBUG|View lookup cache hit for view 'Create' in controller 'Request'.
2016-12-23 16:34:57.4001|2|Microsoft.AspNetCore.Mvc.ViewFeatures.Internal.ViewResultExecutor|DEBUG|The view 'Create' was found.
2016-12-23 16:34:57.4001|1|Microsoft.AspNetCore.Mvc.ViewFeatures.Internal.ViewResultExecutor|INFO|Executing ViewResult, running view at path /Views/Request/Create.cshtml.
2016-12-23 16:34:57.4001|5|Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector|TRACE|Performing unprotect operation to key {0c8426b3-7550-4bd9-9f7a-039549bbdf89} with purposes ('C:\Octopus\Applications\UK - Test\myApplication\2016.12.23.163307', 'Microsoft.AspNetCore.Antiforgery.AntiforgeryToken.v1').
2016-12-23 16:34:57.4001|31|Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector|TRACE|Performing protect operation to key {0c8426b3-7550-4bd9-9f7a-039549bbdf89} with purposes ('C:\Octopus\Applications\UK - Test\myApplication\2016.12.23.163307', 'Microsoft.AspNetCore.Antiforgery.AntiforgeryToken.v1').
2016-12-23 16:34:57.4001|6|Microsoft.AspNetCore.Antiforgery.Internal.DefaultAntiforgery|DEBUG|An antiforgery cookie token was reused.
2016-12-23 16:34:57.4157|2|Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker|INFO|Executed action myApplication.Controllers.RequestController.Create (myApplication) in 25.6799ms
2016-12-23 16:34:57.4157|9|Microsoft.AspNetCore.Server.Kestrel|DEBUG|Connection id "0HL1BA65KRN93" completed keep alive response.
2016-12-23 16:34:57.4157|2|Microsoft.AspNetCore.Hosting.Internal.WebHost|INFO|Request finished in 3891.2205ms 200 text/html; charset=utf-8

Related

RTSP: Not receiving SDP from the server after sending "describe" request

I have a Bosch camera(server) and my end goal is to get the video content description via metadata from it. I am using LwIP Raw API's(1.4.0) for this purpose. At present, I am trying to authenticate with the camera and receive the SDP so I can setup the session. However, after I authenticate by resending the describe request with the digest, I don't get any response from the server and after a while the server resets the connection. Below is the sequence of operations I perform for authentication.
Step 1: Client to Server (mcu sends 1st describe request)
DESCRIBE rtsp://service:PRBUWPCs7*f40j#192.168.1.129/?enablevideo=0&vcd=1 RTSP/1.0
CSeq: 1
User-Agent: rtsp://service:PRBUWPCs7*f40j#192.168.1.129(LIVE555 Streaming Media v2018.02.28)
Accept: application/sdp
Step 2: Server to Client (server responds with nonce for authentication, rx via callback)
Payload:RTSP/1.0 401 Unauthorized
CSeq: 1
WWW-Authenticate: Digest realm="Please log in with a valid
username",nonce="7bd251bb670e45966c415838679f778f",opaque="",stale=FALSE,algorithm=MD5
Step 3: Client to Server (mcu computes the response and resends the describe command )
DESCRIBE rtsp://service:PRBUWPCs7*f40j#192.168.1.129/?enablevideo=0&vcd=1 RTSP/1.0
CSeq: 2
Authorization: Digest username="service", realm="Please log in with a valid username", nonce="7bd251bb670e45966c415838679f778f", uri="rtsp://service:PRBUWPCs7*f40j#192.168.1.129/?enablevideo=0&vcd=1", response="4c87974de2e3ecc3d534beddef9e6962"
User-Agent: rtsp://service:PRBUWPCs7*f40j#192.168.1.129(LIVE555 Streaming Media v2018.02.28)
Accept: application/sdp
Step 4: mcu waiting for SDP, but instead receives pbuf *p as null in the receive call back function.
After a few seconds, also receives a tcp err callback with err code ERR_RST i.e. connection reset.
Could anyone please clarify if my above procedure is correct and if so, any insights on what could likely cause the camera not to respond with the SDP description leading to connection reset and receiving pbuff as NULL in the receive callback? 
Fixed it. There was an issue with md5 module.

SSLError during Robot Framework tests

I have a problem with RF tests. I have built tests with TLS and certificates using "Create Client Cert Session" keywords. I have a specific keywords to do this:
Start Session
Log Creating HTTPS Session
#{client_certs}= Create List ${CERT} ${PRIVKEY}
Create Client Cert Session ${SESSION} ${HOST} client_certs=#{client_certs} verify=${CA} debug=1 timeout=20 disable_warnings=1
Log HTTPS Session has been created!
where ${CERT} and ${PRIVKEY} are respectively the certificate and the private key paths, ${SESSION} is the alias, ${HOST} is the hostname and ${CA} is the certificate authority. I execute this keyword as suite initialization.
This is an example of test:
Test
&{headers}= Create Dictionary &{contentTypeDict}
${response}= Get Request ${SESSION} ${BASEURL}/${endPoint} headers=${headers}
Status Should Be ${expectedResponse} ${response}
When this test is performed, Robot Framework raises the following error:
Test 'Test' failed after retrying for 30 seconds. The last error was: SSLError:
HTTPSConnectionPool(host='myhostname', port=8080): Max retries exceeded with url: /url/to/resource
(Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:852)'),))
This problem doesn't always happen, but sometimes tests are successful and sometimes they fail. What could be the problem? Certificates are correct and have not expired (I check them before executing tests).

IBM WebSphere Portal V8.5 wcm library syndication

I have a WebSphere Portal Version 8.5 Cluster on AIX 7.1 with multiple Virtual Portals, working with managed pages and each Virtual Portal has it's own libraries and one shared library for all VPs using syndication of that library to each VP.
i successfully created the syndication pair between the syndicator (WAS base portal) and the subscriber (Virtual Portal) and tested connection between them and all is good (make sense since VP are local on the same server). however when trying to syndicate the library content it stays on Queued status and in the SystemOut.log i see the following error log:
[4/25/17 9:33:53:201 IDT] 00004163 PackageConsum E Unexpected exception thrown while updating subscription: [IceId: Current State: ], exception: com.ibm.workplace.wcm.services.WCMServiceRuntimeException: code: 400
com.ibm.workplace.wcm.services.WCMServiceRuntimeException: code: 400
at com.aptrix.syndication.business.subscriber.CatalogRetrieverTask.getSourceCatalog(CatalogRetrieverTask.java:330)
at com.aptrix.syndication.business.subscriber.CatalogRetrieverTask.process(CatalogRetrieverTask.java:144)
at com.aptrix.syndication.business.subscriber.PackageConsumerTask.processPackage(PackageConsumerTask.java:513)
at com.aptrix.syndication.business.subscriber.PackageConsumerTask.processUpdate(PackageConsumerTask.java:267)
at com.aptrix.syndication.business.subscriber.PackageConsumerTask$1.run(PackageConsumerTask.java:183)
at com.ibm.wps.ac.impl.UnrestrictedAccessImpl.run(UnrestrictedAccessImpl.java:84)
at com.ibm.wps.command.ac.ExecuteUnrestrictedCommand.execute(ExecuteUnrestrictedCommand.java:90)
at com.aptrix.syndication.business.subscriber.PackageConsumerTask.doManagedWork(PackageConsumerTask.java:195)
at com.aptrix.syndication.business.ManagedTask.runWork(ManagedTask.java:62)
at com.ibm.workplace.wcm.services.workmanager.AbstractWcmWork.runImpl(AbstractWcmWork.java:162)
at com.ibm.workplace.wcm.services.workmanager.AbstractWcmSystemWork.access$001(AbstractWcmSystemWork.java:40)
at com.ibm.workplace.wcm.services.workmanager.AbstractWcmSystemWork$1.run(AbstractWcmSystemWork.java:92)
at com.ibm.wps.ac.impl.UnrestrictedAccessImpl.run(UnrestrictedAccessImpl.java:84)
at com.ibm.wps.command.ac.ExecuteUnrestrictedCommand.execute(ExecuteUnrestrictedCommand.java:90)
at com.ibm.workplace.wcm.services.repository.PACServiceImpl.runAsPrivileged(PACServiceImpl.java:1878)
at com.ibm.workplace.wcm.services.workmanager.AbstractWcmSystemWork.runImpl(AbstractWcmSystemWork.java:87)
at com.ibm.workplace.wcm.services.workmanager.AbstractWcmWork.run(AbstractWcmWork.java:146)
at com.ibm.wps.services.workmanager.impl.WasWorkWrapper.run(WasWorkWrapper.java:44)
at com.ibm.ws.asynchbeans.J2EEContext$RunProxy.run(J2EEContext.java:271)
at java.security.AccessController.doPrivileged(AccessController.java:274)
at com.ibm.ws.asynchbeans.J2EEContext.run(J2EEContext.java:797)
at com.ibm.ws.asynchbeans.WorkWithExecutionContextImpl.go(WorkWithExecutionContextImpl.java:222)
at com.ibm.ws.asynchbeans.ABWorkItemImpl.run(ABWorkItemImpl.java:206)
at java.lang.Thread.run(Thread.java:804)
[4/25/17 9:33:53:222 IDT] 00004163 SyndicationEx W Unsuccessful request to send summary: 400
com.aptrix.deployment.wizard.SyndicatorCommunicationException: Unsuccessful request to send summary: 400
at com.ibm.workplace.wcm.api.syndication.SyndicationExtensionsServiceImpl.sendSummaryToSyndicator(SyndicationExtensionsServiceImpl.java:293)
at com.ibm.workplace.wcm.api.syndication.SyndicationExtensionsServiceImpl.processSubscriberCompleting(SyndicationExtensionsServiceImpl.java:246)
at com.aptrix.syndication.business.subscriber.SubscriberTaskManager.processFailedUpdate(SubscriberTaskManager.java:405)
at com.aptrix.syndication.business.subscriber.PackageConsumerTask.processUpdate(PackageConsumerTask.java:400)
at com.aptrix.syndication.business.subscriber.PackageConsumerTask$1.run(PackageConsumerTask.java:183)
at com.ibm.wps.ac.impl.UnrestrictedAccessImpl.run(UnrestrictedAccessImpl.java:84)
at com.ibm.wps.command.ac.ExecuteUnrestrictedCommand.execute(ExecuteUnrestrictedCommand.java:90)
at com.aptrix.syndication.business.subscriber.PackageConsumerTask.doManagedWork(PackageConsumerTask.java:195)
at com.aptrix.syndication.business.ManagedTask.runWork(ManagedTask.java:62)
at com.ibm.workplace.wcm.services.workmanager.AbstractWcmWork.runImpl(AbstractWcmWork.java:162)
at com.ibm.workplace.wcm.services.workmanager.AbstractWcmSystemWork.access$001(AbstractWcmSystemWork.java:40)
at com.ibm.workplace.wcm.services.workmanager.AbstractWcmSystemWork$1.run(AbstractWcmSystemWork.java:92)
at com.ibm.wps.ac.impl.UnrestrictedAccessImpl.run(UnrestrictedAccessImpl.java:84)
at com.ibm.wps.command.ac.ExecuteUnrestrictedCommand.execute(ExecuteUnrestrictedCommand.java:90)
at com.ibm.workplace.wcm.services.repository.PACServiceImpl.runAsPrivileged(PACServiceImpl.java:1878)
at com.ibm.workplace.wcm.services.workmanager.AbstractWcmSystemWork.runImpl(AbstractWcmSystemWork.java:87)
at com.ibm.workplace.wcm.services.workmanager.AbstractWcmWork.run(AbstractWcmWork.java:146)
at com.ibm.wps.services.workmanager.impl.WasWorkWrapper.run(WasWorkWrapper.java:44)
at com.ibm.ws.asynchbeans.J2EEContext$RunProxy.run(J2EEContext.java:271)
at java.security.AccessController.doPrivileged(AccessController.java:274)
at com.ibm.ws.asynchbeans.J2EEContext.run(J2EEContext.java:797)
at com.ibm.ws.asynchbeans.WorkWithExecutionContextImpl.go(WorkWithExecutionContextImpl.java:222)
at com.ibm.ws.asynchbeans.ABWorkItemImpl.run(ABWorkItemImpl.java:206)
at java.lang.Thread.run(Thread.java:804)
[4/25/17 9:33:53:227 IDT] 00004163 syndication I Syndication Summary - Subscriber
Syndicator: IntShared_Syn, URL=http://'Was_Server':10039/wps/wcm/connect?MOD=Synd
Subscriber: IntShared_Sub, URL=http://'Was_Server':10039/wps/wcm/connect/'VP_URL_Context'?MOD=Subs
Status: FAILED
Failure Detail: Update failed on subscriber
Unexpected exception thrown while updating subscription: [IceId: Current State: ], exception: com.ibm.workplace.wcm.services.WCMServiceRuntimeException: code: 400
Update Type: REBUILD
Start Date: Tue Apr 25 09:33:53 IDT 2017
Finished Date: Tue Apr 25 09:33:53 IDT 2017
Duration:
Total: 0
Total Failed: 0
[4/25/17 9:33:54:613 IDT] 00000136 syndication I Syndication Summary - Syndicator
Syndicator: IntShared_Syn, URL=http://'Was_Server':10039/wps/wcm/connect?MOD=Synd
Subscriber: IntShared_Sub, URL=http://'VP_HostName':10039/wps/wcm/connect?MOD=Subs
Status: FAILED
Failure Detail: Terminated without confirmation
Returned non-confirmed response: Not confirmed. Unable to contact subscriber. Check the subscriber to ensure it is active and error free. Also review your network connections and your syndication configuration to ensure the subscriber details are correct.
Update Type: REBUILD
Start Date: Tue Apr 25 09:33:53 IDT 2017
Finished Date: Tue Apr 25 09:33:54 IDT 2017
Duration: 1 second
Total: 0
Total Failed: 0
WCM Syndication requires HTTP Basis Authentication to be configured and working.
then I needed to make sure that Trust Association is enabled in WAS Console under Security -> Global Security -> Web and SIP security -> Trust association.
confirmed that the box that says Enable trust association is checked.
also ensured the Interceptor com.ibm.portal.auth.tai.HTTPBasicAuthTAI is created and the configuration were correct.
the cause of the error was that in the fields of urlBlackList and urlWhiteList there was use of the variable ${WpsContextRootPath} which i found out that it is not set anywhere so i change it to /wps instead and now the fields are as follow:
urlBlackList = /wps/myportal*
urlWhiteList = /wps/mycontenthandler*
after Restarting the server and retry syndication - it works!.
also you may follow the direction in this link:
https://developer.ibm.com/answers/questions/206675/why-do-i-see-occasionally-see-a-popup-box-with-a-t.html
but setting these parameters disabled the servlet of vieweing all items in the libraries...
You can try using the ip address instead of the hostname. or Try adding the VP context to the syndicator/subscriber URLs.

Apache, mod_ssl "request failed: error reading the headers" for a specific user

Currently we have an Apache 2.2.3 server with mod_ssl 2.2.3 running Django, with users authenticating by using a x509 certificate.
So far the system is running perfectly except for a single user, who when trying to upload a file receives 400 Bad Request error, and the contents of the ssl_error_log regarding this operation are:
[<date>] [error] [client <client ip>] request failed: error reading the headers, referer: <referrer url>
The contents of the ssl_access_log are:
<client ip> - - [<date>] "POST <target page> HTTP/1.1" 400 321
Also, the user's browser is Firefox as far as I know.
I am completely unable to reproduce this bug and so far none of the other users have experienced it. Could you point out some reasons for this to happen?
I've experienced connectivity that stops the upstream after an X amount of bytes is sent. X was a pretty low value, as in enough to request some simple pages, but not to deal with ajax requests much less upload files. As far as I recall, this connectivity problem occurred only when tethering (from a specific Android phone, but I didnt even test other phones).
So if the upstream gets interrupted and the upload stalls, it makes sense apache would return this error, according to this post: "Apache waits a time equal to the Timeout directive (defaults to 5 minutes if not defined) for a response from the client. It is likely Apache is waiting for the CRLF that indicates the end of the headers, yet it is never received.."

LOCK PUT UNLOCK with cURL / WebDAV

My idea is to LOCK a file on an apache/WebDAV server, PUT an updated version of it on the server and UNLOCK it afterwards.
I just tried the following with cadaver:
create a file A.txt with content a file
GET file A.txt which yields a file
edit A.txt to be updated file and save it (in cadaver)
GET file A.txt which yields still yields a file
close edit (VIM) in cadaver
GET file A.txt which yields updated file
I guess internally cadaver LOCKs the file, GETs it and changes it locally. Then it PUTs it and UNLOCKs it.
QUESTION: how can I do this with curl?
PROBLEM: When then connection is slow and I do a PUT for a file, that is not yet completely uploaded, I only get the yet uploaded part. I would like to get the old one as long as the new one isn't complete.
TRIED: I tried the following to LOCK the file by hand (i.e. with cURL):
curl -v -X LOCK --user "user:password" http://myServer/newFile
What I get is:
* About to connect() to myServer port 80 (#0)
* Trying xx.xx.xxx.xxx... connected
* Connected to myServer (xx.xx.xxx.xxx) port 80 (#0)
* Server auth using Basic with user 'user'
> LOCK /newFile HTTP/1.1
> Authorization: Basic xxxxxxxxxxxxxxxxx
> User-Agent: curl/7.21.6 (x86_64-pc-linux-gnu) libcurl/7.21.6 OpenSSL/1.0.0e zlib/1.2.3.4 libidn/1.22 librtmp/2.3
> Host: myServer
> Accept: */*
>
< HTTP/1.1 400 Bad Request
< Date: Wed, 02 May 2012 15:20:55 GMT
< Server: Apache/2.2.3 (CentOS)
< Content-Length: 226
< Connection: close
< Content-Type: text/html; charset=iso-8859-1
<
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>400 Bad Request</title>
</head><body>
<h1>Bad Request</h1>
<p>Your browser sent a request that this server could not understand.<br />
</p>
</body></html>
* Closing connection #0
Looking at the apache log file I find:
[Wed May 02 15:20:55 2012] [error] [client xx.xx.xxx.xxx] The lock refresh for /newFile failed because no lock tokens were specified in an "If:" header. [400, #0]
[Wed May 02 15:20:55 2012] [error] [client xx.xx.xxx.xxx] (20)Not a directory: No locktokens were specified in the "If:" header, so the refresh could not be performed. [400, #103]
Thanks for any hint!!
UPDATE: I added my problem description.. Cheers!
The LOCK method requires a body which contains an XML description of the lock you want to take out. Your cURL test didn't include this body, hence the 400 error response.
But if I understand your question correctly, you want to:
LOCK
PUT
UNLOCK
If that's true, why would you bother with the LOCK and UNLOCK? Just do the PUT! Locks would only be useful if you want to carry out multiple operations while you are holding the lock and avoid having another client see the object in its partially modified state or (perhaps worse) modify the object concurrently with you.
A typical case where locking can be useful is a read-modify-write cycle: you want to GET the object, modify it locally, and PUT it back, but disallow another client from making a competing change between the time you GET it and the time you PUT it. However, for dealing with this specific case, HTTP offers a different method of resolving the issue, without using locks (which are ill-suited for a stateless protocol like HTTP):
GET the object
Modify it locally
PUT the object back with an If-Match header that contains the original ETag returned in step 1
If the PUT results in a 412 error, go back to step 1. Otherwise, you are done.
UPDATE: based on your updated question, I see that you are see a partial truncated or half-uploaded version of the new file if you do a GET concurrent with a PUT. This is unfortunate. The server should treat the PUT as atomic with respect to other requests. Other clients should either see the old version or the new version, never a state in between. There's nothing you should need to do from the client end to make this true. It should be fixed in the server.