I'm trying to diagnose a request failure scenario caused by a possible concurrent access to HttpContext. The thing is that the issue is reproducible only on a server and that too when put under load.
A simple dotnet-dump collect output will not help because that only gives the state of process at a particular point in time. What I'm trying to find out is if a dump can be triggered when a particular condition is met. The intent is to collect detailed stack trace at the point of the exception.
For example, take dump when a null ref exception is thrown when a particular property on a particular type is accessed.
As of now, I've tried to make the app code as lean as possible. Got rid of unnecessary middleware in request pipeline. But the stack trace I see I get of exception is hardly of any use.
Well, based on your comment, It seems you are looking for tool like debug-diag for linux machine. You could consider ProcDump for linux version which similar to debug-diag tool but for linux as no other suitable tool available. However, it has few fature limitaions. Please follow the below insturctions if that meet your requirement.
Installation guideline:
Please check here for details instruction
Minimum Requirements:
Ubuntu 16.04 LTS
Minimum OS:
Red Hat Enterprise Linux / CentOS 7
Fedora 29
Note:
Please note that, Curreent version does not have full feature parity with Windows version of ProcDump, specifically, stay alive functionality, and custom performance counters.
For more details, you could visit our official document as well.
Related
with a new project we encountered some strange behaviour on our ColdFusion application.
Whenever a single request is initiated from the browser, the code of the cfml-templates is
executed multiple times. Upon viewing the corresponding log-files we found out, that indeed
for some reason the same request fires the evaluation in our application multiple times. One request
generates several entries. This is especially the case for long-running requests, such as database imports.
The ColdFusion application implements a REST-service, but even on manually requesting a resource,
such as a certain cfml page, on the same application - the code gets executed an unknown amount of times(variable initializations, database write-operations etc. take place), and if the request runs too long (cap at around ~4-6 seconds) there is no response to the browser.
About the infrastructure:
The application is Coldfusion18 with Tomcat Standard Edition
The webserver is an Apache (2.4.6).
Everything runs on a Linux machine with Cent OS 7.7
The corresponding Java version is 11.0.4
Our best guess is that there might be some misscommunication between the coldfusion connector with
the apache webserver. We actually searched for some configuration parameters that could cause the
problem, without success. Upon an installation on a windows machine we did not encounter that error.
Anyone got any idea?
we just found our answer in the following post:
Link to Solution
I've installed FusionAuth (awesome product) into a Docker Swarm cluster using the official docker-compose.yml file and everything seems to work brilliantly.
EXCEPT
Periodically, when a user goes to login they will be presented with the above error stating that the search engine is not available. If they try again immediately then everything works correctly! I would, obviously, prefer that they never saw the error.
Elasticsearch is definitely running and is responding to API calls correctly, and I can see the fusionauth_user index is present and populated with docs.
I guess my question is two fold:
1) What role does the ElasticSearch engine play in the FusionAuth ecosystem and can it be disabled?
2) Is there a configurable timeout somewhere that is causing the error message and, if so, where can change it?
I've search the docs for answers to the above but I can't seem to find anything :-(
Thanks for the kind feedback.
1) What role does the ElasticSearch engine play in the FusionAuth ecosystem and can it be disabled?
Elasticsearch provides full text search of user data. Each time a user is created or updated the user is re-indexed. In this case during login, we are updating the search index with the last login instant.
This service is required and cannot be disabled. We have had clients request to make this service optional for embedded applications or small scale scenarios where Elasticsearch may not be required. While this is not currently in plan, it is possible we may revisit this option in the future.
2) Is there a configurable timeout somewhere that is causing the error message and, if so, where can change it?
Not currently.
Full disclosure, I am not a Docker or Docker Swarm expert at all - perhaps there are some nuances to Swarm and response time due to spin up and spin down of resources?
Do you see any exceptions in the log when a user sees this error on the login?
Firstly, there are related posts:
GlassFish Server update center installation times out
Java EE 7 updatetool installation fails
I got my Java EE 7 SDK (Update 3) from here: http://www.oracle.com/technetwork/java/javaee/downloads/index.html
I have tried each of the solutions in the above posts and here: https://blogs.oracle.com/dipol/troubleshooting-glassfish-update-center
Including:
In the cmd prompt running set PKG_CLIENT_CONNECT_TIMEOUT=300 and set PKG_CLIENT_READ_TIMEOUT=300 before updatetool in C:\glassfish4\bin\updatetool.bat (c:\glassfish4 in my install directory - all settings were default, including install update tool...).
Set above mentioned timeout to much larger values - doesn't appear to make a difference at all, the process basically bombs immediately.
Running C:\glassfish4\bin\updatetool.bat many times.
Triple checking that I didn't somehow configure a proxy server in my sleep.
Use the update tool via the Glassfish admin console at http://localhost:4848 (seems to show no available update or add-ons, which seems odd..)
I get the following screenshot when I run C:\glassfish4\bin\updatetool.bat
I have no idea why the error would be proxy related, unless it happened to be something on their end. Interestingly, If I go directly to the URL mentioned (via Chrome) I get the following page:
What could possibly be going wrong here?
The updatetool was a commercial feature of Oracle GlassFish. Any update functionality relied on Oracle providing a site where updates could be hosted. Since Oracle GlassFish is no longer supported, this site no longer exists so the updatetool won't work any more.
Rather than downloading GlassFish from Oracle, you should download it from the official open source site, hosted on GitHub. Alternatively, if you really do need support, you could try Payara Server which is open source, and derived from GlassFish, but has support available (disclaimer: I work for Payara)
I am trying to add Apache Nifi in ambari but continuously failing with error Error occured during stack advisor command invocation:
Unable to delete directory /var/run/ambari-server/stack-recommendations/1.
There is a similar thread with the same error in hortonworks community, I have tried everything mentioned in that thread but unable to fix it. My sandbox is installed in vmware workstation 12 player. I also tried to create and remove directory manually but it is failing with the error invalid argument. Created a thread for this error also on stackexchange. Please help!!!
Here is a link to Hortonworks forum thread. And it seems like sandbox is just broken:
This is due to a docker issue in this 2.5 sandbox build. It will be
fixed in next revision of the sandbox.
There are also some workarounds described (like use older HDP 2.4 or establishing own cluser based on the HDP 2.5 docker image)
Updated sandbox arrived: http://hortonworks.com/downloads
Trust me, active member of community see your posts in multiple locations. In a good, no Big Brother ways :) but cross-posting is an old as world ... Well, you got it.
Did you see a notice for this service in Ambari? Telling it's been deprecated? Same note in the github. There's a good reason for that, it's now been implemented properly by the dev team and with many more features. I.e. all the action is there now.
I think I replied a similar question, though not sure it was yours, take a look in HCC.
I have two clusters, one in local virtual machine another in remote cloud. Both clusters in Standalone mode.
My Environment:
Scala: 2.10.4
Spark: 1.5.1
JDK: 1.8.40
OS: CentOS Linux release 7.1.1503 (Core)
The local cluster:
Spark Master: spark://local1:7077
The remote cluster:
Spark Master: spark://remote1:7077
I want to finish this:
Write codes(just simple word-count) in IntelliJ IDEA locally(on my laptp), and set the Spark Master URL to spark://local1:7077 and spark://remote1:7077, then run my codes in IntelliJ IDEA. That is, I don't want to use spark-submit to submit a job.
But I got some problem:
When I use the local cluster, everything goes well. Run codes in IntelliJ IDEA or use spark-submit can submit job to cluster and can finish the job.
But When I use the remote cluster, I got a warning log:
TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
It is sufficient resources not sufficient memory!
And this log keep printing, no further actions. Both spark-submit and run codes in IntelliJ IDEA result the same.
I want to know:
Is it possible to submit codes from IntelliJ IDEA to remote cluster?
If it's OK, does it need configuration?
What are the possible reasons that can cause my problem?
How can I handle this problem?
Thanks a lot!
Update
There is a similar question here, but I think my scene is different. When I run my codes in IntelliJ IDEA, and set Spark Master to local virtual machine cluster, it works. But I got Initial job has not accepted any resources;... warning instead.
I want to know whether the security policy or fireworks can cause this?
Submitting code programatically (e.g. via SparkSubmit) is quite tricky. At the least there is a variety of environment settings and considerations -handled by the spark-submit script - that are quite difficult to replicate within a scala program. I am still uncertain of how to achieve it: and there have been a number of long running threads within the spark developer community on the topic.
My answer here is about a portion of your post: specifically the
TaskSchedulerImpl: Initial job has not accepted any resources; check
your cluster UI to ensure that workers are registered and have
sufficient resources
The reason is typically there were a mismatch on the requested memory and/or number of cores from your job versus what were available on the cluster. Possibly when submitting from IJ the
$SPARK_HOME/conf/spark-defaults.conf
were not properly matching the parameters required for your task on the existing cluster. You may need to update:
spark.driver.memory 4g
spark.executor.memory 8g
spark.executor.cores 8
You can check the spark ui on port 8080 to verify that the parameters you requested are actually available on the cluster.