I have packstack-allinone setup on my RHEL7.1 trial for Juno release.
I am facing problem while launching VM(for ex: cirros) with a disk size mentioned in flavor. If there is 0gb disk size then VM are getting launched but not for higher flavor sizes.
I also observe that when I do this, openstack-nova-compute service goes down which I observed when I checked using nova-manage service list with nova-compute being XXX making me restart the service everytime I try this scenario. The compute logs doesn't throw any error, it just gets stuck at "Creating image".
Is there any Filesystem issue which i missing to be configured? I am new to this, so please help.
PS: I run all commands with "root" user.
The problem was with esxi. Esxi needs to be 5.5v to support RHEL7x Since mine was 5.1v it only supported RHEL6x.
After upgrading esxi5.1 to 5.5v it worked fine.
Related
I'm trying to generate a full heapdump from Weblogic 10.3.6 due to an OutOfMemoryError generated by a Web Application deployed on the Server.
I've setted the following start script:
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/path/to/heapdump
When the OutOfMemoryError occurs, Weblogic generates an empty hprof file (0 bytes size) in /path/to/heapdump folder, and nothing happens: the Server remains in RUNNING mode, even if is not reachable anymore.
The java process is still alive, but with 0% of processor.
Even the server.out log seems completely frozen, without any trace of the OutOfMemoryError.
What's wrong with the configuration?
Probably you can use Java Flight Recorder to save events and check which objects are generating OOM.
(any profiler should work as well).
Been there :( . I remember at the time that we've found it was somewhat logical since there was not enough memory for normal operation, the JVM could not automagically find enough memory to create a heapdump either. If memory serves me well, at that time we did 2 things to debug the memory leak. First we were "lucky" enough that the problem was happening fairly regularly so a close manual monitoring was possible (monitoring of the gc.log looking for repeated FullGC and monitoring of the performance tab in the console). Knowing when the onset of the problem was starting we were doing some kill -3 to get the dump manually. We also used jstack {PID} (JDK 1.6 on Linux) with some luck. With those, at the time, the devs were able to identify the memory leak. Hope that helps.
Okay, your configuration looks alright.. you might want to check if the weblogic process user has the rights to edit the heap dump file.
You can take heap dump by Java tools :
JAVA_HOME/bin/jmap -dump:format=b,file=path_of_the_file
OR
%JROCKIT_HOME%\bin\jrcmd hprofdump filename=path_of_the_file
I am trying to add Apache Nifi in ambari but continuously failing with error Error occured during stack advisor command invocation:
Unable to delete directory /var/run/ambari-server/stack-recommendations/1.
There is a similar thread with the same error in hortonworks community, I have tried everything mentioned in that thread but unable to fix it. My sandbox is installed in vmware workstation 12 player. I also tried to create and remove directory manually but it is failing with the error invalid argument. Created a thread for this error also on stackexchange. Please help!!!
Here is a link to Hortonworks forum thread. And it seems like sandbox is just broken:
This is due to a docker issue in this 2.5 sandbox build. It will be
fixed in next revision of the sandbox.
There are also some workarounds described (like use older HDP 2.4 or establishing own cluser based on the HDP 2.5 docker image)
Updated sandbox arrived: http://hortonworks.com/downloads
Trust me, active member of community see your posts in multiple locations. In a good, no Big Brother ways :) but cross-posting is an old as world ... Well, you got it.
Did you see a notice for this service in Ambari? Telling it's been deprecated? Same note in the github. There's a good reason for that, it's now been implemented properly by the dev team and with many more features. I.e. all the action is there now.
I think I replied a similar question, though not sure it was yours, take a look in HCC.
I have two clusters, one in local virtual machine another in remote cloud. Both clusters in Standalone mode.
My Environment:
Scala: 2.10.4
Spark: 1.5.1
JDK: 1.8.40
OS: CentOS Linux release 7.1.1503 (Core)
The local cluster:
Spark Master: spark://local1:7077
The remote cluster:
Spark Master: spark://remote1:7077
I want to finish this:
Write codes(just simple word-count) in IntelliJ IDEA locally(on my laptp), and set the Spark Master URL to spark://local1:7077 and spark://remote1:7077, then run my codes in IntelliJ IDEA. That is, I don't want to use spark-submit to submit a job.
But I got some problem:
When I use the local cluster, everything goes well. Run codes in IntelliJ IDEA or use spark-submit can submit job to cluster and can finish the job.
But When I use the remote cluster, I got a warning log:
TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
It is sufficient resources not sufficient memory!
And this log keep printing, no further actions. Both spark-submit and run codes in IntelliJ IDEA result the same.
I want to know:
Is it possible to submit codes from IntelliJ IDEA to remote cluster?
If it's OK, does it need configuration?
What are the possible reasons that can cause my problem?
How can I handle this problem?
Thanks a lot!
Update
There is a similar question here, but I think my scene is different. When I run my codes in IntelliJ IDEA, and set Spark Master to local virtual machine cluster, it works. But I got Initial job has not accepted any resources;... warning instead.
I want to know whether the security policy or fireworks can cause this?
Submitting code programatically (e.g. via SparkSubmit) is quite tricky. At the least there is a variety of environment settings and considerations -handled by the spark-submit script - that are quite difficult to replicate within a scala program. I am still uncertain of how to achieve it: and there have been a number of long running threads within the spark developer community on the topic.
My answer here is about a portion of your post: specifically the
TaskSchedulerImpl: Initial job has not accepted any resources; check
your cluster UI to ensure that workers are registered and have
sufficient resources
The reason is typically there were a mismatch on the requested memory and/or number of cores from your job versus what were available on the cluster. Possibly when submitting from IJ the
$SPARK_HOME/conf/spark-defaults.conf
were not properly matching the parameters required for your task on the existing cluster. You may need to update:
spark.driver.memory 4g
spark.executor.memory 8g
spark.executor.cores 8
You can check the spark ui on port 8080 to verify that the parameters you requested are actually available on the cluster.
I am working on transfer large size file, and finally ended with MTOM implementation. we created MTOM enabled web service and client, and tested the client as a plain Java program. and we were able to send 1 GB file successfully. the main point here the heap at client place were not even increasing more than 70 MB.
But when I tried to initiate the same call from web-logic container (means created web client), we end up with below OOM Exception.
at
weblogic.utils.io.UnsyncByteArrayOutputStream.resizeBuffer(UnsyncByteArrayOutputStream.java:59)
at weblogic.utils.io.UnsyncByteArrayOutputStream.write(UnsyncByteArrayOutputStream.java:89)
at javax.activation.DataHandler.writeTo(DataHandler.java:293)
at com.sun.xml.ws.encoding.MtomCodec$ByteArrayBuffer.write(MtomCodec.java:196)
at com.sun.xml.ws.encoding.MtomCodec.encode(MtomCodec.java:163)
at com.sun.xml.ws.encoding.SOAPBindingCodec.encode(SOAPBindingCodec.java:258)
at com.sun.xml.ws.transport.http.client.HttpTransportPipe.process(HttpTransportPipe.java:142)
at com.sun.xml.ws.transport.http.client.HttpTransportPipe.processRequest(HttpTransportPipe.java:86)
at com.sun.xml.ws.api.pipe.Fiber.__doRun(Fiber.java:598)
at com.sun.xml.ws.api.pipe.Fiber._doRun(Fiber.java:557)
at com.sun.xml.ws.api.pipe.Fiber.doRun(Fiber.java:542)
at com.sun.xml.ws.api.pipe.Fiber.runSync(Fiber.java:439)
at com.sun.xml.ws.client.Stub.process(Stub.java:248)
at com.sun.xml.ws.client.sei.SEIStub.doProcess(SEIStub.java:135)
at com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:109)
at com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:89)
at com.sun.xml.ws.client.sei.SEIStub.invoke(SEIStub.java:118)
at $Proxy101.uploadFile(Unknown Source)
anyu one have any idea
UPDATE: it seems the MTOM settings are not effective when we run the program in web-logic container ! but still I am not able to find the solution
UPDATE 2: it seems weblogic is not supporting streaming ! I will update the weblogic version and update the ticket, till them wish me luck..
Add this additional Java/JVM Option in setDomainEnv.sh
EXTRA_JAVA_PROPERTIES="-DUseSunHttpHandler=true ${EXTRA_JAVA_PROPERTIES}"
export EXTRA_JAVA_PROPERTIES
switches from weblogic specific (weblogic.net.http.HttpURLConnection) to sun's HTTP handler.
This solved my issue.
Refer:
Changing HttpURLConnection in running jvm
http://atgtipsandtweaks.blogspot.com/2011/11/weblogicjava-httphandler-issues.html
Thanks!
I am aware that this question is asked many times in forums and I have tried all solutions mentioned in them, but no luck.
Actually, I doubt when last time I was trying to replace the /etc/sysconfig/iptables with my own iptables rules, I mistakenly replaced /etc/init.d/iptables and restarted the machine. And as expected it didn't start. Then I detached the EBS from this instance and attached to a new RHEL instance and fix the mess up by copying back the /etc/init.d/iptables from backup (I used to take backups before replacement :) ) and same for /etc/sysconfig/iptables.
I have also put some custom startup scripts in /etc/init.d folder for our application to start on instance reboot. I have removed those too to make sure any of my script is not causing this. But still system is not allowing me to connect via ssh. AWS console is showing 2/2 checks being successful, but not able to connect via 22.
Here is the last few lines of system log which states that something wrong is happening after or on iptables startup but not showing what. :(
blkfront: xvde1: barriers disabled
Changing capacity of (202, 65) to 62914560 sectors
xvde1: detected capacity change from 0 to 32212254720
EXT4-fs (xvde1): mounted filesystem with ordered data mode. Opts:
dracut: Mounted root filesystem /dev/xvde1
dracut: Loading SELinux policy
type=1404 audit(1398404320.826:2): enforcing=1 old_enforcing=0 auid=4294967295 ses=4294967295
type=1403 audit(1398404321.795:3): policy loaded auid=4294967295 ses=4294967295
dracut:
dracut: Switching root
udev: starting version 147
Initialising Xen virtual ethernet driver.
microcode: CPU0 sig=0x306e4, pf=0x1, revision=0x415
platform microcode: firmware: requesting intel-ucode/06-3e-04
Microcode Update Driver: v2.00 <tigran#aivazian.fsnet.co.uk>, Peter Oruba
NET: Registered protocol family 10
lo: Disabled Privacy Extensions
ip6_tables: (C) 2000-2006 Netfilter Core Team
nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
ip_tables: (C) 2000-2006 Netfilter Core Team
Can anyone help me in identifying what is going wrong here?
Got it fixed.
Actually, it was not the problem of iptables. Again it was due to the known bug in RHEL 6.4 on EC2 which puts wrong entries in sshd_config files. Although, I have checked this file for wrong entries in my first attempt to resolve the issue, somehow it was being created again, may be because every time I start a new machine using my AMI or new RHEL 6.4 AMI. In both cases, AMI is still registered as 6.4, though the OS on the disk is updated to 6.5. May be this was the reason that it was creating wrong entries in sshd_config. Now, again I have fixed this file for wrong entries and created new AMI using RHEL 6.5 and attached the EBS volume from instance created using my RHEL 6.4 AMI, it works fine.