I am going to set up a standalone Hortonworks cluster.
What is the needed system configuration that could process 1TB of data.
Requirement such as:
RAM space
Hard disk space
quad-/hex-/octo-core CPUs running how much GHZ
Cent os which version etc.,
System Configuration depends on the usecase.
Disk - Assuming 1 replication, 1 TB + 25% processing space
If you are using Hive or MapReduce I would start with 16 Gig, 4 or 8 core. CentOS 7.0
Related
I've recently assembled a new AMD Desktop, to replace an older Dell Latitude E7540 laptop.
The AMD Desktop:
Ryzen 3 3100 # 3.8GHz (4C/8T), 32GB DDR4 3600 CL17 RAM, Corsair P600
Gen4 SSD
The DELL Laptop:
Dell Latitude E7540: Intel I7-5600U # 2.6GHz (2C/4T), 16 MB RAM DDR3 1600, Samsung mSATA PM851
On the new AMD Desktop, when executing a docker build command, two situations occur:
The performance is dreadful, even building a simple image, it takes a long time for the command to start. After starting, it takes a long long time to complete (when it completes)
The build window crashes almost 50% of the time.
The benchmarks indicate that the new AMD Desktop is 3.5x faster at single core, and 6x faster at multicore.
As such, I was expecting a much better performance with the new AMD Desktop.
Unfortunately, that's not the case, and for the same Dockerfile (which generates a very big image):
The Dell starts faster
The Dell completes faster (10m vs 30m)
On the Dell, the build window never crashes.
The only difference between both systems is that one is an Intel platform, the new one an Ryzen 3 AMD.
Environment Details are the same on both machines:
Windows Version: Windows 10 Ent. 19049
Docker Desktop Version: Docker 3.0.0
What can explain this abysmal performance on Docker-Desktop on the new AMD system?
After a few troubling days, i can confirm that the problem is not AMD related.
The culprit is the Antivirus, that when ON, its scanning the files used by Docker, which cause all the problems i've described.
Docker documentation states how to disable the antivirus to scan Docker related files:
https://docs.docker.com/engine/security/antivirus/
When antivirus software scans files used by Docker, these files may be locked in a way that causes Docker commands to hang.
One way to reduce these problems is to add the Docker data directory (/var/lib/docker on Linux, %ProgramData%\docker on Windows Server, or $HOME/Library/Containers/com.docker.docker/ on Mac) to the antivirus’s exclusion list. However, this comes with the trade-off that viruses or malware in Docker images, writable layers of containers, or volumes are not detected. If you do choose to exclude Docker’s data directory from background virus scanning, you may want to schedule a recurring task that stops Docker, scans the data directory, and restarts Docker.
here is my scenario: I have a windows VM and it's having 2 runtimes installed on it (Mule1 and Mule2).
Now If i have to distribute 60% of VM CPU to Mule1 and 40% to Mule2. How can it be done?
Is that even possible ?
There is a concept called CPU affinity, when you have more than one core or CPU. The operating system you are using have tools for assigning cores to a process. I'm not aware of a feature to assign or limit a percentage of CPU usage to a process. I don't know about an out of the box feature to limit CPU usage per process.
Linux:
You can use the taskset command to set which cores to assign to the mule process.
Example:
taskset -c 0,1 ./mule
Source: https://help.mulesoft.com/s/article/How-to-set-CPU-affinity-for-Mule-ESB-process
Windows:
In the Task Manager, you can right-click the java.exe and wrapper-windows-x86-64.exe processes, select "Set Affinity" and choose the processors
In this article there are Powershell commands to do the same from the command line: https://help.mulesoft.com/s/article/How-to-set-CPU-affinity-for-a-Mule-ESB-process-in-Windows-as-a-Service
It is completely different topic however Docker allows something similar per container.
I'm trying to run SonarQube 6.2 on VPS but it has only 512mb ram (recommended to use SonarQube is 1gb memory) While I start SonarQube I exceed ram limit to above 1gb ram.
My VPS spec:
FreeBSD 10.3-STABLE
hw.machine: amd64
hw.model: Intel(R) Core(TM) i7-6700 CPU # 3.40GHz
hw.ncpu: 8
So, my question is: It is possible to configure SonarQube proccess limited to 512mb ram to run on this server?
You could give a try by using/adding more swap to compensate the lack of memory, but from the list of requirements seems you need at least 2GB:
https://docs.sonarqube.org/display/SONAR/Requirements
A production tomcat stopped responding after 8 hours of adding more cpu and ram during runtime. There is no traceable log in catalina.out.
java process was still running in the background. There is no load on the system during that time period. No sign of Out of memory.
apache-tomcat-7.0.55
tomcat-native-1.1.31
java version "1.8.0_66"
Red Hat Enterprise Linux Server release 6.6 (Santiago)
Could it be because of the cpu and ram upgrade? Thanks.
I saw a post about it for java 6. I am not sure it is related to my issue.
http://bugs.java.com/bugdatabase/view_bug.do?bug_id=6840239
I'm using a VM with VMware Player to write code and compile.
As my current program is huge, the compilation takes a while to be done (upto 5 minutes)
using 25% of my 4 cores CPU on my host = 100% of one core.
It seems that the VM is limited to use 1 single core.
Is there a way to optimize the number of cores a VM can use?
I'd like to use 50% or 75% of my 4 cores CPU.
Thanks
It sounds like you're limited by the number of parallel build tasks you can run, not the VM CPU configuration, e.g., by default, make will run a single step at a time. Try running several steps in parallel, e.g., run make -j4 or equivalent for your build system.
On a separate note, a VM may be more overhead for you than you might like; consider using Docker to host your development environment.