I am tuning my cluster which has Hive LLAP, According to the below link, https://community.hortonworks.com/articles/215868/hive-llap-deep-dive.html I need to calculate the value of heapsize, but not sure what is the meaning of *?
I also have a question regarding how to calculate the value for hive.llap.daemon.yarn.container.mb other then then default value given by ambari?
I have tried calculating the value by considering this (* as multiplication) and set container value equal to yarn.scheduler.maximum.allocation.mb, However HiveServer 2 interactive does not start after tuning.
Here's the excellent wiki article for setting up hive llap in HDP suite.
https://community.hortonworks.com/articles/149486/llap-sizing-and-setup.html
Your understanding for * is correct, it's used for multiplication.
Rule of thumb here is set hive.llap.daemon.yarn.container.mb to yarn.scheduler.maximum-allocation-mb but if your service is not coming up with that value then I would recommend you to change llap_heap_size to 80% of hive.llap.daemon.yarn.container.mb.
Related
According to the documentation the percentile command should give the exact percentile for every numeric column. At least when my input is composed of floating points - this is not true.
In hive docs, it says the percentile command only works for integers. I didn't understand the relation between spark sql and hive, but it seems spark doesn't just run hive - otherwise it wouldn't have changed it's docs. Also, spark's percentile has a different signature, allowing it to get a frequency parameter, which I also have no idea what purpose it serves.
This is an example with unexpected output:
from pyspark.sql import functions as sf
d = spark.createDataFrame([[35.138071000000004], [34.119932999999996], [34.487992]], ['a'])
d.select(sf.expr('percentile(a, array(0.25,0.5,0.75,0.9,0.95)) AS res')).collect()
Out[1]: [Row(res=[34.3039625, 34.487992, 34.8130315, 35.0080552, 35.0730631])]
If I switch sf.expr content to percentile_approx with a high accuracy, or use high frequency in the current method - I get a reasonable output.
Could you explain what's happening?
Also:
Can you please explain/point me to some resource about the relation between spark sql and apache hive?
Where is the code that spark sql commands run?
Thanks
There is no direct relation between Spark and Hive except Spark's ability to retrieve metadata from Hive MetaStore regarding databases, tables and views defined in Hive. You can get familiar with Spark by reading its online documentation.
SparkSQL is a completely independent (from Hive) implementation of SQL language written in Scala. SparkSQL is one of Spark's modules that uses Spark cluster computing platform. Along with other Spark modules, it can run on Spark's own cluster (aka standalone), or make the use of YARN or Mesos.
Specifically, the percentile function in SparkSQL according to SparkSQL documentation...
Returns the exact percentile value of numeric column col at the given
percentage. The value of percentage must be between 0.0 and 1.0. The
value of frequency should be positive integral.
EDIT
Frequency parameter was added to the percentile function as part of SPARK-18940, to be able to optionally supply extra column (generally speaking, an expression) that contains distribution of the analyzed values. The default value is frequency = 1L.
There is a follow-up SPARK-27929 that will relax the requirement to have it as type Long.
I have a query using to much containers and to much memory. (97% of the memory used).
Is there a way to set the number of containers used in the query and limit the max memory?
The query is running on Tez.
Thanks in advance
Controlling the number of Mappers:
The number of mappers depends on various factors such as how the data is distributed among nodes, input format, execution engine and configuration params. See also How initial task parallelism works
MR uses CombineInputFormat, while Tez uses grouped splits.
Tez:
set tez.grouping.min-size=16777216; -- 16 MB min split
set tez.grouping.max-size=1073741824; -- 1 GB max split
Increase these figures to reduce the number of mappers running.
Also Mappers are running on data nodes where the data is located, that is why manually controlling the number of mappers is not an easy task, not always possible to combine input.
Controlling the number of Reducers:
The number of reducers determined according to
mapreduce.job.reduces
The default number of reduce tasks per job. Typically set to a prime close to the number of available hosts. Ignored when mapred.job.tracker is "local". Hadoop set this to 1 by default, whereas Hive uses -1 as its default value. By setting this property to -1, Hive will automatically figure out what should be the number of reducers.
hive.exec.reducers.bytes.per.reducer - The default in Hive 0.14.0 and earlier is 1 GB.
Also hive.exec.reducers.max - Maximum number of reducers that will be used. If mapreduce.job.reduces is negative, Hive will use this as the maximum number of reducers when automatically determining the number of reducers.
Simply set hive.exec.reducers.max=<number> to limit the number of reducers running.
If you want to increase reducers parallelism, increase hive.exec.reducers.max and decrease hive.exec.reducers.bytes.per.reducer.
Memory settings
set tez.am.resource.memory.mb=8192;
set tez.am.java.opts=-Xmx6144m;
set tez.reduce.memory.mb=6144;
set hive.tez.container.size=9216;
set hive.tez.java.opts=-Xmx6144m;
The default settings mean that the actual Tez task will use the mapper's memory setting:
hive.tez.container.size = mapreduce.map.memory.mb
hive.tez.java.opts = mapreduce.map.java.opts
Read this for more details: Demystify Apache Tez Memory Tuning - Step by Step
I would suggest to optimize query first. Use map-joins if possible, use vectorising execution, add distribute by partitin key if you are writing partitioned table to reduce memory consumption on reducers and write good sql of course.
Using INFO CPU command on Redis, I get the following values back (among other values):
used_cpu_sys:688.80
used_cpu_user:622.75
Based on my understanding, the value indicates the CPU time (expressed in seconds) accumulated since the launch of the Redis instance, as reported by the getrusage() call (source).
What I need to do is calculate the % CPU utilization based on these values. I looked extensively for an approach to do so but unfortunately couldn't find a way.
So my questions are:
Can we actually calculate the % CPU utilization based on these 2 values? If the answer is yes, then I would appreciate some pointers in that direction.
Do we need some extra data points for this calculation? If the answer is yes, I would appreciate if someone can tell me what those data points would be.
P.S. If this question should belong to Server Fault, please let me know and I will post it there (I wasn't 100% sure if it belongs here or there).
You need to read the value twice, calculate the delta, and divide by the time elapsed between the two reads. That should give you the cpu usage in % for that duration.
Im using datastax-4.6. I have created a cassandra table and stored 2crore records. Im trying to read the data using scala. The code works fine for few records but when i try to retrieve all 2crore records it displays me follwing error.
**WARN BlockManagerMasterActor: Removing BlockManager BlockManagerId(1, 172.20.98.17, 34224, 0) with no recent heart beats: 140948ms exceeds 45000ms
15/05/15 19:34:06 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(C15759,34224) not found**
Any help?
This problem is often tied to GC pressure
Tuning your Timeouts
Increase the spark.storage.blockManagerHeartBeatMs so that Spark waits for the GC pause to end.
SPARK-734 recommends setting -Dspark.worker.timeout=30000 -Dspark.akka.timeout=30000 -Dspark.storage.blockManagerHeartBeatMs=30000 -Dspark.akka.retry.wait=30000 -Dspark.akka.frameSize=10000
Tuning your jobs for your JVM
spark.cassandra.input.split.size - will allow you to change the level of parallelization of your cassandra reads. Bigger split sizes mean that more data will have to reside in memory at the same time.
spark.storage.memoryFraction and spark.shuffle.memoryFraction - amount of the heap that will be occupied by RDDs (as opposed to shuffle memory and spark overhead). If you aren't doing any shuffles, you could increase this value. The databricks guys say to make this similar in size to the size of your oldgen.
spark.executor.memory - Obviously this depends on your hardware. Per DataBricks you can do up to 55gb. Make sure to leave enough RAM for C* and for your OS and OS page cache. Remember that long GC pauses happen on larger heaps.
Out of curiosity, are you frequently going to be extracting your entire C* table with Spark? What's the use case?
How do I decide whether a table is small enough for the MAPJOIN optimization?
My guess is that I should look at
du /misc/hdfs/user/hive/warehouse/my_table
and use MAPJOIN if that is below 50% (? 5%?) of RAM.
I am using hive 0.10.
hive-site.xml
hive.mapjoin.smalltable.filesize
Default Value: 25000000
The threshold for the input file size of the small tables; if the file size is smaller than this threshold, it will try to convert the common join into map join.
This is the current release Wiki, but I think this setting goes back to 0.10.