I am trying to configure apache hive server2
My configuration file in hive-site.xml
<configuration>
<property>
<name>hive.server2.thrift.min.worker.threads</name>
<value>5</value>
<description>Minimum number of worker threads</description>
</property>
<property>
<name>hive.server2.thrift.max.worker.threads</name>
<value>500</value>
<description>Maximum number of worker threads</description>
</property>
<property>
<name>hive.server2.thrift.port</name>
<value>10000</value>
<description>TCP port number to listen on</description>
</property>
<property>
<name>hiver.server2.thrift.bind.host</name>
<value>10.89.20.22</value>
<description>TCP interface to bind to</description>
</property>
<property>
<name>hive.server2.transport.mode</name>
<value>binary</value>
<description>Set to http to enable HTTP transport mode</description>
</property>
<property>
<name>hive.server2.thrift.http.port</name>
<value>10001</value>
<description>HTTP port number to listen on</description>
</property>
<property>
<name>hive.server2.thrift.http.max.worker.threads</name>
<value>500</value>
<description>Maximum worker threads in the server pool</description>
</property>
<property>
<name>hive.server2.thrift.http.min.worker.threads</name>
<value>5</value>
<description>Minimum worker threads in the server pool</description>
</property>
<property>
<name>hive.server2.thrift.http.path</name>
<value>cliservice</value>
<description>The service endpoint</description>
</property>
The error that I am receiving.
I dont know what is my error about. Can someone help me to configure this?
Thank you so much
There are no actual error in that screenshot, that information is informational only.
You will also notice in the log these messages are INFO lines which are just informational messages only. If you had any actual errors you might see things like ERROR, FATAL, or even WARN is good to watch out for.
Those properties that are saying deprecated look to be properties from your hadoop site xml configuration files such as hive-site.xml that are no longer used any longer. Hadoop will just ignore those properties since they aren't used anymore. If you remove those properties from their respective configuration xml files these messages should stop if the property is removed from the xml configuration the cluster is looking at. You posted at least a portion of your hive-site.xml, it doesn't look complete but these also may not be in that xml anyway. The hadoop cluster has a number of configuration files, generally at least 1 for each service running on the cluster. It might be another such as core-site.xml, mapred-site.xml, or other xml files which are on each node running the service.
The are INFO messages about the SJF bindings being duplicated in the classpath are likely due to having a duplicate jar file somewhere. There are a few services such as yarn/mapreduce that have a property in their xml for a classpath such as mapreduce.application.classpath which has a list of multiple folders on the nodes operating system that contain jar files the customer uses to run. These messages appear when a node has two jar files that contain an identical class inside the jar file.
The most common thing that happens is you install an update to your hadoop cluster, or a specific service to a new version. Most, if not all, of these hadoop services execute jar files to run their jobs. If you upgrade the cluster, or a service you get new jar files which usually increments the version number in the name of the jar file. For example it will change from test1.1.jar to test1.2.jar. If both the new, and the old are jar file are left on the cluster in the classpath property you will get a classpath confict/warning. Essentially what happens is you now have two jar files with different name yet both have identical classpaths inside them causing these types of messages about classpath having duplicates in it.
In that classpath message it also shows you the duplicate jar files both with the name of this INFO line SJF*. Most likely if you remove the older version from any of the nodes it is still on it should stop telling you about it when it starts up.
It sounds like you had an existing hadoop cluster that you applied an update to which upgraded service to a new version. When you did this whatever version(s) you updated to no longer uses some of the properties the original version used for some reason, which is why it is telling you they are deprecated, or no longer used. You can just remove these from the xml configuration they are in on any node that has them. Then it sounds like your SJF jar file was replaced with the latest version, but the original version wasn't removed at at least somewhere during the upgrade process.
Basically hadoop will keep going without error, but it wants you to clean up after your mess upgrading is what those INFO lines are really trying to tell you.
Recently we decided to add a cache layer to our mule APIs and Redis came to the scope.
We are at Mule3.8.0 and Redis connector 4.0.0. and we met following issues while configuring:
How to separate our keys by Redis DB ? This is not mentioned in document and there is only a 'Default Partition Name' in the configuration seems close but whatever value we put there, seems no effect - it will always be db0 containing all the keys, hence we can't really have "dev", "qa" and "test" key sets in the same redis cluster
The Redis connector document has example as below
<redis:sorted-set-select-range-by-index config- ref="Redis_configuration" key="my_key" start="0" end="-1" />
however when we tried the samething it complains the 'end' value should be >= 0 hence not usable
How to configure a connection pool properly with Redis connector configuration? Not mentioned in document again. The only attribute is the 'Pool Config Reference' and I tried to put a spring bean ref to my own JedisPoolConfig there, seems no effect, and number of the connections remains the same no matter what value I put in that bean.
Thanks in advance If someone could help with these issues above
James
How to separate our keys by Redis DB ?
You can use Redis in cluster mode with sharing data (http://redis.io/topics/cluster-tutorial)
I don't think you need special configuration in Mule.
I think you mix Partition term in Mule and Partition term in Redis.
Regards,
I set up a zookeeper (version 3.4.6) for my solr5.4. my zoo.conf looks like this.I am not able to find error logs in /var/log or var/log/zookeeper. am I missing anything?
Thanks in advance
tickTime=2000
dataDir=/var/zookeeper/
dataLogDir=/var/log/
clientPort=2181
initLimit=5
syncLimit=2
server.1=0.0.0.0:2888:3888
server.2=h1:2888:3888
server.3=h2:2888:3888
ZooKeeper uses log4j for actual logging, so you have to see your log4j.properties. The way ZK is set up by default, you can also override some stuff via JVMs system properties, so you might need to check your startup script as well.
The properties in zoo.cfg (dataDir and dataLogDir) are used for ZK application data, like the transaction log, so don't confuse that with standard log4j logging.
Hi i want to start more than one gemfire cache-server on same host using Spring gemfire 8.1.Please find below gemfire configuration file. I want to start GFServer1 and GFServer2 on same host i.e.HOSTNAME using Spring Gemfire configuration. I want to avoid gfsh command and start everything from eclipse and connect client to these servers on the same host.
Thanks in advance
<util:properties id="gemfireProperties">
<prop key="name">Locator_Dev</prop>
<prop key="mcast-port">0</prop>
<prop key="locators">HOSTNAME[1099]</prop>
<prop key="log-level">warning</prop>
<prop key="http-service-port">8181</prop>
<prop key="jmx-manager">true</prop>
<prop key="jmx-manager-port">1199</prop>
<prop key="jmx-manager-start">true</prop>
<prop key="start-locator">HOSTNAME[1099]</prop>
</util:properties>
<gfe:cache properties-ref="gemfireProperties" />
<gfe:cache-server id="GFServer1" auto-startup="true"
bind-address="HOSTNAME" port="40411" host-name-for-clients="HOSTNAME"
load-poll-interval="2000" max-connections="22" max-threads="16"
max-message-count="1000" max-time-between-pings="30000" >
<gfe:subscription-config eviction-type="ENTRY"
capacity="1000" disk-store="diskStore1" />
</gfe:cache-server>
<gfe:cache-server id="GFServer2" auto-startup="true"
bind-address="HOSTNAME" port="40412" host-name-for-clients="HOSTNAME"
load-poll-interval="2000" max-connections="22" max-threads="16"
max-message-count="1000" max-time-between-pings="30000" >
<gfe:subscription-config eviction-type="ENTRY"
capacity="1000" disk-store="diskStore1" />
</gfe:cache-server>
<gfe:disk-store id="diskStore1" queue-size="50"
auto-compact="true" max-oplog-size="10" time-interval="9999">
<gfe:disk-dir
location="D:\NP\WorkSpace\GemfireRegionSolutionNStart\disk-store\store_1"
max-size="20" />
<gfe:disk-dir
location="D:\NP\WorkSpace\GemfireRegionSolutionNStart\disk-store\store_2"
max-size="20" />
</gfe:disk-store>
<gfe:replicated-region id="customer" name="Customer">
</gfe:replicated-region>
<gfe:replicated-region id="bookMaster" name="BookMaster">
</gfe:replicated-region>
</beans>
The configuration you have posted will create two cache-servers within the same JVM, i.e it will open up two ports within the same process.
If this is not what you want, i.e you want two distinct process, in eclipse you will have to provide two runtime configurations to start the two servers.
Is there a specific question? As #Swapnil points out, this will start 2 GemFire "Cache Servers" (ServerSockets listening for Cache Clients) as you have appropriately configured on the same host within the same JVM. This will work regardless of how this is executed (i.e. IDE, command-line, from Gfsh or from Spring Boot).
Let us know if you have a more specific question, thanks!
So you can configure the LocatorLauncherFactoryBean, for example, like so...
<uti:properties id="gemfireProperties">
<prop key="log-level">config</prop>
<prop key="http-service-port">8181</prop>
<prop key="jmx-manager">true</prop>
<prop key="jmx-manager-port">1199</prop>
<prop key="jmx-manager-start">true</prop>
<prop key="locators">host1[10334],host2[11235],...,hostN[20668]</prop>
</util:properties>
<bean id="locator" class="org.spring.data.gemfire.config.LocatorLauncherFactoryBean">
<property name="gemfireProperties" ref="gemfireProperties"/>
<property name="memberName" value="SpringDataGemFireLocator"/>
<property name="bindAddress" value="10.124.12.24"/>
<property name="port" value="12480"/>
</bean>
As you may have noticed, this Locator can join other Locators in the GemFire Cluster, which were specified in the "gemfireProperties" bean with the "locators" GemFire System property.
NOTE: the "bindAddress" property to the LocatorLauncherFactoryBean is only necessary if the localhost where this Locator will be running has multiple NICs and you want to bind to a specific NIC.
Also, I have set the JMX Manager GemFire System properties to enable the Locator to become and actually start a Manager (on port 1199). This allows you to connect to this Locator from Gfsh either with gfsh>connect --locator=localhost[12480] or with gfsh>connect --jmx-manager=localhost[1199].
Basically, the "gemfireProperties" bean allows you to configure any valid GemFire System property.
Now, since this Locator is running from within your IDE, you will need to configure the "run profile" with a $GEMFIRE environment variable pointing at a GemFire distribution downloaded from Pivotal's website in order to get Pulse running from this Locator. This is expected by the GemFire Manager's ManagementAgent when making a decision of whether to 1. start the embedded HTTP Service (Jetty) running GemFire's out-of-box webapps (e.g. Pulse) and 2. whether it can find Pulse and start the webapp. The ManagementAgent looks for Pulse in the distro.
For instance, I set my $GEMFIRE environment variable to...
/Users/jblum/Downloads/Pivotal/GemStone/Products/GemFire/Pivotal_GemFire_820_b17919_Linux
Now, to get your individiual Spring-configured GemFire Servers to connect to the cluster, that is simple.
Again, you only need a "gemfireProperties" bean defined in each Spring GemFire Server XML configuration file with the "locators" GemFire System property defined, e.g. ...
<uti:properties id="gemfireProperties">
<prop key="log-level">config</prop>
<prop key="locators">localhost[12480]</prop>
</util:properties>
<gfe:cache properties-ref="gemfireProperties"/>
This configuration will enable the GemFire Data Nodes to connect to the cluster, and this cluster will be visible from Pulse, if everything is setup correctly.
Again, hope this helps.
Cheers,
John
OK... so you have a few options from within your IDE (e.g. Eclipse).
If you break the Spring config above into 2 separate Spring (Data GemFire) XML configuration files, each containing 1 of the <gfe:cache-server> elements, call these files, for example, spring-gemfire-server1-context.xml and spring-gemfire-server2-context.xml, you could run the Spring-based/configure GemFire "data nodes" and Cache Servers with the following...
SimpleSpringApplication
Where the argument to the simple little Java "main" program is the file system path to 1 of the Spring XML configuration files noted above.
Better yet, you can launch these Spring (GemFire) configs with a SpringBoot application using something like...
UsefulSpringBootGemFireApplication
As you can see here, you can use either the #Import annotation if you are configuring GemFire from Spring using Java-based Configuration (for example, UsefulSpringBasedGemFireConfiguration), or in your case, with XML using the
#ImportResoruce("/classpath/to/spring-gemfire-server1-context.xml") annotation.
It maybe possible (not sure) to pass a System property value into the #ImportResource annotation, like so...
#SpringBootApplication
#ImportResource("${spring-gemfire-context-xml-location}")
class SpringBootGemFireApplication {
...
}
Then, from within your IDE, you can create 2 separate "run profiles" using the same SpringBootGemFireApplication class and then setting the System property (e.g. -Dspring-gemfire-context-xml-location=/path/to/spring-gemfire-server1-context.xml) appropriately for each configuration.
Remember, you can use Spring's ResourceLoader path qualifiers (e.g. file:, http:, etc) to resolve your Spring GemFire config from different source locations (CLASSPATH, file system, etc)... see the table ("Table 7.1 Resource strings") in hyperlinked section of the Spring Framework Reference Guide.
In the worst case, you need to create 2 separate, but nearly identical SpringBoot application Java main classes to launch your 2 configuration files for each GemFire Data Node (& CacheServer) JVM process. However, hopefully using the System property approach allows you to recycle the same class in 2 separate run profiles.
So, this leaves you with 2 GemFire Data Node JVM processes now. But, what about he Locator?
Well, you can continue to embed the Locator in the GemFire Data Node Server JVM process as before, but if you really want a standalone Locator JVM process, then you can use the following, "experimental" class...
LocatorLauncherFactoryBean
This class uses GemFire's LocatorLauncher class to configure and bootstrap a GemFire Locator from Spring config. I created this class over 2 years ago for a customer POC as example of how a developer might configure a GemFire Locator from Spring config.
The Spring XML configuration (used by the SpringBootGemFireApplication #ImportResoruce annotation, in yet another "run profile" and System property combination), would look similar to the following...
locator.xml
Now, you have effectively achieved 3 separate GemFire JVM processes (2 Servers and 1 Locator). You can scale this from with our IDE to however many Servers and Locators you want, and Pulse will show all of these in the cluster, providing the cluster is configured correctly (Servers pointed at Locators, etc).
Hope this helps.
Cheers!
John
In Knox config file in Ambari we have defined:
<url>http://{{namenode_host}}:{{namenode_http_port}}/webhdfs</url>
The problem is we have 2 namenodes, one active and one passive for high availability. Our active namenode01 failed so namenode02 became active.
This caused problems for a lot scripts as they were hardcoded to point to namenode01. So we used a command to failover namenode02 back to namenode01 using a terminal, not Ambari.
Now, the macro {{namenode_host}} is defined as namenode02 and not namenode01.
So, where is {{namenode_host}} defined?
Or, do we need to failover namenode01 to namenode02, then failover again to namenode01 using Ambari to update the macro?
If we need to failover the namenode using Ambari, I'm assuming we need to select the "Restart" option? There isn't a direct failover command.
See issue here:
https://issues.apache.org/jira/browse/AMBARI-12763
This was committed to Ambari to support HA mode for Knox. However if you're still looking for the location take a look at the file that's edited in the patch. That file is the place where the macros are set. You'll have to find it on your local machine though.
Should be something like params_linux.py