Am using
Apache Hadoop 1.0.4
JDK 1.7
CentOS 5.6
When I install Hadoop Cluster with 2-nodes, it runs fine. I am able to see the status through http://:50070 and http://:50030. However when I build the Hadoop native libraries from source, the cluster continues to work fine, but the monitoring app (Jetty) goes for a toss. The above links dont work.
I compile the native libraries after Hadoop installation (which is basically tar -zxvf). The namenode is already formatted before the compilation, but the cluster is not brought up until native libraries are compiled. Any ideas, how to troubleshoot this? Am using simple ant -Dcompile.native=true compile-native command to build native libraries.
Related
I have an ASP.NET Core 7 application hosted on Azure Web App Linux. I am using a native package that uses GLIBC 2.29 from Linux.
When I run the SSH, I get: ASP .NETCore Version: 7.0.0 and when I run ldd --version, I get:
ldd (Debian GLIBC 2.28-10+deb10u2) 2.28
The native package is looking for GLIBC 2.29 and if I use an older version of the native package that relies on GLIBC 2.28, then it works as expected.
My questions:
Can I bundle the GLIB 2.29 files with my release? Does this even work with native stuff (I can hear C++ developers laughing about bundling files)?
Can I have some setting on Azure Web App to request GLIB 2.29?
N.B. I am aware that I can use Docker for selecting specific components, but I am leaving this as the last resort.
Can I bundle the GLIB 2.29 files with my release?
Yes, it is possible to bundle the GLIB files with the release. But it is not a better approach to do, as we may get the compatibility issues.
Deployed ASP.NET Core 7 to Azure Linux App service.
When I checked the version, Even I have the same 2.28 version.
In the deployed Azure Linux App, open the SSH.
Link to open SSH - https://YourWebAppName.scm.azurewebsites.net/webssh/host.
I have tried to install/ update the latest version by using the below command.
apt-get install libc6
As GLIBC 2.29 is not available in Azure Linux, we can download the source and install from GNU C Library - Sourceware.org.
Check the below steps to install GLIBC 2.29 Version in local Linux.
Run the below commands.
wget https://ftp.gnu.org/gnu/libc/glibc-2.29.tar.gz
tar -xvzf glibc-2.29.tar.gz
Navigate to the glibc-2.29 root directory.
cd glibc-2.29
make
make install
This cannot be installed directly in Azure Linux App Service.We can follow the same process in Linux Virtual Machine.
I can use Docker for selecting specific components
Yes,we can create a custom image in Azure Linux app service and deploy to Azure Container Registry.
Reference taken from zoomadmin.
I work with Cloudera Manager CDH 5.7.1, which supports only Hive 1.1.0.
NiFi 1.0.0-BETA uses Hive 1.2.1.
When I try to use SelectHiveQL processor, I get the following error: Required field 'client_protocol' is unset!, which means that there's a version mismatch between Hive client and server.
Any suggestions to solve this problem?
I thought about building NiFi with hive-jdbc dependency version 1.1.0 instead of the default 1.2.1, but I hope there's a better solution.
Since NiFi is an Apache project, it builds with Apache JARs (such as Hive and Hadoop). However there are vendor-specific profiles and build properties you can use to build NiFi for a particular Hadoop distribution.
For example you could try the following to build a NiFi distro for CDH 5.7.1:
mvn clean install -DskipTests -Pcloudera -Dhadoop.version=2.6.0-cdh5.7.1 -Dhive.version=1.1.0-cdh5.7.1 -Dhbase.version=1.2.0-cdh5.7.1
The Hive processors use Hadoop libraries provided by the NiFi Hadoop Libraries NAR, and other NARs (like the Hadoop/HDFS processors) use this same libraries NAR, so the best approach is to build the whole thing. Otherwise you can try to replace just the Hadoop/Hive/HBase-related NARs and see if that works.
Because NiFi expects the newer version of Hive, it is necessary to remove the unsupported newer features (such as HiveStreaming and ORC support), support the older version of Thrift, and build against the Cloudera-specific libraries.
I have created a branch of the current NiFi-1.1.x release with the necessary changes to get the PutHiveQL and SelectHiveQL processors to work, which you could build as below:
git clone https://github.com/Chaffelson/nifi.git
git checkout nifi-1.1.x-cdhHiveBundle
mvn -T C2.0 clean install -Pcloudera -Dhive.version=1.1.0-cdh5.10.0 -Dhive.hadoop.version=2.6.0-cdh5.10.0 -Dhadoop.version=2.6.0-cdh5.10.0 -DskipTests
I have posted a more complete coverage of this on the Hortonworks Community forum: https://community.hortonworks.com/articles/93771/connecting-nifi-to-cdh-hive.html
I try to configure hadoop 2.3.0 on windows but its give me
Error: Could not find or load main class org.apache.hadoop.hdfs.tools.GetConf
and lots of other error like "winutils.exe","hadoop.dll messing" and "load main class" error.
Please help me to configure Hadoop on windows 8 machine.
I have installed
JDK 1.7
cygwin64
If we directly take the binary distribution of Apache Hadoop 2.3 release and try to run it on Microsoft Windows, then we'll encounter ERROR util.Shell: Failed to locate the winutils binary in the hadoop binary path.
The binary distribution of Apache Hadoop 2.3 release does not contain some windows native components (like winutils.exe, hadoop.dll etc). These are required (not optional) to run Hadoop on Windows.
So you need to build windows native binary distribution of hadoop from source codes following "BUILD.txt" file located inside the source distribution of hadoop. You can follow the following posts (applicable for hadoop 2.3 as well) for step by step guide with screen shot
Build, Install, Configure and Run Apache Hadoop 2.2.0 in Microsoft Windows OS
ERROR util.Shell: Failed to locate the winutils binary in the hadoop binary path
Follow the steps described in the below link which describes the installation of Hadoop 2.3.0 in a windows 8 machine.
http://www.srccodes.com/p/article/38/build-install-configure-run-apache-hadoop-2.2.0-microsoft-windows-os
This works perfectly.
Thanks
Installing in windows:
http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.6.0-Win/bk_installing_hdp_for_windows/content/win-chap2-singlenode.html
Prerrequisites:
http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.6.0-Win/bk_installing_hdp_for_windows/content/win-getting-ready-2-3-2.HTML
In the suggested tutorial above, the build instructions do not work for the most recent version of visual studio/.NET framework/Windows.
http://www.srccodes.com/p/article/38/build-install-configure-run-apache-hadoop-2.2.0-microsoft-windows-os
First download the desired source version and ...
You do not need the windows 7 sdk/build will fail to create the binary. For windows 8, you can build the hadoop-2.5.2-src\hadoop-common-project\hadoop-common\src\main\winutils solution and C:\hfds\hadoop-2.5.2-src\hadoop-common-project\hadoop-common\src\main\native solutions in visual studio
Then download the binary version...
and place the output files in hadoop-2.5.2-src\hadoop-common-project\hadoop-common\target\bin in the bin directory of the downloaded hadoop binary bin folder.
Then follow the remaining steps of the tutorial...
I install the cloudera CDH3 on my machine. Then I try to use eclipse plugin (JIRA MAPREDUCE-1280) to do some MR tasks. However, it seems like the plugin not work with CDH3 for some reason. It cannot connect to the DFS.
Does any get the plugin working?
CDH3 is incompatible to Apache Hadoop 0.20.2.
The Eclipse-Plugin from JIRA MAPREDUCE-1280 is built against Apache Hadoop.
It is not compatible with CDH3.
I'm new to Linux but have to port a asp.net app to Linux platform. (CentOS 5.2)
I downloaded the mono source files and manually build them on my dev box, because there's no aviable Binary package for CentOS 5.2 (almost the same as RedHat), the app works well on the dev box.
The next step is to setup the production server, which has minimal libraries installed.
My question is... how to make the Mono binary files into a install package so I don't need to download and build them in the production server.
(My dev box is the same configuration as the production one)
I have tried to copy all mono related files into the server, but with no luck... May I missed some files or some settings...
You can still get binary RPMS here
EDIT:
Recently, Mono announced support for RHEL/CentOS. You can get packages from them here.