I am writing a simple adapter that reads data from hive server and send it to my application.
This adapter should run on all hive distributions(Horwonworks,Cloudera,MapR and others), so I need to add the local machine(with installed hive client) hive-jdbc standalone jar path to the classpath when running the adapter(along with some hadoop jars it seems), because I don't want to pack the jars along with my application to prevent client server version mismatch.
My question:
How can I find the hive-jdbc standalone jar in the hive client location that will work on all the distributions?
Is there an enviorment varialbe set to the hive libs location, and if so how can i set it?
Thanks
Related
I designed a ktr file for transformation. I need to configure the database connection details of production environment. How can I do this? Any suggestions?
I use environment variables.
KETTLE_HOME
KETTLE_JNDI_ROOT
PATH=$PATH:$KETTLE_HOME
Kettle home is just a link to directory. By default i have directory specially devoted to data-integration suite. It contains several versions of kettle.
Example
/opt/kettle/data-integration-4.4.0 (few old jobs made like several years ago)
/opt/kettle/data-integration-5.2.0 (currenly used)
/opt/kettle/data-integration-6.0.0 (on test)
then there is a link to current default (something like alternatives in debian). It helps to have several versions of kettle on same machine.
ln -s /opt/kettle/data-integration-5.2.0 /opt/kettle/data-integration
now
/opt/kettle/data-integration
is a link to main suite which will be used by default. This is made for jdk compbility reasons. Since 4.4.0 is jdk6 based, 5.2.0 jdk7 (and doesn't work on jdk8 since there is a bug in xstream xml library, but may be fixed already, 6.0 branch works well on jdk8 even it is built on jdk7)
Variables configured in /etc/profile.
File jdbc.properties is located in
$KETTLE_HOME/simple-jndi
and shared by all jobs. This file, with connection settings, generated by puppet if network configuration changes. If administrators do something, puppet automatically will make new file in case of migrations.
For connection definitions I use naming rules
main -connection to productive database
main_slave -connection to slave database (read-only)
test -connection to test database (on separate machine)
test_slave
and so on ...
i am completely new to weblogic and hsql so if the question seems trivial please bear with me, so here's my question.
I have application that uses hsql for backend, the problem is that i am not able to configure hsql for weblogic, whenever i Test Configuration its shows
Connection test failed.
Cannot load driver: org.hsqldb.jdbc.JDBCDriver
I know it requires hsqldb.jar and i have put it in C:\bea\weblogic92\server\lib, i have used this site for reference http://docs.jboss.org/seam/snapshot/en-US/html/weblogic.html go to "39.2.2.1. Setting up the hsql datasource" here it says "Copy hsqldb.jar to the Weblogic domain's shared library directory: cp $SEAM_HOME/lib/hsqldb.jar $BEA_HOME/user_projects/domains/seam_examples/lib" i am not able to understand this also.
Please Help
Thanks
In general, there are two things to check.
Make sure the HSQLDB Jar is the version you need (2.x).
Which directory to put hsqldb.jar. This directory is either the general jar lib directory where all the jars used by the system are stored, or it is the jar lib directory for your application. The example for seam is for the second type.
I am using Axway server which runs on jre 1.5 . So it accepts all classed builded below 1.6 . But problem is I read THIS LINK and I got this line as guide
Environment:
* Java Developer Kit (JDK) 1.4.x or greater for deployment and 1.5.x (Java 5) for compiling/building. As of AMQ 5.5.0 you need JDK 1.6.0 to**
Which says I can use ActiveMQs below version 5.5 for jre 1.5.
when I use ActiveMQ 5.4.3 am getting bad class version error.Please help me.
This is the line where am getting this error
Object o = Class.forName("org.apache.activemq.ActiveMQConnectionFactory",
true,ConnFactory.class.getClassLoader())
.newInstance();
You could extract the jar as a zip file, and then look inside the folder and check inside the META-INF directory. The MANIFEST.MF file will tell you what version of the jvm the jar was compiled with. I just grabbed the 5.4.3 version of the activemq jar and it said this:
Manifest-Version: 1.0
Archiver-Version: Plexus Archiver
Created-By: Apache Maven
Built-By: chirino
**Build-Jdk: 1.6.0_26**
Specification-Title: ActiveMQ :: Jar Bundle
Specification-Version: 5.4.3
Specification-Vendor: The Apache Software Foundation
Implementation-Title: ActiveMQ :: Jar Bundle
Implementation-Version: 5.4.3
Implementation-Vendor-Id: org.apache.activemq
Implementation-Vendor: The Apache Software Foundation
which clearly shows that it was built with the 1.6 jvm. Might need to download the source and build it on 1.5 manually.
In maven repository the last version of activemq built on jdk 1.5 is 5.4.2.
Problem is due to the jvm config file entry for jar newly added.
I don't know and I am not responsible and I don't have permission to do any Admin related privileges.
I put the jar in CLASSPATH exactly.So its ok for all the classes to access that.
But problem is in JVM config file I have to put the entry, I mean I have to give name of the jar with CLASSPATH in jvm.config file of axway.Though I put my jar in class path without put entry in jvm.config, it is worthless.Because while jvm get initialized it checks the config file first to add jar with it.
I am trying to use the MSSQL driver for JDBC. I have followed the instructions on how to add an external JAR file to the Eclipse Windows IDE as follow:
- Download the installation from here http://msdn.microsoft.com/en-us/sqlserver/aa937724.aspx (Microsoft JDBC Driver 4.0 for SQL Server)
- Extract the zip to a file location I have extracted it here (c:\MsJDBCforSqlDriver)
- In Eclipse Package Explorer right click your project -> Build Path -> Configure Build Path -> Java Build Path -> Libraries Tab- Add External JAR file and then I browse to
"C:\MsJDBCforSqlDriver\sqljdbc_4.0\enu\sqljdbc4.jar"
- I then view the successful reference as in image below including SQLServerDriver.class
When I try to execute Class.forName("com.microsoft.sqlserver.jdbc.SQLServerDriver") I get a ClassNotFound Exception as below:
I tried all suggestions I could find on Google including setting the classpath in Windows Environment Variables.
Under the Run -> Run Configurations menu item you should find an entry for a Java application with the same name as the class that contains your main() method.
Have a look at the classpath tab. You should find your project listed under user entries. Expand it and see if the Microsoft JDBC Jar is listed.
The class path you use for compilation need not be the same as you choose for runtime, though Eclipse usually mirrors any changes to your build path to any launch configurations.
Your spelling and case for the package and class names look correct (Java is case-sensitive on names).
And yes, learning Java and Eclipse will take effort - Eclipse is quite different from other IDEs and it takes time to get to know it.
I found the answer here: http://code.google.com/p/android/issues/detail?id=27490
Seems like includes under Referenced Libraries is not available at runtime and I couldn't see how to change this in my version of Eclipse
"
Reported by fred...#mobileinteraction.se, Mar 22, 2012
Host OS: Windows 7 x64
SDK tools version: 17
Eclipse version: 3.7.2
ADT plug-in version:17.0.0v201203161636-291853
Platform targeted by your project: 2,2
Version of the platform running in the emulator: 2.2
STEPS TO REPRODUCE:
1. Create a project that uses external jars (in my case android-support-v4.jar ).
2. Added the android-support-v4.jar using Properties - Java Build Path - Add External Jar. Have the Activity extends from FragmentActivity.
3. Clean and run project
EXPECTED RESULTS: Application would run launching the FragmentActivity
OBSERVED RESULTS: Application crashes, android-support-v4.jar is placed within "References Libraries" and thus Android can't find it at runtime.
SOLUTION:
Manualy create a folder /libs within your project, copy the android-support-v4.jar to that folder and the ADT will place it under 'Android Dependencies' and it works.
This is the case for all external jars added using Properties - Java Build Path - Add External Jar.
"
Adding the JDBC driver of MS SQL Server to Eclipse:
Download the JDBC driver and uncompress it
Secondary click in the root of the Eclipse project and go to "Properties".
Now go to the section "Java Build Path - Libreries" y then click "Add External JARs".
Search the "sqljdbc.jar" file in the directory where you unpacked the downloaded driver, double click y finish pressing "ok".
I faced the problem and then solved the issue i.e. If You are using the Tomcat as a server for your Web application, the same Exception was coming as the Driver was unavailable to the project, later I put the jar file (sqljdbc4-2.0.jar) into the WEB-INF/lib folder then it was working
Using a test app to figure out problems with production application and hsqldb.
If I compile the test app with the hsqldb.jar file as an external jar in eclipse the program works. However, if I only have the jar file in my JRE system library I get a pre-9.0 client attempted to connect error from the database.
The two files are the exact same files. Only difference is one was added as external jar and the other was just in the class path. When production app builds it is using classpath to same file and generating the same pre-9.0 error.
The jar you want to use is an HSQLDB version 2.0 jar. There is a second HSQLDB jar which is version 1.8 and is in your classpath before your intended jar.
This is a classpath issue. You should be able to find where the second jar (1.8) is and how it gets into the classpath by using java -verbose or other debugging options.
[added] An additional step you could take to debug this in your application is to print out the value of the field org.hsqldb.persist.HsqlDatabaseProperties.THIS_FULL_VERSION before connecting to the database. This will print the version of the jar used as the client. A successful connection means the client and server jars are the same version.
The server instance prints out its version to the console when it starts.
I would also recommend using the latest snapshot jar for HSQLDB from the http://hsqldb.org/support/ page.
Sorted out problem by moving hsqldb.jar to first declared in classpath.