Cannot find hive odbc connector error messages using unixODBC - sql

I am trying to setup unixodbc to use the hive driver connector from cloudera (in an Ubuntu machine).
In my ~/.local/lib folder I have links to the .so files provided by cloudera,
also the env variable LD_LIBRARY_PATH contains /home/luca/.local/lib:/opt/cloudera/hiveodbc/lib/64/.
I created the file /etc/odbcinst.ini containing the following:
[hive]
Description = Cloudera ODBC Driver for Apache Hive (64-bit)
Driver = /home/luca/.local/lib/libclouderahiveodbc64.so
ODBCInstLib= /home/luca/.local/lib/libodbcinst.so
UsageCount = 1
DriverManagerEncoding=UTF-16
ErrorMessagesPath=/opt/cloudera/hiveodbc/ErrorMessages/
LogLevel=0
SwapFilePath=/tmp
and in my home folder I have .odbc.ini containing:
[hive]
Driver=hive
HOST=<thehost>
PORT=<theport>
Schema=<theschema>
FastSQLPrepare=0
UseNativeQuery=0
HiveServerType=2
AuthMech=2
#KrbHostFQDN=[Hive Server 2 Host FQDN]
#KrbServiceName=[Hive Server 2 Kerberos service name]
UID=<myuid>
When I test the connection using isql -v hive
I get the following error message:
[S1000][unixODBC][DSI] The error message NoSQLGetPrivateProfileString could not be found in the en-US locale. Check that /en-US/ODBCMessages.xml exists.
[ISQL]ERROR: Could not SQLConnect
How can I fix this issue (why is the path absolute for /en-US/)?

The SQLGetPrivateProfileString was not found in your ODBCInstLib library. Either the library could not be loaded, or the library did not contain the symbol.
Use strace isql -v hive 2>&1 | grep ini to see if your configuration file is being loaded. Use strace isql -v hive 2>&1 | grep odbcinst.so to see where it is looking for the library.
Make sure that the library exists in the given location and has the correct architecture. Use file -L /home/luca/.local/lib/libodbcinst.so to check the architecture. Use nm /home/luca/.local/lib/libodbcinst.so | grep SQLGetPrivateProfilString to check if it has the correct symbol.

Related

Apache Airflow command not found with SSHOperator

I am trying to use the SSHOperator to SSH into a remote machine and run an external application through the command line. I have setup the SSH connection via the admin page.
This section of code is used to define the commands and the SSH connection to the external machine.
sshHook = SSHHook(ssh_conn_id='remote_comp')
command_1 ="""
cd /files/232-065/Rans
bash run.sh
"""
Where 'run.sh' runs the shell script:
#!/bin/sh
starccm+ -batch run_export.java Rans_Model.sim
Which simply runs the commercial software starccm+ with some options I have specified.
This section defines the task:
inlet_profile = SSHOperator(
task_id='inlet_profile',
ssh_hook=sshHook,
command=command_1
)
I have confirmed the SSH connection works by giving a simple 'ls' command and checking the output.
The error that I get is:
bash run.sh, error: run.sh: line 2: starccm+: command not found
The command in 'run.sh' works when I am logged into the machine (it does not require a GUI). This makes me think that there is a problem with the SSH session and it is not the same as the one that Apache Airflow logs into, but I am not sure how to solve this problem.
Does anyone have any experience with this?
There is no issue with SSH connection (at least from the error message). However, the issue is with starccm+ installation path.
Please check the installation path of starccm+ .
Check if the installation path is part of $PATH env variable
$ echo $PATH
If not, then install it in the standard locations like /bin or /usr/bin etc (provided they are included in $PATH variable), or export the installed director into PATH variable like this,
$ export PATH=$PATH:/<absolute_path>
It is not ideal but if you struggle with setting the path variable you can run starccm stating the full path like:
/directory/where/star/is/installed/starccm+ -batch run_export.java Rans_Model.sim

libclouderahiveodbc64.so file not found

I have problem using latest Cloudera Hive ODBC Driver 2.6.9 for 64-bit AIX 7.1. On Linux host such configuration works flawlessly
$ oslevel -s
7100-03-05-1524
$ getconf KERNEL_BITMODE
64
It seems that isql works itself but cannot load driver with following message "file not found"
$ ../bin/isql_64 -v "Cloudera Hive 64-bit" user pass
[01000][unixODBC][Driver Manager]Can't open lib '/build/lib/libclouderahiveodbc64.so' : file not
found
[ISQL]ERROR: Could not SQLConnect
UnixODBC as well as driver are 64-bit. Isql is working itself
$ file /build/lib/libclouderahiveodbc64.so
/build/lib/libclouderahiveodbc64.so: 64-bit XCOFF executable or object module not stripped
$ file ../bin/isql_64
../bin/isql_64: 64-bit XCOFF executable or object module
$ ../bin/isql_64 --version
unixODBC 2.3.9
Library paths are set properly:
echo $LIBPATH
/build/lib
$ echo $LD_LIBRARY_PATH
/build/lib
Environment variables are pointing to my configs which also seem fine:
$ env | grep INI
CLOUDERAHIVEINI=/build/lib/cloudera.hiveodbc.ini
ODBCSYSINI=/build/etc
ODBCINI=/build/etc/odbc.ini
Driver is pointed correctly in *.ini files
$ cat /build/etc/odbc.ini | grep Driver
Cloudera Hive 64-bit=Cloudera ODBC Driver for Apache Hive 64-bit
Description=Cloudera ODBC Driver for Apache Hive (64-bit) DSN
# Driver: The location where the ODBC driver is installed to.
Driver=/build/lib/libclouderahiveodbc64.so
$ cat /build/etc/odbcinst.ini | grep Driver
[ODBC Drivers]
Cloudera ODBC Driver for Apache Hive 64-bit=Installed
[Cloudera ODBC Driver for Apache Hive 64-bit]
Description=Cloudera ODBC Driver for Apache Hive (64-bit)
Driver=/build/lib/libclouderahiveodbc64.so
Neither do ldd or isql show missing dependencies
$ ldd libclouderahiveodbc64.so
libclouderahiveodbc64.so needs:
/usr/lib/libpthread.a(shr_xpg5_64.o)
/usr/lib/libc.a(shr_64.o)
/usr/lib/libC.a(shr_64.o)
/usr/lib/libC.a(shr2_64.o)
/usr/lib/libC.a(shr3_64.o)
/usr/lib/libC.a(ansi_64.o)
/unix
/usr/lib/libcrypt.a(shr_64.o)
/usr/lib/libC.a(shrcore_64.o)
/usr/lib/libC.a(ansicore_64.o)
$ ldd ../bin/isql_64
../bin/isql_64 needs:
/usr/lib/libc.a(shr_64.o)
/usr/lib/libpthread.a(shr_xpg5_64.o)
/build/lib/libreadline.a(libreadline.so.7)
/build/lib/libodbc.a(libodbc.so.2)
/build/lib/libiconv.a(libiconv.so.2)
/unix
/usr/lib/libcrypt.a(shr_64.o)
/usr/lib/libc.a(shr.o)
/usr/lib/libcurses.a(shr42.o)
/build/lib/libgcc_s.a(shr.o)
/usr/lib/libpthread.a(shr_xpg5.o)
/usr/lib/libcrypt.a(shr.o)
/usr/lib/libpthreads.a(shr_comm.o)
Any tips kindly appreciated. Thanks !

Using "Remote SSH" in VSCode on a target machine that only allows inbound SSH connections

Is there a way to use the VSCode Remote SSH extension to interact with a remote host that does not allow outbound internet connections?
Is it possible to download the vscode-server files from another system and copy to host?
I read this but I can't connect the server to internet.
When you connect to a host it executes a bash script that wgets or curls a tarball and extracts it in a directory in your home directory. Here's an offline workaround.
Attempt to connect, let it fail
On server, get the commit id
$ ls ~/.vscode-server/bin
553cfb2c2205db5f15f3ee8395bbd5cf066d357d
Download tarball replacing $COMMIT_ID with the the commit number from the previous step
For Stable Version
https://update.code.visualstudio.com/commit:$COMMIT_ID/server-linux-x64/stable
For Insider Version
https://update.code.visualstudio.com/commit:$COMMIT_ID/server-linux-x64/insider
Move tarball to ~/.vscode-server/bin/$COMMIT_ID/vscode-server-linux-x64.tar.gz
Extract tarball in this directory
$ cd ~/.vscode-server/bin/$COMMIT_ID
$ tar -xvzf vscode-server-linux-x64.tar.gz --strip-components 1
Connect again
You'll still need to install any extensions manually. There's a download button next to all the extensions in the marketplace. Once you have the .vsix file you can install them through the GUI with the Install from VSIX option in the extensions manager.
This is kind of a pain and hopefully they improve this process, but if you have a network-based home directory, you only have to do this once.
open vscode -> about
Version: 1.46.1
Commit: cd9ea6488829f560dc949a8b2fb789f3cdc05f5d
Date: 2020-06-17T21:17:14.222Z
Electron: 7.3.1
Chrome: 78.0.3904.130
Node.js: 12.8.1
V8: 7.8.279.23-electron.0
OS: Darwin x64 17.7.0
$COMMIT_ID = cd9ea6488829f560dc949a8b2fb789f3cdc05f5d
A new feature is being added to support offline install
However, you can now solve this issue by a new user setting in the Remote - SSH extension. If you enable the setting remote.SSH.allowLocalServerDownload, the extension will install the VS Code Server on the client first and then copy it over to the server via SCP.
Note: This is currently an experimental feature but will be turned on by default in the next release
https://code.visualstudio.com/blogs/2019/10/03/remote-ssh-tips-and-tricks
A a work around I have done the following:
Desktop ~/.ssh/config
...
Host *
RemoteForward 54321
...
Remote: ~/bin/wget in which ~/bin is added to PATH via .bashrc
#!/bin/bash
export LD_LIBRARY_PATH=$HOME/opt/lib/tsocks/
export TSOCKS_CONF_FILE=$HOME/opt/tsocks/tsocks.conf
$HOME/bin/tsocks /usr/bin/wget $#
Remote: ~/opt/tsocks/tsocks.conf
server = 127.0.0.1
server_port = 54321
server_type = 5
note tsocks binary has been scp-ed to ~/bin/tsocks and ~/opt/tsocks/ has been created with libtsocks.so which is normally stored in /usr/lib64/libtsocks.so
This is a work around that allows me to have wget functionality with out messing with anything outside my profile to get it to work (eg: no root required ... even though I have it).
Current Version of VS Code: 1.48.2
I just kill the wget process on the server end, and let the client download the archive and transfer it to the server end. That's quite easy as below.
make sure that you set in settings.json
"remote.SSH.allowLocalServerDownload": true,
execute the shell scrpits below.
# to find the <pid>
ps aux | grep wget | grep vscode-server
# kill the process
kill -9 <pid>
# then wait for the client downloading and transferring
# optional: If you want to know the progress, just
cd ~/.vscode-server/bin/<commit-id>/
watch -n 1 -d ls -rthl

I m trying to integrate ldap with devstack and when i did ./stack.sh i got this localrc: line 9: KEYSTONE_IDENTITY_BACKEND: command not found

localrc file
ADMIN_PASSWORD=password2 MYSQL_PASSWORD=password2
RABBIT_PASSWORD=password2 SERVICE_PASSWORD=password2
SERVICE_TOKEN=token2
ENABLED_SERVICES=key,n-api,n-crt,n-obj,n-cpu,n-net,n-cond,cinder,c-sch,c-api,c-vol,n-sch,n-novnc,n-xvnc,n-cauth,horizon,mysql,rabbit,ldap
KEYSTONE_IDENTITY_BACKEND=ldap
KEYSTONE_CLEAR_LDAP=yes LDAP_PASSWORD=9632
I followed this website(http://www.ibm.com/developerworks/cloud/library/cl-ldap-keystone/)
I am assuming the above snippet is from a file written in shell script. Your example looks Ok.
I checked the link you provided and noted that the line you say failed is written in the IBM example as:
KEYSTONE_IDENTITY_BACKEND = ldap
Which is not legal sh (or bash) and would cause the error message you described.
KEYSTONE_IDENTITY_BACKEND = ldap
-bash: KEYSTONE_IDENTITY_BACKEND: command not found
I suspect you copied and pasted the bad example from the link into your localrc file, which caused the error you saw, but somehow when you wrote the SO question, you corrected the mistake by removing the spaces around the "=".
Edit: Investigation
;TLDR
Create a file in the root of the devstack repo, devstack/local.conf with the contents:
[[local|localrc]]
ADMIN_PASSWORD=password2
MYSQL_PASSWORD=password2
RABBIT_PASSWORD=password2
SERVICE_PASSWORD=password2
SERVICE_TOKEN=token2
ENABLED_SERVICES=key,n-api,n-crt,n-obj,n-cpu,n-net,n-cond,cinder,c-sch,c-api,c-vol,n-sch,n-novnc,n-xvnc,n-cauth,horizon,mysql,rabbit,ldap
KEYSTONE_IDENTITY_BACKEND=ldap
KEYSTONE_CLEAR_LDAP=yes
LDAP_PASSWORD=9632
Full Description
I installed devstack on Centos7 (using the Devstack Quick Start Guide):
git clone https://git.openstack.org/openstack-dev/devstack
cd devstack
./stack.sh
I entered passwords as prompted, but eventually it failed with the error:
Error: pg_config executable not found.
Please add the directory containing pg_config to the PATH
or specify the full executable path with the option:
python setup.py build_ext --pg-config /path/to/pg_config build ...
or with the pg_config option in 'setup.cfg'.
I traced the problem to a limited PATH in the sudoers entry, and because my postgreSQL install is in a non-standard location, I linked pg_config into /usr/local/bin and ran stack.sh again:
sudo ln -s /usr/pgsql-9.3/bin/pg_config /usr/local/bin/pg_config
./stack.sh
(You probably won't have to do this if Postgres is in a standard location).
Install took a long time -
This is your host IP address: 192.168.200.181
This is your host IPv6 address: ::1
Horizon is now available at http://192.168.200.181/dashboard
Keystone is serving at http://192.168.200.181/identity/
The default users are: admin and demo
The password: 12345678
2016-07-17 18:16:32.834 | WARNING:
2016-07-17 18:16:32.834 | Using lib/neutron-legacy is deprecated, and it will be removed in the future
2016-07-17 18:16:32.834 | stack.sh completed in 1447 seconds.
I killed the devstack session and did it all again with a clean git repo and with a localrc file.
./unstack.sh
cd ..
git clone https://git.openstack.org/openstack-dev/devstack
cd devstack
cat << __EOF > local.conf
[[local|localrc]]
ADMIN_PASSWORD=password2
MYSQL_PASSWORD=password2
RABBIT_PASSWORD=password2
SERVICE_PASSWORD=password2
SERVICE_TOKEN=token2
ENABLED_SERVICES=key,n-api,n-crt,n-obj,n-cpu,n-net,n-cond,cinder,c-sch,c-api,c-vol,n-sch,n-novnc,n-xvnc,n-cauth,horizon,mysql,rabbit,ldap
KEYSTONE_IDENTITY_BACKEND=ldap
KEYSTONE_CLEAR_LDAP=yes
LDAP_PASSWORD=9632
__EOF
./stack.sh
This time there were no password prompts, so the local config was definitely read.

Windows 7 OSQL Batch

I have just got a new work PC from running Windows 7. I have installed SQL Server 2008 and I have a batch file that runs an OSQL command as follow:
osql -S MyServer -E -d MSDB -n -o results.txt -i MyScript.sql
For some reason it is giving me exception below:
Specified driver could not be loaded due to system error 126: The specified module could not be found. (SQL Server Native Client 10.0, C:\Windows\system32\sqlncli10.dll).
I am not sure what the issue is and I have even tried to run the batch file using 'Run as administrator'.
OSQL is being deprecated; I would consider converting your code to use sqlcmd or PowerShell. However if you're missing SQL native client you might consider adding it. You can download the sqlncli.msi file for your platform from this URL (more than halfway down the page).