How to get one stream from error stream and input stream when calling a script using JSCH - jsch

I am calling a script file (.sh) located on remote machine using JSCH. During execution the script outputs both error and success statements. The JSCH code what I have written exhibits two streams, InputStream & Error Stream
How can I get an single input stream that contains error and output ?
Channel channel=session.openChannel("exec");
((ChannelExec)channel).setCommand("/opt/sdp/SnapShot/bin/dumpSubscribers.ksh");;
InputStream in=channel.getInputStream();
InputStream error=((ChannelExec)channel).getErrStream();
channel.connect();
**Script output:**
\[2014-01-23 19:41:01] SnapShot: Start dumping database
szTimgExtension: sdp511 enabled
functionOfferSupport active
Failed to prepare statements: ODBC Error 'S0022', TimesTen Error 2211, ODBC rc -1
[TimesTen][TimesTen 7.0.6.8.0 ODBC Driver][TimesTen]TT2211: Referenced column O.START_SECONDS not found -- file "saCanon.c", lineno 9501, procedure "sbPtTblScanOfColRef()" [Unable to prepare statement: <Statement for getting subscriber_offer data.>.]
Database error: ODBC Error 'S0022', TimesTen Error 2211, ODBC rc -1
[TimesTen][TimesTen 7.0.6.8.0 ODBC Driver][TimesTen]TT2211: Referenced column O.START_SECONDS not found -- file "saCanon.c", lineno 9501, procedure "sbPtTblScanOfColRef()" [Unable to prepare statement: <Statement for getting subscriber_offer data.>.]
[2014-01-23 19:41:01] SnapShot: Result files:
/var/opt/fds/TT/dump//SDP1.DUMP_subscriber.v3.csv
/var/opt/fds/TT/dump//SDP1.DUMP_usage_counter.v3.csv
[2014-01-23 19:41:01] SnapShot: Finished dumping database

Initialize an Output stream for both to write to, then instead of getInputStream, use setOutputSteam and setErrStream
OutputStream out = new OutputStream();
channel.setOutputStream(out);
channel.setErrStream(out);
Note that 'out' will be closed when the channel disconnects. To prevent that behavior, add a boolean when setting the output stream:
channel.setErrStream(out, true);
channel.setOutputSteam(out, true);
This may be important if the output stream you are using for the JSCH ChannelExec session is being reused elsewhere in your code.
If you need to read the output stream into an input stream, refer to this question.

Related

SAS CLI execute error: [Microsoft][ODBC Excel Driver]String data, right truncated

I have a SaS script and is suppose to get data from a SQL Server. I finally manage to get a connection working but am now getting a new error. Below is a screen shot and also I pasted the text:
Error Screenshot
NOTE: SAS variable labels, formats, and lengths are not written to DBMS tables.
ERROR: CLI execute error: [Microsoft][ODBC Excel Driver]String data, right
truncated
ERROR: A pipe communications routine failed: The pipe has been ended. (109)
ERROR: A pipe communications routine failed: The pipe is being closed. (232)
ERROR: A pipe communications routine failed: The pipe is being closed. (232)
ERROR: A pipe communications routine failed: The pipe is being closed. (232)
ERROR: A pipe communications routine failed: The pipe is being closed. (232)
ERROR: CLI disconnect failed: Server communication failure.
I do not know anything about SaS but I need to get this working this within the next day or 2. Does anyone know how to fix this error? I hope I do not need to add anything to the script, I had inherited this from the previous data guy and it has worked.

How to use the taildir source in Flume to append only newest lines of a .txt file?

I recently asked the question Apache Flume - send only new file contents
I am rephrasing the question in order to learn more and provide more benefitto future users of Flume.
Setup: Two servers, one with a .txt file that gets lines appended to it regularly.
Goal: Use flume TAILDIR source to append the most recently written line to a file on the other server.
Issue: Whenever the source file has a new line of data added, the current configuration appends everything in file on server 1 to the file in server 2. This results in duplicate lines in file 2 and does not properly recreate the file from server 1.
Configuration on server 1:
#configure the agent
agent.sources=r1
agent.channels=k1
agent.sinks=c1
#using memort channel to hold upto 1000 events
agent.channels.k1.type=memory
agent.channels.k1.capacity=1000
agent.channels.k1.transactionCapacity=100
#connect source, channel,sink
agent.sources.r1.channels=k1
agent.sinks.c1.channel=k1
#define source
agent.sources.r1.type=TAILDIR
agent.sources.r1.channels=k1
agent.sources.r1.filegroups=f1
agent.sources.r1.filegroups.f1=/home/tail_test_dir/test.txt
agent.sources.r1.maxBackoffSleep=1000
#connect to another box using avro and send the data
agent.sinks.c1.type=avro
agent.sinks.c1.hostname=10.10.10.4
agent.sinks.c1.port=4545
Configuration on server 2:
#configure the agent
agent.sources=r1
agent.channels=k1
agent.sinks=c1
#using memory channel to hold up to 1000 events
agent.channels.k1.type=memory
agent.channels.k1.capacity=1000
agent.channels.k1.transactionCapacity=100
#connect source, channel, sink
agent.sources.r1.channels=k1
agent.sinks.c1.channel=k1
#here source is listening at the specified port using AVRO for data
agent.sources.r1.type=avro
agent.sources.r1.bind=0.0.0.0
agent.sources.r1.port=4545
#use file_roll and write file at specified directory
agent.sinks.c1.type=file_roll
agent.sinks.c1.sink.directory=/home/Flume_dump
You have to set position json file. Then the source check the position and write only new added lines to sink.
ex) agent.sources.s1.positionFile = /var/log/flume/tail_position.json

Running SQL in Stata

I am trying to load data from SQL server management studio into Stata. How do I get Stata to run the .sql file? I have used the -ado- procedure from another post, but it does not work because my database has a username and password.
Original -ado- code:
program define loadsql
*! Load the output of an SQL file into Stata, version 1.2 (dvmaster#gmail.com)
version 12.1
syntax using/, DSN(string) [CLEAR NOQuote LOWercase SQLshow ALLSTRing DATESTRing]
#delimit;
tempname mysqlfile exec line;
file open `mysqlfile' using `"`using'"', read text;
file read `mysqlfile' `line';
while r(eof)==0 {;
local `exec' `"``exec'' ``line''"';
file read `mysqlfile' `line';
};
file close `mysqlfile';
odbc load, exec(`"``exec''"') dsn(`"`dsn'"') `clear' `noquote' `lowercase' `sqlshow' `allstring' `datestring';
end;
help odbc discusses connect_options for connecting to odbc data sources. Two of which are u(userId) and p(password) which can be added to the original code written by #Dimitriy V. Masterov (see post here).
I believe you should be able to connect using SQL Server authentication by adding the u(string) and p(string) as additional options following syntax in the ado file, and then again down below following
odbc load, exec(`"``exec''"') dsn(`"`dsn'"')
This would also require that you pass these arguments to the program when you call it:
loadsql using "./sqlfile.sql", dsn("mysqlodbcdata") u(userId) p(Password)

pyhs2/hive No files matching path file and file Exists

Using the hive or beeline client, I have no problem executing this statement:
hive -e "LOAD DATA LOCAL INPATH '/tmp/tmpBKe_Mc' INTO TABLE unit_test_hs2"
The data from the file is loaded successfully into hive.
However, when using pyhs2 from the same machine, the file is not found:
import pyhs2
conn_str = {'authMechanism':'NOSASL', 'host':'azus',}
conn = pyhs2.connect(conn_str)
with conn.cursor() as cur:
cur.execute("LOAD DATA LOCAL INPATH '/tmp/tmpBKe_Mc' INTO TABLE unit_test_hs2")
Throws exception:
Traceback (most recent call last):
File "data_access/hs2.py", line 38, in write
cur.execute("LOAD DATA LOCAL INPATH '%s' INTO TABLE %s" % (csv_file.name, table_name))
File "/edge/1/anaconda/lib/python2.7/site-packages/pyhs2/cursor.py", line 63, in execute
raise Pyhs2Exception(res.status.errorCode, res.status.errorMessage)
pyhs2.error.Pyhs2Exception: "Error while compiling statement: FAILED: SemanticException Line 1:23 Invalid path ''/tmp/tmpBKe_Mc'': No files matching path file:/tmp/tmpBKe_Mc"
I've seen similar questions posted about this problem, and the usual answer is that the query is running on a different server that doesn't have the local file '/tmp/tmpBKe_Mc' stored on it. However, if that is the case, why would running the command directly from the CLI work but using pyhs2 not work?
(Secondary question: how can I show which server is trying to handle the query? I've tried cur.execute("set"), which returns all configuration parameters but when grepping for "host" the returned parameters don't seem to contain a real hostname.)
Thanks!
This happens because pyhs2 trying to find file on cluster
Solution is to have your source saved in related hdfs location instead of /tmp

sqlTool.rc contais password in plain text

I want to stop my HSQL DB from command line using command
java -jar hsqldb/lib/sqltool.jar --rcfile=sqltool.rc --sql "shutdown;" localhost-sa-myPassword
This command expects sqlTool.rc file which contains DB related information like user id and password. This RC file contains password in plain text.
Is there any way to hide that password in RC file?
Or is there any way to encrypt this RC file and use it?
Or is it possible to forcibly stop the server without giving id/pwd?
Any other approach is also welcome as long as password is not visible in plain text
Thanks in advance
I got alternative approach for this.
Connection c = DriverManager.getConnection(
"jdbc:hsqldb:file:/opt/db/testdb;shutdown=true", "SA", "");
create the connection as mentioned above.
Then execute sql query as given below
stmt.execute("shutdown");
This will shutdown the HSQL DB