I am trying to query a table that has long-raw() type data which is having a text in it. I need to export this data to a flat file. For a given id, I see there are 14 rows in the table. I am fetching the data using JDBC connection and when fetching the data using ResultSet, I am getting ArrayIndexOutOfBoundsException at 13th row. Not sure why this issue occurs.
The data in the long raw column could be large. I am suspecting it is not able to fetch all the data that is present. I might be wrong too. I cant find much information when could a ArrayIndexOutOfBoundsException occur in this scenario. Complete code is below-
import java.io.*;
import java.sql.*;
import java.util.ArrayList;
public class TestMain {
private static String URL = "jdbc:oracle:thin:#localhost:8080:xe";
private static String USER_NAME = "scott";
private static String PASSWORD = "tiger";
public static void main(String[] args) throws SQLException, IOException {
Connection newConnection = getNewConnection();
BufferedWriter bw = new BufferedWriter(new FileWriter("./extractedFile/RawDataFile.txt"));
PreparedStatement extractableRowCount = getExtractableRowCount(newConnection, (long)34212);
ResultSet foundCountRs = extractableRowCount.executeQuery();
foundCountRs.next();
int foundCount = foundCountRs.getInt(1);
//`here I get 14 as the count`
System.out.println("Available rows for id:: 34212 are "+foundCount);
foundCountRs.close();
extractableRowCount.close();
PreparedStatement fetchBinaryQueryStatement = getExtractableRow(newConnection, (long)34212);
ResultSet fetchedRowsRs = fetchBinaryQueryStatement.executeQuery();
int i=0;
while (fetchedRowsRs.next()) {
i++;
//`I see outputs upto i=13, and then I get ArrayIndexOutOfBoundsException
System.out.println("i = " + i);
String userName = fetchedRowsRs.getString("user_name");
InputStream savedTextData = fetchedRowsRs.getBinaryStream("saved_text");
bw.write(userName + ":: ");
int len = 0;
if (savedTextData != null) {
while ((len = savedTextData.read()) != -1) {
bw.write((char) len);
bw.flush();
}
}
fetchedRowsRs.close();
fetchBinaryQueryStatement.close();
}
bw.close();
}
public static Connection getNewConnection() throws SQLException {
DriverManager.registerDriver(new oracle.jdbc.OracleDriver());
return DriverManager.getConnection(URL, USER_NAME, PASSWORD);
}
public static PreparedStatement getExtractableRow(Connection connection, Long id) throws SQLException {
PreparedStatement statement = connection.prepareStatement("SELECT user_name, saved_text FROM user_email_text_data where id = ?");
statement.setLong(1, id);
return statement;
}
public static PreparedStatement getExtractableRowCount(Connection connection, Long id) throws SQLException {
PreparedStatement statement = connection.prepareStatement("SELECT count(1) FROM user_email_text_data where id = ?");
statement.setLong(1, id);
return statement;
}
}
Full stack trace of error:
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 8
at oracle.jdbc.driver.T4CMAREngineNIO.buffer2Value(T4CMAREngineNIO.java:814)
at oracle.jdbc.driver.T4CMAREngineNIO.unmarshalUB2(T4CMAREngineNIO.java:577)
at oracle.jdbc.driver.T4CMAREngineNIO.unmarshalSB2(T4CMAREngineNIO.java:557)
at oracle.jdbc.driver.T4CMAREngine.processIndicator(T4CMAREngine.java:1573)
at oracle.jdbc.driver.T4CMarshaller$StreamMarshaller.unmarshalOneRow(T4CMarshaller.java:179)
at oracle.jdbc.driver.T4CLongRawAccessor.unmarshalOneRow(T4CLongRawAccessor.java:159)
at oracle.jdbc.driver.T4CTTIrxd.unmarshal(T4CTTIrxd.java:1526)
at oracle.jdbc.driver.T4CTTIrxd.unmarshal(T4CTTIrxd.java:1289)
at oracle.jdbc.driver.T4C8Oall.readRXD(T4C8Oall.java:850)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:543)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:252)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:612)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:226)
at oracle.jdbc.driver.T4CPreparedStatement.fetch(T4CPreparedStatement.java:1023)
at oracle.jdbc.driver.OracleStatement.fetchMoreRows(OracleStatement.java:3353)
at oracle.jdbc.driver.InsensitiveScrollableResultSet.fetchMoreRows(InsensitiveScrollableResultSet.java:736)
at oracle.jdbc.driver.InsensitiveScrollableResultSet.absoluteInternal(InsensitiveScrollableResultSet.java:692)
at oracle.jdbc.driver.InsensitiveScrollableResultSet.next(InsensitiveScrollableResultSet.java:406)
I was getting the exact same exception when calling ResultSet.next() using ojdbc8-12.2.0.1:
java.lang.ArrayIndexOutOfBoundsException: 8
at oracle.jdbc.driver.T4CMAREngineNIO.buffer2Value(T4CMAREngineNIO.java:814)
at oracle.jdbc.driver.T4CMAREngineNIO.unmarshalUB2(T4CMAREngineNIO.java:577)
at oracle.jdbc.driver.T4CMAREngineNIO.unmarshalSB2(T4CMAREngineNIO.java:557)
at oracle.jdbc.driver.T4CMAREngine.processIndicator(T4CMAREngine.java:1573)
at oracle.jdbc.driver.T4CMarshaller$StreamMarshaller.unmarshalOneRow(T4CMarshaller.java:179)
at oracle.jdbc.driver.T4CLongRawAccessor.unmarshalOneRow(T4CLongRawAccessor.java:159)
at oracle.jdbc.driver.T4CTTIrxd.unmarshal(T4CTTIrxd.java:1526)
at oracle.jdbc.driver.T4CTTIrxd.unmarshal(T4CTTIrxd.java:1289)
at oracle.jdbc.driver.T4C8Oall.readRXD(T4C8Oall.java:850)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:543)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:252)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:612)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:226)
at oracle.jdbc.driver.T4CPreparedStatement.fetch(T4CPreparedStatement.java:1023)
at oracle.jdbc.driver.OracleStatement.fetchMoreRows(OracleStatement.java:3353)
The exception disappeared when I upgraded driver version to ojdbc8-18.3.0.0.
Updating JDBC driver might be worth a try, if anyone should find themselves in the same situation.
The below lines are at the end of the outer while loop but they need to be executed after the loop since you can't close the result set object and then call next()
fetchedRowsRs.close();
fetchBinaryQueryStatement.close();
As your size of array is 14 therefore the last index is 13 as it starts from 0.So you are accessing value of index greater than size of array it gives ArrayOutOfBound exception.
Related
I have requirement to read data from a database and analyse the data using pig.
I have written a UDF in java Referring following link
register /tmp/UDFJars/CassandraUDF_1-0.0.1-SNAPSHOT-jar-with-dependencies.jar;
A = Load '/user/sampleFile.txt' using udf.DBLoader('10.xx.xxx.4','username','password','select * from customer limit 10') as (f1 : chararray);
DUMP A;
package udf;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.mapreduce.InputFormat;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.RecordReader;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
import org.apache.pig.LoadFunc;
import org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigSplit;
import org.apache.pig.data.Tuple;
import org.apache.pig.data.TupleFactory;
import com.data.ConnectionCassandra;
import com.datastax.driver.core.ResultSet;
import com.datastax.driver.core.Row;
import com.datastax.driver.core.Session;
public class DBLoader extends LoadFunc {
private final Log log = LogFactory.getLog(getClass());
Session session;
private ArrayList mProtoTuple = null;
private String jdbcURL;
private String user;
private String pass;
private int count = 0;
private String query;
ResultSet result;
List<Row> rows;
int colSize;
protected TupleFactory mTupleFactory = TupleFactory.getInstance();
public DBLoader() {
}
public DBLoader(String jdbcURL, String user, String pass, String query) {
this.jdbcURL = jdbcURL;
this.user = user;
this.pass = pass;
this.query = query;
}
#Override
public InputFormat getInputFormat() throws IOException {
log.info("Inside InputFormat");
// TODO Auto-generated method stub
try {
return new TextInputFormat();
} catch (Exception exception) {
log.error(exception.getMessage());
log.error(exception.fillInStackTrace());
throw new IOException();
}
}
#Override
public Tuple getNext() throws IOException {
log.info("Inside get Next");
Row row = rows.get(count);
if (row != null) {
mProtoTuple = new ArrayList<Object>();
for (int colNum = 0; colNum < colSize; colNum++) {
mProtoTuple.add(row.getObject(colNum));
}
} else {
return null;
}
Tuple t = mTupleFactory.newTuple(mProtoTuple);
mProtoTuple.clear();
return t;
}
#Override
public void prepareToRead(RecordReader arg0, PigSplit arg1) throws IOException {
log.info("Inside Prepare to Read");
session = null;
if (query == null) {
throw new IOException("SQL Insert command not specified");
}
if (user == null || pass == null) {
log.info("Creating Session with user name and password as: " + user + " : " + pass);
session = ConnectionCassandra.connectToCassandra1(jdbcURL, user, pass);
log.info("Session Created");
} else {
session = ConnectionCassandra.connectToCassandra1(jdbcURL, user, pass);
}
log.info("Executing Query " + query);
result = session.execute(query);
log.info("Query Executed :" + query);
rows = result.all();
count = 0;
colSize = result.getColumnDefinitions().asList().size();
}
#Override
public void setLocation(String location, Job job) throws IOException {
log.info("Inside Set Location");
try {
FileInputFormat.setInputPaths(job, location);
} catch (Exception exception) {
log.info("Some thing went wrong : " + exception.getMessage());
log.debug(exception);
}
}
}
Above is my pig script and java code.
Here /user/sampleFile.txt is a dummy file with no data.
I am getting following exception:
Pig Stack Trace
ERROR 1066: Unable to open iterator for alias A
org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias A
at org.apache.pig.PigServer.openIterator(PigServer.java:892)
at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:774)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:372)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
at org.apache.pig.Main.run(Main.java:484)
at org.apache.pig.Main.main(Main.java:158)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.io.IOException: Job terminated with anomalous status FAILED
at org.apache.pig.PigServer.openIterator(PigServer.java:884)
... 13 more
Vivek! Do you even get in prepareToRead? (I see you did some logging, so it would be nice to know what you actually have in log) Also it would be really great to provide full stacktrace as I see you don't have full underlying exception.
Just some thoughts - I never tried writing a LoadFunc without implementing my own InputFormat and RecordReader - TextInputFormat checks for file existence and it's size (and creates a number of InputSplits based on file size(s)), so if your dummy file is empty there is a big possibility that no InputSplits are produced or zero-length InputSplit is produced. As it has zero-length it may cause pig to throw that exception. So the good suggestion is to implement own InputFormat (it's actually pretty easy). Also just as a fast try - try
set pig.splitCombination false
Probably it won't help, but it's easy to try.
I'm trying to learn Selenium WebDriver and have a question that I cannot resolve long time. These are my first steps in Java and I appreciate your help.
I have a code that pools out the values from the DataBase table. This code should put that query result into an array and execute it repeatedly using every next row (For Loop), but I do not know how to do it. Currently it pools all rows, but runs only last row repeatedly (5 times). Could you please help me to create correct array with for loop? Tanks a lot in advance! Here is my code:
public class DB_TFETCHdata {
ProfilesIni listProfiles = new ProfilesIni();
FirefoxProfile profile = listProfiles.getProfile("selenium");
WebDriver oWD = new FirefoxDriver(profile);
String dbZipCode;
String dbDOBMonth;
String dbDOBDay;
String dbDOBYear;
int i = 0;
#Before
public void setUp() throws Exception{
oWD.get("https://www.ehealthinsurance.com/");
String ConnStr = "jdbc:sqlserver://localhost:1433;databaseName=TestData1; user=sa; password=1";
String DatabaseDriver = "com.microsoft.sqlserver.jdbc.SQLServerDriver";
String strSQL = "Select * FROM InfoTbl";
Class.forName(DatabaseDriver);
Connection conn = DriverManager.getConnection(ConnStr);
Statement st = conn.createStatement();
ResultSet rs = st.executeQuery(strSQL);
while(rs.next()){
dbZipCode = rs.getString("ZipCode");
dbDOBMonth = rs.getString("DOB_Month");
dbDOBDay = rs.getString("DOB_Day");
dbDOBYear = rs.getString("DOB_Year");
System.out.println(dbZipCode + "-" + dbDOBMonth + "-" + dbDOBDay + "-" + dbDOBYear);
//System.out.println("There were " + rowCount + " records.");
// rs.close();
//st.close();
//conn.close();
}
}
#Test
public void testLogin() throws Exception {
for (int i = 1; i<5; i++){
/*String strZipCode = oResultset [i][0];
String strDOBMonth = oResultset [i][1];
String strDOBDay = oResultset [i][2];
String strDOBYear = ArrXlDataLocal [i][3];*/
//new Select(oWD.findElement(By.name("insuranceType"))).selectByVisibleText("Dental");
//new Select(oWD.findElement(By.name("insuranceType"))).selectByVisibleText("Dental");
oWD.findElement(By.linkText("Dental")).click();
Thread.sleep(4000);
oWD.findElement(By.id("zipCode")).clear();
oWD.findElement(By.id("zipCode")).sendKeys(dbZipCode);
oWD.findElement(By.id("goBtn")).click();
oWD.findElement(By.id("census_primary_genderMALE")).click();
oWD.findElement(By.id("census_primary_month")).clear();
oWD.findElement(By.id("census_primary_month")).sendKeys(dbDOBMonth);
oWD.findElement(By.id("census_primary_day")).clear();
oWD.findElement(By.id("census_primary_day")).sendKeys(dbDOBDay);
oWD.findElement(By.id("census_primary_year")).clear();
oWD.findElement(By.id("census_primary_year")).sendKeys(dbDOBYear);
oWD.findElement(By.id("census_primary_tobacco")).click();
oWD.findElement(By.id("continue-btn")).click();
Thread.sleep(10000);
String strNumOfPlans = oWD.findElement(By.cssSelector("span.text-pink")).getText();
String strNumOfPlans2 = oWD.findElement(By.xpath("//*[#id='quote-title']/strong")).getText();
System.out.println("Here are the " + strNumOfPlans +" bestselling plans. Plans start at " + strNumOfPlans2);
}
}
#After
public void TearDown(){
}
}
You can use dataProvider feature of TestNG. Create a new function for dataProvider and do the database fetch steps inside this function, while reading each recordset, store the values in the object array. See below example code, did not test this code for errors.
#DataProvider
public Object[][] getData()
{
// open DB connection, get record set and store values in array object
//Rows - Number of times your test has to be repeated.
//Columns - Number of parameters in test data.
Object[][] data = new Object[3][2];
// you can dynamically read the rows/columns of recordset instead of hardcoding(new Object[3][2])
// 1st row
data[0][0] ="value1";
data[0][1] = "value2";
// similarly for all rows in the record set
return data;
}
#Test(DataProvider="getData")
public void testLogin(){
}
I am using unboundid ldap sdk for executing ldap query. I am facing a strange problem while running ldap search query. I am getting a Exception when i run query against a group which contains 50k entries. My Exception :
LDAPException(resultCode=4 (size limit exceeded), errorMessage='size limit exceeded')
at com.unboundid.ldap.sdk.migrate.ldapjdk.LDAPSearchResults.nextElement(LDAPSearchResults.java:254)
at com.unboundid.ldap.sdk.migrate.ldapjdk.LDAPSearchResults.next(LDAPSearchResults.java:279)
Now the strange thing is i already have set the maxResultSize to 100k in search constrains than why i am getting this error ?
My code is
ld = new LDAPConnection();
ld.connect(ldapServer, 389);
LDAPSearchConstraints ldsc = new LDAPSearchConstraints();
ldsc.setMaxResults(100000);
ld.setSearchConstraints(ldsc);
Anybody have any idea ?
Sorry for necroposting, but your topic with no answer is still the first in google.
Using unboundid you actually can get unlimited number of records in paging mode.
public static void main(String[] args) {
try {
int count = 0;
LDAPConnection connection = new LDAPConnection("hostname", 389, "user#domain", "password");
final String path = "OU=Users,DC=org,DC=com";
String[] attributes = {"SamAccountName","name"};
SearchRequest searchRequest = new SearchRequest(path, SearchScope.SUB, Filter.createEqualityFilter("objectClass", "person"), attributes);
ASN1OctetString resumeCookie = null;
while (true)
{
searchRequest.setControls(
new SimplePagedResultsControl(100, resumeCookie));
SearchResult searchResult = connection.search(searchRequest);
for (SearchResultEntry e : searchResult.getSearchEntries())
{
if (e.hasAttribute("SamAccountName"))
System.out.print(count++ + ": " + e.getAttributeValue("SamAccountName"));
if (e.hasAttribute("name"))
System.out.println("->" + e.getAttributeValue("name"));
}
LDAPTestUtils.assertHasControl(searchResult,
SimplePagedResultsControl.PAGED_RESULTS_OID);
SimplePagedResultsControl responseControl =
SimplePagedResultsControl.get(searchResult);
if (responseControl.moreResultsToReturn())
{
resumeCookie = responseControl.getCookie();
}
else
{
break;
}
}
}
catch (Exception e)
{
System.out.println(e.toString());
}
}
Check the server-side size limit setting. It prevails over the client-side setting which is what you're doing in your code.
I am using Tomcat 7, Microsoft SQL Server 2008 RC2 and JTDS driver
I am actually using C3P0 as well to try and solve this problem, but it makes no difference at all. I was using Microsoft's driver but it caused me other problems (the requested operation is not supported on forward only result sets)
I get the following error, always at the same point. I have successfully run other queries before getting to this point:
java.sql.SQLException: Invalid state, the ResultSet object is closed.
at
net.sourceforge.jtds.jdbc.JtdsResultSet.checkOpen(JtdsResultSet.java:287)
at
net.sourceforge.jtds.jdbc.JtdsResultSet.findColumn(JtdsResultSet.java:943)
at
net.sourceforge.jtds.jdbc.JtdsResultSet.getInt(JtdsResultSet.java:968)
at
com.mchange.v2.c3p0.impl.NewProxyResultSet.getInt(NewProxyResultSet.java:2573)
at com.tt.web.WebPosition.createPosition(WebPosition.java:863)
The code is as follows:
public static List getListPositions(String query) {
Connection con = null;
Statement stmt = null;
ResultSet rs = null;
List list = null;
try { //execute the sql query and create the resultSet
con = DBConnection.getInstance().getConnection();
stmt = con.createStatement();
rs = stmt.executeQuery(query);
while(rs.next()) {
if(rs.isFirst()) {
list = new ArrayList();
}
WebPosition webPos = null;
webPos = new WebPosition(rs);
list.add(webPos);
}
} catch (java.sql.SQLException e) {
System.out.println("SQLException in getListPositions");
System.out.print(query);
Log.getInstance().write(query);
Log.getInstance().write(e.toString());
} catch (Exception ex) {
System.out.print(ex);
ex.printStackTrace();
Log.getInstance().write(ex.toString());
} finally {
if (rs != null) {
try {
rs.close();
} catch (SQLException e) {
Log.getInstance().write(e.toString());
}
}
if (stmt != null) {
try {
stmt.close();
} catch (SQLException e) {
Log.getInstance().write(e.toString());
}
}
DBConnection.getInstance().returnConnection(con);
}
return list;
}
public WebPosition(ResultSet rs) {
createPosition( rs);
}
public void createPosition(ResultSet rs) {
try {
this.setCurrentDate4Excel(rs.getString("SYSDATE_4_EXCEL"));
this.setExerciseType(rs.getInt("EXERCISE_STYLE_CD"));
...
The code fails in between the above two lines.
I am at a loss to explain why the Result set would be closed in the middle of a function (i.e. it would retrieve rs.getString("SYSDATE_4_EXCEL") but then fail with the error posted at the line rs.getInt("EXERCISE_STYLE_CD"))
Does anyone have any ideas? I imagine that it is some kind of memory issue, and that the connection is automatically closed after a certain amount of data, but I am not sure how to fix this. I tried increasing the heapsize for the JVM.
I can compile but can't execute the following code
with the error (using Postgres):
Fatal database error
ERROR: syntax error at or near "as"
Position: 13
import java.sql.*;
public class JDBCExample
{
private static final String JDBC_DRIVER = "org.postgresql.Driver";
private static final String URL = "jdbc:postgresql://hostname/database";
private static final String USERNAME = "username";
private static final String PASSWORD = "password";
public static void main(String[] args) throws Exception
{
Connection dbConn = null;
Statement query = null;
ResultSet results = null;
Class.forName(JDBC_DRIVER);
try
{
dbConn = DriverManager.getConnection(URL, USERNAME, PASSWORD);
}
catch (SQLException e)
{
System.out.println("Unable to connect to database\n"+e.getMessage());
System.exit(1);
}
try
{
query = dbConn.createStatement();
results = query.executeQuery("select 20_5 as name from flowshop_optimums");
while (results.next())
{
System.out.println(results.getString("name"));
}
dbConn.close();
}
catch (SQLException e)
{
System.out.println("Fatal database error\n"+e.getMessage());
try
{
dbConn.close();
}
catch (SQLException x) {}
System.exit(1);
}
} // main
} // Example
It's not the underscore, it's the fact that the column name starts with a number. You'd need to escape this.
For MySQL, use backticks.
select `20_5` as name from flowshop_optimums
For SQL Server, use square brackets.
select [20_5] as name from flowshop_optimums
For PostgreSQL, use double quotes.
select "20_5" as name from flowshop_optimums
results = query.executeQuery("select \"20_5\" as name from flowshop_optimums");
But you should really change that column name.
Try this: results = query.executeQuery("select '20_5' as name from flowshop_optimums");
You need to enclose 20_5 with backticks or brackets, depending on whether or not you're using MySQL or SQLServer, respectively.