Hive table load in Parquet format - hive

I have the below input file. I need to load this file in hive table in orc and parquet format.
productID,productCode,name,quantity,price,supplierid
1001,PEN,Pen Red,5000,1.23,501
1002,PEN,Pen Blue,8000,1.25,501
I have pasted my code in the bottom. I am able to successfully create and load in orc hive table but not in parquet.
After creating and loading the parquet table, when i query, i see only NULL values for all fields. Am i missing anything?
val productsupplies = sc.textFile("/user/cloudera/product.csv")
val productfirst = productsupplies.first
val product = productsupplies.filter(f => f != productfirst).map(x => { val a = x.split(",")
(a(0).toInt,a(1),a(2),a(3),a(4).toFloat,a(5))
}).toDF("productID","productCode","name","quantity","price","supplierid")
product.write.orc("/user/cloudera/productsupp.orc")
product.write.parquet("/user/cloudera/productsupp.parquet")
val hc = new org.apache.spark.sql.hive.HiveContext(sc)
hc.sql("create table product_supp_orc ( " +
"product_id int, " +
"product_code string, " +
"product_name string, " +
"product_quatity string, " +
"product_price float, " +
"product_supplier_id string) stored as orc " +
"location \"/user/cloudera/productsupp.orc \" ")
hc.sql("create table product_supp_parquet ( " +
"product_id int, " +
"product_code string, " +
"product_name string, " +
"product_quatity string, " +
"product_price float, " +
"product_supplier_id string) stored as parquet " +
"location \"/user/cloudera/productsupp.parquet\" ")
hc.sql("select * from product_supp_parquet")

Try:
hc.sql("create table product_supp_parquet ( " +
"productid int, " +
"productcode string, " +
"name string, " +
"quantity string, " +
"price float, " +
"supplierid string) stored as parquet " +
"location \"/user/cloudera/products.parquet\" ")
Basically, the names must be the same as what you used in the file for uploading.

Related

How to pass multiple values in a native query and retrieve that data in a Map<String, List<Data>>?

Here I want to pass a list of Ids and date as a parameter to the database query and I want the query to return data in a Map so the key is the id and List is the data against that id.
Query<Data> query = rdsSession.createNativeQuery(DBQueries.DATA_QUERY,
Data.class).setParameterList("performance_id", listOfIds).setParameter("fromdate", fromDate);
List<Data> listDBRecords = null;
Map<String, List<Data> records=listDBRecords.stream().collect(Collectors.groupingBy(Data::getId));
So how can I achieve this result by passing multiple values(ie. ids) in the query and return that data? This is the query I have written
public static final String SELECT_PORTFOLIO_SECURITY_DATA = "select capl.corporate_action_portfolio_level_id, " +
"capl.performance_id, capl.forwardlooking_sequence_no, " +
"capl.as_of_date, capl.price, casl.currency, capl.exchange_rate, " +
"capl.gross_dividend, capl.net_dividend, capl.sub_portfolio_guid, capl.effective_date, capl.adjustment_weighted_factor, capl.price_adjustment_factor, " +
"casl.share_adjustment_factor, casl.float_adjustment_factor, capl.index_shares, capl.input_tos_live, casl.input_float_live, capl.prediction_type from " +
ConfigurationManager.getProperties().getProperty("rdsConnection.schema") +
".corporate_action_portfolio_level capl inner join " +
ConfigurationManager.getProperties().getProperty("rdsConnection.schema") +
".corporate_action_security_level casl " +
"on capl.performance_id = casl.performance_id and capl.as_of_date = casl.as_of_date and capl.prediction_type = casl.prediction_type and " +
"capl.forwardlooking_sequence_no = casl.forwardlooking_sequence_no " +
"where capl.performance_id = :performance_id and capl.as_of_date = :as_of_date and capl.prediction_type = 'PREDICTED' or capl.prediction_type = 'LIVE'";
Can someone tell what is the right way?

Dynamic dates in where parameter

Am I able to use a dynamic date range for the "where" parameter? I'd like to get all the data changed in the last 30 mins.
All I can find in the documentation is using static dates:
https://developer.xero.com/documentation/api/requests-and-responses
You could try using the query string in the Where Clause.
For Ex.
string querystr = "Date >= " +
"DateTime(" + dateFrom.Year.ToString() + ",
" + dateFrom.Month.ToString() + ", " + dateFrom.Day.ToString() + ") " +
"&& Date <= " +
"DateTime(" + dateTo.Year.ToString() + ", "
+ dateTo.Month.ToString() + ", " + dateTo.Day.ToString() + ")";
Passing this querystring to retrieve user defined date rage Invoices

Store Groovy SQL result as a variable?

I am very, very new to Groovy (like two days in). I am trying to take a SQL result which will return a single value, and store that value so that it can be put in the body of an email. However, my SQL result always returns NULL, but the other values in the email work just fine.
I know that the result will only return a single value, so how in the world can I get that value stored as a string?
def sql = getSqlInstance()
def user = "\'" + trmCurrentUser.username + "\'";
def name = "\'" + trmCurrentUser.displayname + "\'";
def email = "\'" + trmCurrentUser.email + "\'";
def comm = "\'" + trmComponent.id + "\'";
def rslt=[];
def cmpgn;
rslt[0] = sql.firstRow("select activity_id from DB.TABLE where communication_id=${comm}");
def cmpgn = rslt[0];
toAddress = trmCurrentUser.email;
fromAddress = "me#email.com";
trmUtilities.sendEmail (
//self explanatory
toAddress, fromAddress,
//body of email
user + " " + name + " " + email + " " + comm + " " + cmpgn[0],
//subject of email
" Action on " + trmComponent.componentType + " " + trmComponent.name
);
The resulting email I get is this (with the appropriate values other than the NULL):
'MYID' 'MY NAME' 'MYEMAIL' 'COMM' null

What is wrong with this SQLite create Statement in AndroidStudio

Can anyone decipher what's wrong with this method? The log says that there's is a syntax error but I'm not able to see it.
Here's the code:
public void onCreate(SQLiteDatabase sqLiteDatabase) {
String createTable = "CREATE_TABLE " + MOVIES_DB + "("
+ TITLE + " TEXT, "
+ OVERVIEW + " TEXT, "
+ POSTER_PATH + " TEXT, "
+ RELEASE_DATE + " TEXT, "
+ ORIGINAL_TITLE + " TEXT, "
+ ORIGINAL_LANGUAGE + " TEXT, "
+ BACKDROP_PATH + " TEXT, "
+ ID + " INTEGER PRIMARY KEY, "
+ VOTE_COUNT + " TEXT, "
+ POPULARITY + " TEXT, "
+ VOTE_AVERAGE + " TEXT, "
+ ADULT + " TEXT, "
+ VIDEO + " TEXT, "
+ GENRE_IDS + " TEXT, "
+ HISTORY_USER + " TEXT, "
+ WATCH_LATER + " TEXT, "
+ FAVORITE_FLAG + " INTEGER DEFAULT 0"
+ ")";
sqLiteDatabase.execSQL(createTable);
}
It used to work. I added the FAVORITE_FLAG and now it doesn't run. I also tried with INTEGER NOT NULL DEFAULT 0 after FAVORITE_FLAG
As said by #Juergen D, CREATE TABLE take a space and not and underscore.
Take a look at the CREATE TABLE Documentation for SQLite.
BTW; try to put your id / PK declaration first, as it's the common practice (and more logical)
SQL Fiddle
SQLite (SQL.js) Schema Setup:
CREATE TABLE MOVIES_DB(
"ID" INTEGER PRIMARY KEY,
"TITLE" TEXT,
"OVERVIEW" TEXT,
"POSTER_PATH" TEXT,
"RELEASE_DATE" TEXT,
"ORIGINAL_TITLE" TEXT,
"ORIGINAL_LANGUAGE" TEXT,
"BACKDROP_PATH" TEXT,
"VOTE_COUNT" TEXT,
"POPULARITY" TEXT,
"VOTE_AVERAGE" TEXT,
"ADULT" TEXT,
"VIDEO" TEXT,
"GENRE_IDS" TEXT,
"HISTORY_USER" TEXT,
"WATCH_LATER" TEXT,
"FAVORITE_FLAG" INTEGER DEFAULT 0
);
Insert into MOVIES_DB Values(1,"test","test","test 7",
"test","test 12","test","test","test","test","test",
"test","test","test","test","test",2);
Query 1:
select ID, TITLE, FAVORITE_FLAG from MOVIES_DB
Results:
| ID | TITLE | FAVORITE_FLAG |
|----|-------|---------------|
| 1 | test | 2 |

'IN' query in fusion table

In the following code, i want to add "IN" +Village+. Where to add this condition in the code. Variable village takes value from a drop down list based on that filter should occur.please help me.Village name is a column in my fusion table.
i.e select 'geometry',villageName from table where querypass > textvalue IN villagename='madurai'
function querymape()
{
/*variable holds the value*/
var village =document.getElementById('village').value.replace(/'/g, "\\'");
var operatore=document.getElementById('operatorstringe').value.replace(/'/g, "\\'");
var textvaluee=document.getElementById("text-valuee").value.replace(/'/g, "\\'");
var querypasse=document.getElementById('query-passe').value.replace(/'/g, "\\'");
{
layer.setQuery("SELECT 'geometry'," + querypasse + " FROM " + tableid + " WHERE " + querypasse + " " + operatore + " '" + textvaluee + "'"+"AND 'VillageName=+village+'");
}
}
/*This is my new code.But its not working.Please help me*/
function querymap()
{
//var villagename='';
var operator=document.getElementById('operatorstring').value.replace(/'/g, "\\'");
var textvalue=document.getElementById("text-value").value.replace(/'/g, "\\'");
var querypass=document.getElementById('query-pass').value.replace(/'/g, "\\'");
var searchStringe = document.getElementById('Search-stringe').value.replace(/'/g, "\\'");
{
layer.setQuery("SELECT 'geometry'," + querypass + " FROM " + tableid + " WHERE " + querypass + " " + operator + " '" + textvalue + "'"+"AND 'VillageName'="+ searchStringe+"");
}
}
Multiple conditions can be combined using the keyword "and"?
You twisted the IN syntax around, it is used when you want to match several values, if you only want to compare to a single value use "=" instead
Applied to your query (with IN syntax):
select 'geometry',villageName from table where querypass > textvalue and villagename IN ('madurai','another village')
With = syntax:
select 'geometry',villageName from table where querypass > textvalue and villagename = 'madurai'