SELECT COL HIVE SQL VALUE WHERE VALUES <5000 - sql

Question

I'm learning about HIVE and I have come across a question I cannot seem to find a workable answer for. I have to extract all of the numeric columns that ONLY contain integer values <5000 from a table and create a space separated text file. I am familiar with creating text files and selecting rows but selecting columns that meet a specific parameter I am not familiar with, any help or guidance will be appreciated! Below I've listed the structure of the table. Also, there is an image attached showing the data in table format. For OUTPUT I need to go through ALL the COLUMNS and RETURN ONLY the the COLUMNS that meet the parameter of integer values LESS THAN 5000.
create table lineorder (
lo_orderkey int,
lo_linenumber int,
lo_custkey int,
lo_partkey int,
lo_suppkey int,
lo_orderdate int,
lo_orderpriority varchar(15),
lo_shippriority varchar(1),
lo_quantity int,
lo_extendedprice int,
lo_ordertotalprice int,
lo_discount int,
lo_revenue int,
lo_supplycost int,
lo_tax int,
lo_commitdate int,
lo_shipmode varchar(10)
)
Data in tbl format

Answer 1

Conditional columns selecting is a terrible, horrible, no good, very bad idea.
Being that said, here is a demo.
with t as
(
select stack
(
3
,10 ,100 ,1000 ,'X' ,null
,20 ,null ,2000 ,'Y' ,200000
,30 ,300 ,3000 ,'Z' ,300000
) as (c1,c2,c3,c4,c5)
)
select regexp_replace
(
printf(concat('%s',repeat(concat(unhex(1),'%s'),field(unhex(1),t.*,unhex(1))-2)),*)
,concat('([^\\x01]*)',repeat('\\x01([^\\x01]*)',field(unhex(1),t.*,unhex(1))-2))
,c.included_columns
) as record
from t
cross join (select ltrim
(
regexp_replace
(
concat_ws(' ',sort_array(collect_set(printf('$%010d',pos+1))))
,concat
(
'( ?('
,concat_ws
(
'|'
,collect_set
(
case
when cast(pe.val as int) >= 5000
or cast(pe.val as int) is null
then printf('\\$%010d',pos+1)
end
)
)
,'))|(?<=\\$)0+'
)
,''
)
) as included_columns
from t
lateral view posexplode(split(printf(concat('%s',repeat(concat(unhex(1),'%s'),field(unhex(1),*,unhex(1))-2)),*),'\\x01')) pe
) c
+---------+
| record |
+---------+
| 10 1000 |
| 20 2000 |
| 30 3000 |
+---------+

Answer 2

I don't think hive supports variable substitution in the function. So you would have to write a shell scripts that executes the first query which returns the required columns.Then you can assign it to a variable in shell script and then create a new query for creating files in local directory and run it via hive -e from bash.
create table t1(x int , y int) ; // table used for below query
Sample bash script :
cols =hive -e 'select concat_ws(',', case when min(x) > 5000 then 'x' end , case when min(y) > 5000 then 'y' end) from t1'
query ="INSERT OVERWRITE LOCAL DIRECTORY <directory name> ROW FORMAT DELIMITED FIELDS TERMINATED BY ' ' select $cols from t1 "
hive -e query