I have a table, the schema is as below:
struct TBL{
1:optional string category;
2:optional string endpoint;
3:optional string priority;
4:optional i64 timestap;
5:optional string traceId;
6:optional string rpcId;
7:optional string businessId;
8:optional string message;
9:optional int date;
}
I only care about 8:optional string message;
message field is log.
I have a list of urls some of which can be found in the message filed. I need find which urls can be found in the table. In the message filed, the url always comes with "getRequestURL" such as
10.128.226.31,[INFO 2019-03-13 10:00:18.770] [resin-port-9300-44] [aaaLiMKyYK9bNbetoyhLw_-301570423] com.example.cashpay.filter.StaticFilter.doFilter(StaticFilter.java:99) [getRequestURL=http://www.example.com/merchant/deduct/query/v3]
The table is very large.
I want to filter the table first, say:
select message from TBL where message like '%getRequestURL%' and date ='20190319'
and then I can use union query to found which urls are in the filtered table above.
say:
select 'com/res/doc/eula/help.html', count(1) from filtered_tbl where message like 'com/res/doc/eula/help.html'
union
select 'com/charge/bank/alipay_wap/', count(1) from filtered_tbl where message like 'com/charge/bank/'
union
select 'com/reductionActivity/queryRegisterStatus', count(1) from filtered_tbl where message like 'com/reductionActivity/queryRegisterStatus'
union
select 'com/charge/pushcmd', count(1) from filtered_tbl where message like 'com/charge/pushcmd'
union......
Does hive support filter first? If it support, then what can I do?
Related
I have a table with a text column and I would like to select all rows that match the list of search parameters that were provided by the user:
select * from value where value.text in ('Mary', 'Steve', 'Walter');
In addition, I want to notify the user if any of his search terms could not be found. Let's say 'Steve' does not exist in the value.text column, how can I write a query that will show 'Steve'? As that information does not exist in any table, I have no idea how it could be done using a SQL query.
The actual Hibernate code looks like this:
List<String> searchItemList = new ArrayList<>();
searchItemList.add("Mary");
searchItemList.add("Steve");
searchItemList.add("Walter");
Query query = em.createQuery("select v from Value as v where v.text in ( :searchitemlist )");
query.setParameter("searchitemlist", searchItemList);
List result = query.getResultList();
log.info("{}", result.size());
log.info("{}", result);
The searchItemList is a list of all search terms provided by the user. Can be a few hundreds lines long. The current workaround is to search the value table once for each searchItem and note all queries that return 0 rows. That is rather inefficient, surely there is a better approach? Please advise.
You can use the following query to get an array of search items that exist in the database
SELECT DISTINCT value.text from value where value.text in ('Mary', 'Steve', 'Walter');
after running this query, If we assume that the answer is stored in an array called result, notExistSearchListItems will give you the final result
IEnumerable<string> notExistSearchListItems = searchItemList.Except(result);
I am having a tons of URL's in my database and want to filter them by user-defined string in format something/*/something, where * stands for "anything". So when user defines checkout/*/complete, it means it filters out url's like:
http://my_url.com/checkout/15/complete
http://my_url.com/checkout/85/complete
http://my_url.com/checkout/something/complete
http://my_url.com/super/checkout/something/complete
etc.
How do I do that in SQL? Or should I filter out all the results and use PHP to do the job?
My SQL request now is
SELECT * FROM custom_logs WHERE pn='$webPage' AND id IN ( SELECT MAX(id) FROM custom_logs WHERE action_clicked_text LIKE '%{$text_value_active}%' GROUP BY token ) order by action_timestamp desc
This filters out all the log messages with user-defined text in column action_clicked_text, but uses LIKE statement, which will not work with * inside.
You want like. Either:
where url like '%checkout/%/complete%'
to get the urls that match he pattern. Or:
where url not like '%checkout/%/complete%'
to get the other urls.
How do I extract the id parameter below using Big Query Regexp_Extract some rows with page urls in them that look similar to :
url.com/id=userIDmadeUPofletterandnumbers&em=MemberType
eg url.com/id=asd1221231sf&em=studentMember
I have tried using:
a. REGEXP_EXTRACT(urlValue,"id=\w+") as Idvalue but I get the error message:
Invalid string literal: "id=\w+"
I am pretty close with this: REGEXP_EXTRACT(urlValue,"(id=.*&em)") however it shows me id=asd1221231sf&em and I want to exclude id= and &em at the end
#standardSQL
WITH `project.dataset.table` AS (
SELECT 'url.com/id=userIDmadeUPofletterandnumbers&em=MemberType' urlValue UNION ALL
SELECT 'url.com/id=asd1221231sf&em=studentMember'
)
SELECT REGEXP_EXTRACT(urlValue, r'id=(\w+)') id, urlValue
FROM `project.dataset.table`
Row id urlValue
1 userIDmadeUPofletterandnumbers url.com/id=userIDmadeUPofletterandnumbers&em=MemberType
2 asd1221231sf url.com/id=asd1221231sf&em=studentMember
{"create_channel":"1","update_comm":"1","channels":"*"}
This is the database field which I want to query.
What would my query look like if I wanted to select all the records that have a "create_channel": "1" and a "update_comm": "1"
Additional question:
View the field below:
{"create_channel":"0","update_comm":"0","channels":[{"ch_id":"33","news":"1","parties":"1","questions ":"1","cam":"1","edit":"1","view_subs":"1","invite_subs":"1"},{"ch_id":"18","news":"1","parties":"1","questions ":"1","cam":"1","edit":"1","view_subs":"1","invite_subs":"1"}]}
How would I go about finding out all those that are subadmins in the News, parties, questions and Cams sections
You can use the ->> operator to return a member as a string:
select *
from YourTable
where YourColumn->>'create_channel' = '1' and
YourColumn->>'update_comm' = '1'
To find a user who has news, parties, questions and cam in channel 33, you can use the #> operator to check if the channels array contains those properties:
select *
from YourTable
where YourColumn->'channels' #> '[{
"ch_id":"33",
"news":"1",
"parties":"1",
"questions ":"1",
"cam":"1"
}]';
In SQL it works fine
SELECT NOID, NO_DOSSOIN, NO_ORDO, POSOLOG FROM dbo.ESPMEDS_ORDO_SORTIR
WHERE NO_DOSSOIN = #NO_DOSSOIN AND NOID IN (#NOIDIN)
example
SELECT NOID, NO_DOSSOIN, NO_ORDO, POSOLOG FROM dbo.ESPMEDS_ORDO_SORTIR
WHERE NO_DOSSOIN = 10 AND NOID IN (16,17)
But as I put this in a dataset.xsd query I don't get the same output, I cannot put more than one id into NOIDIN parameter because the NOID type is integer
so my file DataSet.xsd only work like this:
SELECT NOID, NO_DOSSOIN, NO_ORDO, POSOLOG FROM dbo.ESPMEDS_ORDO_SORTIR
WHERE NO_DOSSOIN = 10 AND NOID IN (16)
the error says I cannot convert data from string to int
You should just separate the NOIDIN. Don't expect to be able to pass an Int32 that looks like 16,17 it will always be seen as a string by this wizard and won't compile at all if you execute it from the code.
The easiest option for you is to pass the range in two values like this :
SELECT NOID, NO_DOSSOIN, NO_ORDO, POSOLOG FROM dbo.ESPMEDS_ORDO_SORTIR
WHERE NO_DOSSOIN = #NO_DOSSOIN AND NOID IN (#NOIDSTART, #NOIDEND)
And then assign :
#NOIDSTART = 16
#NOIDEND = 17
If you're parameters are dynamic you should read this article which pretty much covers the subject.