The Splunk query below returns only the calc_string column and returns blank for the index column, if I remove the ln4 and ln5 it returns index. How can I make this query to return both columns Please advise??
index=vehicle*
sourcetype=info_ssl
splunk_server_group=ALL
| stats values(xx) as XX values(yy) as YY
| eval calc_string=if(isnull(XX), YY, XX)
| table index calc_string
| sort index
With the stats command you'll want to use the BY clause to return one row for each distinct value in the BY clause fields.
In your example when using | stats values(xx) the result set only consists of the values of field yy without the index
So to include the index we'll use the by clause with the stats aggregation (line 2 at the very end)
index=vehicle* sourcetype=info_ssl splunk_server_group=ALL
| stats values(xx) as XX values(yy) as YY BY index
| eval calc_string=if(isnull(XX), YY, XX)
| table index calc_string
| sort index
If this fixes your problem, take a moment to accept the answer. This can be done by clicking on the check mark beside the answer to toggle it from greyed out to filled in!
Related
The objective of the query im trying to write is to take a count of raw data from the previous month and add that to a count from a lookup table (.csv)
What I have attempted to do is…
index=*** source=***
| stats count(_raw) as monthCount
| join
[ | inputlookup Log_Count_YTD.csv]
| eval countYTD = toNumber(monthCount) + toNumber(TOTAL_COUNT_YTD)
| table countYTD
This query doesn’t return any value on a table. The TOTAL_COUNT_YTD is the only field from the inputlookup file. Let me know if there is any other information you need to help me out with this one. Thanks!
The stats command transforms the data so it has only 1 field: monthCount. The inputlookup returns only the TOTAL_COUNT_YTD field. The join command works by comparing values of common fields between the main search and the subsearch. Since there are no common fields no events are joined.
There is no need for join in this case. The appendcols command will do, assuming the CSV contains a single field in a single row.
index=*** source=***
| stats count() as monthCount
| appendcols
[ | inputlookup Log_Count_YTD.csv]
| eval countYTD = toNumber(monthCount) + toNumber(TOTAL_COUNT_YTD)
| table countYTD
FWIW, the tonumber function is unnecessary, but doesn't hurt.
Using this search command
| eventcount summarize=false | dedup index | fields index
I get a list of all indexes I have access to in Splunk. Is it also possible to get another column besides this within which the source for the index is visible too?
EDIT: It seems like I found a solution:
| tstats count WHERE index=* sourcetype=* source=* by index, sourcetype, source | fields - count
This gives back a list with columns for indexes, sourcetypes and sources.
In case the permissions to read sources are not enforced by the tstats, you can join to your original query with an inner join on index, to limit to the indexes that you can see:
| tstats count WHERE index=* OR index=_* by index source
| dedup index source | fields index source
| join type=inner index [| eventcount summarize=false | dedup index | fields index]
I have a table like this:
I want to perform count on different set of columns (all subsets where there is at least one element from X and one element from Y). How can I do that in Postgres?
For example, I may have {x1,x2,y3}, {x4,y1,y2,y3},etc. I want to count number of "id"s having 1 in each set. So for the first set:
SELECT COUNT(id) FROM table WHERE x1=1 AND x2=1 AND x3=1;
and for the second set does the same:
SELECT COUNT(id) FROM table WHERE x4=1 AND y1=1 AND y2=1 AND y3=1;
Is it possible to write a loop that goes over all these sets and query the table accordingly? The array will have more than 10000 sets, so it cannot be done manually.
You should be able convert the table columns to an array using ARRAY[col1, col2,...], then use the array_positions function, setting the second parameter to be the value you're checking for. So, given your example above, this query:
SELECT id, array_positions(array[x1,x2,x3,x4,y1,y2,y3,y4], 1)
FROM tbl
ORDER BY id;
Will yield this result:
+----+-------------------+
| id | array_positions |
+----+-------------------+
| a | {1,4,5} |
| b | {1,2,4,7} |
| c | {1,2,3,4,6,7,8} |
+----+-------------------+
Here's a SQL Fiddle.
I am having an issue and I'm not sure how to solve it.
I have an SSRS report that pulls from a table. I want a parameter filter to show de-duplicated values based on available options in one of the columns.
So my dataset with a query like:
SELECT * FROM table1 WITH (NOLOCK) WHERE col1 IN (#param)
Then I want a parameter called param that gets its available and default values from col1 in the above data set and I want them to be de-duplicated.
From reading online I learned I have to create a dummy param and use VBA code to de-duplicate that list.
So I have these params:
param_dummy that gets its available and default values from col1 in the above dataset
param that gets a de-duplicate list from param_dummy using Code.RemoveDuplicates
But I'm having an issue with circular logic. param gets its value from param_default which gets its value from the dataset/query which uses param.
How can I solve this?
One thought is to remove the WHERE col1 IN (#param) and instead use a filter on the Tablix table in the SSRS report. This works but I am wondering how efficient it is.
And/or if anyone has any other suggestions I am all ears.
Updated to add more details...
So let us say I have a table in my DB like so:
| id | col1 | col2 |
|----|------|--------|
| 1 | a | hello |
| 2 | b | how |
| 3 | a | are |
| 4 | c | you |
| 5 | d | on |
| 6 | a | this |
| 7 | b | lovely |
| 8 | c | day |
What I want is:
a Tablix to show all the fields from the table
a filter where the user can select between the available dropdowns in col1 (de-duplicated)
a text filter that allows nulls where a user can filter on col2
the parameters will have default values so the table will load on page load
So I have a dataset with a query like so:
SELECT
*
FROM dbo.table1
WHERE col1 IN (#col1options) AND (#col2value IS NULL OR col2 = #col2value)
Then for col1options I would make available and default options be Get values from a query and I would use the above dataset and col1.
But this won't work since the query/dataset depends on col1options which gets its default values from the query/dataset.
I can use a second dataset but that means making multiple calls to the SQL server and I want to avoid that.
I'm not sure I understand your issue so this is a guess...
If you mean you want to be able to filter your data by choosing one or more entries from a specific column in the table, but this column has duplicates and you want your parameter list to not show duplicates then this is what do to.
Create a new report
Add dataset dsMain as SELECT * FROM myTable WHERE myColumn IN (#myParam)
Add dataset dsParamValues as SELECT DISTINCT myColumn FROM myTable ORDER BY myColumn
Edit the #myParam parameter properties and set the available and default values to a query, then choose dsParamValues
Add you table/matrix control and set it's dataset property to dsMain
Found an easier solution.
Follow this link to build the "dummy" hidden parameter, the visible paramter and the de-dupe VBA code
Add a tablix properties filter where param is in the visible / non-hidden parameter from above VBA (FYI double click to add parameter)
Adding via double click will append a (0) at the end, remove the (0)
It should work as expected at that point! You should be able to select one, some or all parameters and your report should update accordingly.
I have a query which is performing very badly due to full scan of a table.I have checked the statistics rebuild the indexes but its not working.
SQL Statement:
select distinct NA_DIR_EMAIL d, NA_DIR_EMAIL r
from gcr_items , gcr_deals
where gcr_deals.GCR_DEALS_ID=gcr_items.GCR_DEALS_ID
and
gcr_deals.bu_id=:P0_BU_ID
and
decode(:P55_DIRECT,'ALL','Y',trim(upper(NA_ORG_OWNER_EMAIL)))=
decode(:P55_DIRECT,'ALL','Y',trim(upper(:P55_DIRECT)))
order by 1
Execution Plan :
Plan hash value: 3180018891
-------------------------------------------------------------------------
| Id | Operation | Name | Rows | Time |
-------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 8 | 00:11:42 |
| 1 | SORT ORDER BY | | 8 | 00:11:42 |
| 2 | HASH UNIQUE | | 8 | 00:11:42 |
|* 3 | HASH JOIN | | 7385 | 00:11:42 |
|* 4 | VIEW | index$_join$_002 | 10462 | 00:00:05 |
|* 5 | HASH JOIN | | | |
|* 6 | INDEX RANGE SCAN | GCR_DEALS_IDX12 | 10462 | 00:00:01 |
| 7 | INDEX FAST FULL SCAN| GCR_DEALS_IDX1 | 10462 | 00:00:06 |
|* 8 | TABLE ACCESS FULL | GCR_ITEMS | 7386 | 00:11:37 |
-------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
3 - access("GCR_DEALS"."GCR_DEALS_ID"="GCR_ITEMS"."GCR_DEALS_ID")
4 - filter("GCR_DEALS"."BU_ID"=TO_NUMBER(:P0_BU_ID))
5 - access(ROWID=ROWID)
6 - access("GCR_DEALS"."BU_ID"=TO_NUMBER(:P0_BU_ID))
8 - filter(DECODE(:P55_DIRECT,'ALL','Y',TRIM(UPPER("NA_ORG_OWNER_EMAI
L")))=DECODE(:P55_DIRECT,'ALL','Y',TRIM(UPPER(:P55_DIRECT))))
In the beginning a part of the condition in the WHERE clause must be decomposed (or "decompiled" - or "reengeenered") into a simpler form without using decode function, which a form can be understandable by the query optimizer:
AND
decode(:P55_DIRECT,'ALL','Y',trim(upper(NA_ORG_OWNER_EMAIL)))=
decode(:P55_DIRECT,'ALL','Y',trim(upper(:P55_DIRECT)))
into:
AND (
:P55_DIRECT = 'ALL'
OR
trim(upper(:P55_DIRECT)) = trimm(upper(NA_ORG_OWNER_EMAIL))
)
To find rows in the table based on values stored in the index, Oracle uses an access method named Index scan, see this link for details:
https://docs.oracle.com/cd/B19306_01/server.102/b14211/optimops.htm#i52300
One of the most common access method is Index Range Scan see here:
https://docs.oracle.com/cd/B19306_01/server.102/b14211/optimops.htm#i45075
The documentation says (in the latter link) that:
The optimizer uses a range scan when it finds one or more leading
columns of an index specified in conditions, such as the following:
col1 = :b1
col1 < :b1
col1 > :b1
AND combination of the preceding conditions for leading columns in the
index
col1 like 'ASD%' wild-card searches should not be in a leading
position otherwise the condition col1 like '%ASD' does not result in a
range scan.
The above means that the optimizer is able to use the index to find rows only for query conditions that contain basic comparision operators: = < > <= >= LIKE which are used to comparing simple values with plain column names. What the documentation doesn't clearly say - and you need to deduce it reading between the lines - is a fact that when some function is used in the condition, in a form function( column_name ) or function( expression_involving_column_names ) , then the index range scan cannot be used. In this case the query optimizer must evaluate this expression individually for each row in the table, thus must read all rows (perform a full table scan).
A short conclusion and a rule of thumb:
Functions in the WHERE clause can prevent the optimizer from using
indexes
If you see some function somewhere in the WHERE clause, then it is a sign that you are
running the red light
STOP immediately and think three times how
this function impact the query optimizer and the performance of your
query, and try to rewrite the condition to a form that the optimizer
is able to understand.
Now take a look at our rewritten condition:
AND (
:P55_DIRECT = 'ALL'
OR
trim(upper(:P55_DIRECT)) = trimm(upper(NA_ORG_OWNER_EMAIL))
)
and STOP - there are still two functions trim and upper applied to a column named NA_ORG_OWNER_EMAIL. We need to think how they can impact the query optimizer.
I assume that you have created a plain index on a single column: CREATE INDEX somename ON GCR_ITEMS( NA_ORG_OWNER_EMAIL ).If yes, then the index contains only plain values of NA_ORG_OWNER_EMAIL.
But the query is trying to find trimm(upper(NA_ORG_OWNER_EMAIL)) values, which are not stored in the index, so this index cannot be used in this case.
This condition requires a function based index:
https://docs.oracle.com/cd/E11882_01/appdev.112/e41502/adfns_indexes.htm#ADFNS00505
CREATE INDEX somename ON GCR_ITEMS( trim( upper( NA_ORG_OWNER_EMAIL )))
Unfortunately even the function based index will still not help, because the condition in the query is too general - if a value of :P55_DIRECT = ALL the query must retrieve all rows from the table (perform a full table scan), otherwise must use the index to search value within it.
This is because the query is planned (think of it as "compiled") by the query optimizer only once, during it's first execution. Then the plan is stored in the cache and used to execute the query for all further executions. A value of the parameter is not know in advance, so the plan must consider each possible cases, thus will always perform a full table scan.
In 12c there is a new feature "Adaptive query optimalization":
https://docs.oracle.com/database/121/TGSQL/tgsql_optcncpt.htm#TGSQL94982
where the query optimizer analyses each parameters of the query on each runs, and is able to detect that the plan is not optimal for some runtime parameters, and choose a better "subplans" depending on actual parameter's value ... but you must use 12c, and additionally pay for Enterprise Edition, because only this edition includes that feature. And it's still not certain if the adaptive plan will work in this case or not.
What you can do without paying for 12c EE is to DIVIDE this general query into two separate variants, one for a case where :P55_DIRECT = ALL, and the other for remaining cases, and run an appropriate variant in the client (your application) depending on the value of this parameter.
A version for :P55_DIRECT = ALL, that will perform a full table scan
where gcr_deals.GCR_DEALS_ID=gcr_items.GCR_DEALS_ID
and
gcr_deals.bu_id=:P0_BU_ID
order by 1
and a version for other cases, that will use the function based index:
where gcr_deals.GCR_DEALS_ID=gcr_items.GCR_DEALS_ID
and
gcr_deals.bu_id=:P0_BU_ID
and
trim(upper(:P55_DIRECT)) = trimm(upper(NA_ORG_OWNER_EMAIL))
order by 1