Questions related to splunk builtin macros in correlation search - splunk

I am not sure if this is the appropriate forum to ask this question, but really need help and I am stuck. So here goes : I am exploring splunk enterprise security and was specifically looking into analytic stories and correlation searches.
For example :
Analytic story : Trickbot
Correlation search : Attempt to stop security service
| tstats `security_content_summariesonly` values(Processes.process) as process min(_time) as firstTime max(_time) as lastTime from datamodel=Endpoint.Processes where (Processes.process_name = net.exe OR Processes.process_name = sc.exe) Processes.process="* stop *" by Processes.dest Processes.user Processes.parent_process Processes.process_name Processes.process Processes.process_id Processes.parent_process_id
| `drop_dm_object_name(Processes)`
| `security_content_ctime(firstTime)`
| `security_content_ctime(lastTime)`
|lookup security_services_lookup service as process OUTPUTNEW category, description
| search category=security
| `attempt_to_stop_security_service_filter`
I am trying to understand what exactly this code is doing, but stuck at these macros like security_content_summariesonly, drop_dm_object_name, security_content_ctime, attempt_to_stop_security_service_filter. I can't find definitions for these macros anywhere. I have tried to look into -> settings -> advance search -> macros, but these are not listed there.
Can somebody help ?

If you have access to the host(s) Splunk's running on, you can find the definitions in $SPLUNK_HOME$/etc/*/macros.conf
If you don't have that access, then it's possible you don't have permissions to see the definitions of those macros
However, you can always use the Job Inspector to see how Splunk translates what you type into what it runs

If you have the query in a search window then click on the query and type Shift-CTRL-E to have Splunk expand all of the macros for you.
I can tell you the drop_dm_object_name macro is just rename $1.* as *. The other macros are specific to their app.

Related

Apache Zeppelin: passing parameters between different interpreters

Here's a problem.
I'm building a dashboard in Apache Zeppelin using org.postgresql.Driver to connect to Greenplum database. Usually I use something like:
where country_title like '${country=UK,UK|USA|Canada|%}'
to pass parameters into my query. But I have a long list of other parameters (like "towns") which I also need to pass to my query. And they are different for different country_title values. I can use something like:
select towns from towns_by_country where country_title like '${country=UK,UK|USA|Canada|%}'
to get a list of towns. How can I use it as a parameter in drop-down menu (WITH '%', of course).
I can use z.angularBind or z.put & z.get , of course. But they work in %spark interpreters, not in custom interpreters.
I would be very grateful for an answer or any other constructive feedback.

Downloading all full-text articles in PMC and PubMed databases

According to one of the answered questions by NCBI Help Desk , we cannot "bulk-download" PubMed Central. However, can I use "NCBI E-utilities" to download all full-text papers in PMC database using Efetch or at least find all corresponding PMCids using Esearch in Entrez Programming Utilities? If yes, then how? If E-utilities cannot be used, is there any other way to download all full-text articles?
First of all, before you go downloading files in bulk, I highly recommend you read the E-utilities usage guidelines.
If you want full-text articles, you're going to want to limit your search to open access files. Furthermore, I suggest also restricting your search to Medline articles if you want articles that are any good. Then you can do the search.
Using Biopython, this gives us :
search_query = 'medline[sb] AND "open access"[filter]'
# getting search results for the query
search_results = Entrez.read(Entrez.esearch(db="pmc", term=search_query, retmax=10, usehistory="y"))
You can use the search function on the PMC website and it will display the generated query that you can copy/paste into your code.
Now that you've done the search, you can actually download the files :
handle = Entrez.efetch(db="pmc", rettype="full", retmode="xml", retstart=0, retmax=int(search_results["Count"]), webenv=search_results["WebEnv"], query_key=search_results["QueryKey"])
You might want to download in batches by changing retstart and retmax by variables in a loop in order to avoid flooding the servers.
If handle contains only one file, handle.read() contains the whole XML file as a string. If it contains more, the articles are contained in <article></article> nodes.
The full text is only available in XML, and the default parser available in pubmed doesn't handle XML namespaces, so you're going to be on your own with ElementTree (or an other parser) to parse your XML.
Here, the articles are found thanks to the internal history of E-utilities, which is accessed with the webenv argument and enabled thanks to the usehistory="y" argument in Entrez.read()
A few tips about XML parsing with ElementTree : You can't delete a grandchild node, so you're probably going to want to delete some nodes recursively. node.text returns the text in node, but only up to the first child, so you'll need to do something along the lines of "".join(node.itertext()) if you want to get all the text in a given node.
According to one of the answered questions by NCBI Help Desk , we cannot "bulk-download" PubMed Central.
https://www.nlm.nih.gov/bsd/medline.html + https://www.ncbi.nlm.nih.gov/pmc/tools/openftlist/ will download a good portion of it (I don't know the percentage). It will indeed miss the PMC full-texts articles whose license doesn't allow redistribution as explained on https://www.ncbi.nlm.nih.gov/pmc/tools/openftlist/.

how to show button if there are two roles Lotus Formula script

I have a Page in Lotus, which has an action button.
This button must be not visible when user is not part of role Admin or Supervisor.
The way i did this is by adding this formula:
#IsNotMember("[Admin]"; #UserRoles) | #IsNotMember("[Supervisor]"; #UserRoles)
But it does not work..
If I have only 1, like: #IsNotMember("[Admin]"; #UserRoles) And the role Admin, then as admin I can see it.
But I also would like to have that if user is from role Supervisor and not Admin that he still can see the button.
What to do?
Knut's answer is correct. My answer shows why, and gives an approach to hide-whens that almost always makes them easier to figure out.
Notes has been using hide-when formulas forever, but people really tend to think in terms of see-when in their requirements! We know when we want to see things (when we're Supervisors or Admins in this case). And we're really bad at turning those see-when requirements into hide-when formulas because we're really bad at remembering DeMorgan's Law, which says things like: ^(P & Q) == (^P | ^Q)
So if we state the requirement this way:
Hide when the user isn't a Supervisor or an Admin
we tend to have trouble turning it into the correct formula with two #IsNotMember calls (which are implicitly logical Nots), because we forget that Ors have to turn into Ands in order to get it right. But if we think of it this way:
See when the user is an Admin or the user is an Supervisor
It's easy to see how to express it:
#IsMember("[Supervisor]"; #UserRoles) | #IsMember("[Admin]"; #UserRoles);
Or using the power of formula language lists, we can shorten that to this :
#IsMember("[Supervisor]": "[Admin]" ; #UserRoles);
To turn that into the equivalent hide-when, all you need to do is put a logical Not around it like this:
! (#IsMember("[Supervisor]": "[Admin]" ; #UserRoles));
You can do that with any see-when formula - just surround it with parenthesis and put a ! in front of it, but in the special case of a formula that just uses #IsMember, you can just change it to #IsNotMember, which brings it back to Knut's solution.
Formula
#IsNotMember("[Admin]" : "[Supervisor]"; #UserRoles)
returns #True if user has neither role "[Admin]" nor "[Supervisor]". Use this as hide-when formula for your button. Only Admins and Supervisors will see the button then.

PhpStorm unable to resolve column for multiple database connections

I have only been using PhpStorm a week or so, so far all my SQL queries have been working fine with no errors after setting up the database connection. This current code actually uses a second database (one is for users the other for the specific product) so I added that connection in the database tab too but its still giving me a 'unable to resolve column' warning.
Is there a way to see what database its looking at? Will it work with multiple databases? Or have I done something else wrong?
Error below:
$this->db->setSQL("SELECT T1.*, trunc(sysdate) - trunc(DATE_CHANGED) EXPIRES FROM " . $this->tableName . " T1 WHERE lower(" . $this->primaryKey . ")=lower(:id)")
Also here is what my database settings window looks like as seen some people having problems with parameter patterns causing this error but I'm fairly sure that is not the issue here:
Using PhpStorm 10.0.3
You can set the SQL resolution scope in File -> Settings -> Languages & Frameworks -> SQL Resolution Scopes.
This allows you to provide a default for the entire project and you can optionally define specific mappings to certain paths in the project.
So the short answer is that it cant read the table name as a variable even though its set in a variable above. I thought PhpStorm could work that out. The only way to remove the error would be to either completely turn off SQL inspections (obviously not ideal as I use it throughout my project) or to temporarily disable it for this statement only using the doc comment:
/** #noinspection SqlResolve */
Was hoping to find a more focused comment much like the #var or #method ones to help tell Phpstorm what the table should be so it could still inspect the rest of the statement. Something like:
/** #var $this->tableName TABLE_IM_USING */
Maybe in the future JetBrains will add that or make PhpStorm clever enough to look at the variable 3 lines above.
You can use Nowdoc/Heredoc also instead of using
/** #noinspection SqlResolve */
Here is the example
$name = "your name";
$query = <<<SQL
SELECT * FROM user WHERE name LIKE "%$name%" ORDER BY id
SQL;
Both of the methods will make the warning gone

sharepoint crawl rule to exclude AllItems.aspx , but get an item/document in search resu lts if queried in the search box

I followed this blog Tips 1and created a crawl rule http://.*forms/allitems.aspx and ran full crawl. I no longer get the results with AllItems.aspx. However, if there is any document with name Something.doc in a Document Library , it no longer gets pulled in the search results.
I think what I desire is a basic functionality, like the user should not get to see Allitems.aspx in the search results but should get the item/document with names entered in the search box.
Please let me know if I am missing anything. I have already put in 24 hours...googled the max I could.
It seems that an Index Reset is required. Here's the steps I did:
1. Add the following crawl rule to exclude: *://*allitems.aspx.
2. Index Reset.
3. Full Crawl.
I could not find a good way to do this using crawl rules. Instead, I opted to set up a restriction on the search results web part.
In the search results web part properties, select "Change Query"
Add a property filter to exclude anything with "AllItems" (and any other exclusions you want in place.
Used Steve Mann's blog as a reference and for the images: http://stevemannspath.blogspot.com/2013/04/sharepoint-2013-search-removing-junk.html