How to strip the XHTML tags from the SQL query - sql

I have this below query for Rapid 7 Nexpose SQL database and have been trying to remove xhtl,
SELECT
dp.policy_id, dp.title as policy_title, dpr.rule_id, dpr.title as policy_rule_title,
dp.benchmark_name, da.ip_address, da.host_name, dpr.description, dp.category,
fapr.date_tested, htmlToText(fapr.proof) as proof, fapr.compliance,
dpr.severity, htmlToText(dpr.rationale) as rationale, htmlToText(dpr.remediation) as remediation
FROM fact_asset_policy_rule fapr
JOIN dim_policy dp on dp.policy_id = fapr.policy_id
JOIN dim_policy_rule dpr on dpr.policy_id = fapr.policy_id and fapr.rule_id = dpr.rule_id
JOIN dim_asset da on da.asset_id = fapr.asset_id
WHERE fapr.compliance = false order by dp.title, dpr.title`
However it exports the below content for rationale and remediation with below xhtml tags which i would like to get rid off:
Rationale column example:
<xhtml:p xmlns="http://checklists.nist.gov/xccdf/1.2"
xmlns:ae="http://benchmarks.cisecurity.org/ae/0.5"
xmlns:cc6="http://cisecurity.org/20-cc/v6.1"
xmlns:cc7="http://cisecurity.org/20-cc/v7.0"
xmlns:ciscf="https://benchmarks.cisecurity.org/ciscf/1.0"
xmlns:notes="http://benchmarks.cisecurity.org/notes"
xmlns:xccdf="http://checklists.nist.gov/xccdf/1.2"
xmlns:xhtml="http://www.w3.org/1999/xhtml"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">Attack surface
reduction helps prevent actions and apps that are typically used by
exploit-seeking malware to infect machines.</xhtml:p>
I would only like to get the output with TEXT "Attack surface reduction helps prevent actions and apps that are typically used by exploit-seeking malware to infect machines" and remove rest of the data.
Remediation Example:
<xhtml:div xmlns="http://checklists.nist.gov/xccdf/1.2"
xmlns:ae="http://benchmarks.cisecurity.org/ae/0.5"
xmlns:cc6="http://cisecurity.org/20-cc/v6.1"
xmlns:cc7="http://cisecurity.org/20-cc/v7.0"
xmlns:ciscf="https://benchmarks.cisecurity.org/ciscf/1.0"
xmlns:notes="http://benchmarks.cisecurity.org/notes"
xmlns:xccdf="http://checklists.nist.gov/xccdf/1.2"
xmlns:xhtml="http://www.w3.org/1999/xhtml"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <xhtml:p>
<xhtml:p> To establish the recommended configuration via GP, set the
following UI path to <xhtml:span
class="inline_block">Enabled</xhtml:span> : </xhtml:p> <xhtml:code
class="code_block">Computer Configuration\Policies\Administrative
Templates\Windows Components\Windows Defender Antivirus\Windows Defender
Exploit Guard\Attack Surface Reduction\Configure Attack Surface
Reduction rules </xhtml:code> <xhtml:p> <xhtml:strong>Note:
</xhtml:strong> This Group Policy path may not exist by default. It is
provided by the Group Policy template <xhtml:span
class="inline_block">WindowsDefender.admx/adml</xhtml:span> that is
included with the Microsoft Windows 10 Release 1709 Administrative
Templates (or newer). </xhtml:p> <xhtml:p class="bold">Impact:</xhtml:p>
<xhtml:p> <xhtml:p>When a rule is triggered, a notification will be
displayed from the Action Center.</xhtml:p> </xhtml:p> </xhtml:p>
</xhtml:div>
Again need to output of remediation to be clean without XHTML and xmlns, so the clean text should look like something below:
To establish the recommended configuration via GP, set the following UI path to Enabled : Computer Configuration\Policies\Administrative Templates\Windows Components\Windows Defender Antivirus\Windows Defender Exploit Guard\Attack Surface Reduction\Configure Attack Surface Reduction rules, This Group Policy path may not exist by default. It is provided by the Group Policy template WindowsDefender.admx/adml that is included with the Microsoft Windows 10 Release 1709 Administrative Templates (or newer).
Impact: When a rule is triggered, a notification will be displayed from the Action Center.
Is there a way I can get clean HTML to text ? without the content that I don't need?

Is there a way I can get clean HTML to text?
If it is fully valid XHTML, you can cast it as XML and then select the value of the node that interests you using XPath.
declare #xhtml varchar(max)
set #xhtml = '<xhtml:div xmlns="http://checklists.nist.gov/xccdf/1.2" xmlns:ae="http://benchmarks.cisecurity.org/ae/0.5" xmlns:cc6="http://cisecurity.org/20-cc/v6.1" xmlns:cc7="http://cisecurity.org/20-cc/v7.0" xmlns:ciscf="https://benchmarks.cisecurity.org/ciscf/1.0" xmlns:notes="http://benchmarks.cisecurity.org/notes" xmlns:xccdf="http://checklists.nist.gov/xccdf/1.2" xmlns:xhtml="http://www.w3.org/1999/xhtml" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <xhtml:p> <xhtml:p>To establish the recommended configuration via GP, set the following UI path to <xhtml:span class="inline_block">Enabled</xhtml:span> : </xhtml:p> <xhtml:code class="code_block">Computer Configuration\Policies\Administrative Templates\Windows Components\Windows Defender Antivirus\Windows Defender Exploit Guard\Attack Surface Reduction\Configure Attack Surface Reduction rules </xhtml:code> <xhtml:p> <xhtml:strong>Note:</xhtml:strong> This Group Policy path may not exist by default. It is provided by the Group Policy template <xhtml:span class="inline_block">WindowsDefender.admx/adml</xhtml:span> that is included with the Microsoft Windows 10 Release 1709 Administrative Templates (or newer). </xhtml:p> <xhtml:p class="bold">Impact:</xhtml:p> <xhtml:p> <xhtml:p>When a rule is triggered, a notification will be displayed from the Action Center.</xhtml:p> </xhtml:p> </xhtml:p> </xhtml:div>'
;with xmlnamespaces('http://www.w3.org/1999/xhtml' as xhtml)
select (cast(#xhtml as xml)).value('(//xhtml:p)[1]', 'varchar(max)')
See http://sqlfiddle.com/#!18/9eecb/136834.
Use more specific XPath if you need. For more complex requirements than "value of a single node" you can switch to XQuery, too.

Related

How to skip a testcase if a link is not present and go to next link in Robot framework

Scenario:
There are 5 Links in the Home page:
Link 1
Link 2
Link 3
Link 4
Link 5
Each of the above links are separate test cases, so there are a total of 5 test cases.
All the links may not present in all the sites, according to the requirements.
So I need to write a Robot framework test case which works dynamically for all the sites, Like 1 site may have 3 links only some has all the 5 links. So its like SKIPPING a particular Test case if that lisk is not present.
*** Keywords ***
Go to Manage Client Reports
Click Link link:Manage Client Reports
Can anyone help.
In the upcoming Robot Framework Release 4.0 a new test status skipped will be introduced. Here is a brief status about the release:
Past due by 27 days 87% complete
Major release concentrating on adding the skip status (#3622), IF/ELSE
(#3074) and enhancing the listener API (#3296 and #3538). Last major
release to support Python 2.
So it can be ready any time soon now.
This is what you can have New SKIP status #3622. There will be a Skip If and a Skip keywords and more to be used.
How to skip tests
There are going to be multiple ways:
A special exception that library keywords can use to mark a single test to be skipped. See also #3685.
BuiltIn keyword Skip (or Skip Test and Skip Task) that utilizes the aforementioned exception.
BuiltIn keyowrd Skip If to skip based on condition.
When the skipping exception is used in a suite setup, all tests in the suite are skipped.
Command line option --skip to unconditionally skip tests based on tags. Similar to --exclude but skipped tests are shown in logs/reports
with a skip status and not dropped from execution altogether.
Command line option --skiponfailure to skip tests if they fail. Similar effect than with the current --noncritical.
What about criticality
As already discussed in #2087, the skip status is very similar feature
than Robot's current criticality concept. There are many people who
would like to have both, but I don't think that's a good idea and
believe it's better to remove criticality when skipping is added.
Separate issue #3624 covers removing criticality and explains this in
more detail. Colors
Skip status needs a specific color to match current pass (green) and
fail (red). Yellow feels like a good candidate with a traffic light
metaphor, but I'm open for other ideas and we could possibly change
other colors as well. Probably should make colors configurable too --
currently only report background colors support it.
Report background color mentioned above needs some thinking as well.
Currently it's either green or red, but with the added skip status we
could use also yellow or whatever skip color we decide to use.
Different scenarios where different colors could be used are listed
below (assuming green/yellow/red scheme):
All tests pass. This is naturally green.
Any test fails. This is naturally red.
Any test is skipped (no failures). This probably should be green but could also be yellow.
All tests skipped. This could be yellow. Could also be green but that's a bit odd if all tests are yellow.
Depending on your deadlines you might won't be able to wait this release, nevertheless it is a good to know thing.
There is an advanced solution where you can generate your test cases run-time. To do so you have to implement a small library that also acts as a listener. This way it can have a start_suite method that will be invoked and it will get the suite(s) as Python object(s), robot.running.model.TestSuite. Then you could use this object along with Robot Framework's API to create new test cases. The idea below was inspired by and it is based on this blog post: Dynamically create test cases with Robot Framework.
DynamicTestLibrary.py:
from robot.running.model import TestSuite
class DynamicTestLibrary(object):
ROBOT_LISTENER_API_VERSION = 3
ROBOT_LIBRARY_SCOPE = 'GLOBAL'
ROBOT_LIBRARY_VERSION = 0.1
def __init__(self):
self.ROBOT_LIBRARY_LISTENER = self
self.top_suite = None
def _start_suite(self, suite, result):
self.top_suite = suite
self.top_suite.tests.clear() # remove placeholder test
def add_test_case(self, keyword, *args):
tc = self.top_suite.tests.create(name=keyword)
tc.keywords.create(name=keyword, args=args)
globals()[__name__] = DynamicTestLibrary
UPDATE for Robot Framework 4.0
Due to the backward incompatible changes (Running and result models have been changed) made in the 4.0 release the add_test_case function should be change as below if you are using version above 4.0.
def add_test_case(self, name, keyword, *args):
tc = self.top_suite.tests.create(name=name)
tc.body.create_keyword(name=keyword, args=args)
You can utilize this library in a suite setup, in which you check which links are present and add test cases for the ones that are available.
test.robot
*** Settings ***
Library DynamicTestLibrary
Suite Setup Check Links And Generate Test Cases
*** Variables ***
##{LINKS} Manage Clients # test input 1
#{LINKS} Manage Clients Manage Client Hardware # test input 2
##{LINKS} Manage Clients Manage Client Hardware Manage Client Reports # test input 3
*** Test Cases ***
Placeholder
[Documentation] Placeholder test that will be removed during execution.
No Operation
*** Keywords ***
Check Links And Generate Test Cases
FOR ${link} IN #{LINKS}
DynamicTestLibrary.Add Test Case Go to ${link}
END
Go to Manage Client Reports
Log Many Click Link link:Manage Client Reports
Go to Manage Client Hardware
Log Many Click Link link:Manage Client Hardware
Go to Manage Clients
Log Many Click Link link:Manage Clients
Go to ${link} will give the appropriate keyword name that will be called in a test case with the same name. You can check with each example input list that the number of executed tests will be equal with the length of the list.
Here is the output:
# robot --pythonpath . test.robot
==============================================================================
Test
==============================================================================
Go to Manage Clients | PASS |
------------------------------------------------------------------------------
Go to Manage Client Hardware | PASS |
------------------------------------------------------------------------------
Test | PASS |
2 critical tests, 2 passed, 0 failed
2 tests total, 2 passed, 0 failed
==============================================================================

Omitting the creation of a temporary access path with DB2/400 when accessing DDS-defined tables with SQL

I have two table definitions in DDS, compiled into *FILE objects and filled with data:
Kunpf:
A UNIQUE
A R KUNTBL
A FIRMA 60A ALWNULL
A KUNR 5S 0B
A KUNID 4S 0B
A K KUNR
A K KUNID
Kunsupf:
A R KUNSUTBL
A KUNID R B REFFLD(KUNID KUN/KUNPF)
A
A SUCHSTR 78A
A K SUCHSTR
A K KUNID
I'm using the following statement in interactive SQL (STRSQL):
SELECT DISTINCT FIRMA, KUNR FROM KUN/KUNPF
LEFT JOIN KUN/KUNSUPF ON (KUNPF.KUNID = KUNSUPF.KUNID)
WHERE SUCHSTR LIKE 'Freiburg%'
ORDER BY FIRMA
FOR READ ONLY
Everytime I execute this statement, I'm getting a considerable delay until the answer screen opens up. Beforehand a message is shown, stating that a temporary access path is being created.
How can I find out which/how this temporary access path is created? My goal is to have this access path made permanent so it doesn't need to be rebuilt with every invocation of this query.
I searched the net (especially the IBM site) but what I found out was mostly for DB2 on z/OS. The F4-Prompting facility in STRSQL doesn't provide help: I was searching for something like EXPLAIN SELECT from MySQL. The IBM DB2 Advanced Functions and Administration PDF states that there's a debug mode but it seems that it is only available from some (old) Windows tool I don't remember to have.
I'm utilizing V4R5, if this is relevant.
to see the access path on the green screen...
strdbg
strsql
run your statement
exit f3
enddbg
dspjoblog
the access path messages are at the bottom of the log f10 f18 afaik
v4r5??? That's like 20 years old...
For the IBM i, the "Run SQL Scripts" component of the old Client Access For Windows iSeries Navigator component and the new Access Client Solutions (ACS) contains Visual Explain (VE).
Luckily it seems though it was added to v4r5
http://ibmsystemsmag.com/ibmi/administrator/db2/database-performance-tuning-with-visual-explain/
Just start iNav, right click on "Database" and select "Run SQL Scripts"
Paste your query there and click "Visual Explain" -->"Run and Explain"
(or the corresponding button)
Optionally, in green screen.
Do a STRDBG to enter debug mode, F12 to continue and then go into STRSQL. The Db optimizer will then output additional messages into the joblog giving you more information about what it is doing..

Corrupted MD-SAL queries when trying to read flows on tables: missing flow rules

I am experiencing a glitch from OpenDaylight (using Mininet).
Essentially, I am querying flow rules on specific nodes and on specific tables. The relevant code is the following, and is run by 1 separate thread per node that I am polling:
public static final InstanceIdentifier<Nodes NODES_II = InstanceIdentifier
.builder(Nodes.class).build();
public static InstanceIdentifier<Table> makeTableIId(NodeId nodeId, Short tableId) {
return NODES_IID.child(Node.class, new NodeKey(nodeId))
.augmentation(FlowCapableNode.class)
.child(Table.class, new TableKey(tableId));
}
and
InstanceIdentifier<Table> tableIId = makeTableIId(nodeId, tableId);
Optional<Table> tableOptional = dataBroker.newReadOnlyTransaction()
.read(LogicalDatastoreType.OPERATIONAL, tableIId).get();
if(!tableOptional.isPresent()) {
continue;
}
List<Flow> flows = tableOptional.get().getFlow();
The behavior: tableOptional is present, and getFlow() returns an empty list.
The observation: there ARE flow rules installed on ALL nodes on the tables I am querying, but for some reason, some of these nodes show none of these flows on none of the tables (here, tables 3, 4, 5, and 6).
The weirdness: On one of the problematic nodes, I have four rules, installed on tables 9, 13, 17 and 22 respectively. They timeout simultaneously after 150 seconds. After they disappear, the query suddenly begins to "see" the flows installed on tables 3, 4, 5, and 6, returning these for each table.
Question: How is this even possible?
EDIT I just realized that the rules whose timeout "suddenly fix everything" were also rules that generated warnings in ODL's log (OpenFlowPlugin to be more specific). I did not observe any obvious issue, so I'd sort of brushed it aside.
Here is the code relevant to the error:
https://pastebin.com/yJDZesXU
Here are the errors I get every time I install a rule that walks through these lines:
https://pastebin.com/c9HYLBt6
I must stress that these rules work as intended, and that printing them out reveals no evident formatting issue. Again, they appear fine when dumped.
My hypothesis is that this warning is a symptom of ODL "messing up" trying to store the rules in MD-SAL, which ends up messing a lot of rule-reading queries. On uninstallation of the garbage that ensues, rule-reading queries become functional again.
This makes sense to me, but then... I haven't understood how to fix these warnings, or what these warnings were about in the first place.
EDIT 2: By commenting lines suspecting of causing the warnings in the above pastebin:
//ipv4MatchBuilder.setIpv4SourceAddressNoMask(...);
//ipv4MatchBuilder.setIpv4SourcenArbitraryBitmask(...);
The warnings disappear, AND the flows appear correctly on all tables, when pinged. This confirms my hypothesis that somewhere, something wrong happens in the data store.
EDIT 3: I have found that by setting any non-trivial arbitrary bitmask, this error goes away. That is, I have tried setting an arbitrary bitmask which was neither null nor "255.255.255.255", and this error has gone away. The problem is I might like having a bitmask for the source, but an exact match on the destination. Even setting the bitmask to "127.255.255.255" (as I tried) is still unnerving. It really feels to me like this is an OpenFlowPlugin glitch, though.
EDIT 4: Steps for reproducing the bug
Install a rule with ipv4 arbitrary bitmask match, with the destination ip set, and the destination arbitrary bitmask either null or set to 255.255.255.255.
Ipv4MatchArbitraryBitMaskBuilder ipv4MatchBuilder = new Ipv4MatchArbitraryBitMaskBuilder();
ipv4MatchBuilder.setIpv4DestinationAddressNoMask(new Ipv4Address("10.0.0.1"));
ipv4MatchBuilder.setIpv4DestinationArbitraryBitmask(new DottedQuad("255.255.255.255"));
matchBuilder = new MatchBuilder().setEthernetMatch(ethernetMatchBuilder.build()).setLayer3Match(ipv4MatchBuilder.build());
... and so on ...
Extra optional steps: Install one such rule for the destination, one such rule for the source, and install equivalent rules where the bitmask is set to something else, like 127.255.255.255.
Make a query to MDSal to fetch flow information from the node on which you installed the flow rule.
Now, do "log:display" inside your ODL controller. You should have a warning about a malformed destination address. Additionally, the Table object you queried should contain no flows, so tableObject.getFlow() should return an empty list.

Expert advisor in different trader platform

I would like to know, if I code an EA in a normal metatrader4 platform, can I reuse the .ex4 in other trading platform for example InstaTrader?
The reason is that, when I have created a new EA in InstaTrader, the EA code generated from InstaTrader is different from the one generated from metatrader4. And I couldn't find any documentation regarding to InstaTrader's EA.
Not sure anyone has encountered this before?
No. MQL is the language specifically for meta-editor which is a member of meta-trader platform.
Other trading languages may have their own scripting languages.
Metatrader4
In principle Metatrader4 uses a Metalang.exe that compiles the MQL4-source-code files into an "internally"-executable format EX4
As defined, EX4 is binary-executable on all Metatrader4 terminals.
.
WhiteLabel-ed Terminals
InstaTrader(TM) and many other *-Trader(TM)-s are so called White-Label modifications of the same MetaQuotes, Inc., software product, the [Metatrader 4 Terminal], that are just individually "skinned" in a name of the respective Broker, who has bought from the MetaQuotes, Inc. the license for a suite of [Metatrader 4 Server + Metatrader 4 Risk Management + Metatrader 4 Dealer Desk + ... ], inclusive, but not limited to, the right to re-label the client Terminal program.
Thus under most of the situations your EX4 code shall run on any other re-labeled Terminal
But...
Restrictions on binary-compatibility apply, as Metatrader4 terminals gets released in so called Builds, ( Build 432 -> Build 468 -> Build 509 -> ... -> Build 600 -> Build 624 ) and some of these have also modified the binary-code format.
Thus the EX4 code shall be hosted on a "similar" generation of the Terminal Build
Finally...
The ultimate show-stopper is the MetaQuotes, Inc., licensing policy, which makes a server-side locking to take place [Metatrader 4 Server] has a setting to reject connection requests from client Terminals in case their Build # is lesser than a treshold set on Server side.
There the SLM story ends. Forever.

Need help in Apache Camel multicast/parallel/concurrent processing

I am trying to achieve concurrent/parallel processing in my requirement, but I did not get appropriate help in my multiple attempts in this regard.
I have 5 remote directories ( which may be added or removed) which contains log files, I want to Dow load them for every 15 minutes to my local directory and want to perform Lucene indexing after completion of ftp transfer job, I want to add routers dynamically.
Since all those remote machines are different end points , and different routes. I don't have any particular end point to kickoff all these.
Start
<parallel>
<download remote dir from: sftp1>
<download remote dir from: sftp2>
....
</parallel>
<After above task complete>
<start Lucene indexing>
<end>
Repeat above for every 15 minutes,
I want to download all folders paralally, Kindly suggest the solution if anybody worked on similar requirement.
I would like to know how to start/initiate these multiple routes (like this multiple remote directories) should be kick started when I don't have a starter end point. I would like to start all ftp operations parallel and on completing those then indexing. Thanks for taking time to reading this post , I really appreciate your help.
I tried like this,
from (bean:foo? Method=start).multicast ().to (direct:a).to (direct:b)...
From (direct:a) .from (sftp:xxx).to (localdir)
from (direct:b).from (sftp:xxx).to (localdir)
camel-ftp support periodic polling via the consumer.delay property
add camel-ftp consumer routes dynamically for each server as shown in this unit test
you can then aggregate your results based on a size or timeout value to initiate the Lucene indexing, etc
[todo - put together an example]