How do I loop thought each DB field to see if range is correct - sql

I have this response in soapUI:
<pointsCriteria>
<calculatorLabel>Have you registered for inContact, signed up for marketing news from FNB/RMB Private Bank, updated your contact details and chosen to receive your statements</calculatorLabel>
<description>Be registered for inContact, allow us to communicate with you (i.e. update your marketing consent to 'Yes'), receive your statements via email and keep your contact information up to date</description>
<grades>
<points>0</points>
<value>No</value>
</grades>
<grades>
<points>1000</points>
<value>Yes</value>
</grades>
<label>Marketing consent given and Online Contact details updated in last 12 months</label>
<name>c21_mrktng_cnsnt_cntct_cmb_point</name>
</pointsCriteria>
There are many many many pointsCriteria and I use the below xquery to give me the DB value and Range of what that field is meant to be:
<return>
{
for $x in //pointsCriteria
return <DBRange>
<db>{data($x/name/text())}</db>
<points>{data($x//points/text())}</points>
</DBRange>
}
</return>
And i get the below response
<return><DBRange><db>c21_mrktng_cnsnt_cntct_cmb_point</db><points>0 1000</points></DBRange>
That last bit sits in a property transfer. I need SQL to bring back all rows where that DB field is not in that points range (field can only be 0 or 1000 in this case), my problem is I dont know how to loop through each DBRange/DBrange in this manner? please help

I'm not sure that I really understand your question, however I think that you want to make queries in your DB using specific table with a column name defined in your <db> field of your xml, and using as values the values defined in <points> field of the same xml.
So you can try using a groovy TestStep, first parse your Xml and get back your column name, and your points. To iterate over points if the values are separated with a blank space you can make a split(" ") to get a list and then use each() to iterate over the points on this list. Then using groovy.sql.Sql you can perform the queries in your DB.
Only one more thing, you need to put the JDBC drivers for your vendor DB in $SOAPUI_HOME/bin/ext and then restart SOAPUI in order that it can load the necessary driver classes.
So the follow code approach can achieve your goal:
import groovy.sql.Sql
import groovy.util.XmlSlurper
// soapui groovy testStep requires that first register your
// db vendor drivers, as example I use oracle drivers...
com.eviware.soapui.support.GroovyUtils.registerJdbcDriver( "oracle.jdbc.driver.OracleDriver")
// connection properties db (example for oracle data base)
def db = [
url : 'jdbc:oracle:thin:#db_host:d_bport/db_name',
username : 'yourUser',
password : '********',
driver : 'oracle.jdbc.driver.OracleDriver'
]
// create the db instance
def sql = Sql.newInstance("${db.url}", "${db.username}", "${db.password}","${db.driver}")
def result = '''<return>
<DBRange>
<db>c21_mrktng_cnsnt_cntct_cmb_point</db>
<points>0 1000</points>
</DBRange>
</return>'''
def resXml = new XmlSlurper().parseText(result)
// get the field
def field = resXml.DBRange.db.text()
// get the points
def points = resXml.DBRange.points.text()
// points are separated by blank space,
// so split to get an array with the points
def pointList = points.split(" ")
// for each point make your query
pointList.each {
def sqlResult = sql.rows "select * from your_table where ${field} = ?",[it]
log.info sqlResult
}
sql.close();
Hope this helps,

Thanks again for your help #albciff, I had to add this into a multidimensional array (I renamed field to column and result is a large return from the Xquery above)
def resXml = new XmlSlurper().parseText(result)
//get the columns and points ranges
def Column = resXml.DBRange.db*.text()
def Points = resXml.DBRange.points*.text()
//sorting it all out into a multidimensional array (index per index)
count = 0
bigList = Column.collect
{
[it, Points[count++]]
}
//iterating through the array
bigList.each
{//creating two smaller lists and making it readable for sql part later
def column = it[0]
def points = it[1]
//further splitting the points to test each
pointList = points.split(" ")
pointList.each
{//test each points range per column
def sqlResult = sql.rows "select * from my_table where ${column} <> ",[it]
log.info sqlResult
}
}
sql.close();
return;

Related

How to get row data from web table in Karate framework

The web table has a combination of textbox, span and checkboxes. I need to get first row of all the data in single def and have to verify with DB in order wise.
Ex: First row of table has columns like below.
OrderID(span), EmpName(input), IsHeEligible(checkbox), Address(span)
By using below,
def tebleFirstData = scriptAll("table/tbody/tr/td",'_.textContent')
Able to get only span text data, not able to get input tag test data.
I tried below,
data = attribute("table/tbody/tr/td[5]/input",'value')
But, I'm able to get only single input tag attribute value.
How can I get all the data in single def, i.e span data and input data?
Below is the solution to get all row data from table..!
* def UiFirstRowElements = locateAll("Row xpath")
print UiFirstRowElements
def tableData = []
def RowHtml = UiFirstRowElements[0].html
print RowHtml
eval
"""
for(var i=0; i<UiFirstRowElements.length; i++)
if(UiFirstRowElements[i].html.contains("input") && UiFirstRowElements[i].html.contains("date")){
tableData.push(locate("//table/tbody/tr/td["+(i+1)+"]/div/input").property('value'))
}
else if(UiFirstRowElements[i].html.contains("input") && UiFirstRowElements[i].html.contains("checkbox")){
tableData.push(locate("//table/tbody/tr/td["+(i+1)+"]/div/input").property('checked'))
}
else {
tableData.push(locate("//table/tbody/tr/td["+(i+1)+"]").property('textContent'))
}
"""
* print 'TableName-->', TableName
* print tableData

Passing a list to SQL each row call Groovy

I am currently rendering a list of sql rows from a database using:
Sql sql = new Sql(dataSource)
def list = []
def index = 0
params.mrnaIds.each { mrnaName ->
sql.eachRow ("select value from patient_mrna where mrna_id=$mrnaId") { row ->
list[index] = row.value
index++
}
}
render list
However I would like to avoid assigning the values to a list before rendering them.
The variable params.mrnaIds is coming from a multi select input, so it could either be a single string or a string array containing ids. Is there a way to iterate through these ids inside the eachRow method?
I would like to be able to execute something like:
render sql.eachRow ("select value from patient_mrna where mrna_id=?", params.mrnaIds) { row ->
list[index] = row.value
index++
}
But I'm not completely sure that there is a way to call eachRow with this functionality. If there is not, is there some other way to render the results without storing them in a list?
I think you can render each row:
sql.eachRow( someQuery, someParams ){ row ->
render row as JSON
}
There is rows() to return a list instead ok working with it (like eachRow() is used for). It also shares all the different arguments. E.g.:
render sql.rows("select value from patient_mrna where mrna_id=?", params).collect{ it.value }

Grails query rows to arrays

I'm new to Groovy and Grails. I think this problem probably has an easy answer, I just don't know it.
I have a database table:
id | category | caption | image | link
I have a query that lets me retrieve one row for each distinct item in 'category.'
I'm trying to return a map where each row is an array named by it's category.
e.g., If I select the rows:
[{category='foo', caption='stuff', ...} {category='bar', caption='things', ...}]
I want to be able to:
return [foo:foo, bar:bar]
where:
foo = [caption='stuff', ...]
bar = [caption='things', ...]
Thanks for any help.
You can transform your list using collect in Groovy, however, the result depends on the source data. I could not infer from your post that if you are returning one item per category or multiple.
def ls = Category.list()
def newList = ls.collect {
[(it.category):['caption':it.caption,'image':it.image,'link':it.link]]
}
will result in something like :
[
[bar:[caption:BarCaption-2, image:BarImage-2, link:BarLink-2]],
[bar:[caption:BarCaption-1, image:BarImage-1, link:BarLink-1]],
[bar:[caption:BarCaption-0, image:BarImage-0, link:BarLink-0]],
[foo:[caption:FooCaption-2, image:FooImage-2, link:FooLink-2]],
[foo:[caption:FooCaption-1, image:FooImage-1, link:FooLink-1]],
[foo:[caption:FooCaption-0, image:FooImage-0, link:FooLink-0]]
]
If you have multiple items per each category you probably want to return the list of each.
def bars = newList.findAll { it.containsKey 'bar' }
def foos = newList.findAll { it.containsKey 'foo' }
[foos:foos,bars:bars]
If I understand your question correctly (I think you mean to put ':' instead of '=' for your maps) then I think the following will work. If new to Groovy, you might find the Groovy web console at http://groovyconsole.appspot.com/ useful. You can try snippets of code out easily, like the following:
def listOfMaps = [[category:'foo', caption:'stuff', something:'else'], [category:'bar', caption:'things', another:'thing']]
def mapOfMaps = [:]
listOfMaps.each { mapOfMaps += [(it.remove('category')) : it] }
assert mapOfMaps == [foo:[caption:'stuff', something:'else'], bar:[caption:'things', another:'thing']]

How to convert a list of attribute-value pairs into a flat table whose columns are attributes

I'm trying to convert a csv file containing 3 columns (ATTRIBUTE_NAME,ATTRIBUTE_VALUE,ID) into a flat table whose each row is (ID,Attribute1,Attribute2,Attribute3,....). The samples of such tables are provided at the end.
Either Python, Perl or SQL is fine. Thank you very much and I really appreciate your time and efforts!
In fact, my question is very similar to this post, except that in my case the number of attributes is pretty big (~300) and not consistent across each ID, so hard coding each attribute might not be a practical solution.
For me, the challenging/difficult parts are:
There are approximately 270 millions lines of input, the total size of the input table is about 60 GB.
Some single values (string) contain comma (,) within, and the whole string will be enclosed with double-quote (") to make the reader aware of that. For example "JPMORGAN CHASE BANK, NA, TX" in ID=53.
The set of attributes is not the same across ID's. For example, the number of overall attributes is 8, but ID=53, 17 and 23 has only 7, 6 and 5 respectively. ID=17 does not have attributes string_country and string_address, so output blank/nothing after the comma.
The input attribute-value table looks like this. In this sample input and output, we have 3 ID's, whose number of attributes can be different depending on we can obtain such attributes from the server or not.
ATTRIBUTE_NAME,ATTRIBUTE_VALUE,ID
num_integer,100,53
string_country,US (United States),53
string_address,FORT WORTH,53
num_double2,546.0,53
string_acc,My BankAcc,53
string_award,SILVER,53
string_bankname,"JPMORGAN CHASE BANK, NA, TX",53
num_integer,61,17
num_double,34.32,17
num_double2,200.541,17
string_acc,Your BankAcc,17
string_award,GOLD,17
string_bankname,CHASE BANK,17
num_integer,36,23
num_double,78.0,23
string_country,CA (Canada),23
string_address,VAN COUVER,23
string_acc,Her BankAcc,23
The output table should look like this. (The order of attributes in the columns is not fixed. It can be sorted alphabetically or by order-of-appearance.)
ID,num_integer,num_double,string_country,string_address,num_double2,string_acc,string_award,string_bankname
53,100,,US (United States),FORT WORTH,546.0,My BankAcc,SILVER,"JPMORGAN CHASE BANK, NA, TX"
17,61,34.32,,,200.541,Your BankAcc,GOLD,CHASE BANK
23,36,78.0,CA (Canada),VAN COUVER,,Her BankAcc,,
This program will do as you ask. It expects the name of the input file as a parameter on the command line.
Update Looking more carefully at the data I see that not all of the data fields are available for every ID. That makes things more complex if the fields are to be kept in the same order as they appear in the file.
This program works by scanning the file and accumulating all the data for output into hash %data. At the same time it builds a hash %headers, that keeps the position each header appears in the data for each ID value.
Once the file has been scanned, the collected headers are sorted by finding the first ID for each pair that includes information for both headers. The sort order for that pair within the complete set must be the same as the order they appeared in the data for that ID, so it's just a matter of comparing the two position values using <=>.
Once a sorted set of headers has been created, the %data hash is dumped, accessing the complete list of values for each ID using a hash slice.
Update 2 Now that I realise the sheer size of your data I can see that my second attempt was also flawed, as it tried to read all of the information into memory before outputting it. That isn't going to work unless you have a monster machine with about 1TB of memory!
You may get some mileage from this version. It scans twice through the file, the first time to read the data so that the full set of header names can be created and ordered, then again to read the data for each ID and output it.
Let me know if it's not working for you, as there's still things I can do to make it more memory-efficient.
use strict;
use warnings;
use 5.010;
use Text::CSV;
use Fcntl 'SEEK_SET';
my $csv = Text::CSV->new;
open my $fh, '<', $ARGV[0] or die qq{Unable to open "$ARGV[0]" for input: $!};
my %headers = ();
my $last_id;
my $header_num;
my $num_ids;
while (my $row = $csv->getline($fh)) {
next if $. == 1;
my ($key, $val, $id) = #$row;
unless (defined $last_id and $id eq $last_id) {
++$num_ids;
$header_num = 0;
$last_id = $id;
print STDERR "Processing ID $id\n";
}
$headers{$key}[$num_ids-1] = ++$header_num;
}
sub by_position {
for my $id (0 .. $num_ids-1) {
my ($posa, $posb) = map $headers{$_}[$id], our $a, our $b;
return $posa <=> $posb if $posa and $posb;
}
0;
}
my #headers = sort by_position keys %headers;
%headers = ();
print STDERR "List of headers complete\n";
seek $fh, 0, SEEK_SET;
$. = 0;
$csv->combine('ID', #headers);
print $csv->string, "\n";
my %data = ();
$last_id = undef;
while () {
my $row = $csv->getline($fh);
next if $. == 1;
if (not defined $row or defined $last_id and $last_id ne $row->[2]) {
$csv->combine($last_id, #data{#headers});
print $csv->string, "\n";
%data = ();
}
last unless defined $row;
my ($key, $val, $id) = #$row;
$data{$key} = $val;
$last_id = $id;
}
output
ID,num_integer,num_double,string_country,string_address,num_double2,string_acc,string_award,string_bankname
53,100,,"US (United States)","FORT WORTH",546.0,"My BankAcc",SILVER,"JPMORGAN CHASE BANK, NA, TX"
17,61,34.32,,,200.541,"Your BankAcc",GOLD,"CHASE BANK"
23,36,78.0,"CA (Canada)","VAN COUVER",,"Her BankAcc",,
Use Text::CSV from CPAN:
#!/usr/bin/env perl
use strict;
use warnings;
# --------------------------------------
use charnames qw( :full :short );
use English qw( -no_match_vars ); # Avoids regex performance penalty
use Text::CSV;
my $col_csv = Text::CSV->new();
my $id_attr_csv = Text::CSV->new({ eol=>"\n", });
$col_csv->column_names( $col_csv->getline( *DATA ));
while( my $row = $col_csv->getline_hr( *DATA )){
# do all the keys but skip if ID
for my $attribute ( keys %$row ){
next if $attribute eq 'ID';
$id_attr_csv->print( *STDOUT, [ $attribute, $row->{$attribute}, $row->{ID}, ]);
}
}
__DATA__
ID,num_integer,num_double,string_country,string_address,num_double2,string_acc,string_award,string_bankname
53,100,,US (United States),FORT WORTH,546.0,My BankAcc,SILVER,"JPMORGAN CHASE BANK, NA, TX"
17,61,34.32,,,200.541,Your BankAcc,GOLD,CHASE BANK
23,36,78.0,CA (Canada),VAN COUVER,,Her BankAcc,,

SQL to Magento model understanding

Understanding Magento Models by reference of SQL:
select * from user_devices where user_id = 1
select * from user_devices where device_id = 3
How could I perform the same using my magento models? getModel("module/userdevice")
Also, how can I find the number of rows for each query
Following questions have been answered in this thread.
How to perform a where clause ?
How to retrieve the size of the result set ?
How to retrieve the first item in the result set ?
How to paginate the result set ? (limit)
How to name the model ?
You are referring to Collections
Some references for you:
http://www.magentocommerce.com/knowledge-base/entry/magento-for-dev-part-5-magento-models-and-orm-basics
http://alanstorm.com/magento_collections
http://www.magentocommerce.com/wiki/1_-_installation_and_configuration/using_collections_in_magento
lib/varien/data/collection/db.php and lib/varien/data/collection.php
So, assuming your module is set up correctly, you would use a collection to retrieve multiple objects of your model type.
Syntax for this is:
$yourCollection = Mage::getModel('module/userdevice')->getCollection()
Magento has provided some great features for developers to use with collections. So your example above is very simple to achieve:
$yourCollection = Mage::getModel('module/userdevice')->getCollection()
->addFieldToFilter('user_id', 1)
->addFieldToFilter('device_id', 3);
You can get the number of objects returned:
$yourCollection->count() or simply count($yourCollection)
EDIT
To answer the question posed in the comment: "what If I do not require a collection but rather just a particular object"
This depends if you still require both conditions in the original question to be satisfied or if you know the id of the object you wish to load.
If you know the id of the object then simply:
Mage::getModel('module/userdevice')->load($objectId);
but if you wish to still load based on the two attributes:
user_id = 1
device_id = 3
then you would still use a collection but simply return the first object (assuming that only one object could only ever satisfy both conditions).
For reuse, wrap this logic in a method and place in your model:
public function loadByUserDevice($userId, $deviceId)
{
$collection = $this->getResourceCollection()
->addFieldToFilter('user_id', $userId)
->addFieldToFilter('device_id', $deviceId)
->setCurPage(1)
->setPageSize(1)
;
foreach ($collection as $obj) {
return $obj;
}
return false;
}
You would call this as follows:
$userId = 1;
$deviceId = 3;
Mage::getModel('module/userdevice')->loadByUserDevice($userId, $deviceId);
NOTE:
You could shorten the loadByUserDevice to the following, though you would not get the benefit of the false return value should no object be found:
public function loadByUserDevice($userId, $deviceId)
{
$collection = $this->getResourceCollection()
->addFieldToFilter('user_id', $userId)
->addFieldToFilter('device_id', $deviceId)
;
return $collection->getFirstItem();
}