I am using FileHelpers libary and I have a pipe "|" delimited file that must have only 4 fields, and I need to validate when a record has more than 4 fields and save error.
bla|bla2|bla3|bla4 <- Good Record
bla|bla2|bla3|bla4|bla5 <- Wrong record
File Helpers throw a BadUsageException but the message does not describe well the ocurrence.
Thanks for answer.
You can use the engine.AfterReadRecord event to tell FileHelpers to skip the record:
engine.AfterReadRecord += Engine_AfterReadRecord;
private void Engine_AfterReadRecord(EngineBase engine, FileHelpers.Events.AfterReadEventArgs<object> e)
{
e.SkipThisRecord = true;
}
This would make the engine skip passed every record since I haven't put any criteria in. Just add your own custom logic.
Related
I have the following QlistWidget:
self.myListWidget1 = QListWidget()
os.chdir("./Downloads")
for file in glob.glob("*.pdf"):
QListWidgetItem(file, self.myListWidget1)
self.myListWidget1.sortItems()
How can I add a QLineEdit that is able to filter items in self.myListWidget1 by partial
string?
The currently accepted answer by Papooch is not very good unfortunately. The reason is that it will not keep the current row event if it remains withing filter criteria. You do not want to generate a new list. You actually want to hide items which are not withing filter criteria. Do it like this:
void ItemsListWidget::filterItems(QString filter)
{
for (int row = 0; row < count(); ++row)
{
item(row)->setHidden(!item(row)->text().contains(filter, Qt::CaseInsensitive)); // you can leave out the case insensitivity if you do not want it
}
}
(it is Qt/C++ code but I am pretty sure you can easily adapt it for Python)
You are going to need a separate list on which you will do the filtering and then display the filtered list in the QListWidget.
The most basic example could look like this:
self.myLineEdit = QLineEdit()
self.myListWidget1 = QListWidget()
self.listOfPdfs = [] # this will hold the complete list of files
os.chdir("./Downloads")
for file in glob.glob("*.pdf"):
self.listOfPdfs.append(file) # fill the list
QListWidgetItem(file, self.myListWidget1)
# connect the signal textChanged to the filter function
self.myLineEdit.textChanged.connect(self.filterList)
def filterList(self, text):
self.myListWidget1.clear() # clear the list widget
for file on self.listOfPdfs:
if text in file: # only add the line if it passes the filter
QListWidgetItem(file, self.myListWidget1)
(note that I didn't check the validity of the code, minor modifications might be needed)
Scenario:
I have created transformation to load data into table from csv file and I have following columns in csv file:
Customer_Id
Company_Id
Employee_Name
But user may give input file with column ordering (random order) as
Employee_Name
Company_Id
Customer_Id
so, if I try to load file which has random column ordering, will kettle load correct column values as per column names ... ?
Using ETL Metadata Injection you can use a transformation like this, to either normalize the data, or to store it to your database:
Then you just need to send the correct data to that transformation. You can read the header line from the CSV, and use Row Normaliser to convert to the format used by ETL Metadata Injection.
I have included a quick example here: csv_inject on Dropbox, if you make something like this and run it from something that runs it per csv file it should work.
Ooh, thats some nasty javascript!
The way to do this is with metadata injection. Look at the samples, but basically you need a template which reads the file, and writes it back out. you then use another parent transformation to figure out the headings, configure that template and then execute it.
There are samples in the PDI samples folder, and also take a look at the "figuring out file format" example in matt casters blueprints project on github.
You could try something like this as your JavaScript:
//Script here
var seen;
trans_Status = CONTINUE_TRANSFORMATION;
var col_names = ['Customer_Id','Company_Id','Employee_Name'];
var col_pos;
if (!seen) {
// First line
trans_Status = SKIP_TRANSFORMATION;
seen = 1;
col_pos = [-1,-1,-1];
for (var i = 0; i < col_names.length; i++) {
for (var j = 0; j < row.length; j++) {
if (row[j] == col_names[i]) {
col_pos[i] = j;
break;
}
}
if (col_pos[i] === -1) {
writeToLog("e", "Cannot find " + col_names[i]);
trans_Status = ERROR_TRANSFORMATION;
break;
}
}
}
var Customer_Id = row[col_pos[0]];
var Company_Id = row[col_pos[1]];
var Employee_Name = row[col_pos[2]];
Here is the .ktr I tried: csv_reorder.ktr
(edit, here are the test csv files)
1.csv:
Customer_Id,Company_Id,Employee_Name
cust1,comp1,emp1
2.csv:
Employee_Name,Company_Id,Customer_Id
emp2,comp2,cust2
Assuming rejecting the input file is not an option you basically have 4 solutions.
reorder the fields in an external editor (don't use excel if it contains dates)
Use code within your transformation to detect the column headers and reorder the file.
Use metadata injection as proposed by bolav
Create a job. This need to:
a. load the file into a temporary database.
b. use an sql statement to retrieve the fields (use a SELECT with an ORDER By clause)
c. output the file in the correct order
I have a package which looks as follows:
Notice the two areas which I have marked with a red rectangle: they are identical in every way. Can I make changes to the package so I can avoid this duplication? It seems to me I cannot move them to a Data Flow Task since loops and File System Task do not exists there.
You can create a sub package for the Loop logic as #Bill mentioned, and the way to 'pass the resultset of a query on to another package' is as below (use SSIS 2012 as the example, I did the similar work SSIS 2005, so you only need to change the c# code to vb.net)
In your parent package, create a variable to hold the name of the resultSet variable in the parent package:
In your sub package, create a string variable parentResultSetName:
In your sub package, add a package configuration to mapping the parentResultSetName to the parent package variable resultSetVariableName:
Now, we can read the resultSet variable by the name in the script task of sub package:
public void Main()
{
// TODO: Add your code here
var dsName = Dts.Variables["parentResultSetName"].Value.ToString();
Variables variables = null;
DataSet resultSet = null;
Dts.VariableDispenser.LockForRead(dsName);
Dts.VariableDispenser.GetVariables(ref variables);
try
{
resultSet = variables[dsName].Value as DataSet;
if (resultSet != null)
{
MessageBox.Show("Sub package get: " + resultSet.Tables[0].Rows[0][0].ToString());
}
Dts.TaskResult = (int)ScriptResults.Success;
}
catch (Exception e)
{
Dts.Events.FireError(-1, "", e.Message, "", 0);
}
}
Here is the result:
Just place both your queries from "select batch logins" tasks to recordset and make another foreach loop over that recordset, executing those two queries from a variable.
I can see that in your second foreach loop there are some additional tasks, so you'll have incorporate them to "Select all batch logins" task somehow, or make constraints that'd match in both loops.
An alternative approach is to add an 'action' column to your 'batches' table. (or add an outrigger table). Work out beforehand what you want to do to the records. Then just delete the records and files once.
It looks like you are doing RBAR operations here.
So for example you run a couple of UPDATE statements against your tables that leave the tables in a state where each record is flagged for deletion or not.
Then after that you go through the loop and delete based off what is in the table.
It would make your package a lot simpler. and you can incorporate some 'commit' logic that makes sure a record and file are always deleted at the same time.
I am using JMeter v2.5.
I need to get data from the responses of the test and extract data from it (which I am doing using regular exp extractor). How do I store this extracted data to a file?
Just solved a similar problem. After getting the data using a regular expression extractor, add a BeanShell PostProcessor element. Use the code below to write the variables to a file:
name = vars.get("name");
email = vars.get("email");
log.info(email); // if you want to log something to jmeter.log file
// Pass true if you want to append to existing file
// If you want to overwrite, then don't pass the second argument
f = new FileOutputStream("/my/file/path/result.csv", true);
p = new PrintStream(f);
this.interpreter.setOut(p);
print(name + "," + email);
f.close();
import org.apache.jmeter.services.FileServer;
String path=FileServer.getFileServer().getBaseDir();
name1= vars.get("user_Name_value");
name2= vars.get("UserId_value");
f = new FileOutputStream("E://csvfile/result.csv", true); //spec-ify true if you want to overwrite file. Keep blank otherwise.
p = new PrintStream(f);
this.interpreter.setOut(p);
p.println(name1+"," +name2);
f.close();
this is worked for me i hope it will work for you also
If you just want to write extracted variables to CSV results file, then just add to user.properties the variables you want:
sample_variables=name,email
As per doc:
https://jmeter.apache.org/usermanual/properties_reference.html#results_file_config
They will be appended as last column of CSV results file.
You have a couple options
You can tally the results by adding an aggregate report listener to your thread group => add listener => aggregate report
You can get raw results by adding a simple data writer listener to your thread group => add listener => simple data writer
Hope this helps
You may use https://jmeter-plugins.org/wiki/FlexibleFileWriter/ with sample variables set up.
Or with fake Dummy Sampler.
Anyway Flexible File Writer is good for writing data into file.
I was wondering how I would go about checking to see if a table contains a value in a certain column.
I need to check if the column 'e-mail' contains an e-mail someone is trying to register with, and if something exists, do nothing, however, if nothing exists, insert the data into the database.
All I need to do is check if the e-mail column contains the value the user is registering with.
I'm using the RedBeanPHP ORM, I can do this without using it but I need to use that for program guidelines.
I've tried finding them but if they don't exist it returns an error within the redbean PHP file. Here's the error:Fatal error: Call to a member function find() on a non-object in /home/aeterna/www/user/rb.php on line 2433
Here's the code that I'm using when trying this:
function searchDatabase($email) {
return R::findOne('users', 'email LIKE "' . $email . '"');
}
My approach on the function would be
function searchDatabase($email) {
$data = array('email' => $email);
$user = R::findOne('users', 'email LIKE :email, $data);
if (!empty($user)) {
// do stuff here
} // end if
} // end function
It's a bit more clean and in your function
Seems like you are not connected to a database.
Have you done R::setup() before R::find()?
RedBeanPHP raises this error if it can't find the R::$redbean instance, the facade static functions just route calls to the $redbean object (to hide all object oriented fuzzyness for people who dont like that sort of thing).
However you need to bootstrap the facade using R::setup(). Normally you can start using RB with just two lines:
require('rb.php'); //cant make this any simpler :(
R::setup(); //this could be done in rb.php but people would not like that ;)
//and then go...
R::find( ... );
I recommend to check whether the $redbean object is available or whether for some reason the code flow has skipped the R::setup() boostrap method.
Edited to account for your updated question:
According to the error message, the error is happening inside the function find() in rb.php on line 2433. I'm guessing that rb.php is the RedBean package.
Make sure you've included rb.php in your script and set up your database, according to the instructions in the RedBean Manual.
As a starting point, look at what it's trying to do on line 2433 in rb.php. It appears to be calling a method on an invalid object. Figure out where that object is being created and why it's invalid. Maybe the find function was supplied with bad parameters.
Feel free to update your question by pasting the entirety of the find() function in rb.php and please indicate which line is 2433. If the function is too lengthy, you can paste it on a site like pastebin.com and link to it from here.
Your error sounds like you haven't done R::setup() yet.
My approach to performing the check you want would be something like this:
$count = count(R::find('users', 'email LIKE :email', array(':email' => $email)));
if($count === 0)
{
$user = R::dispense('users');
$user->name = $name;
$user->email = $email;
$user->dob = $dob;
R::store($user);
}
I don't know if it is this basic or not, but with SQL (using PHP for variables), a query could look like
$lookup = 'customerID';
$result = mysql_fetch_array(mysql_query("SELECT columnName IN tableName WHERE id='".$lookup."' LIMIT 1"));
$exists = is_null($result['columnName'])?false:true;
If you're just trying to find a single value in a database, you should always limit your result to 1, that way, if it is found in the first record, your query will stop.
Hope this helps