Create dataframe specific lists in a function - pandas

I have several datasets. I would like to create a list for each one. Is there a way to do this in some kind of function? As of now I write it one below the other, since the code is quite long I don't want to do it manually for each dataset. I want to create a datasource list for each dataset, which contains the information of the respective dataset. As of now, my code looks like this:
datasource = df['Data_source'].tolist()
datasource1 = df1['Data_source'].tolist()
datasource2 = df2['Data_source'].tolist()
datasource3 = df3['Data_source'].tolist()
...
Thank you!

Related

How to send a table names as parameters to a function which performs a join on them?

Currently I used the following code for joining tables.
Booking.joins(:table1, :table2, :table3, :table4).other_queries
However, the number of tables to be joined with depends on certain conditions. The other_queries also form a very large chain. So, I am duplicating a lot of code just because I need to perform joins differently.
So, I want to implement something like this
def method(params)
Booking.joins(params).other_queries
end
How can this be done?
Maybe just Booking.joins(*params).other_queries is what you need?
Operator * transforms array into list of params, for example:
arr = [1,2,3]
any_method(*arr) # is equal to any_method(1,2,3)
However, if params is smth came from user I recommend you not to trust it, it probably could be security issue. But if you trust it or filter it - why not.
SAFE_JOINS = [:table1, :table2, :table3]
def method(params)
booking = Booking.scoped # or Booking.all if you are rails 5
(params[:joins] & SAFE_JOINS.map(&:to_s)).each do |j|
booking = booking.joins(j.intern)
end
end

Search for a Dataset where a associated Dataset has a specific value using Active-Record?

i am currently trying to do a "complex" search inside my Datasets.
I want to find every Dataset that is associated with a specific tool.
I tried something like this:
Dataset.find_by_sql("SELECT * FROM datasets, tools, sectors, products WHERE datasets.id = sectors.dataset_id AND sectors.product_id = products.id AND products.tool_id = tools.id AND tools.name = '#{toolname}'")
This works well. But we want to define scopes for querys, for being able to combine some of them later like: Dataset.find_by_toolname("foo").find_by_sector_name("foo2"). So i defined a scope for this:
scope :with_tool_name, ->(toolname) {find_by_sql("SELECT * FROM datasets, tools, sectors, products WHERE datasets.id = sectors.dataset_id AND sectors.product_id = products.id AND products.tool_id = tools.id AND tools.name = '#{toolname}'")}
The Problem here is that this query will return an Array and not an ActiveRecord-Relation so that i a am not able to do another scope-request on the result of the first.
So how can i fetch for Data where an attribute of some associated model has a specific value .. and then apply another scope on this ?
Here is what the Data-Model looks like:
You can use the activerecord-method joins(). And the result of every join can be mixed together with the method merge(). Here a small example to get all datasets for one tool_id.
Dataset.joins(:sectors).merge(Sector.joins(:product)).merge(Product.joins(:tool).where(:tool_id => 1))

SRSS Report Builder SUM IIF from different datasets

I have two datasets in Report Builder 3.0 with a similar field and I want to put a SUM of the number occurrences of a particular value in that common field across both datasets.
I've got an expression I'm using for each individual dataset:
=SUM(IIF(Fields!caseorigin.Value = "mail",1,0))
and
=SUM(IIF(Fields!cliorigin.Value = "mail",1,0))
But I can't seem to work out a way to sum the values from both datasets. I've tried:
=SUM(IIF((Fields!caseorigin.Value, "caseDS") = "mail",1,0)) + SUM(IIF((Fields!cliorigin.Value, "cliDS") = "mail",1,0))
Is there any way to make this work, or an alternative method?
Just looks like a syntax error here; when specifying a scope it should like something like:
=Sum(Expression, Scope)
Applying this to your example:
=SUM(IIF(Fields!caseorigin.Value = "mail",1,0), "caseDS")
+ SUM(IIF(Fields!cliorigin.Value = "mail",1,0), "cliDS")
Should work for you.

How to create secondary index in Cassandra Hector API programmatically

I have been trying to create indexing using below set of lines.
KeyspaceDefinition fromCluster = cluster.describeKeyspace(KEYSPACE);
ColumnFamilyDefinition cfDef = fromCluster.getCfDefs().get(0);
BasicColumnFamilyDefinition columnFamilyDefinition = newBasicColumnFamilyDefinition(cfDef);
BasicColumnDefinition columnDefinition = new BasicColumnDefinition();
columnDefinition.setName(StringSerializer.get().toByteBuffer("A_NO"));
columnDefinition.setIndexName("A_NO_idx");
columnDefinition.setIndexType(ColumnIndexType.KEYS);
columnDefinition.setValidationClass(ComparatorType.UTF8TYPE.getClassName());
columnFamilyDefinition.addColumnDefinition(columnDefinition);
But i am unable to do so. Actually i am storing the data in the columns dynamically as well as creating those columns dynamically and after that for better query purpose i am trying to put index on some particular columns. Any suggestion please how to do that.
Its eventually quite simple. You just have to create the secondary index while defining your columnfamily. In the above code, all the manipulation are done on the object index which has to be created while defining only. The steps for adding index are
List<ColumnDef> columns = new ArrayList<ColumnDef>();
columns.add(newIndexedColumnDef("columnName", "UTF8Type"));
List<ColumnDefinition> columnMetadata = ThriftColumnDef
.fromThriftList(columns);
cdefs.add(cf_def); //cf_def is your columnfamily definition
The helper method code is from KeyspaceCreationTest
public ColumnDef newIndexedColumnDef(String column_name, String comparer){
ColumnDef cd = new ColumnDef(se.toByteBuffer(column_name), comparer);
cd.setIndex_name(column_name);
cd.setIndex_type(IndexType.KEYS);
return cd;
}
References for comparer can be found here
I hope it will help you.

How do I use Linq-to-sql to iterate db records?

I asked on SO a few days ago what was the simplest quickest way to build a wrapper around a recently completed database. I took the advice and used sqlmetal to build linq classes around my database design.
Now I am having two problems. One, I don't know LINQ. And, two, I have been shocked to realize how hard it is to learn. I have a book on LINQ (Linq In Action by Manning) and it has helped some but at the end of the day it is going to take me a couple of weeks to get traction and I need to make some progress on my project today.
So, I am looking for some help getting started.
Click HERE To see my simple database schema.
Click HERE to see the vb class that was generated for the schema.
My needs are simple. I have a console app. The main table is the SupplyModel table. Most of the other tables are child tables of the SupplyModel table.
I want to iterate through each of Supply Model records. I want to grab the data for a supply model and then DoStuff with the data. And I also need to iterate through the child records, for each supply model, for example the NumberedInventories and DoStuff with that as well.
I need help doing this in VB rather than C# if possible. I am not looking for the whole solution...if you can supply a couple of code-snippets to get me on my way that would be great.
Thanks for your help.
EDIT
For the record I have already written the following code...
Dim _dataContext As DataContext = New DataContext(ConnectionStrings("SupplyModelDB").ConnectionString)
Dim SMs As Table(Of Data.SupplyModels) = _dataContext.GetTable(Of Data.SupplyModels)()
Dim query = From sm In SMs Where sm.SupplyModelID = 1 Select sm
This code is working...I have a query object and I can use ObjectDumper to enumerate and dump the data...but I still can't figure it out...because ObjectDumper uses reflection and other language constructs I don't get. It DOES enumerate both the parent and child data just like I want (when level=2).
PLEASE HELP...I'M stuck. Help!
Seth
in C# it would be:
var result = from s in _dataContent.SupplyModels where s.SupplyModelID==1 select s;
foreach(SupplyModel item in result)
{
// do stuff
foreach(SupplyModelChild child in item.SupplyModelChilds)
{
//do more stuff on the child
}
}
and a VB.NET version (from the Telerik code converter)
Dim result As var = From s In _dataContent.SupplyModels _
Where s.SupplyModelID = 1 _
Select s
For Each item As SupplyModel In result
' do stuff
'do more stuff on the child
For Each child As SupplyModelChild In item.SupplyModelChilds
Next
Next