I am trying to remove some columns from my data frame and would prefer not to return the modified data frame and reassign it to the old. Instead, I would like the function to just modify the data frame. This is what I tried but it does not seem to be doing what I except. I was under the impression arguments as passed as reference and not by value?
function remove_cols! (df::DataFrame, cols)
df = df[setdiff(names(df), cols)];
end
df = DataFrame(x = [1:10], y = [11:20]);
remove_cols!(df, [:y]); # this does not modify the original data frame
Of course the below works but I would prefer if remove_cols! just changed the df in place
df = remove_cols!(df, [:y]);
How can I change the df in place inside my function?
Thanks!
As I understand Julia it uses what is called pass by sharing, meaning that the reference is passed by value. So when you pass the DataFrame to the function a new reference to the DataFrame is created which is local to the function. When you reassign the local df variable with its own reference to the DataFrame it has no effect on the separate global variable and its separate reference to the DataFrame.
There is a function in DataFrames.jl for deleting columns in DataFrames.
To answer the question of how to mutate a dataframe in your own function in general, the key is to use functions and operations that mutate the dataframe within the function. For example, see the function below which builds upon the standard dataframe append! function with some added benefits like it can append from any number of dataframes, the order of columns does not matter and missing columns will be added to the dataframes:
function append_with_missing!(df1::DataFrame, dfs::AbstractDataFrame...)
columns = Dict{Symbol, Type}(zip(names(df1), colwise(eltype, df1)))
for df in dfs
columns_temp = Dict(zip(names(df), colwise(eltype, df)))
merge!(columns, columns_temp)
end
for (n, t) in columns, df in [df1; [i for i in dfs]]
n in names(df) || (df[n] = Vector{Union{Missing,t}}(missing, size(df, 1)))
end
for df in dfs
append!(df1, df[names(df1)])
end
end
Here, the first dataframe passed itself is mutated with rows added from the other dataframes.
(The functionality for adding missing columns is based upon the answer given by #Bogumił Kamiński here: Breaking change on vcat when columns are missing)
Related
I read in the excel file like so:
data = sc.read_excel('/Users/user/Desktop/CSVB.xlsx',sheet= 'Sheet1', dtype= object)
There are 3 columns in this data set that I need to work with as .obs but it looks like everything is in the .X data matrix.
Anyone successfully subset after reading in the file or is there something I need to do beforehand?
Okay, so assuming sc stands for scanpy package, the read_excel just takes the first row as .var and the first column as .obs of the AnnData object.
The data returned by read_excel can be tweaked a bit to get what you want.
Let's say the index of the three columns you want in the .obs are stored in idx variable.
idx = [1,2,4]
Now, .obs is just a Pandas DataFrame, and data.X is just a Numpy matrix (see here). Thus, the job is simple.
# assign some names to the new columns
new_col_names = ['C1', 'C2', 'C3']
# add the columns to data.obs
data.obs[new_col_names] = data.X[:,idx]
If you may wish to remove the idx columns from data.X, I suggest making a new AnnData object for this.
I am conducting an analysis of a dataset. To find my results, I use this line of code:
new_df = df_ncis.groupby(['state', 'year'])['totals'].mean()
The object returned by this statement is a Series, when it should be a dataframe. I don't understand why this happened, or how to solve this issue. Also, one of the columns of the new object is missing its name. Here is the github link for the project: https://github.com/louishrm/gundataUS.
Any help would be great.
You are filtering the result by ['totals'] which is a series.
try this instead
new_df = df_ncis[['state', 'year', 'totals']].groupby(['state', 'year']).mean()
which will give you a dataframe with your 3 columns.
or if you want it as a dataframe of one column (Note the double brackets)
new_df = df_ncis.groupby(['state', 'year'])[['totals']].mean()
when print a pandas dataframe, how to print the first n row by default?
I find myself frequently doing df.head(10) to view the column names and first a couple of rows.
I prefer when I type 'df', it prints the first n row by default, instead of printing the whole df, which in this case I cannot see the column names.
If I understand you correctly, you may set
pd.options.display.max_rows = 10
and whenever you do just df in your notebook, only 10 rows would be displayed.
You can always set back to the default value doing
pd.reset_option('display.max_rows')
Check pd.describe_option('display') for more information
Curry DataFrame.head using functools.partial.
from functools import partial
head10 = partial(pd.DataFrame.head, n=10)
Now you can either call the function passing your DataFrame as an argument,
head10(df)
Or, pass the function to df.pipe (which internally passes df as an argument to your function),
df.pipe(head10)
To get the first 10 rows by default.
The other option is to create a new class that extends DataFrame and add your own function (e.g., headXX) which internally calls df.head(n=10) and returns the result.
See the subclassing DataFrame section in the docs.
Working with a dataframe df I wanted to create a new column A and assign it to a single value (a string in my case)
df['A'] = value
gave a warning and suggested to use loc
however the solution below still gave the same warning:
df.loc[:,'A']=value
Doing some research I found the solution below which does not generate a warning:
df=df.assign(A =value)
Is it the general accepted way of creating a new column and assigning it to a value? Are there other possibilities using loc?
pandas version '0.20.1'
EDIT: this is the warning message obtained for the 2 first methods
"A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead"
As explained by #EdChum and #ScottBoston
Since df was derived using a mask on some original dataframe
df = df_original[boolean_mask]
to avoid the warning with the two first methods, use instead df=df_original[boolean_mask].copy()
df.assign does not need this because it automatically creates a copy of the original dataframe
Now I have written a parser to extract the information from raw html source code, which could return them as a tuple, and I have to loop this function and use the return to construct a DataFrame (each loop's return as a row). Here's what I have done:
import pandas as pd
import leveldb
for key, value in db.RangeIter():
html = db.Get(key)
result = parser(html)
df = df.append(pd.Series(result, index = index), ignore_index = True)
Note that parser and index are already defined, and db is a leveldb object which store all links and corresponding html source code. My problem is what's the more efficient way to construct that DataFrame? THANKS!
I would create a dataframe before the loop starts, then append successive dataframes to that. Note that if result is a tuple, it needs to be converted to a list before being converted into a dataframe. And I assume your index is already a list. So:
df = pd.DataFrame()
for key, value in db.RangeIter():
html = db.Get(key)
result = parser(html)
df = df.append(pd.DataFrame(list(result), index = index).transpose())
df.reset_index(inplace = True)
This is not to say your parser could not more efficiently return data for the creation of a dataframe, but I'm working within the confines of a single returned tuple.
Also, depending on the number of elements in the tuple, it may be more efficient to create simple python lists within the loop then create dataframes from those lists when complete, but you don't state the tuple size.