I have three Dataframes : df1, df2, df3 with the same number of "rows" but different number of "columns" and different "column" labels. I want to "merge" them in one single dataframe with the order df1,df2,df3 and keeping the original column labels.
I've read in Combine a list of pandas dataframes to one pandas dataframe that this can be done by:
df = pd.DataFrame.from_dict(map(dict,df_list))
But I cannot fully understand the code. I assume df_list is:
df_list = [df1,df2,df3]
But what is dict? A dictionary of df_list? How to get it?
I solve this by:
df = pd.concat([df1, df2], axis=1, sort=False)
df = pd.concat([df, df3], axis=1, sort=False)
Related
I need to union 3 dataframes df1, df2, df3, how can I:keep all columns in all 3 dataframes without overlab?
3 dataframes are for 3 different kind of products, one dataframe has less columns than the other two.
step1 = pd.merge_ordered(df1, df2)
all_lob = pd.merge_ordered(step1, df3)
The result seems eliminated some columns, how can i just stack 3 dataframes all together?
Thank you.
I'm not sure what you want to do, but when we are talking about putting the dataframes together on the column-level, you can use the pandas concat method and correspondingly specify the axis:
import pandas as pd
df_1 = pd.DataFrame({'Col_1':[1,2,3], 'Col_2':[4,5,6]})
df_2 = pd.DataFrame({'Col_1':[1,2,3,4], 'Col_2':[4,5,6,7]})
df_3 = pd.DataFrame({'Col_1':[1,2,3,4,5], 'Col_2':[4,5,6,7,8]})
df = pd.concat([df_1,df_2,df_3],axis=1)
print(df)
I want to keep the columns of df if its column name matches the index of df2.
My code below only returns the df.index but I want to return the entire subset of pandas dataframe.
import pandas as pd
df = df[df.columns.intersection(df2.index)]
From my understanding, you want to have datas from both dataframes matching with index of df2. Correct?
You can use Merge to join the dataframes.
df = pd.merge(df1, df2, how='inner', on=[df2.index])
I have 2 DataFrames. DF1 and DF2.
(Please note that DF2 has more entries than DF1)
What I want to do is add the nationality column to DF1 (The script should look up the name in DF1 and find the corresponding nationality in DF2).
I am currently using the below code
final_DF =df1.merge(df2[['PPS','Nationality']], on=['PPS'], how ='left')
Although the nationality column is being added to DF1 the code duplicating entries and also adding additional data from DF2 that I do not want.
Is there a method to get the nationality from DF2 while only keeping the DF1 data?
Thanks
DF1
DF2
OUPUT
2 points, you need to do.
If there is any duplicated in the DF2
You can define 'how' in the merge statement. so it will look like
final_DF = DF1.merge(DF2, on=['Name'], how = 'left')
since you want to keep only to DF1 rows, 'left' should be the ideal option for you.
For more info refer this
is there a way to conveniently merge two data frames side by side?
both two data frames have 30 rows, they have different number of columns, say, df1 has 20 columns and df2 has 40 columns.
how can i easily get a new data frame of 30 rows and 60 columns?
df3 = pd.someSpecialMergeFunct(df1, df2)
or maybe there is some special parameter in append
df3 = pd.append(df1, df2, left_index=False, right_index=false, how='left')
ps: if possible, i hope the replicated column names could be resolved automatically.
thanks!
You can use the concat function for this (axis=1 is to concatenate as columns):
pd.concat([df1, df2], axis=1)
See the pandas docs on merging/concatenating: http://pandas.pydata.org/pandas-docs/stable/merging.html
I came across your question while I was trying to achieve something like the following:
So once I sliced my dataframes, I first ensured that their index are the same. In your case both dataframes needs to be indexed from 0 to 29. Then merged both dataframes by the index.
df1.reset_index(drop=True).merge(df2.reset_index(drop=True), left_index=True, right_index=True)
If you want to combine 2 data frames with common column name, you can do the following:
df_concat = pd.merge(df1, df2, on='common_column_name', how='outer')
I found that the other answers didn't cut it for me when coming in from Google.
What I did instead was to set the new columns in place in the original df.
# list(df2) gives you the column names of df2
# you then use these as the column names for df
df[list(df2)] = df2
There is way, you can do it via a Pipeline.
** Use a pipeline to transform your numerical Data for ex-
Num_pipeline = Pipeline
([("select_numeric", DataFrameSelector([columns with numerical value])),
("imputer", SimpleImputer(strategy="median")),
])
**And for categorical data
cat_pipeline = Pipeline([
("select_cat", DataFrameSelector([columns with categorical data])),
("cat_encoder", OneHotEncoder(sparse=False)),
])
** Then use a Feature union to add these transformations together
preprocess_pipeline = FeatureUnion(transformer_list=[
("num_pipeline", num_pipeline),
("cat_pipeline", cat_pipeline),
])
Read more here - https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.FeatureUnion.html
This solution also works if df1 and df2 have different indices:
df1.loc[:, df2.columns] = df2.to_numpy()
I have two populated DataFrames, df1 and df2. I also have an empty Dataframe (test):
df1 = pd.read_excel(xlpath1, sheetname='Sheet1')
df2 = pd.read_excel(xlpath2, sheetname='Sheet1')
test = pd.DataFrame()
I'd like to iterate through the rows of df1 and add those rows to the empty test Dataframe. When I try the following, I don't get any sort of error, but nothing is added to the test DataFrame:
for i, j in df1.iterrows():
test.append(j)
Any ideas? Do I need to add the proper columns to the test DataFrame first? My total end-goal is to iterate through multiple DataFrames and add only unique items to the empty DataFrame (ex, adding items that appear in one of the many DataFrames).
If are trying to append dataframe df1 on empty dataframe df2 you can use concat function of pandas.
test = pd.concat([df1, test], axis = 0)
axis = 0 ; for appending two dataframes row-wise