I am struggling to replicate the elegant ease - and successful outcome - teasingly promised in the 'Basic Plotting:plot' section of the pandas df.plot() documentation at:
http://pandas.pydata.org/pandas-docs/stable/visualization.html#visualization
There the authors' first image is pretty close to the kind of line-graph I want to plot from my dataframe. Their first df and resulting plot is a single-liner just as I hoped my df below would look when plotted.
My df looks like this:
2014-03-28 2014-04-04 2014-04-11 2014-04-18 \
Jenny Todd 1699.6 1741.6 1710.7 1744.2
2014-04-25 2014-05-02 2014-05-09
Jenny Todd 1764.2 1789.7 1802.3
Their second image is a multi-line graph very similar to what I hoped for when I try to plot a multiple-index version of my df. Eg:
2014-06-13 2014-06-20 2014-06-27 \
William Acer 1674.7 1689.4 1682.0
Katherine Baker 1498.5 1527.3 1530.5
2014-07-04 2014-07-11 2014-07-18 \
William Acer 1700.0 1674.5 1677.8
Katherine Baker 1540.4 1522.3 1537.3
2014-07-25
William Acer 1708.0
Katherine Baker 1557.1
However, they get plots. I get featureless 3.3kb images and a warning:
/home/lee/test/local/lib/python2.7/site-packages/matplotlib/axes/_base.py:2787: UserWarning: Attempting to set identical left==right results in singular transformations; automatically expanding.
left=0.0, right=0.0
'left=%s, right=%s') % (left, right))
The authors of the documentation seem to have the plot() function deducing from the df's indexes the values of the x-axis and the range and values of the y axis.
Searching around, I can find people with different data, different indexes and different scenarios (for example, plotting one column against another or trying to produce multiple subplots) who get this kind of 'axes' error. However, I haven't been able to map their issues to mine.
I wonder if anyone can help resolve what is different about my data or code that leads to a different plot outcome from the documentation's seemingly-similar data and seemingly-similar code.
My code:
print plotting_df # (This produces the df examples I pasted above)
plottest = plotting_df.plot.line(title='Calorie Intake', legend=True)
plottest.set_xlabel('Weeks')
plottest.set_ylabel('Calories')
fig = plt.figure()
plot_name = week_ending + '_' + collection_name + '.png'
fig.savefig(plot_name)
Note this dataframe is being created dynamically many times within the script. On any given run, the script will acquire different sets of dates, differently-named people, and different numbers to plot. So I don't have predictability about what strings will come up for index and legend labels for plotting beforehand. I do have predictability about the format.
I get that my dataframe's date index has differently-formatted dates than the referred documentation describes. Is this the cause? Whether it is or isn't, how should one best handle this issue?
Added on 2016-08-24 to answer the comment below about being unable to recreate my data
plotting_df is created on the fly as a subset of a much larger dataframe. It's simply an index (or sometimes multiple indices) and some of the date columns extracted from the larger dataframe. The code that produces plotting_df works fine and always produces plotting_df with correct indices and columns in a format I expect.
I can simulate creation of a dataset to store in plotting_df with this python code:
plotting_1 = {
'2014-03-28': 1699.6,
'2014-04-04': 1741.6,
'2014-04-11': 1710.7,
'2014-04-18': 1744.2,
'2014-04-25': 1764.2,
'2014-05-02': 1789.7,
'2014-05-09': 1802.3
}
plotting_df = pd.DataFrame(plotting_1, index=['Jenny Todd'])
and I can simulate creation of a multiple-indices plotting_df with this python code:
plotting_2 = {
'Katherine Baker': {
'2014-06-13': 1498.5,
'2014-06-20': 1527.3,
'2014-06-27': 1530.5,
'2014-07-04': 1540.4,
'2014-07-11': 1522.3,
'2014-07-18': 1537.3,
'2014-07-25': 1557.1
},
'William Acer': {
'2014-06-13': 1674.7,
'2014-06-20': 1689.4,
'2014-06-27': 1682.0,
'2014-07-04': 1700.0,
'2014-07-11': 1674.5,
'2014-07-18': 1677.8,
'2014-07-25': 1708.0
}
}
plotting_df = pd.DataFrame.from_dict(plotting_2)
I did try the suggested transform with code:
plotdf = plotting_df.T
plotdf.index = pd.to_datetime(plotdf.index)
so that my original code now looks like:
print plotting_df # (This produces the df examples I pasted above)
plotdf = plotting_df.T # Transform the df - date columns to indices
plotdf.index = pd.to_datetime(plotdf.index) # Convert indices to datetime
plottest = plotdf.plot.line(title='Calorie Intake', legend=True)
plottest.set_xlabel('Weeks')
plottest.set_ylabel('Calories')
fig = plt.figure()
plot_name = week_ending + '_' + collection_name + '.png'
fig.savefig(plot_name)
but I still get the same result (blank 3.3kb images created).
I did note that adding the transform made no difference when I printed out the first instance of plotdf. So should be I doing some other transform?
This is your problem:
fig = plt.figure()
plot_name = week_ending + '_' + collection_name + '.png'
fig.savefig(plot_name)
You are creating a second figure after creating the first one and then you are saving only that second empty figure. Just take out the line fig = plt.figure() and change fig.savefig to plt.savefig
So you should have :
print plotting_df # (This produces the df examples I pasted above)
plotdf = plotting_df.T # Transform the df - date columns to indices
plotdf.index = pd.to_datetime(plotdf.index) # Convert indices to datetime
plottest = plotdf.plot.line(title='Calorie Intake', legend=True)
plottest.set_xlabel('Weeks')
plottest.set_ylabel('Calories')
plot_name = week_ending + '_' + collection_name + '.png'
plt.savefig(plot_name)
Related
I am given a data set that looks something like this
and I am trying to graph all the points with a 1 on the first column separate from the points with a 0, but I want to put them in the same chart.
I know the final result should be something similar to this
But I can't find a way to filter the points in Julia. I'm using LinearAlgebra, CSV, Plots, DataFrames for my project, and so far I haven't found a way to make DataFrames storage types work nicely with Plots functions. I keep running into errors like Cannot convert Float64 to series data for plotting when I try plotting the points individually with a for loop as a filter as shown in the code below
filter = select(data, :1)
newData = select(data, 2:3)
#graph one initial point to create the plot
plot(newData[1,1], newData[1,2], seriestype = :scatter, title = "My Scatter Plot")
#add the additional points with the 1 in front
for i in 2:size(newData)
if filter[i] == 1
plot!(newData[i, 1], newData[i, 2], seriestype = :scatter, title = "My Scatter Plot")
end
end
Other approaches have given me other errors, but I haven't recorded those.
I'm using Julia 1.4.0 and the latest versions of all of the packages mentioned.
Quick Edit:
It might help to know that I am trying to replicate the Nonlinear dimensionality reduction section of this article https://sebastianraschka.com/Articles/2014_kernel_pca.html#principal-component-analysis
With Plots.jl you can do the following (I am passing a fully reproducible code):
julia> df = DataFrame(c=rand(Bool, 100), x = 2 .* rand(100) .- 1);
julia> df.y = ifelse.(df.c, 1, -1) .* df.x .^ 2;
julia> plot(df.x, df.y, color=ifelse.(df.c, "blue", "red"), seriestype=:scatter, legend=nothing)
However, in this case I would additionally use StatsPlots.jl as then you can just write:
julia> using StatsPlots
julia> #df df plot(:x, :y, group=:c, seriestype=:scatter, legend=nothing)
If you want to do it manually by groups it is easiest to use the groupby function:
julia> gdf = groupby(df, :c);
julia> summary(gdf) # check that we have 2 groups in data
"GroupedDataFrame with 2 groups based on key: c"
julia> plot(gdf[1].x, gdf[1].y, seriestype=:scatter, legend=nothing)
julia> plot!(gdf[2].x, gdf[2].y, seriestype=:scatter)
Note that gdf variable is bound to a GroupedDataFrame object from which you can get groups defined by the grouping column (:c) in this case.
I have monthly data of 6 variables from 2014 until 2018 in one dataset.
I'm trying to draw 6 subplots (one for each variable) with monthly X axis (Jan, Feb....) and 5 series (one for each year) with their legend.
This is part of the data:
I created 5 series (one for each year) per variable (30 in total) and I'm getting the expected output but using MANY lines of code.
What is the best way to achieve this using less lines of code?
This is an example how I created the series:
CL2014 = data_total['Charity Lottery'].where(data_total['Date'].dt.year == 2014)[0:12]
CL2015 = data_total['Charity Lottery'].where(data_total['Date'].dt.year == 2015)[12:24]
This is an example of how I'm plotting the series:
axCL.plot(xvals, CL2014)
axCL.plot(xvals, CL2015)
axCL.plot(xvals, CL2016)
axCL.plot(xvals, CL2017)
axCL.plot(xvals, CL2018)
There's no need to litter your namespace with 30 variables. Seaborn makes the job very easy but you need to normalize your dataframe first. This is what "normalized" or "unpivoted" looks like (Seaborn calls this "long form"):
Date variable value
2014-01-01 Charity Lottery ...
2014-01-01 Racecourse ...
2014-04-01 Bingo Halls ...
2014-04-01 Casino ...
Your screenshot is a "pivoted" or "wide form" dataframe.
df_plot = pd.melt(df, id_vars='Date')
df_plot['Year'] = df_plot['Date'].dt.year
df_plot['Month'] = df_plot['Date'].dt.strftime('%b')
import seaborn as sns
plot = sns.catplot(data=df_plot, x='Month', y='value',
row='Year', col='variable', kind='bar',
sharex=False)
plot.savefig('figure.png', dpi=300)
Result (all numbers are randomly generated):
I would try using .groupby(), it is really powerful for parsing down things like this:
for _, group in data_total.groupby([year, month])[[x_variable, y_variable]]:
plt.plot(group[x_variables], group[y_variables])
So here the groupby will separate your data_total DataFrame into year/month subsets, with the [[]] on the end to parse it down to the x_variable (assuming it is in your data_total DataFrame) and your y_variable, which you can make any of those features you are interested in.
I would decompose your datetime column into separate year and month columns, then use those new columns inside that groupby as the [year, month]. You might be able to pass in the dt.year and dt.month like you had before... not sure, try it both ways!
I'm having trouble understanding Pandas subplots - and how to create axes so that all subplots are shown (not over-written by subsequent plot).
For each "Site", I want to make a time-series plot of all columns in the dataframe.
The "Sites" here are 'shark' and 'unicorn', both with 2 variables. The output should be be 4 plotted lines - the time-indexed plot for Var 1 and Var2 at each site.
Make Time-Indexed Data with Nans:
df = pd.DataFrame({
# some ways to create random data
'Var1':pd.np.random.randn(100),
'Var2':pd.np.random.randn(100),
'Site':pd.np.random.choice( ['unicorn','shark'], 100),
# a date range and set of random dates
'Date':pd.date_range('1/1/2011', periods=100, freq='D'),
# 'f':pd.np.random.choice( pd.date_range('1/1/2011', periods=365,
# freq='D'), 100, replace=False)
})
df.set_index('Date', inplace=True)
df['Var2']=df.Var2.cumsum()
df.loc['2011-01-31' :'2011-04-01', 'Var1']=pd.np.nan
Make a figure with a sub-plot for each site:
fig, ax = plt.subplots(len(df.Site.unique()), 1)
counter=0
for site in df.Site.unique():
print(site)
sitedat=df[df.Site==site]
sitedat.plot(subplots=True, ax=ax[counter], sharex=True)
ax[0].title=site #Set title of the plot to the name of the site
counter=counter+1
plt.show()
However, this is not working as written. The second sub-plot ends up overwriting the first. In my actual use case, I have 14 variable number of sites in each dataframe, as well as a variable number of 'Var1, 2, ...'. Thus, I need a solution that does not require creating each axis (ax0, ax1, ...) by hand.
As a bonus, I would love a title of each 'site' above that set of plots.
The current code over-writes the first 'Site' plot with the second. What I missing with the axes here?!
When you are using DataFrame.plot(..., subplot=True) you need to provide the correct number of axes that will be used for each column (and with the right geometry, if using layout=). In your example, you have 2 columns, so plot() needs two axes, but you are only passing one in ax=, therefore pandas has no choice but to delete all the axes and create the appropriate number of axes itself.
Therefore, you need to pass an array of axes of length corresponding to the number of columns you have in your dataframe.
# the grouper function is from itertools' cookbook
from itertools import zip_longest
def grouper(iterable, n, fillvalue=None):
"Collect data into fixed-length chunks or blocks"
# grouper('ABCDEFG', 3, 'x') --> ABC DEF Gxx"
args = [iter(iterable)] * n
return zip_longest(*args, fillvalue=fillvalue)
fig, axs = plt.subplots(len(df.Site.unique())*(len(df.columns)-1),1, sharex=True)
for (site,sitedat),axList in zip(df.groupby('Site'),grouper(axs,len(df.columns)-1)):
sitedat.plot(subplots=True, ax=axList)
axList[0].set_title(site)
plt.tight_layout()
Consider a simple 2x2 dataset with with Series labels prepended as the first column ("Repo")
Repo AllTests Restricted
0 Galactian 1860.0 410.0
1 Forecast-MLib 140.0 47.0
Here are the DataFrame columns:
p(df.columns)
([u'Repo', u'AllTests', u'Restricted']
So we have the first column is the string/label and the second and third columns are data values. We want one series per row corresponding to the Galactian and the Forecast-MLlib repos.
It would seem this would be a common task and there would be a straightforward way to simply plot the DataFrame . However the following related question does not provide any simple way: it essentially throws away the DataFrame structural knowledge and plots manually:
Set matplotlib plot axis to be the dataframe column name
So is there a more natural way to plot these Series - that does not involve deconstructing the already-useful DataFrame but instead infers the first column as labels and the remaining as series data points?
Update Here is a self contained snippet
runtimes = npa([1860.,410.,140.,47.])
runtimes.shape = (2,2)
labels = npa(['Galactian','Forecast-MLlib'])
labels.shape=(2,1)
rtlabels = np.concatenate((labels,runtimes),axis=1)
rtlabels.shape = (2,3)
colnames = ['Repo','AllTests','Restricted']
df = pd.DataFrame(rtlabels, columns=colnames)
ps(df)
df.set_index('Repo').astype(float).plot()
plt.show()
And here is output
Repo AllTests Restricted
0 Galactian 1860.0 410.0
1 Forecast-MLlib 140.0 47.0
And with piRSquared help it looks like this
So the data is showing now .. but the Series and Labels are swapped. Will look further to try to line them up properly.
Another update
By flipping the columns/labels the series are coming out as desired.
The change was to :
labels = npa(['AllTests','Restricted'])
..
colnames = ['Repo','Galactian','Forecast-MLlib']
So the updated code is
runtimes = npa([1860.,410.,140.,47.])
runtimes.shape = (2,2)
labels = npa(['AllTests','Restricted'])
labels.shape=(2,1)
rtlabels = np.concatenate((labels,runtimes),axis=1)
rtlabels.shape = (2,3)
colnames = ['Repo','Galactian','Forecast-MLlib']
df = pd.DataFrame(rtlabels, columns=colnames)
ps(df)
df.set_index('Repo').astype(float).plot()
plt.title("Restricting Long-Running Tests\nin Galactus and Forecast-ML")
plt.show()
p('df columns', df.columns)
ps(df)
Pandas assumes your label information is in the index and columns. Set the index first:
df.set_index('Repo').astype(float).plot()
Or
df.set_index('Repo').T.astype(float).plot()
I have a data frame with perfectly organised timestamps, like below:
It's a web log, and the timestamps go though the whole year. I want to cut them into each day and show the visits within each hour and plot them into the same figure and stack them all together. Just like the pic shown below:
I am doing well on cutting them into days and plot the visits of a day individually, but I am having trouble plotting them and stacking them together. The primary tool I am using is Pandas and Matplotlib.
Any advices and suggestions? Much Appreciated!
Edited:
My Code is as below:
The timestamps are: https://gist.github.com/adamleo/04e4147cc6614820466f7bc05e088ac5
And the dataframe looks like this:
I plotted the timestamp density through the whole period used the code below:
timestamps_series_all = pd.DatetimeIndex(pd.Series(unique_visitors_df.time_stamp))
timestamps_series_all_toBePlotted = pd.Series(1, index=timestamps_series_all)
timestamps_series_all_toBePlotted.resample('D').sum().plot()
and got the result:
I plotted timestamps within one day using the code:
timestamps_series_oneDay = pd.DatetimeIndex(pd.Series(unique_visitors_df.time_stamp.loc[unique_visitors_df["date"] == "2014-08-01"]))
timestamps_series_oneDay_toBePlotted = pd.Series(1, index=timestamps_series_oneDay)
timestamps_series_oneDay_toBePlotted.resample('H').sum().plot()
and the result:
And now I am stuck.
I'd really appreciate all of your help!
I think you need pivot:
#https://gist.github.com/adamleo/04e4147cc6614820466f7bc05e088ac5 to L
df = pd.DataFrame({'date':L})
print (df.head())
date
0 2014-08-01 00:05:46
1 2014-08-01 00:14:47
2 2014-08-01 00:16:05
3 2014-08-01 00:20:46
4 2014-08-01 00:23:22
#convert to datetime if necessary
df['date'] = pd.to_datetime(df['date'] )
#resample by Hours, get count and create df
df = df.resample('H', on='date').size().to_frame('count')
#extract date and hour
df['days'] = df.index.date
df['hours'] = df.index.hour
#pivot and plot
#maybe check parameter kind='density' from http://stackoverflow.com/a/33474410/2901002
#df.pivot(index='days', columns='hours', values='count').plot(rot='90')
#edit: last line change to below:
df.pivot(index='hours', columns='days', values='count').plot(rot='90')