Break text files into multiple datasets at each \newline with Pandas - pandas

Question

I have a text file, textfile.qdp:
Line to skip 1
Line to skip 2
Line to skip 3
1.25 0.649999976 2.24733017E-2 2.07460159E-3 3.01663446 1.89463757E-2 1.48296626E-2 2.98285842
2.0999999 0.199999988 7.33829737E-2 6.63989689E-3 3.48941302 3.8440533E-2 6.34965161E-3 3.44462299
2.5 0.200000048 0.118000358 8.37391801E-3 2.64556909 3.93543094E-2 6.16234308E-3 2.60005236
2.9000001 0.199999928 0.145619139 9.26280301E-3 2.56852388 4.85827066E-2 6.0398886E-3 2.51390147
3.29999995 0.200000048 0.167878062 9.94068757E-3 2.46484375 5.69529012E-2 6.81256084E-3 2.40107822
3.70000005 0.200000048 0.175842062 1.01562217E-2 2.28405786 6.24930188E-2 8.10874719E-3 2.21345592
4.10000038 0.200000048 0.181325018 1.03028165E-2 2.02467489 6.38177395E-2 1.2183371E-2 1.94867384
4.5 0.199999809 0.157546207 9.59824398E-3 1.76375055 6.11177757E-2 6.072836E-2 1.64190447
4.94999981 0.25 0.156071633 8.54758453E-3 1.51925421 5.52904457E-2 0.149736568 1.3142271
5.5 0.300000191 0.125403479 6.9860979E-3 1.52551162 4.61589135E-2 0.511757791 0.967594922
6.10000038 0.299999952 9.54503566E-2 6.10219687E-3 3.56054449 3.59460302E-2 2.85172343 0.672874987
6.86499977 0.464999914 5.7642214E-2 3.80936684E-3 4.10104704 2.42055673E-2 3.67026114 0.406580269
8.28999996 0.960000038 2.10143197E-2 1.60136714E-3 0.142320022 8.9181494E-3 6.96837786E-4 0.132705033
9.48999977 0.239999771 5.72929019E-3 1.6677354E-3 3.82030606E-2 2.56266794E-3 4.94769251E-4 3.51456255E-2
4.13999987 1.99999809E-2 2.47749758 4.67826687E-2 30.4350224 0.973279834 0.754008532 28.7077332
4.17999983 1.99999809E-2 2.44065595 4.64052781E-2 30.5456734 0.99132967 0.677066088 28.8772774
4.21999979 1.99999809E-2 2.4736743 4.67251018E-2 30.8877811 1.01084304 0.807663918 29.0692749
4.26000023 2.00002193E-2 2.48481822 4.68727946E-2 30.9508438 1.02374947 0.834929705 29.092165
4.30000019 1.99999809E-2 2.54010344 4.73690033E-2 31.119503 1.03903878 0.93061626 29.1498489
4.34000015 1.99999809E-2 2.49571872 4.69326451E-2 31.1599998 1.05370748 0.892735004 29.2135563
4.38000011 1.99999809E-2 2.58409572 4.77907397E-2 31.367794 1.06788957 1.05168498 29.2482204
4.42000008 1.99999809E-2 2.6437602 4.83172201E-2 31.5764256 1.08456028 1.1402396 29.3516254
4.46000004 1.99999809E-2 2.65394902 4.84031737E-2 31.5579567 1.09554553 1.1519351 29.3104763
4.5 1.99999809E-2 2.62269425 4.81106751E-2 31.644083 1.11161876 1.12954116 29.4029236
Each column is a different parameter value, and the text file includes several datasets that, in the end, I want to plot with, e.g., different colors. A new line in the text file marks a different set.
import pandas as pd
import matplotlib as plt
names = ['e', 'de', 'y', 'y_err', 'total', 'model1', 'model2', 'model3']
for j in range(8):
names.append('model%i' %j)
df = pd.read_table('textfile.qdp', skiprows=3, names=names, delimiter=' ', skip_blank_lines=True)
fig, ax = plt.subplots(figsize=(10, 6))
ax.errorbar(df.e, df.y, xerr=df.de, yerr=df.y_err, fmt='o', label='data') # Here I want to plot different dfs
How do I do that with Pandas? I think it is related to this question where:
dfs = {
k: pd.read_csv(pd.io.common.StringIO('\n'.join(dat)), delim_whitespace=True)
for k, *dat in map(str.splitlines, open('my.csv').read().split('\n\n'))
}
but I am not sure how this translates with read_table (also *dat returns Invalid syntax).

Answer 1

You could do it this way:
First the basics (note that you should call matplotlib.pyplot and not matplotlib to access the .subplots function at the end):
import pandas as pd
import matplotlib.pyplot as plt
import io
names = ['e', 'de', 'y', 'y_err', 'total']
for j in range(8):
names.append('model%i' %j)
Keep your arguments of the read_table stored:
kwargs = {
"names":names,
"delimiter":' ',
"skip_blank_lines":True
}
Read and store the content of your files, skipping the rows not needed:
skiprows=3
with open('textfile.qdp') as f:
collect_lines_as_list = f.readlines()
selected_lines = collect_lines_as_list[skiprows:]
content = "".join(selected_lines) # Join all lines to get one string only
Then split the content on blank lines (that is, two successive line return) and store it in a temp StringIO to recreate an object that Pandas will manage. Using dict comprehension (quite the same as the other answer you linked), you can collect all data in one shot:
dfs = {
k: pd.read_table(io.StringIO(data), **kwargs)
for k, data
in enumerate(content.split('\n\n'))
}
The plot your dataframes the way you want (by iterating through your dict values):
fig, axs = plt.subplots(len(dfs), figsize=(10, 6))
for ax, df in zip(list(axs), dfs.values()):
ax.errorbar(df.e, df.y, xerr=df.de, yerr=df.y_err, fmt='o', label='data')
plt.show()
You will get something like this:
If you want to plot all data in the same subplot, proceed like this:
fig, ax = plt.subplots(figsize=(10, 6))
for df in dfs.values():
ax.errorbar(df.e, df.y, xerr=df.de, yerr=df.y_err, fmt='o', label='data')
plt.show()]