Read csv with multiline text columns by dask - pandas

Question

I have to read csv which contains full text data, which can be multiline. I am able to read this csv by pure pandas (tested on version 0.25.3 and 1.0.3) without any problems but when i try to read this csv by dask i receive ParserError: Error tokenizing data. C error: EOF inside string starting at row 28 the row number depends on file which i try to read.
I prepared the artificial dataframe to reproduce this error. Can i something tune in dask parameters, preprocess input file or this is dask implementation issue?
multiplication_factor = 71 # 70 works fine, 71 fail
number_of_columns = 100
import pandas as pd
import dask.dataframe as dd
import textwrap
pandas_default_kwargs = {
'cache_dates': True,
# 'chunksize': None, # not support by dask
'comment': None,
# 'compression': 'infer', # not support by dask
'converters': None,
'date_parser': None,
'dayfirst': False,
'decimal': b'.',
'delim_whitespace': False,
'delimiter': None,
'dialect': None,
'doublequote': True,
'dtype': object,
'encoding': None,
'engine': None,
'error_bad_lines': True,
'escapechar': None,
'false_values': None,
'float_precision': None,
'header': 'infer',
# 'index_col': None, # not support by dask
'infer_datetime_format': False,
# 'iterator': False, # not support by dask
'keep_date_col': False,
'keep_default_na': True,
'lineterminator': None,
'low_memory': True,
'mangle_dupe_cols': True,
'memory_map': False,
'na_filter': True,
'na_values': None,
'names': None,
'nrows': None,
'parse_dates': False,
'prefix': None,
'quotechar': '"',
'quoting': 0,
'sep': ',',
'skip_blank_lines': True,
'skipfooter': 0,
'skipinitialspace': False,
'skiprows': None,
'squeeze': False,
'thousands': None,
'true_values': None,
'usecols': None,
'verbose': False,
'warn_bad_lines': True,
}
artificial_df_1_row = pd.DataFrame(
data=[
(
textwrap.dedent(
f"""
some_data_for
column_number_{i}
"""
)
for i
in range(number_of_columns)
)
],
columns=[f'column_name_number_{i}' for i in range(number_of_columns)]
)
path_to_single_line_csv = './single_line.csv'
path_to_multi_line_csv = './multi_line.csv'
# prepare data to save
single_line_df = artificial_df_1_row
multi_line_df = pd.concat(
[single_line_df] * multiplication_factor,
)
# save data
single_line_df.to_csv(path_to_single_line_csv, index=False)
multi_line_df.to_csv(path_to_multi_line_csv, index=False)
# read 1 row csv by dask - works
dask_single_line_df = dd.read_csv(
path_to_single_line_csv,
blocksize=None, # read as single block
**pandas_default_kwargs
)
dask_single_line_df_count = dask_single_line_df.shape[0].compute()
print('[DASK] single line count', dask_single_line_df_count)
# read multiline csv by pandas - works
pandas_multi_line_df = pd.read_csv(
path_to_multi_line_csv,
**pandas_default_kwargs
)
pandas_multi_line_df_shape_0 = pandas_multi_line_df.shape[0]
print('[PANDAS] multi line count', pandas_multi_line_df_shape_0)
# read multine csv by dask - depends on number of rows fails or not
dask_multi_line_df = dd.read_csv(
path_to_multi_line_csv,
blocksize=None, # read as single block
**pandas_default_kwargs
)
dask_multi_line_df_shape_0 = dask_multi_line_df.shape[0].compute()
print('[DASK] multi line count', dask_multi_line_df_shape_0)

Answer 1

The only way you can read such a file is to ensure that the chunk boundaries are not within a quoted string which, unless you know a lot about the data layout, means not chunking a file at all (but you can still parallelise between files).
This is because, the only way to know whether or not you are in a quoted string is to parse a file from the start, and the way dask achieves parallelism is to have each chunk-reading task completely independent, needing only a file offset. In practice, dask reads from the offset and considers the first newline marker as the point to start parsing from.