getting pyarrow.lib.ArrowInvalid: CSV parse error: Expected 9 columns, got 1
13:31 15 Sep 2020

so I am trying apache arrow for the first time and want to read an entire directory of txt files into a pyarrow datastructure. I am getting


pyarrow.lib.ArrowInvalid: CSV parse error: Expected 9 columns, got 1

when I run the code below? no clue how to debug this. any help appreciated. ALSO if there's a book that covers python and pyarrow happy to read it.


import pyarrow.csv as csv

import pyarrow as pa  


l_all_files = ['x08.txt', 'x21.txt', 'x108.txt']

read_options = csv.ReadOptions( column_names= ('Sol' , 'H20', 'H50', 'H100',  'Date', 'Cv', 'Dys', 'Ple' , 'Cl') ,  skip_rows=26)


df_arrow_all = pa.concat_tables(( csv.read_csv(current_filename,read_options=read_options, ) for current_filename in l_all_files))  

when the skip_rows = 26 is done this is what the line looks like


$sss        3431    3232    3249  25523  35.66    57  56le   77.73
csv pyarrow