getting pyarrow.lib.ArrowInvalid: CSV parse error: Expected 9 columns, got 1
so I am trying apache arrow for the first time and want to read an entire directory of txt files into a pyarrow datastructure. I am getting
pyarrow.lib.ArrowInvalid: CSV parse error: Expected 9 columns, got 1
when I run the code below? no clue how to debug this. any help appreciated. ALSO if there's a book that covers python and pyarrow happy to read it.
import pyarrow.csv as csv
import pyarrow as pa
l_all_files = ['x08.txt', 'x21.txt', 'x108.txt']
read_options = csv.ReadOptions( column_names= ('Sol' , 'H20', 'H50', 'H100', 'Date', 'Cv', 'Dys', 'Ple' , 'Cl') , skip_rows=26)
df_arrow_all = pa.concat_tables(( csv.read_csv(current_filename,read_options=read_options, ) for current_filename in l_all_files))
when the skip_rows = 26 is done this is what the line looks like
$sss 3431 3232 3249 25523 35.66 57 56le 77.73