Thanks @Will Jones. Will try that and report back.

I think part of the trick is specifying the schema up front so the reader doesn't have to try inferring it.

Can't you "just" read that csv into a spark dataframe and than save that dataframe as a parquet file?

@Kees Duvekot - yea, but that makes multiple Parquet files. I am looking for a single Parquet file in this case.

Can't you do a coalesc before?

@Will Jones - this worked to convert a 50 GB CSV => a single Parquet file BTW:

writer = None
with pyarrow.csv.open_csv(in_path) as reader:
    for next_chunk in reader:
        if next_chunk is None:
            break
        if writer is None:
            writer = pq.ParquetWriter(out_path, next_chunk.schema)
        next_table = pa.Table.from_batches([next_chunk])
        writer.write_table(next_table)
writer.close()

Thanks for pointing me in the right direction. The resulting Parquet file had 50,476 row groups 😲 The notebook in case you’re interested.