srim.export

Export SRIM results to Parquet, HDF5 and CSV, with metadata.

The public entry points are srim.output.Results.to_parquet(), to_hdf5(), to_csv() and srim.output.Collision.to_parquet() / to_csv(); this module holds the implementation.

Metadata

Every export carries a JSON document (Parquet: schema key-value metadata under the key pysrim2; HDF5: the node attribute pysrim2; CSV: one metadata.yaml for all tables) with the keys pysrim2_version, exported_utc (ISO 8601, UTC), table, metadata (the .metadata dict of the output, with source reduced to the file name so that no local paths are exported) and spec (the run specification if one is passed, else null).

The collision table can have millions of rows, so it is only exported when asked for (include=["collision"]) and is always written in chunks of chunk_rows events streamed from srim.output.Collision.iter_events(): memory stays proportional to one chunk, not to the file. Its columns are fixed (the same in every chunk and file): integer ion, strings kind and atom, and 64-bit floats for everything else; fields that do not apply to an event kind (see iter_events()) are NaN (floats) and, in Parquet, null.

srim.export.collision_to_csv(collision, path, chunk_rows=100000)

Stream the events of collision into one CSV file at path; returns the row count

srim.export.collision_to_parquet(collision, path, chunk_rows=100000, metadata=None, spec=None, name='collision')

Stream the events of collision into one Parquet file at path

One row group per chunk. Returns the number of rows written.

srim.export.write_parquet(df, path, name, metadata=None, spec=None)

Write df to path with the pysrim2 metadata document in the schema