M2C

Anthony C. · Computer Engineering

From COBOL-described binary records to typed Arrow and Parquet data.

Hi, I'm Anthony. I built M2C Binary Engine in Rust to turn what I was learning about binary formats, data representation, columnar storage, failure recovery, reproducible benchmarking, and post-quantum cryptography into one end-to-end project.

Legacy systems still exchange fixed-record files whose layout lives in COBOL Copybooks. Modern analytical tools expect typed, self-describing data. M2C explores the bridge between those representations.

Try the pipeline demo View source

Educational systems/data-engineering prototype — not production software.

Why I built this

Learning systems by following the bytes

I wanted one project where a physical byte layout, a typed in-memory model, a durable file format, and failure behavior all had to agree.

That meant working through CP037 EBCDIC, the supported DISPLAY, COMP, and COMP-3 representations, Arrow schemas, Parquet row groups, bounded batches, filesystem publication, explicit resume, benchmark design, independent validation, and optional artifact protection—without broadening the documented v0.1 subset.

  • Rust
  • Binary parsing
  • EBCDIC
  • DISPLAY · COMP · COMP-3
  • Apache Arrow
  • Apache Parquet
  • Recovery
  • Benchmark design
  • Post-quantum cryptography
Try the pipeline

A small, real transformation

This reference run uses a repository-owned 105-byte fixture. The output was produced by the actual Rust CLI and packaged here so the portfolio stays useful without pretending a static page is live execution.

Verified fixture-backed run

tests/fixtures/sample_fixed.{cpy,bin}

Inspect fixture source

Input

3 fixed records · 35 bytes each

1000100 01 SAMPLE-RECORD.
2000200 05 HEADER-GROUP.
3000300* THIS FIXED-FORMAT COMMENT MUST BE IGNORED.
4000400 10 CUSTOMER-NAME PIC X(10).
5000500 10 FILLER PIC X(2).
6000600 10 ACCOUNT-NUMBER PIC 9(4) DISPLAY.
7000700 10 INTEREST-RATE PIC 9(5)V9(2) DISPLAY.
8000800 10 BALANCE-BIN PIC S9(4) COMP.
9000900 10 RATE-BIN PIC 9(5)V9(2) BINARY.
10001000 10 AMOUNT-PACKED PIC S9(7)V9(2) COMP-3.
11001100 10 FILLER PIC X.

Pipeline

  1. 1Compile Copybook
  2. 2Decode fixed records
  3. 3Build Arrow batches
  4. 4Write Parquet

Typed result

Arrow schema → Parquet file

Load the checked-in reference output to inspect its schema, rows, and Parquet metadata.

Exact CLI command
cargo run --release -- convert --report-json --copybook tests/fixtures/sample_fixed.cpy --input tests/fixtures/sample_fixed.bin --output sample-fixed.parquet --batch-records 2

This interaction reveals a verified reference run; it does not execute Rust on the server. The downloadable file is the real 3,925-byte Parquet artifact generated for this page.

Download Parquet
How it works

Compile once, decode from a typed layout

The Copybook is parsed before record decoding begins. The hot path consumes resolved offsets, physical encodings, signedness, precision, scale, and logical Arrow types.

COBOL Copybook

Documented v0.1 subset

Fixed-record binary

CP037 and supported numerics

Compiled typed layout

Offsets, byte lengths, encodings, precision, and scale

Batch decode

Supported EBCDIC and numeric representations

Apache Arrow

Typed in-memory record batches

Apache Parquet

Incremental local columnar output

Resumable conversion

A separate local mode writes deterministic Parquet parts, a manifest, and immutable commit receipts. An explicit resume validates the committed prefix before continuing.

Artifact protection

An optional Cargo feature protects a produced file separately with ML-KEM-768 key establishment, HKDF-SHA-256, and AES-256-GCM authenticated encryption.

The supported grammar is intentionally narrow. Read the exact Copybook subset.

What I measured

Performance with its boundaries intact

The published suite measures controlled local workloads, separates end-to-end conversion from in-memory decoding, and preserves every measured run.

Read the benchmark report

Batch-size tradeoff

Local synthetic benchmark · 3,000,000 records · 100.14 MiB source input

Median wall-clock time

Batch

256

Runtime12,471.72 ms
Observed peak working set132.18 MiB

Source records

240,544/s

Batch

4,096

Runtime2,148.79 ms
Observed peak working set13.93 MiB

Source records

1,396,136/s

Batch

65,536

Runtime2,096.45 ms
Observed peak working set15.26 MiB

Source records

1,430,990/s

What I learned

Very small batches substantially increased both runtime and observed working set in this test. The behavior is consistent with the large number of Parquet row groups and associated metadata retained by the writer in this workload.

Single-workstation measurements over deterministic synthetic data. They are not an SLA or a universal memory bound. Working-set values are the greatest PeakWorkingSet64 samples observed by the Windows harness, not exact heap or RSS peaks.

Checked against another implementation

100 out of 100 records matched

The same externally generated mainframe-style binary dataset was decoded independently by M2C and by Apache Spark with Cobrix. This is a correctness comparison, not a performance comparison.

100/100

semantically identical records

Fields compared
4 per record
Text encoding
CP037 EBCDIC
Spark
4.0.1
Cobrix
2.9.4

M2C

Rust · Arrow 53

Spark / Cobrix

Independent decode

  • CP037 text matched exactly
  • COMP-3 packed decimals matched exactly
  • Column names and numeric representations normalized as documented
  • All four fields compared across every record

The external Copybook writes PIC S9(7)V99 COMP-3. M2C decoded the same binary data using the semantically equivalent PIC S9(7)V9(2) COMP-3 form accepted by its documented subset.

This does not establish full COBOL support, OCCURS or REDEFINES support, variable-record support, or production-mainframe compatibility in general.

Read the compatibility report
Engineering decisions

Make failure and protection explicit

Two capabilities sit beside the core conversion path. Each has a narrow contract and avoids implying infrastructure the project does not provide.

Resume instead of restart

Large local conversions should not necessarily restart from zero after a process interruption. The recoverable path makes publication state visible and validates it before continuation.

  • Deterministic part names
  • Manifest
  • Immutable commit receipts
  • Validation before continuation
  • Explicit resume

This is local process-failure recovery, not distributed fault tolerance.

Protect artifacts separately

Protection is intentionally separate from conversion. A produced artifact can be read as input to a closed, versioned envelope without coupling cryptographic formats to Arrow or Parquet semantics.

  • ML-KEM-768
  • HKDF-SHA-256
  • AES-256-GCM
  • Optional pqc feature

This is an experimental portfolio implementation. It is not a security audit or security certification.