Skip to main content

2026-03-29: Multi-Source Ingestion Implemented

What Happened​

Completed the Multi-Source Ingestion milestone. The pipeline now handles two fundamentally different data shapes -- RSS content feeds and EDI 834 healthcare enrollment transactions -- through the same infrastructure.

What Was Built​

IPipelineRecord interface -- base type with Id, Timestamp, SourceType. Both FeedItem and EnrollmentRecord implement it. The pipeline operates on this interface; domain-specific types stay in their adapter projects.

Keyed DI adapter routing -- adapters registered by key ("rss", "edi834") and resolved at runtime from SourceSettings.Type. Adding a new source type means registering one more key. The pipeline loop doesn't change.

Concurrent processing -- sources process in parallel via Task.WhenAll with a SemaphoreSlim(4) throttle. RSS and 834 run simultaneously without blocking each other.

Edi834SourceAdapter -- reads 834 files from disk, parses X12 segment structure (ISA/GS/ST envelope, INS/REF/DTP/NM1/HD segments), produces EnrollmentRecord instances. Handles missing files, malformed segments, and incomplete records gracefully.

EnrollmentRecord model -- SubscriberId, MemberName, RelationshipCode, MaintenanceTypeCode, CoverageStartDate, CoverageEndDate, PlanId. Implements IPipelineRecord.

Test Coverage​

  • 22 total tests across 3 test projects
  • Conduit.Sources.Rss.Tests: 6 tests (updated for IPipelineRecord)
  • Conduit.Tests: 4 tests (updated for IPipelineRecord)
  • Conduit.Sources.Edi834.Tests: 12 new tests covering parsing, field extraction, error handling, edge cases

Key Design Decisions​

  • Keyed services over factory pattern -- .NET 8+ keyed services are cleaner than a manual factory. One line to register, one line to resolve.
  • Minimal 834 parser -- built for common enrollment scenarios, not a full X12 parser. A production system would use a library like OopFactory.X12 or EdiFabric.
  • Runtime type serialization -- JsonOutputWriter casts to object before serializing so domain-specific properties are preserved in JSON output.
  • Atom deferred -- auto-detection of RSS vs Atom doesn't prove anything the 834 adapter doesn't prove more forcefully. Deferred to keep focus.

What's Next​

  • Data Transformation milestone is next (dedup, enrichment, pluggable storage)
  • The adapter pattern is proven and ready for additional source types (Arxiv, FHIR, etc.)