Initiative: Frontdoor - Lightweight Integration Platform

The Bet

We believe that by building a lightweight, headless integration platform using workflow automation tools (e.g., n8n), we can: - Enable Lighthouse and D360 to consume integrations without heavy dependencies - Reduce integration development time by 70% - Normalize incoming data from multiple sources (FTP, API, streaming) to a standard model - Deliver batches of clean, normalized data that can feed any downstream system

Background

Current Challenges

Lighthouse and D360 are heavily dependent on flexible integration paths, but our existing integration systems create bottlenecks: - Long development cycles for new integrations - Tight coupling to existing systems - Difficulty supporting diverse integration types (FTP, API, streaming) - Limited ability to normalize data before downstream consumption

The Opportunity

Instead of treating Lighthouse and D360 as projects that depend on existing integrations, we can flip the model: - Build a lightweight integration platform that handles the "front door" for all data sources - Use workflow automation (e.g., n8n) to quickly configure integrations - Move files to S3 with minimal transformation - Apply normalization as a lightweight final step - Output: Batches of normalized integrated data ready for any downstream system

The Frontdoor Model

Architecture Principles

Outward Dependencies: Integration complexity lives at the source (FTP, API, streaming)
Standard Internal Model: We define a normalized data model internally
Lightweight Transformation: Source data → Normalized model is a simple mapping step
Universal Output: Normalized batches can enter the "front door" of any existing system

Example Flow

Partner Data Source (FTP/API/Streaming)
    ↓
n8n Workflow (connection handling)
    ↓
S3 Landing Zone (raw data)
    ↓
Lightweight Normalization (transform to standard model)
    ↓
S3 Normalized Batches
    ↓
Downstream Systems (Lighthouse, D360, existing integrations)

Pilot Approach

Phase 1: POC with Existing Integration

Goal: Prove the concept with minimal risk

Select one existing integration that we already understand
Build entirely headless (no UI, pure workflow automation)
Leverage our team's existing integration experience
Validate that normalized output can feed existing systems

Success Criteria: - POC delivers data to S3 in normalized format - Downstream system consumes normalized data successfully - Integration development time < 3 days (vs ~2 weeks traditional)

Phase 2: Lighthouse Integration

Goal: Deliver first production integration for Lighthouse

Build on POC learnings
Focus on Lighthouse's most critical data source
Iterate based on real usage

Phase 3: Evaluate for D360

Goal: Assess viability for broader rollout

Determine if model works for D360 use cases
Identify any model limitations
Plan scaling approach if successful

Artifacts Requested

1. Engineer Design Document (REQUIRED)

Owner: Platform Engineering Team Due: TBD

A comprehensive design document covering:

Architecture

Detailed system architecture with component diagram
Data flow from source to normalized output
Integration with existing systems (S3, data lake, downstream consumers)
Technology stack recommendation (n8n evaluation vs alternatives)

Data Model

Standard normalized data model specification
Schema definition (format, required fields, versioning strategy)
Transformation mapping examples (source → normalized)
Validation and quality checks

Implementation Plan

POC scope and timeline
Technology selection criteria
Infrastructure requirements (hosting, monitoring, alerting)
Security considerations (credentials, access control, data encryption)

Integration Patterns

FTP integration pattern
API integration pattern
Streaming integration pattern
Error handling and retry logic
Monitoring and observability

Operational Considerations

Deployment strategy
Configuration management (per-integration settings)
Monitoring and alerting (success/failure rates, latency, data quality)
Maintenance and support model

Risk Assessment

Technical risks and mitigation strategies
Dependency risks (external tools, existing systems)
Data quality risks
Scaling considerations

Success Metrics

Metric	Baseline	Target	Measurement
Integration Development Time	~10-14 days	<3 days	Time from request to production
Integration Types Supported	Limited	FTP, API, Streaming	Capability matrix
Data Normalization Success	N/A	>99%	Validation pass rate
Downstream System Adoption	0	Lighthouse + D360	Active consumers

Timeline

Phase	Dates	Focus
Discovery	Feb 20-27	Engineer design document, technology evaluation
POC	Feb 28-Mar 7	Build with existing integration, validate model
Lighthouse Integration	Mar 8-21	First production integration
D360 Evaluation	Mar 22-31	Assess viability for D360, plan next steps

Open Questions

Technology Selection: Is n8n the right choice, or should we evaluate alternatives (Apache NiFi, Prefect, Airbyte)?
Data Model: What level of normalization is "lightweight enough" vs "comprehensive enough"?
Existing System Integration: How do we ensure normalized data can seamlessly enter existing integration systems?
Scaling: If successful, how do we scale to support all integrations across the platform?
Team Ownership: Who owns this platform long-term (Data Platform? Integrations? New team)?

Lighthouse: Primary beneficiary, flexible data integration needs
D360: Secondary beneficiary, similar integration requirements
Existing Integrations: Must ensure compatibility with normalized output

Team & Stakeholders

Executive Sponsor: Robert Matsuoka (CTO)
Engineering Lead: TBD (Platform Engineering)
Product Stakeholders: Lighthouse PM, D360 PM
Technical Reviewers: Cathy Daves (Director, Platform & Analytics), Shiv Yadav (Director, AI-First Initiatives)

Frontdoor: Lightweight Integration Platform