Communication & Observability Ecosystem
A production-grade operational ecosystem built to automate, track, and monitor the complete cycle of institutional communication at scale. The architecture handles synchronous and asynchronous workflows, concurrency control, intelligent queues, operational traceability, and real-time observability—ensuring stability even under high parallel processing load.
Project metrics
4 key metricsMax simultaneous email processing with intelligent concurrency control
Complete traceability per send — batch, operator, timestamp, filter context
Individually validated stages in the automated send pipeline
Async queue-based and sync direct pipeline architectures
Problem statement
Institutional communication with multiple stakeholders lacked automation, traceability, and operational control. The team had no centralized way to send individualized reports, monitor delivery status in real-time, or audit who sent what and when.
Reports and notifications were generated and sent manually, creating bottlenecks and human error
No centralized audit trail — it was impossible to know which communications had been delivered successfully
High-volume sends could overwhelm the email provider without any rate control in place
Customer Success teams had no real-time visibility into delivery failures or sends stuck in the queue
Recipient data (emails, roles) was frequently outdated and required manual correction before every send
Constraints
- Zero downtime during implementation — the platform remained fully operational throughout development
- Strict per-recipient data isolation — each report must contain only the data relevant to that specific person
- Rate limiting compliance with the transactional email provider (Brevo)
Approach & modules
We designed a full operational ecosystem integrating a no-code frontend (Bubble), workflow orchestration (N8N), PDF generation (Gotenberg), file storage (Google Drive), queue management, and a real-time observability dashboard (Next.js) — all connected through a single source of truth in PostgreSQL via Supabase.
Queue Engine
Async scheduler-based queue with 20-concurrent-item limit, anti-duplication via pre-marking, and 3s inter-send delay for provider compliance
PDF Pipeline
Per-recipient HTML generation filtered by context, Gotenberg HTML→PDF conversion, Google Drive upload with direct download links for email attachment
Sync Direct Flow
Immediate synchronous workflow for single-recipient institutional communications, supporting up to 2 attachments with conditional routing per document type
Observability Panel
Real-time CS dashboard with TanStack Table, expandable row lifecycle view, 7 combined filters, URL-persisted filter state, and 4 KPI counters
System design
Four layers with clear separation of concerns: no-code UI for operator capture, N8N for workflow orchestration, Supabase as the operational source of truth, and third-party integrations for PDF generation and email delivery.
Process visualization
Step-by-step workflow showing how data and operations flow through the system.
Operator selects recipients, filters context, and configures the email template in Bubble
N8N validates the payload, normalizes data, generates batch_id and sequential send number
Per-recipient HTML generated (filtered by context), converted by Gotenberg, uploaded to Drive
Records inserted into email_queue with status=pending, awaiting the 1-minute scheduler
Scheduler pulls up to 20 pending items, marks as processing, sends via Brevo with 3s delay
Status updated to sent/error; full audit trail persisted with operator, batch ID, and filter context
Project journey
Key milestones and phases throughout the project lifecycle.
Ecosystem Design
Defined the two flows (async and sync), separation of responsibilities between Bubble, N8N and Supabase, and the email_queue table schema
PDF Generation & Queue
Implemented the Gotenberg pipeline, Drive upload, per-step chain validations and the complete async queue structure
Scheduler & Concurrency Control
Queue consumption workflow with 20-item concurrency cap, 3s inter-send delay and preventive anti-duplication pre-marking
Operational Dashboard
Next.js dashboard with TanStack Table, URL-persisted filters, expandable row lifecycle view and real-time KPI counters
Launch & Refinements
Production stabilization, sticky header scroll fix, post-launch corrections and future improvement mapping
Ecosystem Design
Defined the two flows (async and sync), separation of responsibilities between Bubble, N8N and Supabase, and the email_queue table schema
PDF Generation & Queue
Implemented the Gotenberg pipeline, Drive upload, per-step chain validations and the complete async queue structure
Scheduler & Concurrency Control
Queue consumption workflow with 20-item concurrency cap, 3s inter-send delay and preventive anti-duplication pre-marking
Operational Dashboard
Next.js dashboard with TanStack Table, URL-persisted filters, expandable row lifecycle view and real-time KPI counters
Launch & Refinements
Production stabilization, sticky header scroll fix, post-launch corrections and future improvement mapping
Dive into the documentation
Detailed implementation guides, architecture decisions and technical specifications for this project.