Coniglio

Repository: gitlab.com/publicala/coniglio

Coniglio is Publica.la's event-tracking and session-analytics backend. It ingests millions of client-side tracking events daily, validates and stores them, then aggregates them into reading and listening sessions that other services (for example Farfalla) can query for analytics purposes.

Overview

Coniglio provides:

• HTTP ingestion endpoints for high-volume tracking events
• Validation, storage, and aggregation of events into user sessions
• Analytics data APIs for other services to consume
• Tenant-aware data isolation and aggregation logic
• Environment toggles to pause heavy aggregation during maintenance windows

Technology Stack

Layer	Technology
Runtime	PHP 8.3
Framework	Laravel 10
Database	SingleStoreDB (columnstore, sharded)
Queues	Redis/SQS (Horizon locally, SQS via Vapor in staging and production)
Testing	Composer + Pest
Static Analysis	PHPStan / Larastan, Pint
Observability	Sentry
Deployment	Laravel Vapor (serverless on AWS)

High-Level Data Flow

1. Ingestion: api/v1/track/* routes hit TrackEventHandler for individual events. api/v1/track/batch accepts arrays of events for offline sync scenarios (see Offline Analytics).
2. Validation: Controller dispatches ValidateTrackRequest job. Batch events are validated individually with UUID-based deduplication.
3. Event Dispatch: Job fires Laravel events (SessionStart, Heartbeat, ...).
4. Storage: Listener TrackEventStorer inserts raw events into track_events.
5. Aggregation Scheduler: TrackEventsAggregator runs every minute (configurable).
   • Calculates a safe unprocessed time window (now-5 s).
   • Locks the range with RaceConditionMiddleware to avoid overlaps.
   • Creates a batch of TenantTrackEventsAggregator jobs (one per tenant).
6. Tenant Aggregation: Each job pages through events and feeds them to SessionProjector, creating or updating rows in sessions.
7. Data Access: Analytics endpoints provide access to session data for other services to query.

Offline Analytics

Volpe no longer sends events directly to Coniglio. Events flow through Delfino RPC to the host application (Farfalla for web, Fenice for mobile and desktop). Farfalla forwards events directly; Fenice adds connectivity detection and offline queuing. See Offline Analytics for the full architecture.

Environment Toggles

Variable	Purpose
TRACK_EVENTS_PROCESSING_AGGREGATION_ENABLED	Enable/disable aggregation jobs
TRACK_EVENTS_PROCESSING_SCHEDULED_AGGREGATION	Run aggregator via scheduler
TRACK_EVENTS_PROCESSING_SCHEDULED_SELF_HEALING	Run stuck window recovery hourly

Automatic detection in Kernel.php pauses aggregation every Tuesday 10:00-12:00 UTC, providing a maintenance window.

Local Development Quick-Start

cp .env.example .env && composer install
php artisan horizon            # or set QUEUE_CONNECTION=sync
php artisan schedule:work      # run scheduler
./vendor/bin/pest              # execute tests
# Optionally seed fixtures

Deployment

CI/CD is handled by GitLab:

1. .gitlab-ci.yml runs lint, static analysis, and tests.
2. Artifacts are deployed via Laravel Vapor using vapor.production.yml and vapor.staging.yml.

How It Scales

• Append-Only Writes: track_events table receives inserts only.
• Windowed Aggregation: Lock-based time windows eliminate per-row flags.
• Tenant Partitioning: Workload is split per tenant for linear scalability.
• Paged Processing: Aggregators paginate to keep jobs short-lived.
• Autoscaling: Horizon and Vapor balance worker counts to traffic; SingleStore columnstore optimizes large analytic reads.

With this architecture Coniglio delivers reliable, near-real-time session analytics while maintaining linear throughput as event volume grows.

Related Systems

Coniglio provides analytics data to:

• Farfalla: provides dashboards, reports, and exports by accessing Coniglio's database directly or through its API endpoints
• Medusa: triggers and insights for content automations
• Other services: ecosystem-wide analytics