Intelligent Batching & the Data Flywheel

The best AI pipelines aren't pipelines at all. They're loops. Every cycle produces data that makes the next cycle sharper, and if you design it right, the system compounds in value without anyone having to touch it.

That's what I built. A multi-tenant intelligence platform that ingested domain events (user feedback, support transcripts, survey responses, system alerts) and ran them through a multi-stage AI pipeline with LLM-as-judge quality gates at every step.

The output was targeted: insights for admins, recommendations for team leads, action items for end users, and alerts where needed.

But the pipeline wasn't the interesting part. The flywheel underneath it was.

Context validation identified missing information. Interactive questionnaires collected it from users. Those responses fed back into the system as new events. Every cycle enriched the data pool, and every interaction made future analysis sharper.

This post covers the what: what the platform did, how its intelligent batching system worked, and the data flywheel that made the whole thing compound over time. Two companion posts cover the architectural evolution: the v1 SNS/SQS implementation and the v2 migration to Step Functions.

Not every event deserved immediate processing. A critical alert needed the pipeline to fire within seconds. A routine survey could wait until there was enough volume to make analysis meaningful. I designed the platform's batching system to handle both ends of that spectrum, and everything in between.

Every incoming event got assigned to two batches at once: a tenant-level batch and a user-level batch. Tenant batches aggregated events across all users for organization-wide analysis. User batches collected events for a single individual, producing personalized insights. Both operated independently with their own priority thresholds.

The key design decision was priority cascade. Higher-priority events counted toward lower-priority thresholds too. A HIGH event incremented the HIGH, MEDIUM, and LOW batch counters all at once. That way, high-priority events never sat idle waiting for a lower-priority batch to fill. They contributed everywhere.

Priority Levels

Each priority level mapped to different thresholds, pipeline routes, and use cases. I tuned these over time based on real usage patterns, but here are the production values that worked well:

Here's how priority cascade worked in practice. A single HIGH event immediately fired the pipeline. Meanwhile, it also counted toward the MEDIUM and LOW thresholds, so those batches filled faster too.

The LOW priority pipeline deserves a quick note. Instead of running the full multi-stage AI pipeline, LOW batches got routed through a specialized Sentiment Trend analysis path. With 2,000 events, I had enough data for longitudinal sentiment analysis: detecting gradual shifts in user sentiment, engagement trends, and behavioral patterns that would be invisible at smaller batch sizes.

Solution Throttle

There was one more guardrail in the batching system: the solution throttle. I capped output at a maximum of 100 solutions per 24-hour window per tenant. When a tenant hit that limit, any additional batches queued as PENDING until the window reset.

This was pure cost control. During onboarding spikes or bulk imports, a single tenant could generate hundreds of batches in rapid succession. Without the throttle, that meant runaway LLM costs. Every batch triggered multiple AI stages, each with its own token spend.

The 100-solution cap hit a sweet spot. It was generous enough that no customer ever noticed the limit in normal operation, but tight enough to prevent a single tenant's activity spike from blowing through the monthly inference budget.

This is where things got genuinely interesting. When a batch hit the context validation stage, the system analyzed the events for missing information. Were there references to configurations it didn't have? User context that was never provided? Business goals that were unclear?

If it found blocking gaps, the batch paused. The system generated an interactive questionnaire tailored to exactly what was missing and surfaced it to the relevant user. Once they completed it, the batch resumed with richer context.

Here's the critical detail: questionnaire responses weren't throwaway. They became new events in the system. Every completed questionnaire enriched the data pool for future batches.

So a team lead who answered three questions about their group's objectives didn't just unblock one batch. They gave the system context that improved every subsequent analysis for that team.

Eliminating the Cold Start

The context validation system had a special case I was particularly proud of: the onboarding questionnaire. This was the very first data collection a tenant encountered. It walked them through structured data gathering: business context, user profiles, existing configurations, and known pain points.

Here's the thing I liked most about this. Every onboarding response was classified as HIGH priority. With a tenant-level HIGH threshold of just 1 event, the pipeline fired immediately after each questionnaire step.

The tenant didn't have to wait for some arbitrary onboarding period to end before seeing value. They answered a few questions, and insights started flowing.

No cold start problem. The flywheel began spinning from the first interaction.

Let me zoom out for a second. This wasn't a one-shot processor where events went in and reports came out. It was a compounding system. Each cycle through the pipeline produced four things, and they all reinforced each other:

More events. Questionnaire responses, user actions, and engagement signals all re-entered the system as new data points. The pool of raw material kept growing on its own.

Richer context. Gaps that were filled stayed filled. Every future analysis started from a higher baseline than the one before it.

More targeted context validation. With richer context, the AI got better at identifying what was actually missing versus what it already knew. It stopped asking redundant questions.

More specific questionnaires. Sharper validation meant sharper questions. Sharper questions meant higher-quality responses. And higher-quality responses fed right back into that richer context.

The flywheel effect meant that tenants who used the platform more got disproportionately better results. Not just because they had more data, but because the system actively sought out and filled the gaps in what it knew.

A tenant that had been on the platform for six months wasn't just getting six months of analysis. They were getting analysis informed by a continuously enriched understanding of their entire data history.

This was the architectural idea I kept coming back to when making design decisions. Every feature, every pipeline stage, every batch threshold was evaluated against a single question: does this make the flywheel spin faster?

The batching system and data flywheel were the conceptual core of the platform. Dual-level batching with priority cascade meant urgent events got immediate attention while routine data was processed efficiently at scale.

Context validation and interactive questionnaires turned the pipeline from a passive processor into an active learner. And the onboarding collection eliminated the cold start problem that plagues most AI-powered platforms.

The architectural lesson I took away: if you're building an AI pipeline, don't think of it as a straight line from input to output. Think of it as a loop. Design every stage to produce data that makes the next cycle better. That's where the real value lives.

Happy coding!

– Nate

This post is the first in a three-part series on this AI pipeline architecture. The companion posts cover the infrastructure that made this system real:

• Building an Event-Driven AI Pipeline with SNS/SQS covers the original architecture, LLM-as-judge quality gates, the AI debugger, and the operational lessons from running it in production.
• From SNS/SQS to Step Functions covers why I migrated to Step Functions, how the orchestration model changed, and the measurable improvements in reliability and observability.