recsys-pipeline-architect

recsys-pipeline-architect is a spec-and-scaffold skill for designing composable recommendation, ranking, and feed pipelines. It encodes the six-stage Source→Hydrator→Filter→Scorer→Selector→SideEffect pattern popularized by xAI's open-sourced For You algorithm (Apache 2.0) and applies it to any problem that picks the top K items for a (user, context) pair. Its premise is that most production recommendation systems are not exotic ML but pipelines whose shape stays fixed while the scoring function and items change, so it helps engineers build that plumbing correctly. This skill is an independent reimplementation of the pattern (MIT) with no code copied from the original.

The six stages each have a defined job and parallelism: Source fetches candidates from one or more origins (parallel), Hydrator enriches candidates with metadata needed for filtering and scoring (parallel), Filter drops ineligible candidates (sequential), Scorer assigns one or more scores (sequential so later scorers see earlier scores), Selector sorts by final score and returns top K (single op), and SideEffect caches, logs, and emits events asynchronously so it never blocks the response. The ordering is deliberate: sources before hydration, hydration before filtering, filtering before the expensive scoring stage, a scorer chain rather than a single scorer, selector after scoring for deterministic caching, and async side effects last. When invoked, the skill walks the user through eight steps from clarifying the use case to generating a runnable scaffold in the user's stack, and it surfaces key trade-offs including single score vs multi-action prediction, candidate isolation vs joint scoring, and online vs offline batch.

The skill targets engineers building content feeds, search ranking, RAG rerankers, task prioritizers, notification triage, and ad selection. It enforces hard rules: do not invent benchmark numbers, attribute the pattern to xAI's open-sourced For You algorithm without using its trademarks or "For You" branding, always surface product trade-offs rather than defaulting silently, keep filter order cheap-before-expensive and universal-before-user-specific, ensure side effects never block, and make the generated scaffold actually run rather than pass off pseudocode. Common use cases include a Strapi v5 TypeScript feed plugin, a Python async RAG reranker, a FastAPI task prioritizer, and an offline-batch notification digest. The upstream repository ships five load-on-demand reference docs and three runnable example scaffolds (Strapi v5 / Go / Python).