X algorithm has been open sourced

Summary

X For You Feed Algorithm This repository contains the core recommendation system powering the "For You" feed on X. It combines in-network content (from accounts you follow) with out-of-network content (discovered through ML-based retrieval) and ranks everything using a Grok-based transformer model. Note: The transformer implementation is ported from the Grok-1 open source release by xAI, adapted for recommendation system use cases. Table of Contents Overview The For You feed algorithm retrieves, ranks, and filters posts from two sources: In-Network (Thunder): Posts from accounts you follow Out-of-Network (Phoenix Retrieval): Posts discovered from a global corpus Both sources are combined and ranked together using Phoenix, a Grok-based transformer model that predicts engagement probabilities for each post. The final score is a weighted combination of these predicted engagements. We have eliminated every single hand-engineered feature and most heuristics from the system. The Grok-based transformer does all the heavy lifting by understanding your engagement history (what you liked, replied to, shared, etc.) and using that to determine what content is relevant to you. System Architecture ┌─────────────────────────────────────────────────────────────────────────────────────────────┐ │ FOR YOU FEED REQUEST │ └─────────────────────────────────────────────────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────────────────────────────────┐ │ HOME MIXER │ │ (Orchestration Layer) │ ├─────────────────────────────────────────────────────────────────────────────────────────────┤ │ │ │ ┌─────────────────────────────────────────────────────────────────────────────────────┐ │ │ │ QUERY HYDRATION │ │ │ │ ┌──────────────────────────┐ ┌──────────────────────────────────────────────┐ │ │ │ │ │ User Action Sequence │ │ User Features │ │ │ │ │ │ (engagement history) │ │ (following list, preferences, etc.) │ │ │ │ │ └────────...