Practical AI/ML Playbook 🕘 история названий (1)

We just published our write-up on semantic search in LinkedIn’s Hiring Assistant — the path from "no relevance labels exist" to a production system scoring 1B+ profiles in real time. The core problem was familiar: engagement signals (InMail Sends/Accepts) measure interest, not qualification fit. The principled fix was a supervision flywheel — product policy encoded into an Expert Judge, a scalable reasoning-based teacher bootstrapped from its labels, and Matryoshka embeddings distilling that multi-step reasoning into vector scoring usable at both retrieval and ranking. A few things surprised us along the way. ⁃ High-confidence LLM judgments were often more reliable than human annotation. ⁃ For embedding fine-tuning, contrastive post-training alignment mattered more than raw model size. ⁃ And relevance supervision at retrieval plus engagement optimization at ranking turned out to be complementary, not competing — MUSE embeddings [unintentional naming collision with recent Meta’s release!] became the ranker's strongest feature group once integrated properly. The blog describes the IVFPQ serving stack we had in production at review time; since then, MUSE retrieval has moved onto the GPU flat-scan backbone I wrote about earlier https://t.me/ai_review/162 — pre-filter (our use case as any enterprise search is still ABM-heavy), then exhaustive scoring, no approximation loss and cluster-filter mismatch, where Matryoshka shines as well as offering a flexible GPU cost-quality trade-off. https://www.linkedin.com/blog/engineering/ai/semantic-search-for-ai-agents-at-scale-retrieval-and-ranking-for-linkedins-hiring-assistant

First time doing AI video generation end-to-end during an internal LinkedIn hackathon — and it was way more “filmmaking” than I expected. I also caught myself slipping into Lean Startup mode for customer validation — quick iterations, real user reactions. Honestly, it reminded me of the good old startup days. Check out the generated video samples: 1) https://www.linkedin.com/feed/update/urn:li:activity:7422747055862878208/ 2) https://www.linkedin.com/posts/yuliyaru_big-thanks-to-nikita-zhiltsov-for-creating-ugcPost-7423047030442237952-R-I4 3) https://www.linkedin.com/posts/salikh_building-has-always-been-significant-to-me-ugcPost-7423066900798099456-hI_Y A couple surprises / lessons: • Script + timing is everything. If audio isn’t locked, the edit falls apart. • Consistency is the hard part (style/identity). Progress came from iteration, not a “perfect prompt”. I recorded a ~3 min video with slides with the main takeaways. Worth a watch if you’re curious about the real workflow behind the hype.

All set for ACL2025 in Vienna 🇦🇹 and KDD2025 in Toronto 🇨🇦! If you’re attending and into industry-scale embedding-based retrieval, LLMs for search and recsys, LLMs-as-judge, or would like to know about LinkedIn‘s initiatives around semantic search and agentic experience in general and particularly for Recruiter Search, let’s connect

GPU Retrieval Large-scale retrieval once required elaborate ANN indexes — HNSW, IVFPQ, PQ-compressed graphs — because scanning billions of vectors exactly seemed impractical. That assumption has flipped in the last two years. Commodity A100/H100/H200 GPUs now perform flat matrix-multiplication over > 100 M embeddings in only a few milliseconds while still honoring business-facet filters. In effect, it is cheaper to spend compute on embedding-based retrieval (EBR) than to maintain complex index structures. Public results confirm that a flat GPU scan can match or surpass ANN graphs in both latency and recall: LinkedIn’s LiNR for feed post recommendation [1] and Semantic Job Search [3] and Meta’s Mixture-of-Logits (MoL) retriever [2] both demonstrate billion-scale exhaustive search running in production. *** Operational Impact - Shorter release cycles. A full refresh of LinkedIn content embeddings used to involve IVFPQ training, product quantization, and a Galene (Lucene) index build — a process of roughly two weeks. With a flat GPU index, vectors are copied to device memory and validated; the productionization loop now completes in < 1 day. - Simpler deployment surface. Retrieval, filtering, and model weights ship as a single binary. Attribute masks execute inside the CUDA kernel, eliminating extra infrastructure for hard-filter enforcement. Qualitative Improvements - Exact recall. Removing IVF clustering avoids the post-filter mismatch that wastes candidates in ANN pipelines. - Higher-dimensional representations. GPU matmul latency is nearly dimension-invariant until VRAM is full; LinkedIn moved from 256-D to 1024-D embeddings with measurable relevance gains. - Model-based similarity. Flat scans lift the restriction to cosine distance. MoL expresses similarity as a learnable mixture of logits and reports up to 77% hit-rate improvement at 100 M scale, still within single-GPU latency budgets [2]. These traits form a stable foundation for LinkedIn's semantic-search iterations [3], allowing simultaneous optimization for relevance and downstream engagement. Implementation Notes - Memory & quantization. INT8 distance accumulation usually retains > 99 % recall and quadruples capacity; shard-by-time or coarse hashing helps once the corpus exceeds VRAM. - Batching. Grouping 32–128 queries per matmul call meets < 10 ms latency by hiding PCIe overhead. - Filter semantics. Encode hard constraints as 1-bit masks passed to the fused kernel (One Permutation One Random Projection, a count-sketch variant); avoid host-side post-filtering to preserve recall. Secondary Benefits - Near-real-time freshness. Streaming micro-updates every few seconds keeps new items searchable without centroid rebuilds. - RAG readiness. Deterministic recall plus strict filtering yields cleaner evidence for retrieval-augmented generation (RAG). - End-to-end optimization. Because the index lives in the model graph (e.g., an embedding matrix with a lightweight MLP head), future work can back-propagate through retrieval and ranking jointly, or optimise business metrics directly at the retrieval stage. The combined evidence from LinkedIn and Meta indicates that exhaustive GPU k-NN is a practical alternative to ANN graphs in 2025. It replaces weeks-long index operations with day-scale releases, improves recall integrity, and supports richer similarity functions — without sacrificing production latency budgets. Teams limited by ANN rebuilds or post-filter loss may now find a flat scan the simpler, faster, and more accurate choice. *** [1] Borisyuk et al. LiNR: Model Based Neural Retrieval on GPUs at LinkedIn. https://arxiv.org/abs/2407.13218 [2] Zhai et al. Revisiting Neural Retrieval on Accelerators. https://arxiv.org/pdf/2306.04039 [3] Building the next generation of job search at LinkedIn. https://www.linkedin.com/blog/engineering/ai/building-the-next-generation-of-job-search-at-linkedin

This is our team’s work on LLM productionization from a year ago. Since September 2024, it has powered the most member experience in job recommendations and search. A strong example of thoughtful ML system design, it may be particularly relevant for ML/AI practitioners. https://www.linkedin.com/blog/engineering/ai/jude-llm-based-representation-learning-for-linkedin-job-recommendations

This article by Dan Fu, a co-author of very popular FlashAttention layer, surveys new approaches to increasing sequence length in machine learning models, with a focus on large language models (LLMs). This ability was recognized by Sam Altman, CEO of Open.ai, as key for the GPT4 breakthrough. Covering the bleeding edge research, he introduced Hyena, a model that uses convolutional filters and gates to achieve near-linear scaling in sequence length, while matching the performance of traditional Transformer models on perplexity and downstream tasks. The article also explores the use of learned Butterfly matrices for faster and more parameter-efficient convolutional operations. This work has implications for applications that require longer sequences, such as high-resolution imaging and language models that can read entire books. For AI/ML practitioners, while this article offers insights into cutting-edge techniques for sequence modeling and the potential for increased model capabilities in a range of domains, one caveat though is that those methods would require changes in checkpoints, pre-trained at web scale upstream, (the enormous work outcome that we as people dealing with fine-tuning usually get for free), which may not necessarily be practical for usage at the time being. However, if those methods are taking off, one can expect another BLOOM-like initiative by HuggingFace and open source community to deliver compatible checkpoints benefitting from the both worlds. 🌐 From Deep to Long Learning? https://hazyresearch.stanford.edu/blog/2023-03-27-long-learning

At LinkedIn, our vision is to create economic opportunities for every member of the global workforce. With the help of cutting-edge technology and human creativity, we work tirelessly to make that vision a reality. Recently, we've harnessed advanced generative AI and large language models, specifically the OpenAI GPT family, to enhance our offerings even further. We're excited to introduce new AI-powered experiences that include personalized writing suggestions for LinkedIn profiles, AI-powered job descriptions to streamline hiring processes, and a wealth of AI courses aimed at helping professionals stay ahead of the curve in a rapidly evolving job market. These innovations are just the beginning, as we actively explore more ways to create value for our members and customers. I am personally eager to see the positive impact these innovations will have on the professional lives of our users. https://www.linkedin.com/pulse/unlock-opportunities-elevate-your-career-power-ai-tomer-cohen

Google publishes a very insightful paper that describes the ML part in their production ads ranking system. Here I spotlight just some of their lessons, providing some notes💡 from our practice at Linkedin: 1. They employ online learning motivated by non-stationary data • The model is trained with a single sequential pass through all data + progressive validation • Importance sampling to emphasize recent examples • 2nd order optimization methods for faster convergence. Distributed Shampoo optimizer is a relatively new optimizer from the family of pre-conditioning methods (Newton, quasi-Newton, AdaGrad), keeping the second-moment information of the accumulated gradients. • Large batch size training for better GPU/TPU utilization 💡 Large batch size training is not trivial, since it tends to converge to sharp minima. In DNNs we prefer going to flat regions with low loss for better generalization. What works in our practice: learning rate schedules with warmup and decay even when adaptive optimizers, such as Adam, are used (larger batch size, warm up for more epochs); gradient clipping combined. 2. Operational metrics they are focused on:  • Bandwidth (number of models that can be trained concurrently) • Latency (end-to-end evaluation time for a new model), • and throughput (models that can be trained per unit time). • Efficient management of computational resources for ML training is implemented via maximizing model throughput, subject to constraints on minimum bandwidth and maximum training latency. 💡 ML infra reliability is essential for development, experiments, and maintenance. At our scale, the available compute always remains a bottleneck. 3. Model distillation is useful for controlling inference costs or amortizing training costs. • 2-pass online distillation setup: a teacher records its predictions progressively before training on examples in the first pass; a student takes in the teacher’s predictions while training on the second pass. 💡 very common for large language models, when one just cannot productionize them without model complexity reduction. Typically the model-specific distillation (like TinyBERT for Transformers), where the standard soft label supervision is enhanced with additional losses incorporating the internal model weight matrices, works better. Other distillation use cases may include creation of relatively lightweight models used for early stages in ranking. 4. Efficiency in matrix multiplication • Bottleneck layers between non-linearities to reduce matrix multiplication complexity • SVD for downprojection • To increase layer sides (making networks wider) without compromising throughput 💡 this bottleneck technique is well aligned with our way of fine-tuning LiBERT (domain-adapted Google’s BERT used at Linkedin) with Transformer adapters, while rather aiming for reduction of the number of trainable parameters. 5. Auto-ML to find optimal layer/embedding size • Neural architecture search based on weight sharing to explore network configurations efficiently (e.g., varying layer width, embedding dimension) to find versions of our model that provide neutral accuracy with decreased training and serving cost. • This is achieved by three means: a weight-sharing network, an RL controller, and constraints. 6. Loss engineering • MTL with joint rank loss and log loss • Calibration and bias correction: rank loss while effective is not calibrated for the task • Curriculum of losses: start training on easier tasks first, then fine-tuning on harder tasks to avoid the gradient instability. 💡 similarly in our practice we have towers (sub-networks) that represent certain business entities and are pre-trained on relatively easy tasks, being fine-tuned as a part of the larger models in downstream applications with arbitrarily complex MTL losses, and eventually being pre-computed and served runtime to meet latency requirements. 7. Quantifying and alleviating irreproducibility • Metrics to evaluate difference in model predictions

V. Vapnik’s statistical learning with invariants [1] [2] is a fascinating read on the machine learning theory. It is the completion of his statistical learning theory aka VC-theory developed in 70s (Vapnik is 85yo!). Unfairly it caught less attention in the…

V. Vapnik’s statistical learning with invariants [1] [2] is a fascinating read on the machine learning theory. It is the completion of his statistical learning theory aka VC-theory developed in 70s (Vapnik is 85yo!). Unfairly it caught less attention in the community than fancy tricks with Transformers. 1) Unlike emotional debates on AGI, he proves that “intelligence is reduced to the selection of predicates preserving invariants”, and since there's only 2 forms of convergence (strong over functions & weak over functionals) there is no other mechanisms for functions in Hilbert space. 2) His proof shows that such model essentially factors in the mutual position between input vectors as an additional statistical evidence. Which is kinda understood as he considers the conditional probability on class given data vs just an indicator function for decision rule. 3) Predicates are arbitrary functions preserving statistical invariants defined on a training set. Their selection in the form of a finite set is the essence of true intelligence. This opens a research direction for new sample efficient learning methods. 4) Unlike feature addition (which just increases VC dimension or model complexity/diversity; therefore, increasing the set of admissible functions, requiring more training data, slowing down the convergence), the predicate addition narrows down the subset of admissible functions. 5) NNs are crippled as approximating just smoothed piecewise linear functions. Non-linearity does not guarantee the convergence to the global optimum. However, he shows the effectiveness of predicates based on tangent distance and Lie derivatives for image recognition. 6) This "unreasonable effectiveness of mathematics" reveals that there can be a finite set of predicates. Perhaps, widely used LeCun’s convolution is such a high-fidelity translation invariant. Self-attention (w/o positional embeddings) for NLP is a good permutation invariant. 7) Numerical optimization algorithms and their properties are left out of scope. In the realm of NNs, unlike convex kernel SVMs, the invariant method has to be coupled with the empirically robust optimizers. While acknowledging that the nonlinear global optimization is unsolvable. 8 ) Last but not least, machine learning people interested in philosophy will likely enjoy his connections to the philosophy of Plato (vault of ideas and forms) and Hegel's dialectics (invariants to combat the method contradictions to reality). [1] https://proceedings.mlr.press/v128/vapnik20a.html [2] https://youtube.com/watch?v=Ow25mjFjSmg