Article
27494 discoveries · Hand-picked discoveries from the open web, newest first.
-
Building Aligned Intelligence System. Part II. Improving Large Language Models
In this post we will look at different techniques for steering LLMs behaviour to get desired outcomes, starting with some basic general principles such as writing a good prompt and ending with fine-tuning and augmenting models with external knowledge. Methods discussed in this post are mainly aimed at improving LLMs reliability and ensuring the consistency and factual accuracy of their...
AstraBlog · EN
-
Exploring Parallel Strategies with Jax
Training large language models either like GPT, LlaMa or Mixtral requires immense computational resources. With model sizes ballooning into the billions or sometimes even trillions of parameters, specialized parallelization techniques are essential to make training feasible. In this post, we’ll explore implementing some of these scaling strategies in Jax - a Python framework designed for hig...
AstraBlog · EN
-
Transformers Inference Optimization Toolset
Large Language Models are pushing the boundaries of artificial intelligence, but their immense size poses significant computational challenges. As these models grow, so does the need for smart optimization techniques to keep them running efficiently on modern hardware. In this post, we’ll explore key optimization strategies that are making LLMs faster and more memory-efficient. We’ll star...
AstraBlog · EN
-
Speculative Decoding: The Art of Guessing Well
Large Language Models spend most of their decoding time not computing, but waiting: every new token requires reading all of the model weights from memory, only to multiply them by a single vector. What if the model could check several tokens for the price of one? In this post we’ll derive speculative decoding from first principles, build a general framework that covers every modern drafte...
AstraBlog · EN
-
Running Kubernetes with a GPU Inside WSL2 on My RTX Laptop
Published: 2026-04-22 I wanted a local environment where I could: run Kubernetes schedule GPU workloads experiment with CUDA / inference / device plugins avoid renting cloud GPUs I used a Lenovo Legion laptop with an RTX GPU. The working stack was: Windows 11 + NVIDIA driver (WSL-enabled) WSL2 (Ubuntu 24.04) K3s (containerd) NVIDIA Container Toolkit NVIDIA device plugin The non-obvious part was that the NVIDIA device plugin itself had to use runtimeClassName: nvidia . Prerequisites You need: Windows 11 NVIDIA GPU (RTX in my case) Latest NVIDIA driver with WSL support WSL2 installed Inside WSL:
Andrey Krisanov · EN
-
My Modern C++ Setup on macOS and Apple Silicon
Published: 2026-05-19 I’m starting to learn modern C++, so I wanted a repeatable development setup on Apple Silicon: use a modern LLVM/Clang toolchain build projects with CMake and Ninja get proper language-server support in VS Code and Helix use sanitizers and static analysis from the beginning keep the setup practical This cheatsheet assumes that Homebrew is already installed.
Andrey Krisanov · EN
-
Monitoring vLLM in Production: Metrics, PromQL, Alerts, and Runbooks
Published: 2026-01-28 Updated: 2026-06-18 Version scope: This guide targets vLLM 0.23.x and its default V1 engine. Metric names and semantics change between releases, so verify the /metrics output of the exact version and serving configuration you run before copying queries or alerts into production. A vLLM server can pass Kubernetes health checks while users experience slow or unstable responses. Requests may wait in the scheduler, long prompts may make prefill expensive, or token streaming may become uneven. This guide focuses on the metrics needed to diagnose latency, scheduler backlog, KV-
Andrey Krisanov · EN
-
Kubernetes Model Serving in 2026: What Changed Since 2024
Published: 2026-07-26 In October 2024, Yuan Tang published AI/ML Innovation in the Kubernetes Ecosystem . The article described three important developments: Kubeflow Model Registry, KServe ModelCars, and TrustyAI. It also pointed toward multi-node serving, inference-aware gateways, speculative decoding, low-rank adaptation (LoRA) adapters, and APIs designed for generative AI. Since then, Kubernetes model serving has developed into a stack of specialized control-plane and data-plane components. Kubernetes now has better primitives for distributing model files and allocating accelerators. Gatew
Andrey Krisanov · EN