|
Over 80% of container spend is wasted. Here’s how to fix it. (Sponsored)
 |
Many teams over-provision containers, underuse spot instances, and have no visibility into which pods are burning budget. Get the eBook from Datadog, which covers five practical optimizations for Kubernetes and ECS environments with specific techniques your team can apply today.
You’ll learn how to:
Pinpoint idle containers, over-provisioned pods, and unused clusters draining your cloud budget.
Right-size CPU and memory with resource requests, limits, and automated cost recommendations.
Cut costs up to 90% with spot instances and savings plans and know exactly when to use each
This week’s system design refresher:
Ollama vs vLLM vs SGLang
How does Claude’s text watermark work?
Top 12 Agent Skills You Should Know
Git Workflow: Essential Commands
Apache Kafka vs. RabbitMQ
Ollama vs vLLM vs SGLang
To use open-weight models on your machine, you have three main options: Ollama, vLLM, and SGLang. But each engine handles requests differently. The diagram below shows the differences and the main techniques behind each engine.
 |
Ollama: A local user calls the OpenAI-compatible API, and requests line up in a FIFO queue. Then Ollama runs a pre-quantized GGUF model, a compressed format it pulls, and the response comes back to the user.
Ollama is best for local dev, prototyping, and laptop-scale hardware.
vLLM: Many users hit the server at once, and continuous batching slots new requests into the running batch instead of making them wait for it to finish. PagedAttention stores the KV cache, the memory a model keeps for tokens it has already processed.
vLLM is best for high-traffic serving, max GPU utilization, and thousands of concurrent requests.
SGLang: Agents and multi-turn chats send requests whose prompts overlap heavily. A prefix-aware scheduler routes them through the RadixAttention cache, a radix tree that reuses every shared prefix instead of recomputing it.
SGLang is best for AI agents and tool loops, multi-turn chats, and JSON/regex outputs.
How does Claude's text watermark work?
Anthropic recently shared their intent to watermark text so they can identify AI-generated text. This post is based on my understanding of how it works.
 |
LLMs produce text word by word. At each step, they generate probabilities for the next likely word. Instead of sampling randomly from those words, the watermarking trick changes which words are allowed to be picked.
How to watermark a response?
Step 1: The model produces probabilities for the next word.
Step 2: Normally a random number generator picks one of the good candidates. With watermarking, a keyed function takes a secret key plus the previous few words and decides which candidates are valid to pick from.
Step 3: This repeats for the whole response. Places where there are multiple plausible choices carry the watermark signal.
How to detect a watermarked text?
Step 1: For any candidate word in the text, we check whether it is a valid choice based on the secret key and the few preceding words. If the word is valid, that is counted as a match.
Step 2: Run this across the entire text. Watermarked text matches far more often. The overall match rate can be treated as an AI-generated score.
I’m personally getting quite annoyed by the false negatives from all these AI text detection techniques, especially for technical writing.
What's your thoughts on AI text detection? Do you think AI text detection is useful, or will it create more problems?
Top 12 Agent Skills You Should Know
Agent skills are instructions and scripts that teach your LLM agent a new skill. The diagram below shows the 12 most-starred