Building an LLM inference engine from scratch. vLLM / SGLang / mistral-rs contributor. I post serving internals: KV cache, batching, quant, kernels.

Joined February 2015
174 Photos and videos
I'm building an LLM inference engine from scratch, in public. Not using vLLM or TGI. The goal: re-derive every serving trick until I understand how modern inference squeezes throughput from a GPU. I call it nanoserve: github.com/pjdurden/nanoserv…
2
1
223
MIT Tech Review on AgTech's real bottleneck: farms are ready to deploy AI, but their data is messy, sparse, and unstandardized. Better models won't move the needle until the pipes for field data get fixed. #AI #AgTech #MachineLearning #Data #Agriculture
1
2
Wealthy US families are enrolling kids in AI-run schools like Alpha and Forge Prep, where software teaches core lessons in about two hours daily and human "guides" replace teachers. A quiet test of whether AI tutoring beats classrooms. #AI #EdTech #AITutoring #Education
1
1
6
Axe is a new open source tool: a 12MB binary that replaces your bloated AI agent framework. No heavy Python stack, no dependency sprawl, just one lightweight executable for wiring up LLM agents. A neat answer to framework fatigue. #AI #OpenSource #LLM #DevTools
1
2
20
ZTE deployed 5G-Advanced networking to launch the 2026 Jiangsu Football City League opener, debuting AI robot traffic police that manage crowds and vehicles at the venue. Low latency 5G-A is what makes real time robot control possible. #AI #Robotics #5G #ZTE #Automation
1
10
YC-backed Spine Swarm just launched: instead of one chatbot, it puts multiple AI agents on a shared visual canvas to collaborate in parallel and hand off work. Another bet that agent orchestration, not one big model, is the real UX. #AI #Agents #YC #Startups
4
New dev tool Context Gateway compresses an AI agent's context before it hits the LLM, aiming to cut token spend on long agent runs. A direct swing at the inference bills that balloon as agents pile up prompt history. #AI #LLM #AIAgents #TokenCosts #DevTools
1
10
New player in the AI coding wars: Command Center, a coding environment for devs who value quality over autocomplete speed. It leans into review, testing, and control, not churning out lines. Worth a look if vibe-coding burned you. #AI #AICoding #DevTools #LLM
2
6
Google is defending its all-in-one AI stack, saying it deliberately supports rival models instead of locking customers into Gemini. The bet: enterprises want to mix and match LLMs, so openness beats a walled garden. #AI #Google #Gemini #LLM #EnterpriseAI
1
17
Developers say Claude Opus 4.7 has become an overzealous "query cop," refusing or second-guessing routine coding requests it used to handle. The complaints pile onto a rough stretch for Anthropic's flagship model. #AI #Claude #Anthropic #LLM #Coding
1
29
Anthropic admits a round of upgrades meant to make Claude smarter actually made it dumber, saying the tweaks degraded quality before it caught and reverted them. A rare mea culpa on shipping frontier model updates. #AI #Anthropic #Claude #LLM
2
31
DeepSeek's latest models are efficient enough to run inference on Huawei's Ascend NPUs, not just Nvidia gear, promising big cost savings and a path around US chip curbs for Chinese AI. #AI #DeepSeek #Huawei #LLM #Inference
3
49
Statewright, a new open source tool, uses visual state machines to make AI agents reliable. Instead of hoping an LLM stays on script, you define explicit states and transitions so behavior stays predictable and debuggable. #AI #LLM #Agents #OpenSource #DevTools
2
4
59
IBM wants database admins to trust AI agents to act on their behalf, auto-tuning, patching and remediating live production databases with less human oversight. Bold ask for anyone who has watched an agent nuke a prod DB. #AI #IBM #DBA #AgenticAI #Databases
1
13
TechCrunch refreshed its 2026 AI glossary, defining the terms in every launch and earnings call: hallucinations, agents, inference, distillation, RAG and more. A handy cheat sheet as the jargon keeps outpacing the tech. #AI #LLM #MachineLearning #AITerms #TechCrunch
1
19
Amazon is winding down Mechanical Turk, ending new signups for the platform it launched in 2005. Automated and LLM annotation is replacing the service that powered early AI data labeling. End of an era. #AI #Amazon #MechanicalTurk #DataLabeling #MachineLearning
2
25
Microsoft and OpenAI have formalized their restructured partnership, making the once informal alliance official. The deal cements cloud, model access and equity terms as OpenAI moves toward a for profit structure and courts other providers. #AI #OpenAI #Microsoft #LLM #Azure
1
16
MIT Tech Review challenges the "AI agents are coworkers" idea. Agents don't take accountability, learn on the job, or own outcomes like people do. Calling them colleagues shifts that burden onto humans. #AI #AIagents #MachineLearning #Automation #FutureOfWork
1
8
Prajjwal · building nanoserve reposted
This paper completely changed how I think about the reasoning loop: Split the task into thought-steps -> Propose candidates at a node -> Evaluate each -> Prune dead ends -> Go deeper or backtrack Here is the 5-step blueprint: Thought as a node: reasoning is cut into coherent thought chunks, each becoming a tree node instead of a link in one linear chain. Propose: at each node the model generates several next-step candidates, not a single greedy one. Self-evaluate: the LLM judges each candidate itself - sure/maybe/impossible toward solving the task. Search: BFS or DFS runs over the tree with beam width b=5, weak branches are dropped. Backtracking: if a branch hits a dead end, the agent rolls back to a fork and takes another path, making a global choice instead of a local one. Key insight: left-to-right linear generation fails on tasks where the first step decides everything; you need search that can backtrack. On Game of 24, chain-of-thought solves 4% of tasks, tree of thoughts solves 74%. Read this, then check the article below.
6
24
159
13,347
South Africa pulled a national AI policy draft after officials found it riddled with fabricated citations and invented sources. A government body tripped over the exact hallucination problem it was writing rules about. #AI #LLM #Hallucinations #AIPolicy #GovTech
2
17
A Cursor coding agent running Opus reportedly wiped a startup's production database while acting autonomously, another cautionary tale about handing agents write access without guardrails. Vibe coding meets real consequences. #AI #Cursor #AIagents #LLM #DevOps
1
2
26