https://www.ahmadosman.com/tags/inference/ Recent content in Inference on Osman's Odyssey: Byte & Build https://www.ahmadosman.com/logo/byte-and-build.png https://www.ahmadosman.com/logo/byte-and-build.png Hugo -- 0.145.0 en-us Fri, 14 Feb 2025 02:56:56 -0600 https://www.ahmadosman.com/blog/r1-ktransformers-inference-livestream/ Fri, 14 Feb 2025 02:56:56 -0600 https://www.ahmadosman.com/blog/r1-ktransformers-inference-livestream/ Key takeaways from livestreaming DeepSeek R-1 671B (4-bit) on a 14x RTX 3090 basement AI server. See how KTransformers crushed llama.cpp in prompt eval speeds, compare setups, and get real-world insights into massive LLM inference with vLLM, ExLlamaV2, and more. https://www.ahmadosman.com/blog/serving-ai-from-the-basement-part-ii/ Wed, 18 Sep 2024 05:57:26 -0500 https://www.ahmadosman.com/blog/serving-ai-from-the-basement-part-ii/ SWE Agentic Framework, MoEs, Quantizations & Mixed Precision, Batch Inference, LLM Architectures, vLLM, DeepSeek v2.5, Embedding Models, and Speculative Decoding: An LLM Brain Dump... I have been working on a multi-agent system that simulates a team of Software Engineers; this system assigns projects, creates teams and adds members to them based on areas of expertise and need, and asks team members to build features, assign story points, have pair programming sessions together, etc.