https://www.ahmadosman.com/tags/ai-server/ Recent content in AI Server on Osman's Odyssey: Byte & Build https://www.ahmadosman.com/logo/byte-and-build.png https://www.ahmadosman.com/logo/byte-and-build.png Hugo -- 0.145.0 en-us Fri, 07 Feb 2025 05:06:36 -0600 https://www.ahmadosman.com/blog/do-not-use-llama-cpp-or-ollama-on-multi-gpus-setups-use-vllm-or-exllamav2/ Fri, 07 Feb 2025 05:06:36 -0600 https://www.ahmadosman.com/blog/do-not-use-llama-cpp-or-ollama-on-multi-gpus-setups-use-vllm-or-exllamav2/ Exploring the intricacies of Inference Engines and why llama.cpp should be avoided when running Multi-GPU setups. Learn about Tensor Parallelism, the role of vLLM in batch inference, and why ExLlamaV2 has been a game-changer for GPU-optimized AI serving since it introduced Tensor Parallelism. https://www.ahmadosman.com/blog/serving-ai-from-the-basement-part-ii/ Wed, 18 Sep 2024 05:57:26 -0500 https://www.ahmadosman.com/blog/serving-ai-from-the-basement-part-ii/ SWE Agentic Framework, MoEs, Quantizations & Mixed Precision, Batch Inference, LLM Architectures, vLLM, DeepSeek v2.5, Embedding Models, and Speculative Decoding: An LLM Brain Dump... I have been working on a multi-agent system that simulates a team of Software Engineers; this system assigns projects, creates teams and adds members to them based on areas of expertise and need, and asks team members to build features, assign story points, have pair programming sessions together, etc. https://www.ahmadosman.com/blog/serving-ai-from-the-basement-part-i/ Fri, 06 Sep 2024 16:37:23 -0500 https://www.ahmadosman.com/blog/serving-ai-from-the-basement-part-i/ Dedicated LLM server powered by 8x RTX 3090 Graphic Cards, boasting a total of 192GB of VRAM.