Articles tagged "Self-hosted"

Two months after our [April 2026 local-LLM landscape column](../columns/local-llm-landscape-2026-april-comprehensive-comparison), here is the primary-source update on what has changed. **Three big shifts**: **(1) Open-weights have closed the gap with closed-source.** [GLM-5.2](https://simonwillison.net/2026/Jun/17/glm-52/) (Z.ai, MIT, June 16, 2026) tops the Intelligence Index v4.1 at **51** (MiniMax M3 44 / DeepSeek V4 Pro 44 / Kimi K2.6 43). [MiniMax M3](https://kilo.ai/open-source-models) ships **1M context + native multimodality + SWE-Bench Pro 59.0% + Terminal-Bench 2.1 66.0% + MCP Atlas 74.2%**. [NVIDIA Nemotron 3 Ultra](https://research.nvidia.com/labs/nemotron/Nemotron-3/) (revealed by Jensen Huang at Computex 2026) is a **550B-parameter** US-flag open-weight leader. [VibeThinker-3B](https://arxiv.org/pdf/2606.16140) (WeiboAI, MIT, Qwen2.5-Coder-3B fine-tune) reaches **frontier-reasoner parity at 3B**. **(2) Blackwell makes 30–70B models practical on consumer GPUs.** The RTX 5090 has **32GB GDDR7 and 1,792 GB/s bandwidth** (+77% vs 4090) with **native MXFP4 — GGUF Q4 runs with zero emulation overhead**, hitting **5,841 tok/s** on Qwen 2.5-Coder-7B at batch 8 (2.6× A100 80GB). The RTX PRO 6000 Blackwell reaches **~8,425 tok/s** on 30B; the B200 ships **192GB HBM3e at 8 TB/s** (4–5× H100). **(3) Japan's SI market is maturing.** **Intec** (TIS group) launched local-LLM deployment SI on January 29, 2026 — **minimum 1 month, from ¥5,000,000+ ex tax** — targeting manufacturing and finance. **Ricoh's 'RICOH On-Prem LLM Starter Kit'** won the **2025 Nikkei Excellent Product/Service Award grand prize** (Qwen2.5-VL-32B-Instruct base). PFN's [PLaMo 3.0 Prime](../columns/plamo-3-0-prime-pfn-japanese-llm-2026-06) was selected for the Japanese **Digital Agency 'Gennai'** common generative-AI platform — alongside the Mizuho / Lion Qwen on-domestic-infrastructure precedent. The column also covers concurrent moves on [Kimi K2.7-Code](../columns/kimi-k2-7-code-moonshot-ai-2026-06), [Sakana Fugu](../columns/sakana-fugu-orchestration-model-2026-06), [DiffusionGemma](../columns/diffusiongemma-google-text-diffusion-2026-06), and [Liquid AI LFM2.5-J](../columns/liquid-ai-lfm25-japanese-models-2026-06). Inference-engine selection (**AWQ + vLLM for GPU, GGUF + llama.cpp for CPU/edge, SGLang for agents, TensorRT-LLM for NVIDIA clusters**), quantization (BitNet 1.58-bit / MXFP4 / AWQ), regulation (**EU AI Act GPAI enforcement from August 2, 2026; systemic-risk threshold of 10^25 FLOPs**, US [Fable 5 export-control precedent](../columns/claude-fable-5-export-control-suspension-2026-06), Chinese-model cross-border data), typical GPU configurations by workload, and a three-step Oflight-recommended adoption path are all covered. The article closes with **three direct inquiry funnels** for local-LLM evaluation, build, and ongoing maintenance.

A primary-source deep dive on **Kimi K2.7-Code**, released June 12, 2026 by Moonshot AI (Beijing). Grounded in the [Hugging Face model card](https://huggingface.co/moonshotai/Kimi-K2.7-Code), [MarkTechPost](https://www.marktechpost.com/2026/06/12/moonshot-ai-releases-kimi-k2-7-code-a-coding-model-reporting-21-8-on-kimi-code-bench-v2-over-k2-6/), and [VentureBeat's skepticism piece](https://venturebeat.com/technology/kimi-k2-7-code-cuts-thinking-tokens-30-practitioners-say-benchmarks-dont-check-out). Covers the 1T-total / 32B-active MoE architecture (384 experts, 8 routed + 1 shared), 256K context, MoonViT ~400M vision encoder, native INT4, forced-on thinking mode. License is **Modified MIT** (attribution required only above 100M MAU or $20M MRR), API pricing is $0.95 input / $0.19 cache-hit / $4.00 output per 1M tokens — roughly **1/18 of Claude Opus 4.8's output price**. OpenAI + Anthropic-compatible endpoints drop straight into Claude Code / Cursor / Aider / Cline / cmux. Moonshot self-reports **+21.8% vs K2.6 on its own Kimi Code Bench v2 and -30% reasoning tokens**, but **all public benchmarks are Moonshot's own proprietary suites; independent SWE-bench Verified / Pro / FrontierCode scores are not yet available as of June 15, 2026** (VentureBeat). For Japanese enterprises the column flags two critical caveats: **(1) both `api.moonshot.cn` and the Singapore-subsidiary-run `api.moonshot.ai` remain exposed to PRC National Intelligence Law Article 7 compelled disclosure (set against Japan's PPC DeepSeek alert of February 3, 2025 and the Digital Agency notice of February 6, 2025), and (2) the only reliable mitigation is Hugging Face self-hosting (~4-8 H100, ~595GB INT4) following the Mizuho / Lion Qwen-on-domestic-infrastructure precedent**.