Articles tagged "Local LLM"

A deep dive into PrismML's Bonsai 27B (July 14, 2026): post-hoc ternary (1.71 effective bits, 5.9GB, 94.6% retention) and 1-bit (1.125 bits, 3.9GB, 89.5%) builds of Qwen3.6-27B that run on an iPhone 17 Pro — with diagrams covering the quantization scheme, benchmarks, and what it means for on-device AI.

**Qwen3.6-35B-A3B-Uncensored / Abliterated** is a family of **community-produced derivatives of Alibaba's Qwen 3.6-35B-A3B** (a 35B MoE with 3B active parameters, 262K context, and hybrid attention) with **refusal behaviors surgically removed** ([HackerNoon overview](https://hackernoon.com/qwen36-35b-a3b-uncensored-a-35b-moe-model-with-262k-context) / [HauhauCS Aggressive](https://huggingface.co/HauhauCS/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive) / [huihui-ai abliterated](https://huggingface.co/huihui-ai/Huihui-Qwen3.6-35B-A3B-abliterated) / [wangzhang abliterated](https://huggingface.co/wangzhang/Qwen3.6-35B-A3B-abliterated) / [prithivMLmods Aggressive](https://huggingface.co/prithivMLmods/Qwen3.6-35B-A3B-Uncensored-Aggressive)). **Base model specs**: **35B total / 3B active parameters** (MoE, sparse experts), **40 layers**, **hybrid attention** in a **3:1 ratio** (linear + full softmax), **native 262K-token context**, and **native text / image / video** multimodal input. Alibaba positions it as a flagship of its open-weights strategy. **The abliteration technique**: **the refusal direction is removed with LoRA-based steering** on attention and MLP projections. It layers on **Expert-Granular Abliteration (EGA)** (abliterating per-expert `down_proj` slices per layer) and **MoE router suppression** (deactivating safety experts at the router stage) — techniques adapted for the MoE architecture. HauhauCS reports **0 refusals across 465 test prompts**. The philosophy: preserve 100% of the base Qwen 3.6-35B's capability, remove only refusal. **Available variants**: - **HauhauCS-Aggressive** ([HF](https://huggingface.co/HauhauCS/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive) / [Ollama](https://ollama.com/fredrezones55/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive)): the most aggressive refusal removal - **huihui-ai Huihui-Qwen3.6-abliterated** ([HF](https://huggingface.co/huihui-ai/Huihui-Qwen3.6-35B-A3B-abliterated) / [Ollama](https://ollama.com/huihui_ai/Qwen3.6-abliterated)): from the established huihui-ai team - **wangzhang abliterated** ([HF](https://huggingface.co/wangzhang/Qwen3.6-35B-A3B-abliterated)) - **prithivMLmods Uncensored-Aggressive** ([HF](https://huggingface.co/prithivMLmods/Qwen3.6-35B-A3B-Uncensored-Aggressive)) Each ships with **quantization variants** (GGUF Q4 / Q5 / Q8 / FP16), spanning consumer GPUs (RTX 5090 32GB) up to H100 servers. **Ethical and legal considerations**: abliterated models **can produce content that Qwen would normally refuse** (illegal drugs, offensive-security code, dangerous-material synthesis, etc.). Legitimate research, jailbreak-resistance testing, roleplay, and adult-content use cases exist, but **enterprise and commercial adoption carries meaningful legal risk**. Compliance with the EU AI Act (effective August 2026) and Japanese PPC guidance is also in question. **The responsibility sits entirely with the user**; Alibaba's Qwen team is not involved. **Positioning**: alongside our [Local LLM June 2026 Update](../columns/local-llm-landscape-2026-june-update), [Kimi K2.7-Code](../columns/kimi-k2-7-code-moonshot-ai-2026-06), and [Ornith-1.0](../columns/ornith-1-0-deepreinforce-agentic-coding-2026-06), this is a case study showing that **"safety-stripping techniques" scale into the MoE era** in the open-weights ecosystem.

Two months after our [April 2026 local-LLM landscape column](../columns/local-llm-landscape-2026-april-comprehensive-comparison), here is the primary-source update on what has changed. **Three big shifts**: **(1) Open-weights have closed the gap with closed-source.** [GLM-5.2](https://simonwillison.net/2026/Jun/17/glm-52/) (Z.ai, MIT, June 16, 2026) tops the Intelligence Index v4.1 at **51** (MiniMax M3 44 / DeepSeek V4 Pro 44 / Kimi K2.6 43). [MiniMax M3](https://kilo.ai/open-source-models) ships **1M context + native multimodality + SWE-Bench Pro 59.0% + Terminal-Bench 2.1 66.0% + MCP Atlas 74.2%**. [NVIDIA Nemotron 3 Ultra](https://research.nvidia.com/labs/nemotron/Nemotron-3/) (revealed by Jensen Huang at Computex 2026) is a **550B-parameter** US-flag open-weight leader. [VibeThinker-3B](https://arxiv.org/pdf/2606.16140) (WeiboAI, MIT, Qwen2.5-Coder-3B fine-tune) reaches **frontier-reasoner parity at 3B**. **(2) Blackwell makes 30–70B models practical on consumer GPUs.** The RTX 5090 has **32GB GDDR7 and 1,792 GB/s bandwidth** (+77% vs 4090) with **native MXFP4 — GGUF Q4 runs with zero emulation overhead**, hitting **5,841 tok/s** on Qwen 2.5-Coder-7B at batch 8 (2.6× A100 80GB). The RTX PRO 6000 Blackwell reaches **~8,425 tok/s** on 30B; the B200 ships **192GB HBM3e at 8 TB/s** (4–5× H100). **(3) Japan's SI market is maturing.** **Intec** (TIS group) launched local-LLM deployment SI on January 29, 2026 — **minimum 1 month, from ¥5,000,000+ ex tax** — targeting manufacturing and finance. **Ricoh's 'RICOH On-Prem LLM Starter Kit'** won the **2025 Nikkei Excellent Product/Service Award grand prize** (Qwen2.5-VL-32B-Instruct base). PFN's [PLaMo 3.0 Prime](../columns/plamo-3-0-prime-pfn-japanese-llm-2026-06) was selected for the Japanese **Digital Agency 'Gennai'** common generative-AI platform — alongside the Mizuho / Lion Qwen on-domestic-infrastructure precedent. The column also covers concurrent moves on [Kimi K2.7-Code](../columns/kimi-k2-7-code-moonshot-ai-2026-06), [Sakana Fugu](../columns/sakana-fugu-orchestration-model-2026-06), [DiffusionGemma](../columns/diffusiongemma-google-text-diffusion-2026-06), and [Liquid AI LFM2.5-J](../columns/liquid-ai-lfm25-japanese-models-2026-06). Inference-engine selection (**AWQ + vLLM for GPU, GGUF + llama.cpp for CPU/edge, SGLang for agents, TensorRT-LLM for NVIDIA clusters**), quantization (BitNet 1.58-bit / MXFP4 / AWQ), regulation (**EU AI Act GPAI enforcement from August 2, 2026; systemic-risk threshold of 10^25 FLOPs**, US [Fable 5 export-control precedent](../columns/claude-fable-5-export-control-suspension-2026-06), Chinese-model cross-border data), typical GPU configurations by workload, and a three-step Oflight-recommended adoption path are all covered. The article closes with **three direct inquiry funnels** for local-LLM evaluation, build, and ongoing maintenance.

A primary-source deep dive on **DiffusionGemma** (`google/diffusiongemma-26B-A4B-it`, 25.2B total / 3.8B active MoE), released June 10, 2026 by Google DeepMind in coordination with NVIDIA. Grounded in the [official Google blog](https://blog.google/innovation-and-ai/technology/developers-tools/diffusion-gemma-faster-text-generation/), [ai.google.dev model card](https://ai.google.dev/gemma/docs/diffusiongemma/model_card), [Hugging Face card](https://huggingface.co/google/diffusiongemma-26B-A4B-it), and [NVIDIA's blog](https://blogs.nvidia.com/blog/rtx-ai-garage-local-gemma-diffusion/). Where autoregressive (AR) models generate one token at a time left-to-right, diffusion language models (DLMs) **denoise a 256-token canvas in parallel into final text**. 15-20 tokens commit per forward pass, up to 48 denoising steps, 1,000+ tok/sec on H100, 700+ on RTX 5090, ~3.5–4× the throughput of the AR Gemma 4 counterpart. Crucially, Google **openly states that quality lags AR**: MMLU Pro 77.6 vs 82.6, GPQA 73.2 vs 82.3, MMMU Pro 54.3 vs 73.8. Apache 2.0, distributed via Hugging Face / Vertex AI / NVIDIA NIM — the first large-scale open-weight diffusion LLM in the industry. The column covers practical implications for Japanese enterprises (on-prem internal agents, code editing, low-latency workflows) and positioning against Mercury (Inception Labs), LLaDA, and Gemini Diffusion.

A deep dive into Gemma 4 12B, released by Google DeepMind on June 3, 2026, grounded in the [official announcement](https://blog.google/innovation-and-ai/technology/developers-tools/introducing-gemma-4-12b/) and [Developer Guide](https://developers.googleblog.com/gemma-4-12b-the-developer-guide/). The standout property is **encoder-free multimodal architecture** — replacing the prior vision encoder (~550M parameters) with a 35M-parameter lightweight embedder plus a single matrix multiplication, and removing the 12-layer Conformer audio encoder entirely by projecting raw audio straight into the LLM's embedding space. Runs on a 16GB VRAM laptop (Copilot+ PC or Apple Silicon Mac), shipped under Apache 2.0, available through Hugging Face / Ollama / LM Studio / MLX / Vertex AI on day one. Covers the architectural rationale, the "approaches 26B MoE at less than half the memory" benchmark claim, positioning within the Gemma 4 family (E2B / E4B / 26B / 31B), competitive comparison against Llama 4 / Qwen 3.5 / Phi-5, and the fit with Japanese enterprise on-prem AI, voice workflows, and data-sovereignty requirements.