Building Internal Knowledge Search with Qwen3.5-9B & RAG: Enterprise Data AI Guide

Why Local SLM + RAG Is Ideal for Internal Knowledge Search

Every organization accumulates vast internal documentation — employment regulations, product manuals, past proposals, meeting minutes, technical specifications — that represents invaluable knowledge assets. In reality, however, most companies find this knowledge buried deep in file servers with no effective way to search it, leaving critical institutional knowledge dormant and underutilized. The solution is combining a local SLM (Small Language Model) with RAG (Retrieval-Augmented Generation). Released on March 2, 2026, Qwen3.5-9B features a 262K-token context window, a 248K-token vocabulary covering 201 languages, and a highly efficient hybrid architecture based on Gated Delta Networks plus sparse MoE, making it an ideal generator for RAG systems. For small and medium-sized businesses in Shinagawa, Minato, and across the Tokyo area, building a natural-language knowledge search system with zero monthly cloud API costs and zero data leakage risk represents a transformative first step in AI-driven business improvement.

RAG Architecture: Core Components and How It Works

RAG (Retrieval-Augmented Generation) is an architecture that retrieves relevant documents from a database in response to a user's question, then provides that information to a language model to generate an answer. The fundamental system consists of five components: a document ingestion pipeline, an embedding model, a vector database, a retriever (search engine), and a generator (language model). When a user asks 'How do I apply for paid leave?', the question is first converted into a vector by the embedding model, and similar document chunks are retrieved from the vector database. These relevant text passages are then input to Qwen3.5-9B along with the prompt, generating an accurate answer grounded in the actual document content. This mechanism enables accurate responses even for the latest internal information that the model was never trained on, while dramatically reducing hallucination — the generation of information that does not correspond to facts.

Building a Document Ingestion Pipeline

The most quality-critical stage in any RAG system is the document ingestion pipeline. Enterprise internal documents exist in diverse formats including PDF, Word (.docx), Excel (.xlsx), email (.eml/.msg), PowerPoint, and plain text files. A parsing infrastructure is needed that handles each format and extracts text in a unified manner. In Python, pdfplumber or PyMuPDF handles PDFs, Python-docx processes Word files, Openpyxl reads Excel spreadsheets, and the standard email library parses mail messages. For PDFs containing tables and charts, Qwen3.5-9B's multimodal capabilities can be leveraged to read them as images for more accurate extraction. Extracted text is stored alongside metadata such as file name, creation date, last modified date, category, and department. For a law firm in Shinagawa, this might mean tracking client names and dates on contracts; for a manufacturer in Ota, it could be specification version numbers and product model codes. Thoughtful metadata design aligned with actual business workflows is key to maximizing search accuracy.

Selecting Embedding Models Optimized for Japanese

The choice of embedding model fundamentally determines RAG search accuracy. Embedding models convert text into numerical vectors that capture semantic meaning, and the quality of these vectors directly impacts search result relevance. When processing Japanese documents, selecting a model optimized for Japanese is critically important. As of 2026, top recommended models include multilingual-e5-large, which excels at Japanese semantic understanding with easy integration into LlamaIndex and LangChain frameworks. BGE-M3, developed by BAAI, supports Dense, Sparse, and ColBERT retrieval modes simultaneously, making it optimal for Japanese hybrid search scenarios. Additional Japanese-specialized options include intfloat/multilingual-e5-base and pkshatech/simcse. All of these models run locally, meaning that just like Qwen3.5-9B, no document data needs to be sent to external servers. Companies in Shibuya and Setagaya can confidently vectorize their internal documents while maintaining full data sovereignty.

Choosing a Vector Database: Chroma, Milvus, and Qdrant

A vector database is required to store the vectors generated by embedding models and execute high-speed similarity searches. In 2026, three leading options stand out. ChromaDB is a Python-native vector database that offers the easiest onboarding experience, installable via pip with no additional infrastructure. It is ideal for small to medium document collections of up to tens of thousands of entries and is recommended for proof-of-concept projects. Milvus is a production-grade distributed vector database capable of handling millions to billions of entries, deployable via Docker with native hybrid search support combining vector and keyword approaches. Qdrant is a high-performance vector database written in Rust that excels at filtered searches such as narrowing results by department or category, supporting both REST API and gRPC interfaces. For startups and SMBs in Shinagawa, starting with ChromaDB and gradually migrating to Milvus or Qdrant as data volumes grow represents an effective staged approach.

Chunking Strategies for Japanese Text

Chunking — how text is divided into segments — is a critical design element that directly impacts RAG search accuracy. While English text chunking commonly relies on token-count-based splitting using spaces as delimiters, Japanese text lacks spaces and requires different approaches. Sentence-level splitting forms the foundation, using Japanese periods and paragraph breaks as delimiters to create chunks that preserve semantic coherence. The recommended chunk size is approximately 300 to 600 characters with 20 to 30 percent overlap to prevent loss of context at boundaries. A more advanced technique is semantic chunking, which uses the embedding model to calculate semantic similarity between sentences and splits at points where meaning shifts. LangChain's SemanticChunker and LlamaIndex's SentenceWindowNodeParser both support this method. For consulting firms in Meguro and Minato that handle diverse report types, hierarchical chunking based on document heading structure (H1, H2, H3) is also effective, preserving heading context information during search retrieval.

Retrieval Optimization: Hybrid Search and Reranking

Maximizing RAG answer quality requires optimizing the retriever component. The most effective approach is hybrid search. Vector similarity search (semantic search) excels at finding conceptually related documents but tends to be weak at exact matching for proper nouns and model numbers. Combining it with keyword search such as BM25 creates a hybrid approach that captures both semantic and lexical relevance. For example, given the question 'What is the heat resistance temperature for model ABC-123?', keyword search precisely matches 'ABC-123' while vector search discovers semantically related chunks about heat resistance specifications. Merging both result sets using Reciprocal Rank Fusion (RRF) significantly improves overall search accuracy. A reranking step using a Cross-Encoder model then re-evaluates the merged results, pushing the most relevant chunks to the top and maximizing the quality of context provided to Qwen3.5-9B. Recommended multilingual reranking models include the multilingual ms-marco-MiniLM-L-12-v2 variant and BGE-reranker-v2-m3.

Qwen3.5-9B as Generator: Leveraging the 262K Context Window

Qwen3.5-9B is particularly well-suited as a RAG generator because of its extraordinary 262K-token context window. Previous small models were limited to 4K to 32K token contexts, which sometimes proved insufficient for incorporating multiple retrieved document chunks. With Qwen3.5-9B's 262K context, approximately 130,000 to 200,000 Japanese characters can be input at once, comfortably accommodating the top 20 to 30 retrieved chunks. This enables generating comprehensive answers that synthesize information across multiple internal documents. For example, a cross-cutting question like 'What are the sales trends over the past three years and related strategic changes?' can be answered by integrating information from multiple annual reports and strategy documents. Furthermore, the reasoning capabilities strengthened by Scaled RL ensure logically consistent answers assembled from multiple sources, making it ideal for knowledge search at financial firms and consulting companies in Shinagawa.

Implementing Citation and Source Tracking

Citation and source tracking is indispensable when operating a RAG system for business use. The ability for users to verify answer reliability and trace back to original documents is essential for building trust in AI within professional workflows. Implementation involves retaining source document names, page numbers, and section headings as metadata for each retrieved chunk, then instructing the model via prompt to explicitly cite the source material and page for each claim. Qwen3.5-9B demonstrates strong instruction-following capability and can reliably format responses with embedded citations. For more granular tracking, fine-grained attribution can be implemented programmatically to map each statement in the answer to its specific source chunk. LlamaIndex's CitationQueryEngine supports this functionality, making cited answer generation straightforward to implement. This capability is particularly powerful for law firms in Shibuya and accounting firms in Setagaya where evidentiary clarity is paramount.

Incremental Updates and Deployment Architecture

Since internal knowledge bases are continually updated, an incremental update mechanism that handles new document additions and existing document modifications is essential. The recommended design monitors file servers or SharePoint for changes, automatically triggering the parse, chunk, embed, and store pipeline for new or updated files. Removing vectors corresponding to deleted files must also be addressed. Two deployment architecture patterns exist: single-server and distributed configurations. For SMBs with document collections up to tens of thousands, a single-server setup housing Qwen3.5-9B, ChromaDB, and a web frontend on one machine (Apple Mac Studio or Linux GPU server) is entirely sufficient. For large-scale deployments, a distributed architecture separating embedding generation, vector search, and LLM inference across multiple servers is more effective. For companies in Minato and Ota building knowledge bases serving hundreds of employees, starting with a single-server configuration and scaling out as usage grows represents the most pragmatic approach.

Accuracy Evaluation Methodology and the RAGAS Framework

Objective accuracy evaluation is essential before trusting a RAG system for business operations. The most widely adopted evaluation framework is RAGAS, which provides four comprehensive metrics: Faithfulness (whether answers are grounded in retrieved documents), Answer Relevancy (whether responses directly address the question), Context Recall (whether search retrieves the information needed for correct answers), and Context Precision (whether retrieved information is actually relevant). Conducting evaluation requires creating a test set of 50 to 100 question-answer pairs. Collect realistic questions from various departments, prepare correct answers, and document the source materials for each. Running this evaluation set through the RAG system periodically and monitoring score trends enables quantitative measurement of improvements from chunking strategy changes or reranking model additions. For IT companies in Shinagawa and SaaS businesses in Shibuya deploying RAG systems to production, incorporating this evaluation process from the initial deployment phase onward is strongly recommended.

Let Oflight Build Your Internal Knowledge Search AI

An internal knowledge search system powered by Qwen3.5-9B and RAG transforms dormant company data into living intellectual assets, dramatically reducing the time employees spend searching for information. Since no data is sent to cloud APIs, businesses across Shinagawa, Minato, Shibuya, Setagaya, Meguro, and Ota can confidently build knowledge bases that include confidential documents. If you are interested in leveraging your company's document data with AI, building a system that helps employees instantly find the information they need, or creating a search solution integrated with your existing file server, please feel free to reach out to Oflight Inc. From document ingestion design and embedding model selection to vector database construction, Qwen3.5-9B optimization, and accuracy evaluation, we provide comprehensive one-stop support for building internal knowledge search AI tailored to your business needs.