# RAG Research for Personal Knowledge Base ## Research Summary (Phase 1: Initial Repository Analysis) This phase focuses on analyzing the information retrieved from the `https://github.com/NirDiamant/RAG_Techniques` repository. ### Search Strategy - **Initial Query (Implicit):** User provided `RAG.md` which has general RAG info. - **Targeted Retrieval:** Focused on `https://github.com/NirDiamant/RAG_Techniques` as a comprehensive source of advanced RAG techniques. - **Focus Areas:** Identifying RAG techniques, frameworks, evaluation methods, and architectural patterns relevant to building a personal, evolving knowledge base for an AI agent. ### Key Resources 1. **`RAG.md` (User-provided):** General overview of RAG, basic steps, common frameworks, databases, and evaluation metrics. 2. **`https://github.com/NirDiamant/RAG_Techniques` (Retrieved via Puppeteer):** An extensive collection of advanced RAG techniques, tutorials, and links to further resources. ### Critical Findings from `NirDiamant/RAG_Techniques` (Relevant to Personal KB) The repository lists numerous techniques. The following are initially highlighted for their potential relevance to a personal knowledge base that needs to be accurate, adaptable, and efficiently queried by an AI agent. **I. Foundational Techniques:** * **Basic RAG:** Essential starting point. * **Optimizing Chunk Sizes:** Crucial for balancing context and retrieval efficiency. A personal KB might have diverse document lengths. * **Proposition Chunking:** Breaking text into meaningful sentences could be highly beneficial for factual recall from personal notes or scraped articles. **II. Query Enhancement:** * **Query Transformations (Rewriting, Step-back, Sub-query):** Useful if the AI agent needs to formulate complex queries or if initial queries are too narrow. * **HyDE (Hypothetical Document Embedding):** Could improve retrieval for nuanced personal queries by generating a hypothetical answer first. * **HyPE (Hypothetical Prompt Embedding):** Precomputing hypothetical questions at indexing could speed up retrieval and improve alignment for a personal KB where query patterns might emerge. **III. Context Enrichment:** * **Contextual Chunk Headers:** Adding document/section context to chunks can improve retrieval accuracy, especially for diverse personal documents. * **Relevant Segment Extraction:** Dynamically constructing multi-chunk segments could provide more complete context to the LLM from personal notes or longer articles. * **Semantic Chunking:** More meaningful than fixed-size, good for understanding topics within personal data. * **Contextual Compression:** Useful for fitting more relevant information into the LLM's context window, especially if personal documents are verbose. * **Document Augmentation (Question Generation):** Could enhance retrieval for a personal KB by pre-generating potential questions the user/agent might ask. **IV. Advanced Retrieval Methods:** * **Fusion Retrieval (Keyword + Vector):** Could be powerful for a personal KB, allowing both semantic and exact-match searches (e.g., finding a specific term in notes). This is distinct from RAG-Fusion which focuses on multiple generated queries. * **Reranking (LLM-based, Cross-Encoder):** Important for ensuring the most relevant personal information is prioritized. * **Multi-faceted Filtering (Metadata, Similarity, Content, Diversity):** Essential for a personal KB to filter by date, source, type of note, or to avoid redundant information. * **Hierarchical Indices:** Summaries + detailed chunks could be useful for navigating a large personal KB. **V. Iterative and Adaptive Techniques:** * **Retrieval with Feedback Loops:** If the agent can provide feedback, this could continuously improve the personal RAG's performance. * **Adaptive Retrieval:** Tailoring strategies based on query types (e.g., "summarize my notes on X" vs. "find specific detail Y"). * **Iterative Retrieval:** Useful for complex questions requiring information from multiple personal documents. **VI. Evaluation:** * **DeepEval, GroUSE:** While potentially complex for a purely personal setup, understanding these metrics can guide manual evaluation or simpler automated checks. Key aspects for personal KB: Faithfulness (is it true to my notes?), Contextual Relevancy. **VII. Advanced Architectures:** * **Graph RAG (General & Microsoft's):** Potentially very powerful for connecting disparate pieces of information in a personal KB (e.g., linking project notes, contacts, and related articles). Might be complex to set up initially. * **RAPTOR (Recursive Abstractive Processing):** Tree-organized retrieval could be excellent for hierarchical personal notes or structured scraped content. * **Self-RAG:** Dynamically deciding to retrieve or not could make the agent more efficient with its personal KB. * **Corrective RAG (CRAG):** Evaluating and correcting retrieval, possibly using web search if personal KB is insufficient, aligns well with an AI agent's needs. * **Integrating Structured Personal Data (Conceptual):** While the primary focus of RAG is often unstructured text, a truly unified personal knowledge base (PUKB) should consider how to incorporate structured personal data like calendar events, to-do lists, and contacts. Conceptual approaches include: 1. **Conversion to Natural Language for Vector Search:** * **Approach:** Transform structured items into descriptive sentences or paragraphs. For example, a calendar event "Project Alpha Sync, 2025-05-15 10:00 AM" could become "A meeting for Project Alpha Sync is scheduled on May 15, 2025, at 10:00 AM." This text is then embedded and indexed alongside other unstructured data. * **Pros:** Allows for a single, unified search interface using the existing vector RAG pipeline; enables semantic querying across all data types. * **Cons:** Potential loss of precise structured query capabilities (e.g., exact date range filtering might be less reliable than on structured fields); overhead of accurately converting structured data to natural language; may require careful template design or LLM prompting for conversion. 2. **Hybrid Search (Vector DB + Dedicated Structured Store):** * **Approach:** Maintain structured personal data in a simple, local structured datastore (e.g., an SQLite database, or by directly parsing standard PIM files like iCalendar `.ics` or vCard `.vcf`). The AI agent, based on query analysis (e.g., detecting keywords like "calendar," "task," "contact," or specific date/time phrases), would decide to query the vector DB, the structured store, or both. Results would then be synthesized. * **Pros:** Preserves the full fidelity and precise queryability of structured data; potentially more efficient for purely structured queries (e.g., "What are my appointments next Monday?"). * **Cons:** Increases agent complexity (query routing, results fusion); requires maintaining and querying two distinct types of data stores; may necessitate Text-to-SQL or Text-to-API-call capabilities for the agent to interact with the structured store based on natural language. 3. **Knowledge Graph Augmentation (Advanced):** * **Approach:** Model structured entities (people, events, tasks, projects) and their interrelationships within a local knowledge graph. Link these graph nodes to relevant unstructured text chunks in the vector DB. * **Pros:** Enables complex, multi-hop queries that traverse relationships (e.g., "Show me notes related to people I met with last week concerning Project X"). Provides a richer, interconnected view of personal knowledge. * **Cons:** Significantly more complex to design, implement, and maintain. Steeper learning curve. Best considered as a future enhancement. * **Initial PUKB Strategy:** For initial development, converting structured data to natural language for vector search offers the simplest path to unification. A hybrid approach can be a powerful V2 enhancement for more precise structured queries. **VIII. Special Technique:** * **Sophisticated Controllable Agent:** The ultimate goal, using a graph as a "brain" for complex RAG tasks on personal data. ### Technical Considerations for Personal KB * **Data Ingestion:** How easily can new notes, scraped web content (`crawl4ai`), PDFs, etc., be added and indexed? * **Update Mechanisms, Frequency, and Synchronization for Local Files:** Personal knowledge, especially from local files, evolves. The PUKB must efficiently detect changes and update its vector index to maintain relevance and accuracy. Strategies include: * **Change Detection Methods for Local Files:** 1. **File System Watchers (Real-time/Near Real-time):** * **Tool:** Python's `watchdog` library is a common cross-platform solution. * **Mechanism:** Monitors specified directories for events like file creation, deletion, modification, and moves. * **Pros:** Provides immediate or near-immediate updates to the PUKB. * **Cons:** Can be resource-intensive if watching a vast number of files/folders; setup might have platform-specific nuances; requires a persistent process. 2. **Checksum/Hash Comparison (Periodic Scan):** * **Tool:** Python's `hashlib` (e.g., for SHA256). * **Mechanism:** During initial ingestion, store a hash of each local file's content in its metadata (e.g., `file_hash`). Periodically (e.g., on agent startup, scheduled background task), scan monitored directories. For each file, recalculate its hash and compare it to the stored hash. If different, or if the file is new, trigger re-ingestion. If a file path from the DB is no longer found, its chunks can be marked as stale or deleted. * **Pros:** Robust against content changes; platform-independent logic. * **Cons:** Not real-time; scans can be time-consuming for very large knowledge bases if not optimized (e.g., only hashing files whose `last_modified_date_os` has changed since last scan). 3. **Last Modified Timestamp (Periodic Scan):** * **Tool:** Python's `os.path.getmtime()`. * **Mechanism:** Store the `last_modified_date_os` in metadata. Periodically scan and compare the current timestamp with the stored one. If different, trigger re-ingestion. * **Pros:** Simpler and faster to check than full hashing for an initial pass. * **Cons:** Less robust than hashing, as timestamps can sometimes be misleading or not change even if content does (rare, but possible with some tools/operations). 4. **Manual Trigger:** * **Mechanism:** The user explicitly commands the AI agent or PUKB system to re-index a specific file or folder. * **Pros:** Simple, user-controlled, good for ad-hoc updates. * **Cons:** Relies on the user to remember and initiate; not suitable for automatic synchronization. * **Recommended Hybrid Approach:** Combine a periodic scan (e.g., daily or on startup) using last modified timestamps as a quick check, followed by hash comparison for files whose timestamps have changed, with an optional file system watcher for designated 'hot' or frequently updated directories if near real-time sync is critical for those. * **Efficient Vector DB Update Strategy (Document-Level Re-indexing):** * When a change in a local file is detected (or a file is deleted): 1. Identify the `parent_document_id` associated with the changed/deleted file (e.g., using `original_path` from metadata). 2. Delete all existing chunks from the vector database that share this `parent_document_id`. 3. If the file was modified (not deleted), re-ingest it: perform full text extraction, preprocessing, chunking, and embedding for the updated content. 4. Add the new chunks and their updated metadata (including new `file_hash` and `last_modified_date_os`) to the vector database. * This document-level re-indexing is far more efficient than a full re-index of the entire knowledge base, especially for local file changes. * **Frequency of Updates:** * For watchers: Near real-time. * For periodic scans: Configurable (e.g., on application start, hourly, daily). Balance between freshness and resource usage. * Web-scraped content (`crawl4ai`): Updates depend on re-crawling schedules, which is a separate consideration from local file sync. * **Scalability:** While "personal," the KB could grow significantly. The chosen techniques and database should handle this. * **Privacy & Security in a Unified PUKB:** Ensuring the privacy and security of potentially sensitive personal or mixed (work/personal) data is paramount. A multi-layered approach is recommended: * **Data Segregation within the Vector DB:** * **ChromaDB:** Supports a hierarchy of Tenants -> Databases -> Collections. For PUKB, different personas or data categories (e.g., 'personal_notes', 'work_project_X', 'sensitive_health_info') can be stored in separate **Collections** or even separate **Databases** under a single tenant. Application logic is crucial for ensuring queries are directed to the correct collection(s) based on user context or active persona. * **SQLite-VSS:** Data segregation relies on standard SQLite practices. This typically involves adding a `persona_id` column to vector tables and ensuring all application queries filter by this ID. Alternatively, separate SQLite database files per persona offer stronger isolation. * **Qdrant (If considered for scalability):** Recommends using a single collection with **payload-based partitioning**. A `persona_id` field in each vector's payload would be used for filtering, ensuring each persona only accesses their data. * **Weaviate (If considered for scalability):** Offers robust **built-in multi-tenancy**. Each persona could be a distinct tenant, providing strong data isolation at the database level, with data stored on separate shards. * **Encryption:** * **At Rest:** * **OS-Level Full-Disk Encryption:** A baseline security measure for the machine hosting the PUKB. * **Database-Level Encryption:** If the chosen vector DB supports it (e.g., encrypting database files or directories). For embedded DBs like ChromaDB (persisted) or SQLite, encrypting the parent directory or the files themselves using OS tools or libraries like `cryptography` can be an option. * **Chunk-Level Encryption (Application-Side):** For highly sensitive data, consider encrypting individual chunks *before* storing them in the vector DB using libraries like Python's `cryptography`. Decryption would occur after retrieval and before sending to the LLM. Secure key management (e.g., OS keychain, environment variables for simpler setups, or dedicated secrets manager for advanced use) is critical. * **In Transit:** Primarily relevant if any PUKB component (LLM, embedding model, or the DB itself if run as a separate server) is accessed over a network. Ensure all communications use TLS/HTTPS. * **Role-Based Access Control (RBAC) / Persona-Based Filtering (Application Layer):** * If the PUKB is used by an agent supporting multiple 'personas' (e.g., 'personal', 'work_project_A'), the application (agent) must enforce access control. * The agent maintains an 'active persona' context. * All queries to the PUKB (vector DB, structured stores) MUST be filtered based on this active persona, using the segregation mechanisms of the chosen DB (e.g., querying specific Chroma collections, filtering by `persona_id` in Qdrant/SQLite-VSS, or targeting a specific Weaviate tenant). * **Implications of Local vs. Cloud LLMs/Embedding Models:** * **Local Models (Ollama, Sentence-Transformers, etc.):** Offer the highest privacy as data (prompts, chunks for embedding) does not leave the user's machine. Consider performance and model capability trade-offs. * **Cloud Models (OpenAI, Anthropic, etc.):** Data is sent to third-party servers. Users must review and accept the provider's data usage, retention, and privacy policies. There's an inherent risk of data exposure or use for model training (unless explicitly opted out and guaranteed by the provider). * **Data Minimization:** Only ingest data that is necessary for the PUKB's purpose. For particularly sensitive information, evaluate if a summary, an anonymized version, or just metadata can be ingested instead of the full raw content. * **Secure Deletion:** Implement a reliable process for deleting data. This involves removing chunks from the vector DB (and ensuring the DB reclaims space/updates indexes correctly) and deleting the original source file if requested. Some vector DBs might require specific commands or compaction processes for permanent deletion. * **Input Sanitization:** If user queries are used to construct dynamic filters or other database interactions, ensure proper sanitization to prevent injection-style attacks, even in a local context. * **Regular Backups:** Securely back up the PUKB (vector DB, configuration, local source files if not backed up elsewhere) to prevent data loss. Ensure backups are also encrypted. * **Ease of Implementation/Maintenance:** For a personal system, overly complex solutions might be burdensome. * **Query Types:** The system should handle diverse queries: factual recall, summarization, comparison, open-ended questions. * **Integration with AI Agent & Query Handling for Unified KB:** The RAG output must be easily consumable, and the agent needs sophisticated logic to interact effectively with a unified PUKB containing diverse data types. This involves: * **Query Intent Recognition:** The agent should employ Natural Language Processing (NLP) techniques (e.g., keyword analysis, intent classification models, or LLM-based analysis) to understand the user's query. For example, distinguishing between: * `\"Summarize my notes on Project Titan.\"` (targets local notes, specific project) * `\"What are the latest web articles on quantum computing?\"` (targets web scrapes, specific topic, recency) * `\"Find emails from John Doe about the Q3 budget.\"` (targets emails, specific sender, topic) * `\"What's on my calendar for next Monday?\"` (targets structured calendar data) * **Metadata-Driven Query Routing & Filtering:** Based on the recognized intent, the agent dynamically constructs queries for the vector database, leveraging the Unified Metadata Schema: * **Source Filtering:** If intent clearly points to a source (e.g., \"my notes\"), the agent adds a filter like `metadata.source_type IN ['manual_note', 'local_md']`. * **Topic/Keyword Filtering:** Standard semantic search combined with keyword filters on `document_title` or `chunk_text` (if the vector DB supports hybrid search or if keywords are part of the metadata). * **Date Filtering:** For time-sensitive queries (e.g., \"latest articles,\" \"notes from last week\"), use `ingestion_date`, `creation_date_os`, or `last_modified_date_os` fields. * **Tag Filtering:** `user_tags` become powerful for personalized retrieval (e.g., `\"Find documents tagged 'urgent' and 'project_alpha'\"`). * **Author/Sender Filtering:** For queries like `\"emails from Jane\"` or `\"documents by Smith\"`. * **Handling Ambiguity:** If a query is ambiguous (e.g., `\"Tell me about Project X\"` could be notes, emails, or web data): * The agent might initially query across a broader set of relevant `source_type`s and present categorized results. * Alternatively, it could ask for clarification: `\"Are you looking for your notes, web articles, or emails regarding Project X?\"` * **Structured Data Querying (Conceptual - see 'Integrating Structured Personal Data'):** If the query targets structured data (e.g., `\"Show my tasks due today\"`), the agent would need to route this to the appropriate structured data store/handler (e.g., an SQLite DB, iCalendar parser) instead of, or in addition to, the vector DB. * **Consuming RAG Output:** The agent receives retrieved chunks (with their metadata) and passes them as context to an LLM for answer generation, summarization, or other tasks. The metadata itself can be valuable context for the LLM. * **Unified Metadata Schema:** A flexible and comprehensive metadata schema is crucial for managing diverse data sources within the PUKB, enabling robust filtering, source tracking, and providing context to the LLM. * **Core Metadata Fields (Applicable to all chunks):** * `chunk_id`: (String) Unique identifier for this specific chunk (e.g., UUID). * `parent_document_id`: (String) Unique identifier for the original source document/file/email/note this chunk belongs to (e.g., hash of file path, URL, or a UUID for the document). * `source_type`: (String, Enum-like) Type of the original source. Examples: 'web_crawl4ai', 'local_txt', 'local_md', 'local_pdf_text', 'local_pdf_image_ocr', 'local_docx', 'local_image_ocr', 'local_email_eml', 'local_email_mbox', 'local_email_msg', 'local_code_snippet', 'manual_note', 'structured_calendar_event', 'structured_todo_item', 'structured_contact'. * `document_title`: (String) Title of the source document (e.g., web page ``, filename, email subject, first heading in a note). * `original_path`: (String, Nullable) Absolute file system path for local files. * `url`: (String, Nullable) Original URL for web-scraped content. * `file_hash`: (String, Nullable) Hash (e.g., SHA256) of the original local file content, for change detection. * `creation_date_os`: (ISO8601 Timestamp, Nullable) Original creation date of the file/document from the OS or source metadata. * `last_modified_date_os`: (ISO8601 Timestamp, Nullable) Original last modified date of the file/document from the OS or source metadata. * `ingestion_date`: (ISO8601 Timestamp) Date and time when the content was ingested into the PUKB. * `author`: (String, Nullable) Author(s) of the document, if available/extractable. * `user_tags`: (List of Strings, Nullable) User-defined tags or keywords associated with the document or chunk. * `chunk_sequence_number`: (Integer, Nullable) If the document is split into multiple chunks, this indicates the order of the chunk within the document. * `text_preview`: (String, Nullable) A short preview (e.g., first 200 chars) of the chunk's text content for quick inspection (optional, can increase storage). * **Source-Specific Optional Fields (Examples):** These can be stored as a nested JSON object (e.g., `source_specific_details`) or flattened with prefixes if the vector DB prefers. * For `source_type` starting with `'local_email_'`: `email_sender`, `email_recipients_to`, `email_recipients_cc`, `email_recipients_bcc`, `email_date_sent`, `email_message_id`. * For `source_type: 'web_crawl4ai'`: `web_domain`, `crawl_depth` (if applicable). * For `source_type: 'local_code_snippet'`: `code_language`. * For `source_type: 'structured_calendar_event'`: `event_start_time`, `event_end_time`, `event_location`, `event_attendees`. * **Population:** Metadata is populated during the ingestion pipeline. `chunk_id` and `parent_document_id` are generated. OS/file metadata is gathered for local files. Web metadata comes from crawl results. Email headers provide email-specific fields. `ingestion_date` is timestamped. `user_tags` can be added later. ### Best Practices Identified (Preliminary) * **Modular Design:** Separate components for ingestion, chunking, embedding, storage, retrieval, and generation. * **Experimentation:** Chunking strategy, embedding models, and retrieval methods often require experimentation. * **Evaluation is Key:** Even for a personal system, periodically checking relevance and accuracy is important. * **Start Simple:** Begin with a basic RAG and iteratively add advanced techniques. ### Common Pitfalls to Avoid for Personal KB * **Over-chunking or Under-chunking:** Finding the right balance is critical. * **Stale Index:** Not updating the RAG with new information regularly. * **Ignoring Metadata:** Not using source, date, tags for filtering and context. * **Choosing an Overly Complex System:** Starting with something too difficult to maintain for personal use. * **Vendor Lock-in:** If using cloud services, consider portability. ### Impact on Approach for Personal KB The `NirDiamant/RAG_Techniques` repository provides a rich set of options. For a personal knowledge base, the initial focus should be on: 1. **Solid Foundational RAG:** Good chunking (semantic or proposition), reliable embedding model. 2. **Effective Retrieval:** Fusion retrieval (keyword + semantic) and reranking seem highly valuable. 3. **Contextual Understanding:** Techniques like contextual chunk headers and relevant segment extraction. 4. **Manageable Complexity:** Prioritize techniques that can be implemented and maintained without excessive effort for a personal system. GraphRAG and RAPTOR are powerful but might be later-stage enhancements. 5. **Data Ingestion:** Needs to be seamless with `crawl4ai` outputs and manual note entry. ### Frameworks and Tools Mentioned (from NirDiamant/RAG_Techniques & RAG.md) * **RAG Frameworks:** * LangChain (has `crawl4ai` loader) * LlamaIndex * Haystack * Semantic Kernel * Dify, Cognita, Verba, Mastra, Letta, Flowise, Swiftide, CocoIndex * **Evaluation Tools:** * DeepEval * GroUSE * LangFuse, Ragas, LangSmith (more for production LLM apps but principles apply) * **Vector Databases (also from `RAG.md`):** * **Open Source / Local-friendly:** ChromaDB, Milvus (can be local), Qdrant, Weaviate (can be local), pgvector (PostgreSQL extension), FAISS (library), LlamaIndex (in-memory default), SQLite-VSS. * **Cloud/Managed:** Pinecone, MongoDB Atlas, Vespa, Elasticsearch, OpenSearch, Oracle AI Vector Search, Azure Cosmos DB, Couchbase. * **Graph-based:** Neo4j (can store vectors). This initial analysis of the `NirDiamant/RAG_Techniques` repository provides a strong foundation. The next steps will involve deeper dives into the most promising techniques, vector databases, and frameworks suitable for a personal knowledge base. ## Phase 2: Unified Ingestion Strategy for Local Files This section details the tools and methods for ingesting various local file types into the Personal Unified Knowledge Base (PUKB), prioritizing local-first, open-source solutions. ### 1. Plain Text (.txt) * **Tool(s):** Python's built-in `open()` function. * **Workflow:** 1. Use `with open('filepath.txt', 'r', encoding='utf-8') as f:` to open the file (specify encoding if known, UTF-8 is a good default). 2. Read content using `f.read()`. 3. Basic cleaning: Strip leading/trailing whitespace, normalize newlines if necessary. ### 2. Markdown (.md) * **Tool(s):** * Python's built-in `open()` for reading raw Markdown text. * `markdown` library (e.g., `pip install Markdown`) if conversion to HTML is desired as an intermediate step (less common for direct RAG ingestion of Markdown). * **Workflow (Raw Text):** 1. Use `with open('filepath.md', 'r', encoding='utf-8') as f:` to open. 2. Read content using `f.read()`. This raw Markdown is often suitable for direct chunking. 3. Basic cleaning: Similar to .txt files. ### 3. PDF (Text-Based) * **Tool(s):** `PyMuPDF` (also known as `fitz`, `pip install PyMuPDF`). `pypdf2` (`pip install pypdf2`) is an alternative. * **Workflow (`PyMuPDF`):** 1. Import `fitz`. 2. Open PDF: `doc = fitz.open('filepath.pdf')`. 3. Initialize an empty string for all text: `full_text = ""`. 4. Iterate through pages: `for page_num in range(len(doc)): page = doc.load_page(page_num); full_text += page.get_text()`. 5. Close document: `doc.close()`. 6. Clean extracted text: Remove excessive newlines, ligatures, or broken words if present. * **Feeding to Advanced Chunkers:** * **Semantic Chunking:** The preprocessed `raw_markdown` can be directly fed. Semantic chunkers often leverage Markdown structure (headings, paragraphs, lists) to identify meaningful boundaries for chunks. * **Proposition Chunking:** The preprocessed `raw_markdown` is suitable. The proposition extractor (often LLM-based) will then parse this text to identify and extract atomic factual statements. ### 4. PDF (Scanned/Image-Based OCR) * **Tool(s):** * Option A: `OCRmyPDF` (`pip install ocrmypdf`, requires Tesseract installed system-wide). * Option B: `PyMuPDF` (to extract images) + `pytesseract` (`pip install pytesseract`, requires Tesseract) + `Pillow` (`pip install Pillow`). `EasyOCR` (`pip install easyocr`) is an alternative to `pytesseract`. * **Workflow (Option A - `OCRmyPDF`):** 1. Use command line: `ocrmypdf input.pdf output_ocr.pdf`. 2. Process `output_ocr.pdf` as a text-based PDF using `PyMuPDF` (see above) to extract the OCRed text layer. * **Workflow (Option B - `PyMuPDF` + `pytesseract`):** * **Feeding to Advanced Chunkers:** * **Semantic Chunking:** The clean text extracted from `cleaned_html` (after HTML-to-text conversion and further cleaning) is fed to the semantic chunker. The quality of semantic segmentation will depend on how well the original HTML structure (paragraphs, sections) was translated into logical text blocks during the HTML-to-text conversion. * **Proposition Chunking:** The clean text extracted from `cleaned_html` is suitable. The proposition extractor will parse this text for factual statements. 1. Open PDF with `PyMuPDF`: `doc = fitz.open('filepath.pdf')`. 2. Iterate through pages. For each page: * Extract images: `pix = page.get_pixmap()`. Convert `pix` to a `Pillow` Image object. * Perform OCR: `text_on_page = pytesseract.image_to_string(pil_image)`. * Append `text_on_page` to `full_text`. 3. Clean OCRed text: This often requires more significant cleaning for OCR errors, layout artifacts, etc. ### 5. DOCX (Microsoft Word) * **Tool(s):** `python-docx` (`pip install python-docx`). * **Workflow:** 1. Import `Document` from `docx`. 2. Open document: `doc = Document('filepath.docx')`. 3. Initialize an empty string: `full_text = ""`. 4. Iterate through paragraphs: `for para in doc.paragraphs: full_text += para.text + '\n'`. 5. (Optional) Extract text from tables, headers, footers if needed, using respective `python-docx` APIs. `python-docx2txt` might simplify this. ### 6. Common Image Formats (PNG, JPG, etc. for OCR) * **Tool(s):** `Pillow` (`pip install Pillow`) + `pytesseract` (`pip install pytesseract`). `EasyOCR` as an alternative. * **Workflow:** 1. Import `Image` from `PIL` and `pytesseract`. 2. Open image: `img = Image.open('imagepath.png')`. 3. Extract text: `text_content = pytesseract.image_to_string(img)`. 4. Clean OCRed text. ### 7. Email (.eml) * **Tool(s):** Python's built-in `email` module (specifically `email.parser.Parser` or `email.message_from_file`). `eml_parser` for a higher-level API. * **Workflow (built-in `email`):** 1. Import `email.parser`. 2. Open file: `with open('filepath.eml', 'r') as f: msg = email.parser.Parser().parse(f)`. 3. Extract body: Iterate `msg.walk()`. For each part, check `part.get_content_type()`. * If `text/plain`, get payload: `body = part.get_payload(decode=True).decode(part.get_content_charset() or 'utf-8')`. * If `text/html`, get payload and strip HTML tags (e.g., using `BeautifulSoup`). 4. Extract other relevant fields: `msg['Subject']`, `msg['From']`, `msg['To']`, `msg['Date']`. ### 8. Email (.msg - Microsoft Outlook) * **Tool(s):** `extract_msg` (`pip install extract-msg`). * **Workflow:** 1. Import `Message` from `extract_msg`. 2. Open file: `msg = Message('filepath.msg')`. 3. Access properties: `body = msg.body`, `subject = msg.subject`, `sender = msg.sender`, `to = msg.to`, `date = msg.date`. 4. The `body` usually contains the main text content. ### 9. Email (mbox) * **Tool(s):** Python's built-in `mailbox` module. * **Workflow:** 1. Import `mailbox`. 2. Open mbox file: `mbox_archive = mailbox.mbox('filepath.mbox')`. 3. Iterate through messages: `for message in mbox_archive:`. 4. Each `message` is an `email.message.Message` instance. Process it similarly to .eml files (see above) to extract text from payloads. 5. Extract other relevant fields: `message['Subject']`, etc. ### 10. Code Snippets (e.g., .py, .js, .java) * **Tool(s):** Python's built-in `open()`. Language-specific parsers like `ast` for Python for deeper analysis (optional). * **Workflow (Raw Text):** 1. Treat as plain text: `with open('filepath.py', 'r', encoding='utf-8') as f: code_text = f.read()`. 2. This raw code text can be directly used for embedding and RAG. 3. Further preprocessing might involve stripping comments or formatting, depending on the desired RAG behavior. --- ## Preprocessing and Advanced Chunking for Diverse Data Effective RAG relies on high-quality chunks. This section details preprocessing steps for various data types before applying advanced chunking techniques. ### Part 1: Preprocessing Specifics by Data Type (Assumes initial text extraction as per "Phase 2: Unified Ingestion Strategy for Local Files") * **`crawl4ai` - `markdown.raw_markdown` Output:** * **Goal:** Ensure clean, consistent Markdown. * **Steps:** 1. Normalize newlines (e.g., ensure all are `\n`). 2. Trim leading/trailing whitespace from the entire document and from individual lines. 3. Collapse multiple consecutive blank lines into a maximum of one or two. 4. Optionally, strip or handle any rare HTML remnants if `crawl4ai`'s conversion wasn't perfect (though `raw_markdown` should be fairly clean). * **`crawl4ai` - `cleaned_html` Output:** * **Goal:** Convert to well-structured plain text. * **Steps:** 1. Use a robust HTML-to-text converter (e.g., `BeautifulSoup(html_content, 'html.parser').get_text(separator='\n', strip=True)` or `html2text`). 2. Ensure paragraph breaks and list structures from HTML are preserved as newlines or appropriate text formatting. 3. Remove any residual non-content elements (e.g., rare JavaScript snippets, CSS, boilerplate not fully caught by `crawl4ai`). 4. Normalize whitespace (trim, collapse multiple spaces). * **Local Plain Text (.txt) & Markdown (.md):** * **Goal:** Clean and standardize plain text. * **Steps:** 1. Normalize newlines. 2. Trim leading/trailing whitespace. 3. Collapse multiple blank lines. 4. For Markdown, ensure its syntax is preserved for chunkers that might leverage it. * **PDF (Text-Based, output from `PyMuPDF` etc.):** * **Goal:** Clean artifacts from PDF text extraction. * **Steps:** 1. Replace common ligatures (e.g., "fi" to "fi", "fl" to "fl"). 2. Attempt to rejoin hyphenated words broken across lines (can be challenging; may use heuristics or dictionary-based approaches). 3. Identify and remove repetitive headers/footers if not handled by the extraction library (e.g., using pattern matching or positional analysis if page layout is consistent). 4. Normalize whitespace and remove excessive blank lines. * **PDF (Scanned/Image-Based, OCR output from `pytesseract`, `EasyOCR`):** * **Goal:** Correct OCR errors and improve readability. * **Steps:** 1. Apply spelling correction (e.g., using `pyspellchecker` or a similar library). 2. Filter out OCR noise or gibberish (e.g., sequences of non-alphanumeric characters unlikely to be valid text, very short isolated "words"). 3. Attempt to reconstruct paragraph/section structure if lost during OCR (e.g., based on analyzing vertical spacing if available from OCR engine, or using NLP techniques to group related sentences). 4. Normalize all forms of whitespace. * **DOCX (Output from `python-docx`):** * **Goal:** Clean text extracted from Word documents. * **Steps:** 1. Normalize whitespace (trim, collapse multiple spaces/newlines). 2. Remove artifacts from complex Word formatting if they translate poorly to plain text. 3. Handle lists and table text appropriately if extracted. * **Image OCR Output (from standalone images):** * **Goal:** Similar to scanned PDF OCR. * **Steps:** 1. Spelling correction. 2. Noise removal. 3. Whitespace normalization. * **Emails (Text extracted from .eml, .msg, .mbox):** * **Goal:** Isolate main content and standardize. * **Steps:** 1. Remove or clearly demarcate headers (From, To, Subject, Date), footers, and common disclaimers. 2. Normalize quoting styles for replies (e.g., convert `>` prefixes consistently, or attempt to strip quoted history if only the latest message is desired). 3. If original was HTML, ensure clean conversion to text, preserving paragraph structure. 4. Standardize signature blocks or remove them. 5. Normalize whitespace. * **Code Snippets (Raw text):** * **Goal:** Prepare code for embedding, preserving semantic structure. * **Steps:** 1. Normalize newlines. 2. Ensure consistent indentation (e.g., convert tabs to spaces, or vice-versa, though often best left as-is if the chunker can use indentation for semantic grouping). 3. Decide on handling comments: strip them, or preserve them as they provide valuable context for RAG. 4. For some languages, normalizing case for keywords might be considered, but generally not required for modern embedding models. ### Part 2: Application of Advanced Chunking Techniques to Preprocessed Data Once the data from various sources has been preprocessed into clean text, the following advanced chunking strategies can be applied: * **Semantic Chunking:** * **Principle:** Divides text into chunks based on semantic similarity or topic coherence, rather than fixed sizes. Often uses embedding models to measure sentence/paragraph similarity. * **Application to Diverse Data:** * **`crawl4ai` (Web Articles, Docs):** Effectively groups paragraphs or sections discussing the same sub-topic. Can leverage existing HTML structure (like `<section>`, `<h2>`) if translated well into the text. * **Local Notes (.txt, .md):** Identifies coherent thoughts or topics within longer notes. Markdown headings can provide strong hints. * **PDF/DOCX (Reports, Chapters):** Groups related paragraphs within sections, even if original formatting cues are subtle in the extracted text. * **Emails:** Can separate distinct topics if an email covers multiple subjects, or keep a single coherent discussion together. * **Code Snippets:** Can group entire functions, classes, or logically related blocks of code, especially if comments and docstrings are included in the preprocessed text. * **Conceptual Example (Python-like pseudo-code):** ```python # from semantic_text_splitter import CharacterTextSplitter # Example library # text_splitter = CharacterTextSplitter.from_huggingface_tokenizer( # "sentence-transformers/all-MiniLM-L6-v2", # Example embedding model # chunk_size=512, # Target, but will split semantically # chunk_overlap=50 # ) # chunks = text_splitter.split_text(preprocessed_text_from_any_source) ``` * **Proposition Chunking:** * **Principle:** Breaks down text into atomic, factual statements or propositions. This often involves using an LLM to rephrase or extract these propositions. * **Application to Diverse Data:** * **`crawl4ai` (Factual Articles):** Ideal for extracting key facts, figures, and claims from news or informational web pages. * **Local Notes (.txt, .md):** Converts personal notes or meeting minutes into a list of distinct facts, ideas, or action items. * **PDF/DOCX (Dense Documents):** Extracts core assertions from academic papers, technical manuals, or reports. * **Emails:** Isolates key information, decisions, or requests from email conversations. * **Code Snippets:** Less about the code logic itself, but can extract factual statements from docstrings or high-level comments (e.g., "Function `calculate_sum` returns the total of a list."). * **Conceptual Example (Python-like pseudo-code):** ```python # llm_client = YourLLMClient() # Interface to an LLM # prompt = f"Extract all distinct factual propositions from the following text. Each proposition should be a complete sentence and stand alone:\n\n{preprocessed_text_from_any_source}" # response = llm_client.generate(prompt) # propositions = response.splitlines() # Assuming LLM returns one prop per line # # Each proposition in 'propositions' is a chunk ``` * **Contextual Chunk Headers:** * **Principle:** Adds a small piece of contextual metadata (e.g., source, title, section) as a prefix to each chunk's text before embedding. This helps the retrieval system and LLM understand the chunk's origin and context. * **Application to Diverse Data (Header Examples):** * **`crawl4ai` Output:** `"[Source: Web Article | URL: {url} | Title: {page_title} | Section: {nearest_heading_text}]\n{chunk_text}"` * **Local Markdown Note:** `"[Source: Local Note | File: {filename} | Path: {relative_path} | Title: {document_title_from_h1_or_filename}]\n{chunk_text}"` * **PDF Document:** `"[Source: PDF Document | File: {filename} | Title: {pdf_title_metadata} | Page: {page_number}]\n{chunk_text}"` * **Email:** `"[Source: Email | From: {sender} | Subject: {subject} | Date: {date}]\n{chunk_text}"` * **Code Snippet:** `"[Source: Code File | File: {filename} | Language: {language} | Context: {function/class_name}]\n{chunk_text}"` * **Implementation:** This is typically done by constructing the header string from the document's metadata (see Unified Metadata Schema) and prepending it to the chunk content before it's passed to the embedding model. ## Phase 3: Deep Dive into Specific RAG Techniques ### 1. Semantic Chunking ... (content as before) ... ### 2. Fusion Retrieval (Hybrid Search) & Reranking (Initial Summary) ... (content as before) ... ### 3. RAG-Fusion (Query Generation & Reciprocal Rank Fusion) ... (content as before) ... ### 4. Reranking with Cross-Encoders and LLMs ... (content as before) ... ### 5. RAPTOR (Recursive Abstractive Processing for Tree-Organized Retrieval) ... (content as before) ... ### 6. Corrective RAG (CRAG) ... (content as before) ... --- ## Phase 4: Deep Dive into Vector Databases (for Local RAG) This section will explore various vector database options suitable for a local personal knowledge base, focusing on ease of setup, Python integration, performance for typical personal KB sizes, and maintenance. ### 1. ChromaDB ... (content as before) ... ### 2. FAISS (Facebook AI Similarity Search) ... (content as before) ... ### 3. Qdrant ... (content as before) ... ### 4. Weaviate ... (content as before) ... ### 5. SQLite-VSS (SQLite Vector Similarity Search) ... (content as before) ... --- ## Phase 5: Deep Dive into RAG Frameworks This section will explore RAG orchestration frameworks, focusing on their suitability for building a custom RAG pipeline for a personal knowledge base. ### 1. LangChain vs. LlamaIndex ... (content as before) ... ### 2. Haystack (by deepset) ... (content as before) ... ### 3. Semantic Kernel (by Microsoft) ... (content as before) ... --- ## Phase 6: Synthesis and Recommendations for Personal Knowledge Base RAG This section synthesizes the research on RAG techniques, vector databases, and frameworks to provide recommendations for building a personal knowledge base for an AI agent, with `crawl4ai` as a primary data ingestion tool. ### I. Core Goal Recap & Decision on Unified vs. Separate RAG * **Primary Goal:** Create a robust, adaptable, and efficiently queryable personal knowledge base for an AI agent. This KB will store diverse information, including scraped web content, project notes, and potentially code. * **Secondary Goal:** Consider if this personal KB can be unified with a work-related KB or if they should remain separate. * **Recommendation:** For initial development and given the "personal" focus, **start with a separate RAG for the personal knowledge base.** * **Reasoning:** * **Simplicity:** Reduces initial complexity in terms of data siloing, access control, and differing update cadences. * **Privacy:** Keeps personal data distinctly separate, which is crucial. * **Focus:** Allows tailoring the RAG system (chunking, embedding, retrieval strategies) specifically to the nature of personal data and queries. * **Future Unification:** A well-designed personal RAG can potentially be integrated or federated with a work RAG later if designed with modularity in mind. The core challenge would be managing context and preventing data leakage between the two. ### II. Recommended RAG Framework Based on the research, here's a comparison and recommendation: | Feature/Aspect | LangChain | LlamaIndex | Haystack | Semantic Kernel | | :---------------------- | :------------------------------------------ | :--------------------------------------------- | :--------------------------------------------- | :------------------------------------------ | | **Primary Focus** | General LLM App Dev, Agents, Chains | Data Framework for LLM Apps, RAG focus | LLM Orchestration, Production RAG, Pipelines | Agentic AI, Planning, Plugins | | **Ease of Basic RAG** | Moderate | High (RAG-centric abstractions) | Moderate (explicit pipeline setup) | Moderate (RAG is a pattern to build) | | **Advanced RAG** | High (many components, flexible) | High (many advanced RAG modules) | High (flexible pipelines, diverse components) | Moderate (via plugins, memory) | | **`crawl4ai` Integration** | Native `Crawl4aiLoader` | Custom loader needed (easy to adapt) | Custom integration needed (easy to adapt) | Custom plugin/memory integration needed | | **Local Deployment** | Excellent (many local model/DB integrations) | Excellent (many local model/DB integrations) | Excellent (many local model/DB integrations) | Good (supports local models, memory stores) | | **Modularity** | High (Chains, LCEL) | High (Engines, Indices, Retrievers) | Very High (Pipelines, Components) | Very High (Kernel, Plugins) | | **Community/Docs** | Very Large | Large, RAG-focused | Good, Growing | Good, Microsoft-backed | | **Personal KB Fit** | Good, flexible | **Excellent**, RAG-first design simplifies setup | Very Good, robust for evolving needs | Good, esp. if agent needs more than RAG | **Primary Recommendation: LlamaIndex** * **Reasons:** * **RAG-Centric Design:** LlamaIndex is built from the ground up for connecting custom data to LLMs, making RAG its core competency. This often leads to more intuitive and quicker setup for RAG-specific tasks. * **Ease of Use for Core RAG:** High-level abstractions for indexing, retrieval, and query engines simplify building a basic-to-intermediate personal RAG. * **Advanced RAG Features:** Offers a rich set of modules for advanced RAG techniques (e.g., various retrievers, node postprocessors/rerankers, query transformations) that can be incrementally added. * **Strong Local Ecosystem:** Excellent support for local embedding models, LLMs (via Ollama, HuggingFace), and local vector stores. * **Data Ingestion Flexibility:** While no direct `crawl4ai` loader exists *yet*, creating one or simply passing `crawl4ai`'s text output to LlamaIndex's `Document` objects for indexing is straightforward. * **Python Native:** Aligns well with `crawl4ai` and common data science/AI workflows. **Runner-Up Recommendation: LangChain** * **Reasons:** * **Mature and Flexible:** A very versatile framework with a vast ecosystem of integrations. * **Native `Crawl4aiLoader`:** Simplifies the initial data ingestion step from `crawl4ai`. * **Strong for Complex Chains/Agents:** If the AI agent's capabilities extend significantly beyond RAG, LangChain's strengths in building complex chains and agents become more prominent. * **Large Community:** Extensive documentation, tutorials, and community support. * Can achieve everything LlamaIndex can for RAG, but sometimes with more boilerplate or a less RAG-specific API. **Why not Haystack or Semantic Kernel as primary for *this specific* goal?** * **Haystack:** Very powerful and modular, excellent for production and complex pipelines. However, for a *personal* KB, its explicitness in pipeline definition might be slightly more setup than LlamaIndex for common RAG patterns. It's a strong contender if the personal RAG is expected to become very complex or integrate deeply with other Haystack-specific tooling. * **Semantic Kernel:** Its primary strength lies in agentic AI, planning, and function calling. While RAG is achievable via its "Memory" and plugin system, it's more of a capability you build *into* a Semantic Kernel agent rather than the framework's central focus. If the AI agent's core is complex task execution and RAG is just one tool, SK is excellent. If RAG *is* the core, LlamaIndex or LangChain might be more direct. ### III. Recommended Local Vector Database **Primary Recommendation: ChromaDB** * **Reasons:** * **Ease of Use & Setup:** `pip install chromadb`, runs in-memory by default or can persist to disk easily. Very developer-friendly. * **Python-Native:** Designed with Python applications in mind. * **Good Integration:** Well-supported by LlamaIndex and LangChain. * **Sufficient for Personal KB:** Scales well enough for typical personal knowledge base sizes. * **Metadata Filtering:** Supports filtering by metadata, which is crucial for a personal KB (e.g., by source, date, tags). **Runner-Up Recommendation: SQLite-VSS** * **Reasons:** * **Simplicity of SQLite:** If already using SQLite for other application data, adding vector search via an extension is very convenient. No separate database server to manage. * **Good Enough Performance:** For many personal KB use cases, performance will be adequate. * **Growing Ecosystem:** Gaining traction and support in frameworks. **Why not others for initial setup?** * **FAISS:** A library, not a full database. Requires more manual setup for persistence and serving, though powerful for raw similarity search. Often used *under the hood* by other vector DBs or frameworks. * **Qdrant/Weaviate:** More feature-rich and scalable, potentially overkill for a basic personal KB's initial setup. They are excellent choices if the KB grows very large or requires more advanced features not easily met by ChromaDB. They can be considered for a "version 2" of the personal RAG. * **Scalability Considerations for a Unified PUKB:** Given that a "Personal Unified Knowledge Base" might grow significantly with diverse local files (text, PDFs, images, emails, code) and web scrapes, the scalability of the chosen vector database becomes more pertinent. * **ChromaDB & SQLite-VSS:** While excellent for starting and for moderately sized KBs due to their ease of setup, their performance might degrade with many millions of diverse vectors or very complex metadata filtering if the PUKB becomes extremely large. SQLite-VSS, being embedded, also shares resources with the main application. * **Qdrant & Weaviate:** These are designed for larger scale and offer more advanced features like optimized filtering, quantization, and potentially better performance under heavy load or with massive datasets. They typically require running as separate services (often via Docker), which adds a small layer of setup complexity compared to embedded solutions. * **Recommendation Adjustment:** For a PUKB envisioned to be *very large and diverse from the outset*, or if initial prototypes with ChromaDB/SQLite-VSS show performance bottlenecks with representative data volumes, considering Qdrant or Weaviate *earlier* in the development lifecycle (perhaps as an alternative for Phase 1 or a direct step into Phase 2 for database setup) would be prudent. The trade-off is initial simplicity versus future-proofing for scale and advanced features. Migration later is possible but involves effort. ### IV. Recommended Initial RAG Techniques Start with a solid foundation and iteratively add complexity: 1. **Data Ingestion (`crawl4ai` + Manual):** * Use `crawl4ai` to scrape web content. * Develop a simple way to add manual notes (e.g., Markdown files in a directory). 2. **Chunking Strategy:** * **Start with:** Recursive Character Text Splitting (available in LlamaIndex/LangChain) with sensible chunk size (e.g., 500-1000 tokens) and overlap (e.g., 50-100 tokens). * **Consider for V2:** Semantic Chunking or Proposition-based chunking for more meaningful segments, especially for diverse personal data. 3. **Embedding Model:** * **Start with:** A good open-source sentence-transformer model (e.g., `all-MiniLM-L6-v2` for a balance of speed and quality, or a model from the MTEB leaderboard). LlamaIndex/LangChain make these easy to use. * **Alternative:** If API costs are not a concern, OpenAI's `text-embedding-3-small` is a strong performer. 4. **Vector Storage:** * ChromaDB (persisted to disk). 5. **Retrieval Strategy:** * **Start with:** Basic semantic similarity search (top-k retrieval). * **Add Early:** Metadata filtering (e.g., filter by source URL, date added, custom tags). 6. **Reranking:** * **Consider adding soon after basic retrieval:** A simple reranker like a Cross-Encoder (e.g., `ms-marco-MiniLM-L-6-v2`) to improve the relevance of top-k results before sending to the LLM. LlamaIndex has `SentenceTransformerRerank`. 7. **LLM for Generation:** * A local LLM via Ollama (e.g., Llama 3, Mistral) or a smaller, efficient model. * Or an API-based model if preferred (OpenAI, Anthropic). 8. **Prompting:** * Standard RAG prompt: "Use the following context to answer the question. Context: {context_str} Question: {query_str} Answer:" ### V. Phased Implementation Approach 1. **Phase 1: Core RAG Pipeline Setup** * Choose framework (LlamaIndex recommended). * Set up `crawl4ai` for data ingestion from a few sample websites. * Implement basic chunking, embedding (local model), and storage in ChromaDB. * Build a simple query engine for semantic search. * Integrate a local LLM for generation. * Test with basic queries. 2. **Phase 2: Enhancing Retrieval Quality** * Implement metadata storage and filtering. * Add a reranking step (e.g., Cross-Encoder). * Experiment with different chunking strategies and embedding models. * Develop a simple way to add/update manual notes. 3. **Phase 3: Advanced Features & Agent Integration** * Explore more advanced retrieval techniques if needed (e.g., HyDE, fusion retrieval if keyword search is important). * Consider query transformations. * Integrate the RAG system as a tool for the AI agent. * Develop a basic UI or CLI for interaction. * Start thinking about evaluation (even if manual). 4. **Phase 4: Long-Term Enhancements** * Explore techniques like CRAG or RAPTOR if complexity is justified. * Implement more robust update/synchronization mechanisms for the knowledge base. * Consider GraphRAG if relationships between personal data points become important. ### VI. Addressing Work vs. Personal Knowledge Base * The recommendation to start with a separate personal KB allows focused development. * If a unified system is desired later: * **Data Separation:** Use distinct metadata tags (e.g., `source_type: "work"` vs. `source_type: "personal"`) within a single vector store. * **Query-Time Filtering:** Ensure queries to the "work" aspect only retrieve from work-tagged documents, and vice-versa. This is critical. * **Agent Context:** The AI agent must be aware of which "mode" it's in (work or personal) to apply the correct filters. * This adds complexity but is feasible with careful design in frameworks like LlamaIndex or LangChain. This synthesis provides a roadmap. The next practical step would be to start implementing Phase 1.