vmvv: Intelligent File Scanner for Language Models

vmvv is an advanced file scanner designed to enrich language models (such as GPT or similar) with comprehensive project context by extracting, organizing, and contextualizing project files. By providing structured insights, vmvv enables language models to generate more accurate, insightful, and relevant outputs tailored to the unique setup of each project.

Content

• Project Overview
• Key Features
• Use Cases
• Main Technologies
• Installation
• Add Script in package.json
• Usage
• Customization: Excluding Files and Directories
• Excluding Files
• Excluding Directories
• Important Notes
• Default .env File
• Token Count

Project Overview

vmvv scans all files and directories within a project, identifying and parsing key information to construct a clear and detailed context for language models. This facilitates the language model’s understanding of the project's architecture, dependencies, and workflows, enhancing the precision and relevance of the responses it generates.

Key Features

• Comprehensive Scanning: Detects and scans all files and directories, filtering out unnecessary or irrelevant files to focus on core project components.
• Intelligent Parsing: Parses critical information from configuration files, dependencies, and code structure, transforming it into an accessible format for language model consumption.
• Customizable Exclusions: Allows users to specify file and directory exclusions, ensuring only relevant files are processed.
• Optimized Contextualization: Prepares extracted data in a structured manner, enabling the language model to better understand complex project setups and generate contextually aware responses.

Use Cases

1. Enhanced Code Assistance
   Scenario: A developer needs assistance refactoring a large codebase with multiple dependencies and complex configurations.
   Solution: By using vmvv to scan the project, a language model can understand the entire codebase structure, making recommendations that respect existing dependencies and configurations, thus improving code quality and coherence.

2. Automated Documentation
   Scenario: The team wants to automate the documentation process for a newly developed API.
   Solution: vmvv scans through relevant project files, such as code and configuration files, to provide the language model with the necessary context for generating accurate, project-specific documentation.

3. Dependency Management and Troubleshooting
   Scenario: A developer encounters compatibility issues within a multi-module project.
   Solution: vmvv scans and organizes dependency files and configurations, allowing the language model to identify and troubleshoot compatibility issues based on the overall project setup.

4. Architecture Analysis and Insights
   Scenario: A system architect is analyzing a legacy project to propose improvements.
   Solution: vmvv provides the language model with a detailed view of the project’s architecture, dependencies, and modules, enabling it to suggest potential architectural improvements with a holistic understanding of the system.

5. Cross-Project Context Sharing
   Scenario: A consultant is working across several projects and needs to compare code structures and setups.
   Solution: vmvv scans each project individually, allowing the language model to access and compare structured data from multiple projects, facilitating insights on best practices and architectural consistencies.

vmvv is ideal for development teams, technical writers, consultants, and anyone looking to leverage language models for enhanced project insights, documentation, and troubleshooting. It is a powerful tool for creating a bridge between raw project data and meaningful, context-aware model outputs.

Main Technologies

• Node.js
• JavaScript

Installation

To install the package from NPM, run the following command:

npm install vmvv

Add Script in package.json

To add a script for scanning your project, edit your package.json file and add the following entry:

{
  "scripts": {
    "scan": "node node_modules/vmvv/lib/project_files.js"
  }
}

Usage

After installing the package and adding the script, you can scan your project by running:

npm run scan

Detailed Instructions

When you run the scan command, you will be prompted to choose between two options:

🔍 Please select the scan type:

📄 Scan and generate JSON

📃 Scan and generate TXT

📁 Scan a specific directory

🚪 Exit

Customization: Excluding Files and Directories

To tailor the scanning process to your project's specific needs, you can exclude certain files or directories. This is particularly useful to prevent the scanner from processing unnecessary or sensitive files, or to avoid re-scanning files generated by the scanner itself.

Excluding Files

1. Open the .env File:

The .env file is located in the root directory of your project. If it does not exist, it will be created automatically when you run the scanner.

2. Define Exclusion Patterns:

The EXCLUDE_FILES variable allows you to specify patterns for files you want to exclude from the scan. You can use exact filenames or wildcard patterns.

Examples:

EXCLUDE_FILES=package-lock.json,yarn.lock,.env,.env.example,project_files.json,project_files.txt,project_files_*.json,project_files_*.txt,*.json,*.txt,.DS_Store

• Exact Filenames:
  • package-lock.json, yarn.lock, .env, .env.example
• Specific Patterns:
  • project_files.json, project_files.txt
• Wildcard Patterns:
  • project_files_*.json (excludes any JSON file starting with project_files_)
  • project_files_*.txt (excludes any TXT file starting with project_files_)
  • *.json (excludes all JSON files)
  • *.txt (excludes all TXT files)
• Other Exclusions:
  • .DS_Store (a common macOS metadata file)

3. Save the .env File:

After adding your exclusion patterns, save the .env file. The scanner will automatically use these patterns in subsequent scans.

Excluding Directories

1. Open the .env File:

As with file exclusions, the .env file is your configuration hub.

2. Define Exclusion Directories:

The EXCLUDE_DIRS variable allows you to specify directories to exclude.

Examples:

EXCLUDE_DIRS=node_modules,.git,.vscode,dist,build,.nuxt,public

• Common Exclusions:
  • node_modules (dependencies)
  • .git (Git repository)
  • .vscode (Visual Studio Code settings)
  • dist, build (build directories)
  • .nuxt, public (framework-specific directories)

3. Save the .env File:

After specifying the directories to exclude, save the .env file. The scanner will respect these exclusions in future scans.

Important Notes:

• Wildcard Support:
  Currently, the scanner supports wildcard patterns (*) in file exclusions, allowing for flexible and broad exclusion criteria.
• Order of Exclusions:
  The scanner processes exclusions based on the patterns defined. More specific patterns should be placed before general ones to ensure precise exclusions.
• Case Insensitivity:
  Exclusion patterns are case-insensitive, meaning Project_Files_*.json and project_files_*.json are treated the same.

Default .env File:

EXCLUDE_DIRS=node_modules,.git,.vscode,dist,build,.nuxt,public
EXCLUDE_FILES=package-lock.json,yarn.lock,.env,.env.example,project_files.json,project_files.txt,project_files_*.json,project_files_*.txt,*.json,*.txt,.DS_Store
PROJECT_DESCRIPTION=This is a project description
MAIN_TECHNOLOGIES=node.js

Token Count

New (ver 2.7.1):

The console indicates the number of tokens once the scan is performed

Number of tokens in the generated file: 3623

After generating the report (either JSON or TXT), you can copy its content and use the OpenAI Tokenizer to count the tokens. This can help ensure that the project context fits within the token limits of your chosen language model.