diff --git a/README.md b/README.md index 20fe805..3ec1b37 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,17 @@ # s3-genindex -Generate HTML directory indexes with file type icons and responsive design. +Generate HTML directory indexes with file type icons and responsive design for local directories and S3-compatible storage. +This is particularly useful for S3 buckets that are publicly readable. + +## Features + +- **Local directory indexing** with recursive traversal +- **S3-compatible storage support** (MinIO, AWS S3, etc.) +- **Hierarchical directory structure** for S3 buckets +- **Responsive HTML design** with file type icons +- **Dry run mode** for testing +- **Flexible filtering** with glob patterns and regex exclusion +- **Hidden file control** and index.html visibility options ## Install @@ -8,17 +19,20 @@ Generate HTML directory indexes with file type icons and responsive design. go install git.ipng.ch/ipng/s3-genindex/cmd/s3-genindex@latest ``` -## Usage +## Quick Start ```bash -# Generate index.html in current directory -s3-genindex +# Local directory +s3-genindex -d /path/to/dir -# Generate recursively with custom output -s3-genindex -r -o listing.html /path/to/dir +# S3 bucket (requires AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY) +s3-genindex -s3 http://minio.example.com:9000/bucket -# Exclude files by regex -s3-genindex -x "(build|node_modules|\.tmp)" +# Dry run to see what would be generated +s3-genindex -d /path/to/dir -n + +# Show index.html files in listings +s3-genindex -d /path/to/dir -i ``` ## Build @@ -28,4 +42,4 @@ make build make test ``` -See [docs/DETAILS.md](docs/DETAILS.md) for complete documentation. \ No newline at end of file +See [docs/DETAILS.md](docs/DETAILS.md) for complete documentation and examples. diff --git a/docs/DETAILS.md b/docs/DETAILS.md index 8a75bcd..bfaf936 100644 --- a/docs/DETAILS.md +++ b/docs/DETAILS.md @@ -2,18 +2,20 @@ ## Overview -s3-genindex is a Go rewrite of the original Python genindex.py script. It generates HTML directory listings with file type icons, responsive design, and dark mode support. +s3-genindex is a Go rewrite of the original Python genindex.py script. It generates HTML directory listings with file type icons, responsive design, and dark mode support for both local directories and S3-compatible storage systems. ## Features +- **Local Directory Indexing**: Recursive traversal of filesystem directories +- **S3-Compatible Storage**: Support for MinIO, AWS S3, and other S3-compatible systems +- **Hierarchical Structure**: Creates proper directory navigation for S3 buckets - **File Type Detection**: Recognizes 100+ file extensions with appropriate icons - **Responsive Design**: Works on desktop and mobile devices - **Dark Mode**: Automatic dark mode support based on system preferences -- **Recursive Processing**: Generate indexes for entire directory trees -- **File Filtering**: Include/exclude files by pattern or regex +- **Dry Run Mode**: Preview what would be generated without writing files +- **File Filtering**: Include/exclude files by glob patterns or regex - **Symlink Support**: Special handling for symbolic links -- **Custom Output**: Configurable output filename -- **Breadcrumb Navigation**: Parent directory navigation +- **Index File Control**: Show/hide index.html files in directory listings ## Installation @@ -34,66 +36,118 @@ go install git.ipng.ch/ipng/s3-genindex/cmd/s3-genindex@latest ## Command Line Options ``` -Usage: s3-genindex [OPTIONS] [directory] +Usage: s3-genindex [OPTIONS] - -d append output file to directory href + -d string + local directory to process + -s3 string + S3 URL to process -f string only include files matching glob (default "*") - -i include dot hidden files - -o string - custom output file (default "index.html") - -r recursively process nested dirs + -i show index.html files in directory listings + -n dry run: show what would be written without actually writing -v verbosely list every processed file -x string exclude files matching regular expression ``` +**Note**: Either `-d ` or `-s3 ` must be specified (mutually exclusive). + ## Usage Examples -### Basic Usage +### Local Directory Processing (`-d`) ```bash -# Generate index.html in current directory -s3-genindex +# Generate index.html for a local directory +s3-genindex -d /var/www/html -# Generate index for specific directory -s3-genindex /path/to/directory +# Process with verbose output to see all files +s3-genindex -d /home/user/documents -v -# Generate with custom output filename -s3-genindex -o listing.html +# Dry run to preview what would be generated +s3-genindex -d /path/to/dir -n + +# Show index.html files in directory listings +s3-genindex -d /var/www -i ``` -### Recursive Processing +### S3 Storage Processing (`-s3`) ```bash -# Process directory tree recursively -s3-genindex -r +# Basic S3 bucket processing (MinIO) +export AWS_ACCESS_KEY_ID="your-access-key" +export AWS_SECRET_ACCESS_KEY="your-secret-key" +s3-genindex -s3 http://minio.example.com:9000/my-bucket -# Process recursively with verbose output -s3-genindex -rv /var/www +# AWS S3 bucket processing +export AWS_ACCESS_KEY_ID="your-aws-key" +export AWS_SECRET_ACCESS_KEY="your-aws-secret" +s3-genindex -s3 https://s3.amazonaws.com/my-bucket + +# S3 with verbose output and dry run +s3-genindex -s3 http://localhost:9000/test-bucket -v -n + +# S3 processing with file filtering +s3-genindex -s3 http://minio.local:9000/logs -f "*.log" ``` -### File Filtering +### File Filtering Examples ```bash -# Include only Python files -s3-genindex -f "*.py" +# Include only specific file types +s3-genindex -d /var/log -f "*.log" +s3-genindex -s3 http://minio:9000/images -f "*.{jpg,png,gif}" -# Exclude build artifacts and dependencies -s3-genindex -x "(build|dist|node_modules|__pycache__|\\.tmp)" +# Exclude build artifacts and temporary files +s3-genindex -d /home/dev/project -x "(build|dist|node_modules|__pycache__|\\.tmp)" -# Include hidden files -s3-genindex -i +# Exclude version control and system files +s3-genindex -d /var/www -x "(\.git|\.svn|\.DS_Store|Thumbs\.db)" + +# Complex filtering with multiple patterns +s3-genindex -d /data -f "*.{json,xml,csv}" -x "(backup|temp|cache)" ``` -### Advanced Usage +### Advanced Usage Scenarios ```bash -# Recursive with custom output and exclusions -s3-genindex -r -o index.html -x "(\.git|\.svn|node_modules)" /var/www +# Documentation site generation for local directory +s3-genindex -d /var/www/docs -i -v -# Verbose processing with directory appending -s3-genindex -r -v -d /home/user/public +# Log file indexing on S3 with size filtering +s3-genindex -s3 http://minio:9000/application-logs -f "*.log" -v + +# Website asset indexing (excluding index files) +s3-genindex -d /var/www/assets -x "(index\.html|\.htaccess)" + +# Backup verification with dry run +s3-genindex -s3 https://backup.s3.amazonaws.com/daily-backups -n -v + +# Development file browsing with hidden files +s3-genindex -d /home/dev/src -i -x "(\.git|node_modules|vendor)" + +# Media gallery generation +s3-genindex -d /var/media -f "*.{jpg,jpeg,png,gif,mp4,mov}" -i +``` + +### Integration Examples + +```bash +# Automated documentation updates (cron job) +#!/bin/bash +export AWS_ACCESS_KEY_ID="docs-access-key" +export AWS_SECRET_ACCESS_KEY="docs-secret-key" +s3-genindex -s3 https://docs.s3.amazonaws.com/api-docs -v + +# Local web server directory indexing +s3-genindex -d /var/www/html -i +nginx -s reload + +# CI/CD artifact indexing +s3-genindex -s3 http://artifacts.internal:9000/build-artifacts -f "*.{tar.gz,zip}" -v + +# Photo gallery with metadata +s3-genindex -d /var/photos -f "*.{jpg,jpeg,png,heic}" -i -v ``` ## File Type Support @@ -206,11 +260,40 @@ make lint # Run golangci-lint (if installed) make check # Run all quality checks ``` +## S3 Configuration + +### Environment Variables for S3 + +When using S3 storage (`-s3` flag), the following environment variables are **required**: + +- `AWS_ACCESS_KEY_ID`: Your S3 access key ID +- `AWS_SECRET_ACCESS_KEY`: Your S3 secret access key + +### S3 URL Format + +S3 URLs should follow this format: +``` +http://host:port/bucket # For MinIO or custom S3-compatible storage +https://host/bucket # For HTTPS endpoints +``` + +Examples: +- `http://minio.example.com:9000/my-bucket` +- `https://s3.amazonaws.com/my-bucket` +- `http://localhost:9000/test-bucket` + +### S3 Features + +- **Hierarchical Processing**: Creates index.html files for each directory level in S3 +- **Path-Style URLs**: Uses path-style S3 URLs for MinIO compatibility +- **Bucket Navigation**: Generates proper directory navigation within S3 buckets +- **No Parent Directory at Root**: Root bucket index doesn't show parent (..) link + ## Configuration No configuration files are needed. All options are provided via command-line arguments. -### Environment Variables +### Standard Environment Variables The tool respects standard Go environment variables: - `GOOS` and `GOARCH` for cross-compilation @@ -240,25 +323,55 @@ This Go version provides the same functionality as the original Python script wi ### Common Issues -**Permission Errors** +**Local Directory Permission Errors** ```bash # Ensure read permissions on target directory chmod +r /path/to/directory ``` -**Large Directories** +**S3 Connection Issues** ```bash -# Use verbose mode to monitor progress -s3-genindex -v /large/directory +# Verify credentials are set +echo $AWS_ACCESS_KEY_ID +echo $AWS_SECRET_ACCESS_KEY -# Exclude unnecessary files -s3-genindex -x "(\.git|node_modules|__pycache__)" +# Test connection with dry run +s3-genindex -s3 http://minio.example.com:9000/bucket -n -v + +# Check S3 endpoint connectivity +curl http://minio.example.com:9000/ ``` -**Memory Usage** +**S3 Permission Errors** ```bash -# Process directories individually for very large trees -for dir in */; do s3-genindex "$dir"; done +# Verify bucket access permissions +aws s3 ls s3://your-bucket/ --endpoint-url http://minio.example.com:9000 + +# Check if bucket exists and is accessible +s3-genindex -s3 http://minio.example.com:9000/bucket -v +``` + +**Large Directory/Bucket Processing** +```bash +# Use verbose mode to monitor progress +s3-genindex -d /large/directory -v +s3-genindex -s3 http://minio:9000/large-bucket -v + +# Exclude unnecessary files to reduce processing time +s3-genindex -d /data -x "(\.git|node_modules|__pycache__|\.tmp)" +s3-genindex -s3 http://minio:9000/bucket -x "(backup|temp|cache)" + +# Use dry run to estimate processing time +s3-genindex -s3 http://minio:9000/bucket -n +``` + +**Network Timeout Issues (S3)** +```bash +# For slow connections, use verbose mode to see progress +s3-genindex -s3 http://slow-endpoint:9000/bucket -v + +# Test with smaller buckets first +s3-genindex -s3 http://endpoint:9000/small-test-bucket -n ``` ### Debug Mode @@ -266,7 +379,27 @@ for dir in */; do s3-genindex "$dir"; done Enable verbose output to see detailed processing: ```bash -s3-genindex -v /path/to/debug +# Local directory debugging +s3-genindex -d /path/to/debug -v + +# S3 debugging with dry run +s3-genindex -s3 http://minio:9000/bucket -n -v + +# Full S3 processing with verbose output +s3-genindex -s3 http://minio:9000/bucket -v +``` + +### S3-Specific Debugging + +```bash +# Test S3 connectivity without processing +curl -I http://minio.example.com:9000/bucket/ + +# List S3 objects directly (if aws-cli is available) +aws s3 ls s3://bucket/ --endpoint-url http://minio.example.com:9000 + +# Verify S3 URL format +s3-genindex -s3 http://wrong-format -n # Will show URL parsing errors ``` ## License @@ -283,8 +416,21 @@ Licensed under the Apache License 2.0. See original Python script for full licen ## Changelog +### v2.0.0 (Current) +- **S3 Support**: Complete S3-compatible storage support (MinIO, AWS S3) +- **Hierarchical S3 Processing**: Creates proper directory navigation for S3 buckets +- **Dry Run Mode**: Preview functionality with `-n` flag +- **Index File Control**: Show/hide index.html files with `-i` flag +- **Mutual Exclusive Flags**: Clean separation between `-d` and `-s3` modes +- **Enhanced Error Handling**: Better error messages and validation +- **Comprehensive Testing**: Extended test suite covering S3 functionality +- **URL Handling Fix**: Proper S3 navigation without URL encoding issues + ### v1.0.0 -- Initial Go rewrite -- Complete feature parity with Python version +- Initial Go rewrite from Python genindex.py +- Complete feature parity with Python version for local directories - Comprehensive test suite -- Modern Go project structure \ No newline at end of file +- Modern Go project structure +- Recursive directory processing +- File type detection and icons +- Responsive HTML design with dark mode support \ No newline at end of file