Refresh README

2025-12-03 12:27:14 +01:00
parent 16fa899b91
commit 60a149b669
2 changed files with 218 additions and 58 deletions
--- a/README.md
+++ b/README.md
@@ -1,6 +1,17 @@
 # s3-genindex

-Generate HTML directory indexes with file type icons and responsive design.
+Generate HTML directory indexes with file type icons and responsive design for local directories and S3-compatible storage.
+This is particularly useful for S3 buckets that are publicly readable.
+
+## Features
+
+- **Local directory indexing** with recursive traversal
+- **S3-compatible storage support** (MinIO, AWS S3, etc.)
+- **Hierarchical directory structure** for S3 buckets
+- **Responsive HTML design** with file type icons
+- **Dry run mode** for testing
+- **Flexible filtering** with glob patterns and regex exclusion
+- **Hidden file control** and index.html visibility options

 ## Install

@@ -8,17 +19,20 @@ Generate HTML directory indexes with file type icons and responsive design.
 go install git.ipng.ch/ipng/s3-genindex/cmd/s3-genindex@latest
 ```

-## Usage
+## Quick Start

 ```bash
-# Generate index.html in current directory
-s3-genindex
+# Local directory
+s3-genindex -d /path/to/dir

-# Generate recursively with custom output
-s3-genindex -r -o listing.html /path/to/dir
+# S3 bucket (requires AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY)
+s3-genindex -s3 http://minio.example.com:9000/bucket

-# Exclude files by regex
-s3-genindex -x "(build|node_modules|\.tmp)"
+# Dry run to see what would be generated
+s3-genindex -d /path/to/dir -n
+
+# Show index.html files in listings
+s3-genindex -d /path/to/dir -i
 ```

 ## Build
@@ -28,4 +42,4 @@ make build
 make test
 ```

-See [docs/DETAILS.md](docs/DETAILS.md) for complete documentation.
+See [docs/DETAILS.md](docs/DETAILS.md) for complete documentation and examples.
--- a/docs/DETAILS.md
+++ b/docs/DETAILS.md
@@ -2,18 +2,20 @@

 ## Overview

-s3-genindex is a Go rewrite of the original Python genindex.py script. It generates HTML directory listings with file type icons, responsive design, and dark mode support.
+s3-genindex is a Go rewrite of the original Python genindex.py script. It generates HTML directory listings with file type icons, responsive design, and dark mode support for both local directories and S3-compatible storage systems.

 ## Features

+- **Local Directory Indexing**: Recursive traversal of filesystem directories
+- **S3-Compatible Storage**: Support for MinIO, AWS S3, and other S3-compatible systems
+- **Hierarchical Structure**: Creates proper directory navigation for S3 buckets
 - **File Type Detection**: Recognizes 100+ file extensions with appropriate icons
 - **Responsive Design**: Works on desktop and mobile devices
 - **Dark Mode**: Automatic dark mode support based on system preferences
- **Recursive Processing**: Generate indexes for entire directory trees
- **File Filtering**: Include/exclude files by pattern or regex
+- **Dry Run Mode**: Preview what would be generated without writing files
+- **File Filtering**: Include/exclude files by glob patterns or regex
 - **Symlink Support**: Special handling for symbolic links
- **Custom Output**: Configurable output filename
- **Breadcrumb Navigation**: Parent directory navigation
+- **Index File Control**: Show/hide index.html files in directory listings

 ## Installation

@@ -34,66 +36,118 @@ go install git.ipng.ch/ipng/s3-genindex/cmd/s3-genindex@latest
 ## Command Line Options

 ```
-Usage: s3-genindex [OPTIONS] [directory]
+Usage: s3-genindex [OPTIONS]

-  -d	append output file to directory href
+  -d string
+    	local directory to process
+  -s3 string
+    	S3 URL to process
  -f string
    	only include files matching glob (default "*")
-  -i	include dot hidden files
-  -o string
-    	custom output file (default "index.html")
-  -r	recursively process nested dirs
+  -i	show index.html files in directory listings
+  -n	dry run: show what would be written without actually writing
  -v	verbosely list every processed file
  -x string
    	exclude files matching regular expression
 ```

+**Note**: Either `-d <directory>` or `-s3 <url>` must be specified (mutually exclusive).
+
 ## Usage Examples

-### Basic Usage
+### Local Directory Processing (`-d`)

 ```bash
-# Generate index.html in current directory
-s3-genindex
+# Generate index.html for a local directory
+s3-genindex -d /var/www/html

-# Generate index for specific directory
-s3-genindex /path/to/directory
+# Process with verbose output to see all files
+s3-genindex -d /home/user/documents -v

-# Generate with custom output filename
-s3-genindex -o listing.html
+# Dry run to preview what would be generated
+s3-genindex -d /path/to/dir -n
+
+# Show index.html files in directory listings
+s3-genindex -d /var/www -i
 ```

-### Recursive Processing
+### S3 Storage Processing (`-s3`)

 ```bash
-# Process directory tree recursively
-s3-genindex -r
+# Basic S3 bucket processing (MinIO)
+export AWS_ACCESS_KEY_ID="your-access-key"
+export AWS_SECRET_ACCESS_KEY="your-secret-key"
+s3-genindex -s3 http://minio.example.com:9000/my-bucket

-# Process recursively with verbose output
-s3-genindex -rv /var/www
+# AWS S3 bucket processing
+export AWS_ACCESS_KEY_ID="your-aws-key"
+export AWS_SECRET_ACCESS_KEY="your-aws-secret"
+s3-genindex -s3 https://s3.amazonaws.com/my-bucket
+
+# S3 with verbose output and dry run
+s3-genindex -s3 http://localhost:9000/test-bucket -v -n
+
+# S3 processing with file filtering
+s3-genindex -s3 http://minio.local:9000/logs -f "*.log"
 ```

-### File Filtering
+### File Filtering Examples

 ```bash
-# Include only Python files
-s3-genindex -f "*.py"
+# Include only specific file types
+s3-genindex -d /var/log -f "*.log"
+s3-genindex -s3 http://minio:9000/images -f "*.{jpg,png,gif}"

-# Exclude build artifacts and dependencies
-s3-genindex -x "(build|dist|node_modules|__pycache__|\\.tmp)"
+# Exclude build artifacts and temporary files
+s3-genindex -d /home/dev/project -x "(build|dist|node_modules|__pycache__|\\.tmp)"

-# Include hidden files
-s3-genindex -i
+# Exclude version control and system files
+s3-genindex -d /var/www -x "(\.git|\.svn|\.DS_Store|Thumbs\.db)"
+
+# Complex filtering with multiple patterns
+s3-genindex -d /data -f "*.{json,xml,csv}" -x "(backup|temp|cache)"
 ```

-### Advanced Usage
+### Advanced Usage Scenarios

 ```bash
-# Recursive with custom output and exclusions
-s3-genindex -r -o index.html -x "(\.git|\.svn|node_modules)" /var/www
+# Documentation site generation for local directory
+s3-genindex -d /var/www/docs -i -v

-# Verbose processing with directory appending
-s3-genindex -r -v -d /home/user/public
+# Log file indexing on S3 with size filtering
+s3-genindex -s3 http://minio:9000/application-logs -f "*.log" -v
+
+# Website asset indexing (excluding index files)
+s3-genindex -d /var/www/assets -x "(index\.html|\.htaccess)"
+
+# Backup verification with dry run
+s3-genindex -s3 https://backup.s3.amazonaws.com/daily-backups -n -v
+
+# Development file browsing with hidden files
+s3-genindex -d /home/dev/src -i -x "(\.git|node_modules|vendor)"
+
+# Media gallery generation
+s3-genindex -d /var/media -f "*.{jpg,jpeg,png,gif,mp4,mov}" -i
+```
+
+### Integration Examples
+
+```bash
+# Automated documentation updates (cron job)
+#!/bin/bash
+export AWS_ACCESS_KEY_ID="docs-access-key"
+export AWS_SECRET_ACCESS_KEY="docs-secret-key"
+s3-genindex -s3 https://docs.s3.amazonaws.com/api-docs -v
+
+# Local web server directory indexing
+s3-genindex -d /var/www/html -i
+nginx -s reload
+
+# CI/CD artifact indexing
+s3-genindex -s3 http://artifacts.internal:9000/build-artifacts -f "*.{tar.gz,zip}" -v
+
+# Photo gallery with metadata
+s3-genindex -d /var/photos -f "*.{jpg,jpeg,png,heic}" -i -v
 ```

 ## File Type Support
@@ -206,11 +260,40 @@ make lint     # Run golangci-lint (if installed)
 make check    # Run all quality checks
 ```

+## S3 Configuration
+
+### Environment Variables for S3
+
+When using S3 storage (`-s3` flag), the following environment variables are **required**:
+
+- `AWS_ACCESS_KEY_ID`: Your S3 access key ID
+- `AWS_SECRET_ACCESS_KEY`: Your S3 secret access key
+
+### S3 URL Format
+
+S3 URLs should follow this format:
+```
+http://host:port/bucket    # For MinIO or custom S3-compatible storage
+https://host/bucket        # For HTTPS endpoints
+```
+
+Examples:
+- `http://minio.example.com:9000/my-bucket`
+- `https://s3.amazonaws.com/my-bucket`
+- `http://localhost:9000/test-bucket`
+
+### S3 Features
+
+- **Hierarchical Processing**: Creates index.html files for each directory level in S3
+- **Path-Style URLs**: Uses path-style S3 URLs for MinIO compatibility
+- **Bucket Navigation**: Generates proper directory navigation within S3 buckets
+- **No Parent Directory at Root**: Root bucket index doesn't show parent (..) link
+
 ## Configuration

 No configuration files are needed. All options are provided via command-line arguments.

-### Environment Variables
+### Standard Environment Variables

 The tool respects standard Go environment variables:
 - `GOOS` and `GOARCH` for cross-compilation
@@ -240,25 +323,55 @@ This Go version provides the same functionality as the original Python script wi

 ### Common Issues

-**Permission Errors**
+**Local Directory Permission Errors**
 ```bash
 # Ensure read permissions on target directory
 chmod +r /path/to/directory
 ```

-**Large Directories**
+**S3 Connection Issues**
 ```bash
-# Use verbose mode to monitor progress
-s3-genindex -v /large/directory
+# Verify credentials are set
+echo $AWS_ACCESS_KEY_ID
+echo $AWS_SECRET_ACCESS_KEY

-# Exclude unnecessary files
-s3-genindex -x "(\.git|node_modules|__pycache__)"
+# Test connection with dry run
+s3-genindex -s3 http://minio.example.com:9000/bucket -n -v
+
+# Check S3 endpoint connectivity
+curl http://minio.example.com:9000/
 ```

-**Memory Usage**
+**S3 Permission Errors**
 ```bash
-# Process directories individually for very large trees
-for dir in */; do s3-genindex "$dir"; done
+# Verify bucket access permissions
+aws s3 ls s3://your-bucket/ --endpoint-url http://minio.example.com:9000
+
+# Check if bucket exists and is accessible
+s3-genindex -s3 http://minio.example.com:9000/bucket -v
+```
+
+**Large Directory/Bucket Processing**
+```bash
+# Use verbose mode to monitor progress
+s3-genindex -d /large/directory -v
+s3-genindex -s3 http://minio:9000/large-bucket -v
+
+# Exclude unnecessary files to reduce processing time
+s3-genindex -d /data -x "(\.git|node_modules|__pycache__|\.tmp)"
+s3-genindex -s3 http://minio:9000/bucket -x "(backup|temp|cache)"
+
+# Use dry run to estimate processing time
+s3-genindex -s3 http://minio:9000/bucket -n
+```
+
+**Network Timeout Issues (S3)**
+```bash
+# For slow connections, use verbose mode to see progress
+s3-genindex -s3 http://slow-endpoint:9000/bucket -v
+
+# Test with smaller buckets first
+s3-genindex -s3 http://endpoint:9000/small-test-bucket -n
 ```

 ### Debug Mode
@@ -266,7 +379,27 @@ for dir in */; do s3-genindex "$dir"; done
 Enable verbose output to see detailed processing:

 ```bash
-s3-genindex -v /path/to/debug
+# Local directory debugging
+s3-genindex -d /path/to/debug -v
+
+# S3 debugging with dry run
+s3-genindex -s3 http://minio:9000/bucket -n -v
+
+# Full S3 processing with verbose output
+s3-genindex -s3 http://minio:9000/bucket -v
+```
+
+### S3-Specific Debugging
+
+```bash
+# Test S3 connectivity without processing
+curl -I http://minio.example.com:9000/bucket/
+
+# List S3 objects directly (if aws-cli is available)
+aws s3 ls s3://bucket/ --endpoint-url http://minio.example.com:9000
+
+# Verify S3 URL format
+s3-genindex -s3 http://wrong-format -n  # Will show URL parsing errors
 ```

 ## License
@@ -283,8 +416,21 @@ Licensed under the Apache License 2.0. See original Python script for full licen

 ## Changelog

+### v2.0.0 (Current)
+- **S3 Support**: Complete S3-compatible storage support (MinIO, AWS S3)
+- **Hierarchical S3 Processing**: Creates proper directory navigation for S3 buckets
+- **Dry Run Mode**: Preview functionality with `-n` flag
+- **Index File Control**: Show/hide index.html files with `-i` flag
+- **Mutual Exclusive Flags**: Clean separation between `-d` and `-s3` modes
+- **Enhanced Error Handling**: Better error messages and validation
+- **Comprehensive Testing**: Extended test suite covering S3 functionality
+- **URL Handling Fix**: Proper S3 navigation without URL encoding issues
+
 ### v1.0.0
- Initial Go rewrite
- Complete feature parity with Python version
+- Initial Go rewrite from Python genindex.py
+- Complete feature parity with Python version for local directories
 - Comprehensive test suite
- Modern Go project structure
+- Modern Go project structure
+- Recursive directory processing
+- File type detection and icons
+- Responsive HTML design with dark mode support