s3-genindex/docs/DETAILS.md

s3-genindex - Detailed Documentation

Overview

s3-genindex is a program that generates HTML directory listings with file type icons, responsive design, and dark mode support for both local directories and S3-compatible storage systems.

Features

• Local Directory Indexing: Recursive traversal of filesystem directories
• S3-Compatible Storage: Support for MinIO, AWS S3, and other S3-compatible systems
• Hierarchical Structure: Creates proper directory navigation for S3 buckets
• File Type Detection: Recognizes 100+ file extensions with appropriate icons
• Responsive Design: Works on desktop and mobile devices
• Dark Mode: Automatic dark mode support based on system preferences
• Dry Run Mode: Preview what would be generated without writing files
• File Filtering: Include/exclude files by glob patterns or regex
• Symlink Support: Special handling for symbolic links
• Index File Control: Show/hide index.html files in directory listings

Installation

From Source

git clone https://git.ipng.ch/ipng/s3-genindex.git
cd s3-genindex
make build

Using Go Install

go install git.ipng.ch/ipng/s3-genindex/cmd/s3-genindex@latest

Command Line Options

Usage: s3-genindex [OPTIONS]

  -d string
    	local directory to process
  -s3 string
    	S3 URL to process
  -f string
    	only include files matching glob (default "*")
  -i	show index.html files in directory listings
  -n	dry run: show what would be written without actually writing
  -v	verbosely list every processed file
  -x string
    	exclude files matching regular expression

Note: Either -d <directory> or -s3 <url> must be specified (mutually exclusive).

Usage Examples

Local Directory Processing (-d)

# Generate index.html for a local directory
s3-genindex -d /var/www/html

# Process with verbose output to see all files
s3-genindex -d /home/user/documents -v

# Dry run to preview what would be generated
s3-genindex -d /path/to/dir -n

# Show index.html files in directory listings
s3-genindex -d /var/www -i

S3 Storage Processing (-s3)

# Basic S3 bucket processing (MinIO)
export AWS_ACCESS_KEY_ID="your-access-key"
export AWS_SECRET_ACCESS_KEY="your-secret-key"
s3-genindex -s3 http://minio.example.com:9000/my-bucket

# AWS S3 bucket processing
export AWS_ACCESS_KEY_ID="your-aws-key"
export AWS_SECRET_ACCESS_KEY="your-aws-secret"
s3-genindex -s3 https://s3.amazonaws.com/my-bucket

# S3 with verbose output and dry run
s3-genindex -s3 http://localhost:9000/test-bucket -v -n

# S3 processing with file filtering
s3-genindex -s3 http://minio.local:9000/logs -f "*.log"

File Filtering Examples

# Include only specific file types
s3-genindex -d /var/log -f "*.log"
s3-genindex -s3 http://minio:9000/images -f "*.{jpg,png,gif}"

# Exclude build artifacts and temporary files
s3-genindex -d /home/dev/project -x "(build|dist|node_modules|__pycache__|\\.tmp)"

# Exclude version control and system files
s3-genindex -d /var/www -x "(\.git|\.svn|\.DS_Store|Thumbs\.db)"

# Complex filtering with multiple patterns
s3-genindex -d /data -f "*.{json,xml,csv}" -x "(backup|temp|cache)"

Advanced Usage Scenarios

# Documentation site generation for local directory
s3-genindex -d /var/www/docs -i -v

# Log file indexing on S3 with size filtering
s3-genindex -s3 http://minio:9000/application-logs -f "*.log" -v

# Website asset indexing (excluding index files)
s3-genindex -d /var/www/assets -x "(index\.html|\.htaccess)"

# Backup verification with dry run
s3-genindex -s3 https://backup.s3.amazonaws.com/daily-backups -n -v

# Development file browsing with hidden files
s3-genindex -d /home/dev/src -i -x "(\.git|node_modules|vendor)"

# Media gallery generation
s3-genindex -d /var/media -f "*.{jpg,jpeg,png,gif,mp4,mov}" -i

Integration Examples

# Automated documentation updates (cron job)
#!/bin/bash
export AWS_ACCESS_KEY_ID="docs-access-key"
export AWS_SECRET_ACCESS_KEY="docs-secret-key"
s3-genindex -s3 https://docs.s3.amazonaws.com/api-docs -v

# Local web server directory indexing
s3-genindex -d /var/www/html -i
nginx -s reload

# CI/CD artifact indexing
s3-genindex -s3 http://artifacts.internal:9000/build-artifacts -f "*.{tar.gz,zip}" -v

# Photo gallery with metadata
s3-genindex -d /var/photos -f "*.{jpg,jpeg,png,heic}" -i -v

File Type Support

The tool recognizes and provides appropriate icons for:

Programming Languages

• Go (.go)
• Python (.py, .pyc, .pyo)
• JavaScript/TypeScript (.js, .ts, .json)
• HTML/CSS (.html, .htm, .css, .scss)
• Shell scripts (.sh, .bash, .bat, .ps1)
• SQL (.sql)

Documents

• PDF (.pdf)
• Text/Markdown (.txt, .md, .markdown)
• Office documents (.doc, .docx, .xls, .xlsx, .ppt, .pptx)
• CSV (.csv)

Media Files

• Images (.jpg, .png, .gif, .svg, .webp)
• Videos (.mp4, .mov, .avi, .webm)
• Audio (.mp3, .wav, .flac, .ogg)

Archives

• Zip files (.zip, .tar, .gz, .7z, .rar)
• Package files (.deb, .rpm, .dmg, .pkg)

System Files

• Certificates (.crt, .pem, .key)
• Configuration files
• License files (LICENSE, README)

HTML Output Features

Responsive Design

• Mobile-friendly layout
• Collapsible columns on small screens
• Touch-friendly navigation

Dark Mode

• Automatic detection of system preference
• Clean dark color scheme
• Proper contrast ratios

File Information

• File sizes in human-readable format
• Last modified timestamps
• File type icons
• Sorting (directories first, then alphabetical)

Navigation

• Parent directory links
• Breadcrumb-style navigation
• Clickable file/directory entries

Project Structure

s3-genindex/
├── cmd/s3-genindex/        # Main application
│   ├── main.go            # CLI entry point
│   └── main_test.go       # CLI tests
├── internal/indexgen/      # Core library
│   ├── indexgen.go        # Main functionality
│   ├── indexgen_test.go   # Unit tests
│   └── integration_test.go # Integration tests
├── docs/                  # Documentation
├── Makefile              # Build automation
├── README.md             # Quick start guide
└── go.mod               # Go module definition

Development

Building

make build          # Build binary
make test           # Run tests
make test-coverage  # Run tests with coverage
make check          # Run all checks (fmt, vet, lint, test)

Testing

The project includes comprehensive tests:

• Unit tests: Test individual functions and utilities
• Integration tests: Test complete directory processing workflows
• Template tests: Verify HTML template generation
• CLI tests: Test command-line argument parsing

Run tests with:

make test                    # Basic test run
make test-coverage          # With coverage report
go test -v ./...           # Verbose output
go test -short ./...       # Skip integration tests

Code Quality

make fmt      # Format code
make vet      # Vet for issues
make lint     # Run golangci-lint (if installed)
make check    # Run all quality checks

S3 Configuration

Environment Variables for S3

When using S3 storage (-s3 flag), the following environment variables are required:

• AWS_ACCESS_KEY_ID: Your S3 access key ID
• AWS_SECRET_ACCESS_KEY: Your S3 secret access key

S3 URL Format

S3 URLs should follow this format:

http://host:port/bucket    # For MinIO or custom S3-compatible storage
https://host/bucket        # For HTTPS endpoints

Examples:

• http://minio.example.com:9000/my-bucket
• https://s3.amazonaws.com/my-bucket
• http://localhost:9000/test-bucket

S3 Features

• Hierarchical Processing: Creates index.html files for each directory level in S3
• Path-Style URLs: Uses path-style S3 URLs for MinIO compatibility
• Bucket Navigation: Generates proper directory navigation within S3 buckets
• No Parent Directory at Root: Root bucket index doesn't show parent (..) link

Configuration

No configuration files are needed. All options are provided via command-line arguments.

Standard Environment Variables

The tool respects standard Go environment variables:

• GOOS and GOARCH for cross-compilation
• GOPATH and GOBIN for installation paths

Comparison with Original

This Go version provides the same functionality as the original Python script with these improvements:

Performance

• Faster execution, especially for large directory trees
• Lower memory usage
• Single binary deployment

Maintenance

• Comprehensive test suite
• Type safety
• Better error handling
• Structured codebase

Compatibility

• Identical HTML output
• Same command-line interface
• Cross-platform binary

Troubleshooting

Common Issues

Local Directory Permission Errors

# Ensure read permissions on target directory
chmod +r /path/to/directory

S3 Connection Issues

# Verify credentials are set
echo $AWS_ACCESS_KEY_ID
echo $AWS_SECRET_ACCESS_KEY

# Test connection with dry run
s3-genindex -s3 http://minio.example.com:9000/bucket -n -v

# Check S3 endpoint connectivity
curl http://minio.example.com:9000/

S3 Permission Errors

# Verify bucket access permissions
aws s3 ls s3://your-bucket/ --endpoint-url http://minio.example.com:9000

# Check if bucket exists and is accessible
s3-genindex -s3 http://minio.example.com:9000/bucket -v

Large Directory/Bucket Processing

# Use verbose mode to monitor progress
s3-genindex -d /large/directory -v
s3-genindex -s3 http://minio:9000/large-bucket -v

# Exclude unnecessary files to reduce processing time
s3-genindex -d /data -x "(\.git|node_modules|__pycache__|\.tmp)"
s3-genindex -s3 http://minio:9000/bucket -x "(backup|temp|cache)"

# Use dry run to estimate processing time
s3-genindex -s3 http://minio:9000/bucket -n

Network Timeout Issues (S3)

# For slow connections, use verbose mode to see progress
s3-genindex -s3 http://slow-endpoint:9000/bucket -v

# Test with smaller buckets first
s3-genindex -s3 http://endpoint:9000/small-test-bucket -n

Debug Mode

Enable verbose output to see detailed processing:

# Local directory debugging
s3-genindex -d /path/to/debug -v

# S3 debugging with dry run
s3-genindex -s3 http://minio:9000/bucket -n -v

# Full S3 processing with verbose output
s3-genindex -s3 http://minio:9000/bucket -v

S3-Specific Debugging

# Test S3 connectivity without processing
curl -I http://minio.example.com:9000/bucket/

# List S3 objects directly (if aws-cli is available)
aws s3 ls s3://bucket/ --endpoint-url http://minio.example.com:9000

# Verify S3 URL format
s3-genindex -s3 http://wrong-format -n  # Will show URL parsing errors

License

Licensed under the Apache License 2.0. See original Python script for full license text.

Contributing

1. Fork the repository
2. Create a feature branch
3. Add tests for new functionality
4. Run make check to ensure code quality
5. Submit a pull request

Changelog

v2.0.0 (Current)

• S3 Support: Complete S3-compatible storage support (MinIO, AWS S3)
• Hierarchical S3 Processing: Creates proper directory navigation for S3 buckets
• Dry Run Mode: Preview functionality with -n flag
• Index File Control: Show/hide index.html files with -i flag
• Mutual Exclusive Flags: Clean separation between -d and -s3 modes
• Enhanced Error Handling: Better error messages and validation
• Comprehensive Testing: Extended test suite covering S3 functionality
• URL Handling Fix: Proper S3 navigation without URL encoding issues

v1.0.0

• Initial Go rewrite from Python genindex.py
• Complete feature parity with Python version for local directories
• Comprehensive test suite
• Modern Go project structure
• Recursive directory processing
• File type detection and icons
• Responsive HTML design with dark mode support

Acknowledgement

This tool was inspired by [index-html-generator] on GitHub.