s3-genindex/docs/DETAILS.md

# s3-genindex - Detailed Documentation

## Overview

s3-genindex is a program that generates HTML directory listings with file type icons, responsive design, and dark mode support for both local directories and S3-compatible storage systems.

## Features

- **Local Directory Indexing**: Recursive traversal of filesystem directories
- **S3-Compatible Storage**: Support for MinIO, AWS S3, and other S3-compatible systems
- **Hierarchical Structure**: Creates proper directory navigation for S3 buckets
- **File Type Detection**: Recognizes 100+ file extensions with appropriate icons
- **Responsive Design**: Works on desktop and mobile devices
- **Dark Mode**: Automatic dark mode support based on system preferences
- **Dry Run Mode**: Preview what would be generated without writing files
- **File Filtering**: Include/exclude files by glob patterns or regex
- **Symlink Support**: Special handling for symbolic links
- **Index File Control**: Show/hide index.html files in directory listings

## Installation

### From Source

```bash
git clone https://git.ipng.ch/ipng/s3-genindex.git
cd s3-genindex
make build
```

### Using Go Install

```bash
go install git.ipng.ch/ipng/s3-genindex/cmd/s3-genindex@latest
```

## Command Line Options

```
Usage: s3-genindex [OPTIONS]

  -d string
    	local directory to process
  -s3 string
    	S3 URL to process
  -f string
    	only include files matching glob (default "*")
  -i	show index.html files in directory listings
  -n	dry run: show what would be written without actually writing
  -v	verbosely list every processed file
  -x string
    	exclude files matching regular expression
```

**Note**: Either `-d <directory>` or `-s3 <url>` must be specified (mutually exclusive).

## Usage Examples

### Local Directory Processing (`-d`)

```bash
# Generate index.html for a local directory
s3-genindex -d /var/www/html

# Process with verbose output to see all files
s3-genindex -d /home/user/documents -v

# Dry run to preview what would be generated
s3-genindex -d /path/to/dir -n

# Show index.html files in directory listings
s3-genindex -d /var/www -i
```

### S3 Storage Processing (`-s3`)

```bash
# Basic S3 bucket processing (MinIO)
export AWS_ACCESS_KEY_ID="your-access-key"
export AWS_SECRET_ACCESS_KEY="your-secret-key"
s3-genindex -s3 http://minio.example.com:9000/my-bucket

# AWS S3 bucket processing
export AWS_ACCESS_KEY_ID="your-aws-key"
export AWS_SECRET_ACCESS_KEY="your-aws-secret"
s3-genindex -s3 https://s3.amazonaws.com/my-bucket

# S3 with verbose output and dry run
s3-genindex -s3 http://localhost:9000/test-bucket -v -n

# S3 processing with file filtering
s3-genindex -s3 http://minio.local:9000/logs -f "*.log"
```

### File Filtering Examples

```bash
# Include only specific file types
s3-genindex -d /var/log -f "*.log"
s3-genindex -s3 http://minio:9000/images -f "*.{jpg,png,gif}"

# Exclude build artifacts and temporary files
s3-genindex -d /home/dev/project -x "(build|dist|node_modules|__pycache__|\\.tmp)"

# Exclude version control and system files
s3-genindex -d /var/www -x "(\.git|\.svn|\.DS_Store|Thumbs\.db)"

# Complex filtering with multiple patterns
s3-genindex -d /data -f "*.{json,xml,csv}" -x "(backup|temp|cache)"
```

### Advanced Usage Scenarios

```bash
# Documentation site generation for local directory
s3-genindex -d /var/www/docs -i -v

# Log file indexing on S3 with size filtering
s3-genindex -s3 http://minio:9000/application-logs -f "*.log" -v

# Website asset indexing (excluding index files)
s3-genindex -d /var/www/assets -x "(index\.html|\.htaccess)"

# Backup verification with dry run
s3-genindex -s3 https://backup.s3.amazonaws.com/daily-backups -n -v

# Development file browsing with hidden files
s3-genindex -d /home/dev/src -i -x "(\.git|node_modules|vendor)"

# Media gallery generation
s3-genindex -d /var/media -f "*.{jpg,jpeg,png,gif,mp4,mov}" -i
```

### Integration Examples

```bash
# Automated documentation updates (cron job)
#!/bin/bash
export AWS_ACCESS_KEY_ID="docs-access-key"
export AWS_SECRET_ACCESS_KEY="docs-secret-key"
s3-genindex -s3 https://docs.s3.amazonaws.com/api-docs -v

# Local web server directory indexing
s3-genindex -d /var/www/html -i
nginx -s reload

# CI/CD artifact indexing
s3-genindex -s3 http://artifacts.internal:9000/build-artifacts -f "*.{tar.gz,zip}" -v

# Photo gallery with metadata
s3-genindex -d /var/photos -f "*.{jpg,jpeg,png,heic}" -i -v
```

## File Type Support

The tool recognizes and provides appropriate icons for:

### Programming Languages
- Go (`.go`)
- Python (`.py`, `.pyc`, `.pyo`)
- JavaScript/TypeScript (`.js`, `.ts`, `.json`)
- HTML/CSS (`.html`, `.htm`, `.css`, `.scss`)
- Shell scripts (`.sh`, `.bash`, `.bat`, `.ps1`)
- SQL (`.sql`)

### Documents
- PDF (`.pdf`)
- Text/Markdown (`.txt`, `.md`, `.markdown`)
- Office documents (`.doc`, `.docx`, `.xls`, `.xlsx`, `.ppt`, `.pptx`)
- CSV (`.csv`)

### Media Files
- Images (`.jpg`, `.png`, `.gif`, `.svg`, `.webp`)
- Videos (`.mp4`, `.mov`, `.avi`, `.webm`)
- Audio (`.mp3`, `.wav`, `.flac`, `.ogg`)

### Archives
- Zip files (`.zip`, `.tar`, `.gz`, `.7z`, `.rar`)
- Package files (`.deb`, `.rpm`, `.dmg`, `.pkg`)

### System Files
- Certificates (`.crt`, `.pem`, `.key`)
- Configuration files
- License files (LICENSE, README)

## HTML Output Features

### Responsive Design
- Mobile-friendly layout
- Collapsible columns on small screens
- Touch-friendly navigation

### Dark Mode
- Automatic detection of system preference
- Clean dark color scheme
- Proper contrast ratios

### File Information
- File sizes in human-readable format
- Last modified timestamps
- File type icons
- Sorting (directories first, then alphabetical)

### Navigation
- Parent directory links
- Breadcrumb-style navigation
- Clickable file/directory entries

## Project Structure

```
s3-genindex/
├── cmd/s3-genindex/        # Main application
│   ├── main.go            # CLI entry point
│   └── main_test.go       # CLI tests
├── internal/indexgen/      # Core library
│   ├── indexgen.go        # Main functionality
│   ├── indexgen_test.go   # Unit tests
│   └── integration_test.go # Integration tests
├── docs/                  # Documentation
├── Makefile              # Build automation
├── README.md             # Quick start guide
└── go.mod               # Go module definition
```

## Development

### Building

```bash
make build          # Build binary
make test           # Run tests
make test-coverage  # Run tests with coverage
make check          # Run all checks (fmt, vet, lint, test)
```

### Testing

The project includes comprehensive tests:

- **Unit tests**: Test individual functions and utilities
- **Integration tests**: Test complete directory processing workflows
- **Template tests**: Verify HTML template generation
- **CLI tests**: Test command-line argument parsing

Run tests with:

```bash
make test                    # Basic test run
make test-coverage          # With coverage report
go test -v ./...           # Verbose output
go test -short ./...       # Skip integration tests
```

### Code Quality

```bash
make fmt      # Format code
make vet      # Vet for issues
make lint     # Run golangci-lint (if installed)
make check    # Run all quality checks
```

## S3 Configuration

### Environment Variables for S3

When using S3 storage (`-s3` flag), the following environment variables are **required**:

- `AWS_ACCESS_KEY_ID`: Your S3 access key ID
- `AWS_SECRET_ACCESS_KEY`: Your S3 secret access key

### S3 URL Format

S3 URLs should follow this format:
```
http://host:port/bucket    # For MinIO or custom S3-compatible storage
https://host/bucket        # For HTTPS endpoints
```

Examples:
- `http://minio.example.com:9000/my-bucket`
- `https://s3.amazonaws.com/my-bucket`
- `http://localhost:9000/test-bucket`

### S3 Features

- **Hierarchical Processing**: Creates index.html files for each directory level in S3
- **Path-Style URLs**: Uses path-style S3 URLs for MinIO compatibility
- **Bucket Navigation**: Generates proper directory navigation within S3 buckets
- **No Parent Directory at Root**: Root bucket index doesn't show parent (..) link

## Configuration

No configuration files are needed. All options are provided via command-line arguments.

### Standard Environment Variables

The tool respects standard Go environment variables:
- `GOOS` and `GOARCH` for cross-compilation
- `GOPATH` and `GOBIN` for installation paths

## Comparison with Original

This Go version provides the same functionality as the original Python script with these improvements:

### Performance
- Faster execution, especially for large directory trees
- Lower memory usage
- Single binary deployment

### Maintenance
- Comprehensive test suite
- Type safety
- Better error handling
- Structured codebase

### Compatibility
- Identical HTML output
- Same command-line interface
- Cross-platform binary

## Troubleshooting

### Common Issues

**Local Directory Permission Errors**
```bash
# Ensure read permissions on target directory
chmod +r /path/to/directory
```

**S3 Connection Issues**
```bash
# Verify credentials are set
echo $AWS_ACCESS_KEY_ID
echo $AWS_SECRET_ACCESS_KEY

# Test connection with dry run
s3-genindex -s3 http://minio.example.com:9000/bucket -n -v

# Check S3 endpoint connectivity
curl http://minio.example.com:9000/
```

**S3 Permission Errors**
```bash
# Verify bucket access permissions
aws s3 ls s3://your-bucket/ --endpoint-url http://minio.example.com:9000

# Check if bucket exists and is accessible
s3-genindex -s3 http://minio.example.com:9000/bucket -v
```

**Large Directory/Bucket Processing**
```bash
# Use verbose mode to monitor progress
s3-genindex -d /large/directory -v
s3-genindex -s3 http://minio:9000/large-bucket -v

# Exclude unnecessary files to reduce processing time
s3-genindex -d /data -x "(\.git|node_modules|__pycache__|\.tmp)"
s3-genindex -s3 http://minio:9000/bucket -x "(backup|temp|cache)"

# Use dry run to estimate processing time
s3-genindex -s3 http://minio:9000/bucket -n
```

**Network Timeout Issues (S3)**
```bash
# For slow connections, use verbose mode to see progress
s3-genindex -s3 http://slow-endpoint:9000/bucket -v

# Test with smaller buckets first
s3-genindex -s3 http://endpoint:9000/small-test-bucket -n
```

### Debug Mode

Enable verbose output to see detailed processing:

```bash
# Local directory debugging
s3-genindex -d /path/to/debug -v

# S3 debugging with dry run
s3-genindex -s3 http://minio:9000/bucket -n -v

# Full S3 processing with verbose output
s3-genindex -s3 http://minio:9000/bucket -v
```

### S3-Specific Debugging

```bash
# Test S3 connectivity without processing
curl -I http://minio.example.com:9000/bucket/

# List S3 objects directly (if aws-cli is available)
aws s3 ls s3://bucket/ --endpoint-url http://minio.example.com:9000

# Verify S3 URL format
s3-genindex -s3 http://wrong-format -n  # Will show URL parsing errors
```

## License

Licensed under the Apache License 2.0. See original Python script for full license text.

## Contributing

1. Fork the repository
2. Create a feature branch
3. Add tests for new functionality
4. Run `make check` to ensure code quality
5. Submit a pull request

## Changelog

### v2.0.0 (Current)
- **S3 Support**: Complete S3-compatible storage support (MinIO, AWS S3)
- **Hierarchical S3 Processing**: Creates proper directory navigation for S3 buckets
- **Dry Run Mode**: Preview functionality with `-n` flag
- **Index File Control**: Show/hide index.html files with `-i` flag
- **Mutual Exclusive Flags**: Clean separation between `-d` and `-s3` modes
- **Enhanced Error Handling**: Better error messages and validation
- **Comprehensive Testing**: Extended test suite covering S3 functionality
- **URL Handling Fix**: Proper S3 navigation without URL encoding issues

### v1.0.0
- Initial Go rewrite from Python genindex.py
- Complete feature parity with Python version for local directories
- Comprehensive test suite
- Modern Go project structure
- Recursive directory processing
- File type detection and icons
- Responsive HTML design with dark mode support

## Acknowledgement

This tool was inspired by
[[index-html-generator](https://github.com/glowinthedark/index-html-generator)] on GitHub.