ipng/s3-genindex
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Refresh README

Browse Source
This commit is contained in:
Pim van Pelt
2025-12-03 12:27:14 +01:00
parent 16fa899b91
commit 60a149b669
2 changed files with 218 additions and 58 deletions
Download Patch File Download Diff File Expand all files Collapse all files

244
docs/DETAILS.md
View File

@@ -2,18 +2,20 @@
## Overview
s3-genindex is a Go rewrite of the original Python genindex.py script. It generates HTML directory listings with file type icons, responsive design, and dark mode support.
s3-genindex is a Go rewrite of the original Python genindex.py script. It generates HTML directory listings with file type icons, responsive design, and dark mode support for both local directories and S3-compatible storage systems.
## Features
- **Local Directory Indexing**: Recursive traversal of filesystem directories
- **S3-Compatible Storage**: Support for MinIO, AWS S3, and other S3-compatible systems
- **Hierarchical Structure**: Creates proper directory navigation for S3 buckets
- **File Type Detection**: Recognizes 100+ file extensions with appropriate icons
- **Responsive Design**: Works on desktop and mobile devices
- **Dark Mode**: Automatic dark mode support based on system preferences
- **Recursive Processing**: Generate indexes for entire directory trees
- **File Filtering**: Include/exclude files by pattern or regex
- **Dry Run Mode**: Preview what would be generated without writing files
- **File Filtering**: Include/exclude files by glob patterns or regex
- **Symlink Support**: Special handling for symbolic links
- **Custom Output**: Configurable output filename
- **Breadcrumb Navigation**: Parent directory navigation
- **Index File Control**: Show/hide index.html files in directory listings
## Installation
@@ -34,66 +36,118 @@ go install git.ipng.ch/ipng/s3-genindex/cmd/s3-genindex@latest
## Command Line Options
```
Usage: s3-genindex [OPTIONS] [directory]
Usage: s3-genindex [OPTIONS]
-d append output file to directory href
-d string
local directory to process
-s3 string
S3 URL to process
-f string
only include files matching glob (default "*")
-i include dot hidden files
-o string
custom output file (default "index.html")
-r recursively process nested dirs
-i show index.html files in directory listings
-n dry run: show what would be written without actually writing
-v verbosely list every processed file
-x string
exclude files matching regular expression
```
**Note**: Either `-d <directory>` or `-s3 <url>` must be specified (mutually exclusive).
## Usage Examples
### Basic Usage
### Local Directory Processing (`-d`)
```bash
# Generate index.html in current directory
s3-genindex
# Generate index.html for a local directory
s3-genindex -d /var/www/html
# Generate index for specific directory
s3-genindex /path/to/directory
# Process with verbose output to see all files
s3-genindex -d /home/user/documents -v
# Generate with custom output filename
s3-genindex -o listing.html
# Dry run to preview what would be generated
s3-genindex -d /path/to/dir -n
# Show index.html files in directory listings
s3-genindex -d /var/www -i
```
### Recursive Processing
### S3 Storage Processing (`-s3`)
```bash
# Process directory tree recursively
s3-genindex -r
# Basic S3 bucket processing (MinIO)
export AWS_ACCESS_KEY_ID="your-access-key"
export AWS_SECRET_ACCESS_KEY="your-secret-key"
s3-genindex -s3 http://minio.example.com:9000/my-bucket
# Process recursively with verbose output
s3-genindex -rv /var/www
# AWS S3 bucket processing
export AWS_ACCESS_KEY_ID="your-aws-key"
export AWS_SECRET_ACCESS_KEY="your-aws-secret"
s3-genindex -s3 https://s3.amazonaws.com/my-bucket
# S3 with verbose output and dry run
s3-genindex -s3 http://localhost:9000/test-bucket -v -n
# S3 processing with file filtering
s3-genindex -s3 http://minio.local:9000/logs -f "*.log"
```
### File Filtering
### File Filtering Examples
```bash
# Include only Python files
s3-genindex -f "*.py"
# Include only specific file types
s3-genindex -d /var/log -f "*.log"
s3-genindex -s3 http://minio:9000/images -f "*.{jpg,png,gif}"
# Exclude build artifacts and dependencies
s3-genindex -x "(build|dist|node_modules|__pycache__|\\.tmp)"
# Exclude build artifacts and temporary files
s3-genindex -d /home/dev/project -x "(build|dist|node_modules|__pycache__|\\.tmp)"
# Include hidden files
s3-genindex -i
# Exclude version control and system files
s3-genindex -d /var/www -x "(\.git|\.svn|\.DS_Store|Thumbs\.db)"
# Complex filtering with multiple patterns
s3-genindex -d /data -f "*.{json,xml,csv}" -x "(backup|temp|cache)"
```
### Advanced Usage
### Advanced Usage Scenarios
```bash
# Recursive with custom output and exclusions
s3-genindex -r -o index.html -x "(\.git|\.svn|node_modules)" /var/www
# Documentation site generation for local directory
s3-genindex -d /var/www/docs -i -v
# Verbose processing with directory appending
s3-genindex -r -v -d /home/user/public
# Log file indexing on S3 with size filtering
s3-genindex -s3 http://minio:9000/application-logs -f "*.log" -v
# Website asset indexing (excluding index files)
s3-genindex -d /var/www/assets -x "(index\.html|\.htaccess)"
# Backup verification with dry run
s3-genindex -s3 https://backup.s3.amazonaws.com/daily-backups -n -v
# Development file browsing with hidden files
s3-genindex -d /home/dev/src -i -x "(\.git|node_modules|vendor)"
# Media gallery generation
s3-genindex -d /var/media -f "*.{jpg,jpeg,png,gif,mp4,mov}" -i
```
### Integration Examples
```bash
# Automated documentation updates (cron job)
#!/bin/bash
export AWS_ACCESS_KEY_ID="docs-access-key"
export AWS_SECRET_ACCESS_KEY="docs-secret-key"
s3-genindex -s3 https://docs.s3.amazonaws.com/api-docs -v
# Local web server directory indexing
s3-genindex -d /var/www/html -i
nginx -s reload
# CI/CD artifact indexing
s3-genindex -s3 http://artifacts.internal:9000/build-artifacts -f "*.{tar.gz,zip}" -v
# Photo gallery with metadata
s3-genindex -d /var/photos -f "*.{jpg,jpeg,png,heic}" -i -v
```
## File Type Support
@@ -206,11 +260,40 @@ make lint # Run golangci-lint (if installed)
make check # Run all quality checks
```
## S3 Configuration
### Environment Variables for S3
When using S3 storage (`-s3` flag), the following environment variables are **required**:
- `AWS_ACCESS_KEY_ID`: Your S3 access key ID
- `AWS_SECRET_ACCESS_KEY`: Your S3 secret access key
### S3 URL Format
S3 URLs should follow this format:
```
http://host:port/bucket # For MinIO or custom S3-compatible storage
https://host/bucket # For HTTPS endpoints
```
Examples:
- `http://minio.example.com:9000/my-bucket`
- `https://s3.amazonaws.com/my-bucket`
- `http://localhost:9000/test-bucket`
### S3 Features
- **Hierarchical Processing**: Creates index.html files for each directory level in S3
- **Path-Style URLs**: Uses path-style S3 URLs for MinIO compatibility
- **Bucket Navigation**: Generates proper directory navigation within S3 buckets
- **No Parent Directory at Root**: Root bucket index doesn't show parent (..) link
## Configuration
No configuration files are needed. All options are provided via command-line arguments.
### Environment Variables
### Standard Environment Variables
The tool respects standard Go environment variables:
- `GOOS` and `GOARCH` for cross-compilation
@@ -240,25 +323,55 @@ This Go version provides the same functionality as the original Python script wi
### Common Issues
**Permission Errors**
**Local Directory Permission Errors**
```bash
# Ensure read permissions on target directory
chmod +r /path/to/directory
```
**Large Directories**
**S3 Connection Issues**
```bash
# Use verbose mode to monitor progress
s3-genindex -v /large/directory
# Verify credentials are set
echo $AWS_ACCESS_KEY_ID
echo $AWS_SECRET_ACCESS_KEY
# Exclude unnecessary files
s3-genindex -x "(\.git|node_modules|__pycache__)"
# Test connection with dry run
s3-genindex -s3 http://minio.example.com:9000/bucket -n -v
# Check S3 endpoint connectivity
curl http://minio.example.com:9000/
```
**Memory Usage**
**S3 Permission Errors**
```bash
# Process directories individually for very large trees
for dir in */; do s3-genindex "$dir"; done
# Verify bucket access permissions
aws s3 ls s3://your-bucket/ --endpoint-url http://minio.example.com:9000
# Check if bucket exists and is accessible
s3-genindex -s3 http://minio.example.com:9000/bucket -v
```
**Large Directory/Bucket Processing**
```bash
# Use verbose mode to monitor progress
s3-genindex -d /large/directory -v
s3-genindex -s3 http://minio:9000/large-bucket -v
# Exclude unnecessary files to reduce processing time
s3-genindex -d /data -x "(\.git|node_modules|__pycache__|\.tmp)"
s3-genindex -s3 http://minio:9000/bucket -x "(backup|temp|cache)"
# Use dry run to estimate processing time
s3-genindex -s3 http://minio:9000/bucket -n
```
**Network Timeout Issues (S3)**
```bash
# For slow connections, use verbose mode to see progress
s3-genindex -s3 http://slow-endpoint:9000/bucket -v
# Test with smaller buckets first
s3-genindex -s3 http://endpoint:9000/small-test-bucket -n
```
### Debug Mode
@@ -266,7 +379,27 @@ for dir in */; do s3-genindex "$dir"; done
Enable verbose output to see detailed processing:
```bash
s3-genindex -v /path/to/debug
# Local directory debugging
s3-genindex -d /path/to/debug -v
# S3 debugging with dry run
s3-genindex -s3 http://minio:9000/bucket -n -v
# Full S3 processing with verbose output
s3-genindex -s3 http://minio:9000/bucket -v
```
### S3-Specific Debugging
```bash
# Test S3 connectivity without processing
curl -I http://minio.example.com:9000/bucket/
# List S3 objects directly (if aws-cli is available)
aws s3 ls s3://bucket/ --endpoint-url http://minio.example.com:9000
# Verify S3 URL format
s3-genindex -s3 http://wrong-format -n # Will show URL parsing errors
```
## License
@@ -283,8 +416,21 @@ Licensed under the Apache License 2.0. See original Python script for full licen
## Changelog
### v2.0.0 (Current)
- **S3 Support**: Complete S3-compatible storage support (MinIO, AWS S3)
- **Hierarchical S3 Processing**: Creates proper directory navigation for S3 buckets
- **Dry Run Mode**: Preview functionality with `-n` flag
- **Index File Control**: Show/hide index.html files with `-i` flag
- **Mutual Exclusive Flags**: Clean separation between `-d` and `-s3` modes
- **Enhanced Error Handling**: Better error messages and validation
- **Comprehensive Testing**: Extended test suite covering S3 functionality
- **URL Handling Fix**: Proper S3 navigation without URL encoding issues
### v1.0.0
- Initial Go rewrite
- Complete feature parity with Python version
- Initial Go rewrite from Python genindex.py
- Complete feature parity with Python version for local directories
- Comprehensive test suite
- Modern Go project structure
- Modern Go project structure
- Recursive directory processing
- File type detection and icons
- Responsive HTML design with dark mode support