ipng/s3-genindex
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Refresh README

Browse Source
This commit is contained in:
Pim van Pelt
2025-12-03 12:27:14 +01:00
parent 16fa899b91
commit 60a149b669
2 changed files with 218 additions and 58 deletions
Download Patch File Download Diff File Expand all files Collapse all files

32
README.md
View File

@@ -1,6 +1,17 @@
# s3-genindex # s3-genindex
Generate HTML directory indexes with file type icons and responsive design. Generate HTML directory indexes with file type icons and responsive design for local directories and S3-compatible storage.
This is particularly useful for S3 buckets that are publicly readable.
## Features
- **Local directory indexing** with recursive traversal
- **S3-compatible storage support** (MinIO, AWS S3, etc.)
- **Hierarchical directory structure** for S3 buckets
- **Responsive HTML design** with file type icons
- **Dry run mode** for testing
- **Flexible filtering** with glob patterns and regex exclusion
- **Hidden file control** and index.html visibility options
## Install ## Install
@@ -8,17 +19,20 @@ Generate HTML directory indexes with file type icons and responsive design.
go install git.ipng.ch/ipng/s3-genindex/cmd/s3-genindex@latest go install git.ipng.ch/ipng/s3-genindex/cmd/s3-genindex@latest
``` ```
## Usage ## Quick Start
```bash ```bash
# Generate index.html in current directory # Local directory
s3-genindex s3-genindex -d /path/to/dir
# Generate recursively with custom output # S3 bucket (requires AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY)
s3-genindex -r -o listing.html /path/to/dir s3-genindex -s3 http://minio.example.com:9000/bucket
# Exclude files by regex # Dry run to see what would be generated
s3-genindex -x "(build|node_modules|\.tmp)" s3-genindex -d /path/to/dir -n
# Show index.html files in listings
s3-genindex -d /path/to/dir -i
``` ```
## Build ## Build
@@ -28,4 +42,4 @@ make build
make test make test
``` ```
See [docs/DETAILS.md](docs/DETAILS.md) for complete documentation. See [docs/DETAILS.md](docs/DETAILS.md) for complete documentation and examples.

242
docs/DETAILS.md
View File

@@ -2,18 +2,20 @@
## Overview ## Overview
s3-genindex is a Go rewrite of the original Python genindex.py script. It generates HTML directory listings with file type icons, responsive design, and dark mode support. s3-genindex is a Go rewrite of the original Python genindex.py script. It generates HTML directory listings with file type icons, responsive design, and dark mode support for both local directories and S3-compatible storage systems.
## Features ## Features
- **Local Directory Indexing**: Recursive traversal of filesystem directories
- **S3-Compatible Storage**: Support for MinIO, AWS S3, and other S3-compatible systems
- **Hierarchical Structure**: Creates proper directory navigation for S3 buckets
- **File Type Detection**: Recognizes 100+ file extensions with appropriate icons - **File Type Detection**: Recognizes 100+ file extensions with appropriate icons
- **Responsive Design**: Works on desktop and mobile devices - **Responsive Design**: Works on desktop and mobile devices
- **Dark Mode**: Automatic dark mode support based on system preferences - **Dark Mode**: Automatic dark mode support based on system preferences
- **Recursive Processing**: Generate indexes for entire directory trees - **Dry Run Mode**: Preview what would be generated without writing files
- **File Filtering**: Include/exclude files by pattern or regex - **File Filtering**: Include/exclude files by glob patterns or regex
- **Symlink Support**: Special handling for symbolic links - **Symlink Support**: Special handling for symbolic links
- **Custom Output**: Configurable output filename - **Index File Control**: Show/hide index.html files in directory listings
- **Breadcrumb Navigation**: Parent directory navigation
## Installation ## Installation
@@ -34,66 +36,118 @@ go install git.ipng.ch/ipng/s3-genindex/cmd/s3-genindex@latest
## Command Line Options ## Command Line Options
``` ```
Usage: s3-genindex [OPTIONS] [directory] Usage: s3-genindex [OPTIONS]
-d append output file to directory href -d string
local directory to process
-s3 string
S3 URL to process
-f string -f string
only include files matching glob (default "*") only include files matching glob (default "*")
-i include dot hidden files -i show index.html files in directory listings
-o string -n dry run: show what would be written without actually writing
custom output file (default "index.html")
-r recursively process nested dirs
-v verbosely list every processed file -v verbosely list every processed file
-x string -x string
exclude files matching regular expression exclude files matching regular expression
``` ```
**Note**: Either `-d <directory>` or `-s3 <url>` must be specified (mutually exclusive).
## Usage Examples ## Usage Examples
### Basic Usage ### Local Directory Processing (`-d`)
```bash ```bash
# Generate index.html in current directory # Generate index.html for a local directory
s3-genindex s3-genindex -d /var/www/html
# Generate index for specific directory # Process with verbose output to see all files
s3-genindex /path/to/directory s3-genindex -d /home/user/documents -v
# Generate with custom output filename # Dry run to preview what would be generated
s3-genindex -o listing.html s3-genindex -d /path/to/dir -n
# Show index.html files in directory listings
s3-genindex -d /var/www -i
``` ```
### Recursive Processing ### S3 Storage Processing (`-s3`)
```bash ```bash
# Process directory tree recursively # Basic S3 bucket processing (MinIO)
s3-genindex -r export AWS_ACCESS_KEY_ID="your-access-key"
export AWS_SECRET_ACCESS_KEY="your-secret-key"
s3-genindex -s3 http://minio.example.com:9000/my-bucket
# Process recursively with verbose output # AWS S3 bucket processing
s3-genindex -rv /var/www export AWS_ACCESS_KEY_ID="your-aws-key"
export AWS_SECRET_ACCESS_KEY="your-aws-secret"
s3-genindex -s3 https://s3.amazonaws.com/my-bucket
# S3 with verbose output and dry run
s3-genindex -s3 http://localhost:9000/test-bucket -v -n
# S3 processing with file filtering
s3-genindex -s3 http://minio.local:9000/logs -f "*.log"
``` ```
### File Filtering ### File Filtering Examples
```bash ```bash
# Include only Python files # Include only specific file types
s3-genindex -f "*.py" s3-genindex -d /var/log -f "*.log"
s3-genindex -s3 http://minio:9000/images -f "*.{jpg,png,gif}"
# Exclude build artifacts and dependencies # Exclude build artifacts and temporary files
s3-genindex -x "(build|dist|node_modules|__pycache__|\\.tmp)" s3-genindex -d /home/dev/project -x "(build|dist|node_modules|__pycache__|\\.tmp)"
# Include hidden files # Exclude version control and system files
s3-genindex -i s3-genindex -d /var/www -x "(\.git|\.svn|\.DS_Store|Thumbs\.db)"
# Complex filtering with multiple patterns
s3-genindex -d /data -f "*.{json,xml,csv}" -x "(backup|temp|cache)"
``` ```
### Advanced Usage ### Advanced Usage Scenarios
```bash ```bash
# Recursive with custom output and exclusions # Documentation site generation for local directory
s3-genindex -r -o index.html -x "(\.git|\.svn|node_modules)" /var/www s3-genindex -d /var/www/docs -i -v
# Verbose processing with directory appending # Log file indexing on S3 with size filtering
s3-genindex -r -v -d /home/user/public s3-genindex -s3 http://minio:9000/application-logs -f "*.log" -v
# Website asset indexing (excluding index files)
s3-genindex -d /var/www/assets -x "(index\.html|\.htaccess)"
# Backup verification with dry run
s3-genindex -s3 https://backup.s3.amazonaws.com/daily-backups -n -v
# Development file browsing with hidden files
s3-genindex -d /home/dev/src -i -x "(\.git|node_modules|vendor)"
# Media gallery generation
s3-genindex -d /var/media -f "*.{jpg,jpeg,png,gif,mp4,mov}" -i
```
### Integration Examples
```bash
# Automated documentation updates (cron job)
#!/bin/bash
export AWS_ACCESS_KEY_ID="docs-access-key"
export AWS_SECRET_ACCESS_KEY="docs-secret-key"
s3-genindex -s3 https://docs.s3.amazonaws.com/api-docs -v
# Local web server directory indexing
s3-genindex -d /var/www/html -i
nginx -s reload
# CI/CD artifact indexing
s3-genindex -s3 http://artifacts.internal:9000/build-artifacts -f "*.{tar.gz,zip}" -v
# Photo gallery with metadata
s3-genindex -d /var/photos -f "*.{jpg,jpeg,png,heic}" -i -v
``` ```
## File Type Support ## File Type Support
@@ -206,11 +260,40 @@ make lint # Run golangci-lint (if installed)
make check # Run all quality checks make check # Run all quality checks
``` ```
## S3 Configuration
### Environment Variables for S3
When using S3 storage (`-s3` flag), the following environment variables are **required**:
- `AWS_ACCESS_KEY_ID`: Your S3 access key ID
- `AWS_SECRET_ACCESS_KEY`: Your S3 secret access key
### S3 URL Format
S3 URLs should follow this format:
```
http://host:port/bucket # For MinIO or custom S3-compatible storage
https://host/bucket # For HTTPS endpoints
```
Examples:
- `http://minio.example.com:9000/my-bucket`
- `https://s3.amazonaws.com/my-bucket`
- `http://localhost:9000/test-bucket`
### S3 Features
- **Hierarchical Processing**: Creates index.html files for each directory level in S3
- **Path-Style URLs**: Uses path-style S3 URLs for MinIO compatibility
- **Bucket Navigation**: Generates proper directory navigation within S3 buckets
- **No Parent Directory at Root**: Root bucket index doesn't show parent (..) link
## Configuration ## Configuration
No configuration files are needed. All options are provided via command-line arguments. No configuration files are needed. All options are provided via command-line arguments.
### Environment Variables ### Standard Environment Variables
The tool respects standard Go environment variables: The tool respects standard Go environment variables:
- `GOOS` and `GOARCH` for cross-compilation - `GOOS` and `GOARCH` for cross-compilation
@@ -240,25 +323,55 @@ This Go version provides the same functionality as the original Python script wi
### Common Issues ### Common Issues
**Permission Errors** **Local Directory Permission Errors**
```bash ```bash
# Ensure read permissions on target directory # Ensure read permissions on target directory
chmod +r /path/to/directory chmod +r /path/to/directory
``` ```
**Large Directories** **S3 Connection Issues**
```bash ```bash
# Use verbose mode to monitor progress # Verify credentials are set
s3-genindex -v /large/directory echo $AWS_ACCESS_KEY_ID
echo $AWS_SECRET_ACCESS_KEY
# Exclude unnecessary files # Test connection with dry run
s3-genindex -x "(\.git|node_modules|__pycache__)" s3-genindex -s3 http://minio.example.com:9000/bucket -n -v
# Check S3 endpoint connectivity
curl http://minio.example.com:9000/
``` ```
**Memory Usage** **S3 Permission Errors**
```bash ```bash
# Process directories individually for very large trees # Verify bucket access permissions
for dir in */; do s3-genindex "$dir"; done aws s3 ls s3://your-bucket/ --endpoint-url http://minio.example.com:9000
# Check if bucket exists and is accessible
s3-genindex -s3 http://minio.example.com:9000/bucket -v
```
**Large Directory/Bucket Processing**
```bash
# Use verbose mode to monitor progress
s3-genindex -d /large/directory -v
s3-genindex -s3 http://minio:9000/large-bucket -v
# Exclude unnecessary files to reduce processing time
s3-genindex -d /data -x "(\.git|node_modules|__pycache__|\.tmp)"
s3-genindex -s3 http://minio:9000/bucket -x "(backup|temp|cache)"
# Use dry run to estimate processing time
s3-genindex -s3 http://minio:9000/bucket -n
```
**Network Timeout Issues (S3)**
```bash
# For slow connections, use verbose mode to see progress
s3-genindex -s3 http://slow-endpoint:9000/bucket -v
# Test with smaller buckets first
s3-genindex -s3 http://endpoint:9000/small-test-bucket -n
``` ```
### Debug Mode ### Debug Mode
@@ -266,7 +379,27 @@ for dir in */; do s3-genindex "$dir"; done
Enable verbose output to see detailed processing: Enable verbose output to see detailed processing:
```bash ```bash
s3-genindex -v /path/to/debug # Local directory debugging
s3-genindex -d /path/to/debug -v
# S3 debugging with dry run
s3-genindex -s3 http://minio:9000/bucket -n -v
# Full S3 processing with verbose output
s3-genindex -s3 http://minio:9000/bucket -v
```
### S3-Specific Debugging
```bash
# Test S3 connectivity without processing
curl -I http://minio.example.com:9000/bucket/
# List S3 objects directly (if aws-cli is available)
aws s3 ls s3://bucket/ --endpoint-url http://minio.example.com:9000
# Verify S3 URL format
s3-genindex -s3 http://wrong-format -n # Will show URL parsing errors
``` ```
## License ## License
@@ -283,8 +416,21 @@ Licensed under the Apache License 2.0. See original Python script for full licen
## Changelog ## Changelog
### v2.0.0 (Current)
- **S3 Support**: Complete S3-compatible storage support (MinIO, AWS S3)
- **Hierarchical S3 Processing**: Creates proper directory navigation for S3 buckets
- **Dry Run Mode**: Preview functionality with `-n` flag
- **Index File Control**: Show/hide index.html files with `-i` flag
- **Mutual Exclusive Flags**: Clean separation between `-d` and `-s3` modes
- **Enhanced Error Handling**: Better error messages and validation
- **Comprehensive Testing**: Extended test suite covering S3 functionality
- **URL Handling Fix**: Proper S3 navigation without URL encoding issues
### v1.0.0 ### v1.0.0
- Initial Go rewrite - Initial Go rewrite from Python genindex.py
- Complete feature parity with Python version - Complete feature parity with Python version for local directories
- Comprehensive test suite - Comprehensive test suite
- Modern Go project structure - Modern Go project structure
- Recursive directory processing
- File type detection and icons
- Responsive HTML design with dark mode support