Uploaded code
This commit is contained in:
375
docs/STORAGE.md
Normal file
375
docs/STORAGE.md
Normal file
@@ -0,0 +1,375 @@
|
||||
# Storage Configuration Guide
|
||||
|
||||
Sharey supports multiple storage backends for maximum flexibility. You can choose between cloud storage (Backblaze B2) or local filesystem storage.
|
||||
|
||||
## 🔄 How to Choose Storage Type
|
||||
|
||||
### **Quick Method: Edit config.json**
|
||||
1. Open your `config.json` file
|
||||
2. Find the `"storage"` section
|
||||
3. Change `"backend"` to either `"local"` or `"b2"`
|
||||
|
||||
```json
|
||||
{
|
||||
"storage": {
|
||||
"backend": "local" // Change this to "local" or "b2"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### **Interactive Method**
|
||||
```bash
|
||||
python src/config_util.py set # Interactive setup
|
||||
python src/config_util.py validate # Verify configuration
|
||||
python test_storage.py # Test the backend
|
||||
```
|
||||
|
||||
### **Environment Variable Method**
|
||||
```bash
|
||||
export STORAGE_BACKEND=local # or "b2"
|
||||
```
|
||||
|
||||
## 📊 Storage Backend Comparison
|
||||
|
||||
| Feature | **Local Storage** | **B2 Cloud Storage** |
|
||||
|---------|------------------|-------------------|
|
||||
| **Setup Difficulty** | ✅ Easy | ⚠️ Requires B2 account |
|
||||
| **Cost** | ✅ Free | 💰 ~$0.005/GB/month |
|
||||
| **Speed** | ✅ Very fast | ⚠️ Network dependent |
|
||||
| **Reliability** | ⚠️ Single machine | ✅ Cloud redundancy |
|
||||
| **Scalability** | ⚠️ Disk space limited | ✅ Unlimited |
|
||||
| **Best for** | Development, self-hosting | Production, scaling |
|
||||
|
||||
## Storage Backends
|
||||
|
||||
### 1. Backblaze B2 (Cloud Storage)
|
||||
- ✅ **Best for production deployments**
|
||||
- ✅ **Scalable and reliable**
|
||||
- ✅ **No local disk space requirements**
|
||||
- ❌ **Requires B2 account and credentials**
|
||||
|
||||
### 2. Local Filesystem
|
||||
- ✅ **No external dependencies**
|
||||
- ✅ **Fast access**
|
||||
- ✅ **No cloud costs**
|
||||
- ❌ **Limited by local disk space**
|
||||
- ❌ **Not suitable for distributed deployments**
|
||||
|
||||
## Configuration Details
|
||||
|
||||
## Configuration Details
|
||||
|
||||
### Option 1: config.json (Recommended)
|
||||
|
||||
```json
|
||||
{
|
||||
"storage": {
|
||||
"backend": "b2", // "b2" or "local"
|
||||
"local_path": "storage" // Path for local storage (only used if backend is "local")
|
||||
},
|
||||
"b2": {
|
||||
"application_key_id": "your_key_id_here",
|
||||
"application_key": "your_application_key_here",
|
||||
"bucket_name": "your_bucket_name_here"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Option 2: Environment Variables
|
||||
|
||||
```bash
|
||||
# Storage configuration
|
||||
export STORAGE_BACKEND=local
|
||||
export STORAGE_LOCAL_PATH=./my_storage
|
||||
|
||||
# B2 configuration (only needed if using B2)
|
||||
export B2_APPLICATION_KEY_ID=your_key_id_here
|
||||
export B2_APPLICATION_KEY=your_application_key_here
|
||||
export B2_BUCKET_NAME=your_bucket_name_here
|
||||
```
|
||||
|
||||
## 🚀 Quick Start Guide
|
||||
|
||||
### **For Development/Testing (Local Storage)**
|
||||
```bash
|
||||
# 1. Edit config.json
|
||||
{
|
||||
"storage": {
|
||||
"backend": "local"
|
||||
}
|
||||
}
|
||||
|
||||
# 2. Test it
|
||||
python test_storage.py
|
||||
|
||||
# 3. Start the app
|
||||
python run.py
|
||||
```
|
||||
Files will be stored in the `storage/` directory.
|
||||
|
||||
### **For Production (B2 Cloud Storage)**
|
||||
```bash
|
||||
# 1. Get B2 credentials from backblaze.com
|
||||
# 2. Edit config.json
|
||||
{
|
||||
"storage": {
|
||||
"backend": "b2"
|
||||
},
|
||||
"b2": {
|
||||
"application_key_id": "your_key_id",
|
||||
"application_key": "your_key",
|
||||
"bucket_name": "your_bucket"
|
||||
}
|
||||
}
|
||||
|
||||
# 3. Test it
|
||||
python test_storage.py
|
||||
|
||||
# 4. Start the app
|
||||
python run.py
|
||||
```
|
||||
|
||||
## Setup Examples
|
||||
|
||||
### Using Local Storage
|
||||
|
||||
1. **Edit config.json:**
|
||||
```json
|
||||
{
|
||||
"storage": {
|
||||
"backend": "local",
|
||||
"local_path": "storage"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
2. **Start the application:**
|
||||
```bash
|
||||
python run.py
|
||||
```
|
||||
|
||||
The application will automatically create the `storage` directory and subdirectories as needed.
|
||||
|
||||
### Using Backblaze B2
|
||||
|
||||
1. **Create a B2 account** at [backblaze.com](https://www.backblaze.com/b2/)
|
||||
|
||||
2. **Create a bucket** and application key with read/write permissions
|
||||
|
||||
3. **Edit config.json:**
|
||||
```json
|
||||
{
|
||||
"storage": {
|
||||
"backend": "b2"
|
||||
},
|
||||
"b2": {
|
||||
"application_key_id": "your_actual_key_id",
|
||||
"application_key": "your_actual_key",
|
||||
"bucket_name": "your_bucket_name"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
4. **Start the application:**
|
||||
```bash
|
||||
python run.py
|
||||
```
|
||||
|
||||
## Interactive Configuration
|
||||
|
||||
Use the configuration utility for easy setup:
|
||||
|
||||
```bash
|
||||
# Configure storage interactively
|
||||
python src/config_util.py set
|
||||
|
||||
# Validate your configuration
|
||||
python src/config_util.py validate
|
||||
|
||||
# Show current configuration
|
||||
python src/config_util.py show
|
||||
```
|
||||
|
||||
## Testing Storage
|
||||
|
||||
Test your storage configuration:
|
||||
|
||||
```bash
|
||||
# Test storage backend
|
||||
python test_storage.py
|
||||
```
|
||||
|
||||
This will:
|
||||
- ✅ Initialize the storage backend
|
||||
- ✅ Upload a test file
|
||||
- ✅ Download and verify the file
|
||||
- ✅ Test file operations (exists, size, list)
|
||||
- ✅ Clean up test files
|
||||
|
||||
## Migration Between Backends
|
||||
|
||||
### From B2 to Local Storage
|
||||
|
||||
1. **Download your data** from B2 (optional - keep as backup)
|
||||
2. **Update config.json:**
|
||||
```json
|
||||
{
|
||||
"storage": {
|
||||
"backend": "local",
|
||||
"local_path": "storage"
|
||||
}
|
||||
}
|
||||
```
|
||||
3. **Restart the application**
|
||||
|
||||
**Note:** Existing files in B2 won't be automatically transferred. URLs will break unless you migrate the data.
|
||||
|
||||
### From Local to B2 Storage
|
||||
|
||||
1. **Set up B2 bucket and credentials**
|
||||
2. **Update config.json:**
|
||||
```json
|
||||
{
|
||||
"storage": {
|
||||
"backend": "b2"
|
||||
},
|
||||
"b2": {
|
||||
"application_key_id": "your_key_id",
|
||||
"application_key": "your_key",
|
||||
"bucket_name": "your_bucket"
|
||||
}
|
||||
}
|
||||
```
|
||||
3. **Restart the application**
|
||||
|
||||
**Migration Script (Optional):**
|
||||
If you want to migrate existing local files to B2, create a simple script:
|
||||
|
||||
```python
|
||||
#!/usr/bin/env python3
|
||||
import os
|
||||
from pathlib import Path
|
||||
from src.storage import LocalStorageBackend, B2StorageBackend
|
||||
|
||||
# Initialize both backends
|
||||
local = LocalStorageBackend("storage")
|
||||
b2 = B2StorageBackend("key_id", "key", "bucket_name")
|
||||
|
||||
# Migrate files
|
||||
for file_path in local.list_files():
|
||||
print(f"Migrating: {file_path}")
|
||||
content = local.download_file(file_path)
|
||||
b2.upload_file(content, file_path)
|
||||
print(f"✅ Migrated: {file_path}")
|
||||
```
|
||||
|
||||
## Directory Structure
|
||||
|
||||
### Local Storage Structure
|
||||
```
|
||||
storage/
|
||||
├── files/
|
||||
│ ├── abc123.jpg
|
||||
│ ├── def456.pdf
|
||||
│ └── ...
|
||||
└── pastes/
|
||||
├── xyz789.txt
|
||||
├── uvw012.txt
|
||||
└── ...
|
||||
```
|
||||
|
||||
### B2 Storage Structure
|
||||
```
|
||||
bucket/
|
||||
├── files/
|
||||
│ ├── abc123.jpg
|
||||
│ ├── def456.pdf
|
||||
│ └── ...
|
||||
└── pastes/
|
||||
├── xyz789.txt
|
||||
├── uvw012.txt
|
||||
└── ...
|
||||
```
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
### Local Storage
|
||||
- **Pros:** Very fast access, no network latency
|
||||
- **Cons:** Limited by disk I/O, single point of failure
|
||||
|
||||
### B2 Storage
|
||||
- **Pros:** Unlimited capacity, built-in redundancy, CDN capabilities
|
||||
- **Cons:** Network latency, dependent on internet connection
|
||||
|
||||
## Security
|
||||
|
||||
### Local Storage
|
||||
- Files are stored with filesystem permissions
|
||||
- Secure as your server's filesystem
|
||||
- Access controlled by Sharey application
|
||||
|
||||
### B2 Storage
|
||||
- Files stored with B2's encryption at rest
|
||||
- Access controlled by application keys
|
||||
- Private bucket - files not publicly accessible
|
||||
- All access proxied through Sharey application
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
**"Storage backend not initialized"**
|
||||
- Check your configuration syntax
|
||||
- Verify credentials (for B2)
|
||||
- Run `python test_storage.py`
|
||||
|
||||
**"B2 authentication failed"**
|
||||
- Verify your application key ID and key
|
||||
- Check bucket name spelling
|
||||
- Ensure key has read/write permissions
|
||||
|
||||
**"Permission denied" (Local storage)**
|
||||
- Check filesystem permissions on storage directory
|
||||
- Ensure Sharey has write access to the path
|
||||
|
||||
**"File not found"**
|
||||
- Check if storage backend was changed without migration
|
||||
- Verify file was uploaded successfully
|
||||
|
||||
### Debug Commands
|
||||
|
||||
```bash
|
||||
# Test storage backend
|
||||
python test_storage.py
|
||||
|
||||
# Validate configuration
|
||||
python src/config_util.py validate
|
||||
|
||||
# Check application logs
|
||||
tail -f logs/sharey.log
|
||||
|
||||
# Test B2 connection (if using B2)
|
||||
python tests/test_b2.py
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Production:** Use B2 for scalability and reliability
|
||||
2. **Development:** Local storage is fine for testing
|
||||
3. **Backup:** Consider periodic backups regardless of backend
|
||||
4. **Monitoring:** Monitor disk space (local) or B2 costs (cloud)
|
||||
5. **Security:** Use environment variables for sensitive credentials in production
|
||||
|
||||
## Cost Considerations
|
||||
|
||||
### Local Storage
|
||||
- **Cost:** Server disk space and bandwidth
|
||||
- **Scaling:** Limited by hardware
|
||||
|
||||
### B2 Storage
|
||||
- **Storage:** $0.005/GB/month
|
||||
- **Download:** $0.01/GB
|
||||
- **API calls:** Mostly free (10,000 free per day)
|
||||
- **Scaling:** Pay as you grow
|
||||
|
||||
For most small to medium deployments, B2 costs are minimal (under $5/month for moderate usage).
|
||||
Reference in New Issue
Block a user