diff --git a/CLOUDFLARE_SETUP.md b/CLOUDFLARE_SETUP.md deleted file mode 100644 index 9b78dd9..0000000 --- a/CLOUDFLARE_SETUP.md +++ /dev/null @@ -1,146 +0,0 @@ -# Cloudflare + nginx Setup Guide for ircquotes - -## Overview -This setup ensures that your ircquotes application can see real client IP addresses even when behind: -1. **Cloudflare** (CDN/Proxy) -2. **nginx** (Reverse Proxy) -3. **Gunicorn** (WSGI Server) - -## Architecture -``` -Client → Cloudflare → nginx → Gunicorn → ircquotes -``` - -## Setup Steps - -### 1. Cloudflare Configuration - -#### Enable Proxy (Orange Cloud) -- Set your DNS record to "Proxied" (orange cloud icon) -- This routes traffic through Cloudflare's edge servers - -#### Recommended Cloudflare Settings: -- **SSL/TLS**: Full (Strict) if you have SSL on origin -- **Security Level**: Medium -- **Bot Fight Mode**: Enabled -- **Rate Limiting**: Configure as needed -- **Page Rules**: Optional caching rules - -#### Important Headers: -Cloudflare automatically adds these headers: -- `CF-Connecting-IP`: Real client IP address -- `CF-Ray`: Request identifier -- `CF-Visitor`: Visitor information - -### 2. nginx Configuration - -Copy the provided `nginx-ircquotes.conf` to your nginx sites: - -```bash -sudo cp nginx-ircquotes.conf /etc/nginx/sites-available/ircquotes -sudo ln -s /etc/nginx/sites-available/ircquotes /etc/nginx/sites-enabled/ -sudo nginx -t -sudo systemctl reload nginx -``` - -**Key nginx features:** -- ✅ Cloudflare IP range restoration -- ✅ Real IP detection via CF-Connecting-IP -- ✅ Additional rate limiting layer -- ✅ Security headers -- ✅ Gzip compression -- ✅ Static file optimization - -### 3. Application Configuration - -The ircquotes app is already configured to: -- ✅ Use `CF-Connecting-IP` header (Cloudflare's real IP) -- ✅ Fall back to `X-Forwarded-For` and `X-Real-IP` -- ✅ Handle 2-proxy setup (Cloudflare + nginx) -- ✅ Rate limit by real client IP - -### 4. Verification - -To verify real IPs are being detected: - -1. **Check application logs**: - ```bash - tail -f /var/log/ircquotes/access.log - ``` - -2. **Test from different locations**: - - Visit your site from different networks - - Check admin panel for real IPs in quote submissions - - Verify rate limiting works per real IP - -3. **Debug headers** (temporary debug route): - ```python - @app.route('/debug-headers') - def debug_headers(): - return jsonify({ - 'real_ip': get_real_ip(), - 'cf_connecting_ip': request.headers.get('CF-Connecting-IP'), - 'x_forwarded_for': request.headers.get('X-Forwarded-For'), - 'x_real_ip': request.headers.get('X-Real-IP'), - 'remote_addr': request.remote_addr - }) - ``` - -### 5. Security Considerations - -#### Cloudflare Settings: -- Enable **DDoS Protection** -- Configure **WAF Rules** for your application -- Set up **Rate Limiting** at Cloudflare level -- Enable **Bot Management** if available - -#### nginx Security: -- Keep Cloudflare IP ranges updated -- Monitor for suspicious patterns -- Implement additional rate limiting -- Regular security updates - -#### Application Security: -- All security features already implemented -- Rate limiting per real IP -- CSRF protection enabled -- Input validation active - -## Troubleshooting - -### IPs showing as 127.0.0.1: -1. Check nginx is passing headers correctly -2. Verify Cloudflare IP ranges in nginx config -3. Ensure ProxyFix is configured for 2 proxies -4. Check `CF-Connecting-IP` header presence - -### Rate limiting not working: -1. Verify real IP detection is working -2. Check rate limiting configuration -3. Monitor nginx and application logs -4. Test with different source IPs - -### Performance issues: -1. Enable nginx caching for static files -2. Configure Cloudflare caching rules -3. Monitor Gunicorn worker count -4. Check database connection pooling - -## Monitoring - -Recommended monitoring: -- **Application logs**: Real IP addresses in logs -- **nginx access logs**: Request patterns -- **Cloudflare Analytics**: Traffic patterns -- **Rate limiting metrics**: Blocked vs allowed requests - -## Production Checklist - -- [ ] Cloudflare proxy enabled (orange cloud) -- [ ] nginx configuration deployed -- [ ] Real IP detection working -- [ ] Rate limiting functional -- [ ] Security headers present -- [ ] SSL/TLS configured -- [ ] Monitoring in place -- [ ] Backup and recovery tested \ No newline at end of file diff --git a/CONFIG_GUIDE.md b/CONFIG_GUIDE.md new file mode 100644 index 0000000..0aa1cb1 --- /dev/null +++ b/CONFIG_GUIDE.md @@ -0,0 +1,148 @@ +# Configuration Guide + +This guide explains how to configure the ircquotes application by editing `config.json` manually. + +## Configuration File Structure + +The `config.json` file is organized into sections: + +### App Section +```json +"app": { + "name": "ircquotes", // Application name + "host": "127.0.0.1", // Host to bind to (use 0.0.0.0 for all interfaces) + "port": 6969, // Port number to run on + "debug": false // Enable debug mode (set to true for development) +} +``` + +### Gunicorn Section (Production Settings) +```json +"gunicorn": { + "workers": 4, // Number of worker processes + "timeout": 30, // Request timeout in seconds + "keepalive": 5, // Keep-alive timeout + "max_requests": 1000, // Max requests per worker before restart + "preload": true // Preload application code +} +``` + +### Database Section +```json +"database": { + "uri": "sqlite:///quotes.db?timeout=20", // Database connection string + "pool_timeout": 20, // Connection pool timeout + "pool_recycle": -1, // Connection recycle time (-1 = disabled) + "pool_pre_ping": true // Test connections before use +} +``` + +### Security Section +```json +"security": { + "csrf_enabled": true, // Enable CSRF protection + "csrf_time_limit": null, // CSRF token time limit (null = no limit) + "session_cookie_secure": false, // Require HTTPS for session cookies + "session_cookie_httponly": true, // Prevent JavaScript access to session cookies + "session_cookie_samesite": "Lax", // SameSite policy for session cookies + "security_headers": { + "x_content_type_options": "nosniff", + "x_frame_options": "DENY", + "x_xss_protection": "1; mode=block", + "strict_transport_security": "max-age=31536000; includeSubDomains", + "content_security_policy": "default-src 'self'; script-src 'self' 'unsafe-inline'; style-src 'self' 'unsafe-inline'; img-src 'self' data:; connect-src 'self'" + } +} +``` + +### Admin Section +```json +"admin": { + "username": "admin", // Admin username + "password_hash": "..." // Argon2 password hash (use generate_password.py) +} +``` + +### Quotes Section +```json +"quotes": { + "min_length": 1, // Minimum quote length in characters + "max_length": 5000, // Maximum quote length in characters + "per_page": 25, // Quotes displayed per page + "auto_approve": false, // Automatically approve new quotes + "allow_html": false // Allow HTML in quotes (not recommended) +} +``` + +### Features Section +```json +"features": { + "voting_enabled": true, // Enable voting on quotes + "flagging_enabled": true, // Enable flagging inappropriate quotes + "copy_quotes_enabled": true, // Enable copy-to-clipboard feature + "dark_mode_enabled": true, // Enable dark mode toggle + "api_enabled": true, // Enable JSON API endpoints + "bulk_moderation_enabled": true // Enable bulk moderation actions +} +``` + +### Logging Section +```json +"logging": { + "level": "DEBUG", // Logging level (DEBUG, INFO, WARNING, ERROR) + "format": "%(asctime)s [%(levelname)s] %(message)s" // Log message format +} +``` + +## Common Configuration Tasks + +### Change Admin Password +1. Run: `python generate_password.py` +2. Edit `config.json` and update `admin.password_hash` with the generated hash +3. Restart the application + +### Change Port +Edit the `app.port` value in `config.json`: +```json +"app": { + "port": 8080 +} +``` + +### Adjust Quote Limits +Edit the `quotes` section: +```json +"quotes": { + "min_length": 10, + "max_length": 2000, + "per_page": 50 +} +``` + +### Disable Features +Set feature flags to `false`: +```json +"features": { + "voting_enabled": false, + "flagging_enabled": false +} +``` + +### Adjust Rate Limits +Modify the `rate_limiting.endpoints` section: +```json +"rate_limiting": { + "endpoints": { + "submit": "10 per minute", + "vote": "120 per minute" + } +} +``` + +## Important Notes + +- Always restart the application after making configuration changes +- Use valid JSON syntax (no trailing commas, proper quotes) +- Test configuration changes in a development environment first +- Keep backups of your working configuration +- Use `python generate_password.py` to create secure password hashes \ No newline at end of file diff --git a/DEPLOYMENT.md b/DEPLOYMENT.md index 3fab514..217c0da 100644 --- a/DEPLOYMENT.md +++ b/DEPLOYMENT.md @@ -13,24 +13,30 @@ All application settings are now centralized in `config.json`. You can easily mo - **Admin credentials** - **Feature toggles** -### Viewing Current Configuration +### Configuration Management +All configuration is done by editing `config.json` directly. This file contains all application settings organized in sections: + +- **app**: Basic application settings (name, host, port, debug) +- **database**: Database connection settings +- **security**: Security headers, CSRF, proxy settings +- **rate_limiting**: Rate limiting configuration for different endpoints +- **admin**: Admin username and password hash +- **quotes**: Quote submission settings (length limits, pagination) +- **features**: Feature toggles (voting, flagging, dark mode, etc.) +- **logging**: Logging configuration + +### Example Configuration Changes ```bash -python config_manager.py -``` +# Edit config.json in any text editor +nano config.json -### Updating Configuration -```bash -# Change port -python config_manager.py app.port 8080 +# Example changes: +# - Change port: "port": 8080 in the "app" section +# - Change quotes per page: "per_page": 50 in the "quotes" section +# - Disable CSRF: "csrf_enabled": false in the "security" section +# - Change rate limits: "login": "10 per minute" in rate_limiting.endpoints -# Change quotes per page -python config_manager.py quotes.per_page 50 - -# Disable CSRF (not recommended) -python config_manager.py security.csrf_enabled false - -# Change rate limits -python config_manager.py rate_limiting.endpoints.login "10 per minute" +# After making changes, restart the application ``` ## Running with Gunicorn (Production) diff --git a/app.py b/app.py index 779c7ea..ef7f464 100644 --- a/app.py +++ b/app.py @@ -1,7 +1,5 @@ from flask import Flask, render_template, request, redirect, url_for, flash, abort, make_response, session, jsonify from flask_sqlalchemy import SQLAlchemy -from flask_limiter import Limiter -from flask_limiter.util import get_remote_address from flask_cors import CORS from flask_wtf.csrf import CSRFProtect import datetime @@ -9,59 +7,109 @@ import json import random from argon2 import PasswordHasher from argon2.exceptions import VerifyMismatchError -from werkzeug.middleware.proxy_fix import ProxyFix # Import ProxyFix import logging from sqlalchemy import event from sqlalchemy.engine import Engine import sqlite3 +import time +import ipaddress from config_loader import config # Import configuration system -def get_real_ip(): +def db_retry_operation(operation, max_retries=2, delay=0.01): """ - Get the real client IP address considering Cloudflare and nginx reverse proxy. - Checks headers in order of priority: - 1. CF-Connecting-IP (Cloudflare's real IP header) - 2. X-Forwarded-For (standard proxy header) - 3. X-Real-IP (nginx real IP header) - 4. request.remote_addr (fallback) + Retry database operations that might fail due to database locks. + Includes session cleanup for better reliability. + + Args: + operation: A callable that performs the database operation + max_retries: Maximum number of retry attempts + delay: Initial delay between retries + + Returns: + The result of the operation if successful + + Raises: + The last exception if all retries fail """ - # Cloudflare provides the real IP in CF-Connecting-IP header - cf_ip = request.headers.get('CF-Connecting-IP') - if cf_ip: - return cf_ip + last_exception = None - # Check X-Forwarded-For (may contain multiple IPs, first is original client) - forwarded_for = request.headers.get('X-Forwarded-For') - if forwarded_for: - # Take the first IP in the chain (original client) - return forwarded_for.split(',')[0].strip() + for attempt in range(max_retries + 1): + try: + return operation() + except Exception as e: + last_exception = e + error_msg = str(e).lower() + + # Handle specific database errors that benefit from retry + if ('database is locked' in error_msg or + 'sqlite3.operationalerror' in error_msg or + 'transaction has been rolled back' in error_msg): + + try: + # Only rollback, don't close session to avoid unbound objects + db.session.rollback() + except: + pass # Ignore cleanup errors + + if attempt < max_retries: + logging.warning(f"Database error detected, rollback and retry (attempt {attempt + 1}/{max_retries + 1})") + time.sleep(delay) + continue + + # For non-database errors or final attempt, re-raise immediately + raise - # Check X-Real-IP (nginx header) - real_ip = request.headers.get('X-Real-IP') - if real_ip: - return real_ip - - # Fallback to request.remote_addr or default - return request.remote_addr or '127.0.0.1' + # This should never be reached due to the logic above, but just in case + if last_exception: + raise last_exception + else: + raise RuntimeError("Database operation failed for unknown reasons") -# Configure SQLite for better concurrency +def validate_ip_address(ip_str): + """ + Validate that an IP address string is a valid IPv4 or IPv6 address. + Returns a sanitized IP address string or '127.0.0.1' if invalid. + """ + try: + # This will raise ValueError if the IP is invalid + ip_obj = ipaddress.ip_address(ip_str) + return str(ip_obj) + except (ValueError, TypeError): + # If IP is invalid, return localhost as fallback + app.logger.warning(f"Invalid IP address detected: {ip_str}") + return '127.0.0.1' + +# Configure SQLite for better concurrency and performance @event.listens_for(Engine, "connect") def set_sqlite_pragma(dbapi_connection, connection_record): if isinstance(dbapi_connection, sqlite3.Connection): cursor = dbapi_connection.cursor() # Set WAL mode for better concurrency cursor.execute("PRAGMA journal_mode=WAL") - # Set timeout for locked database - cursor.execute("PRAGMA busy_timeout=30000") # 30 seconds + # Reduce timeout for faster failures instead of long waits + cursor.execute("PRAGMA busy_timeout=1000") # 1 second - faster failure # Optimize for performance cursor.execute("PRAGMA synchronous=NORMAL") - cursor.execute("PRAGMA cache_size=1000") + cursor.execute("PRAGMA cache_size=20000") # Larger cache cursor.execute("PRAGMA temp_store=memory") + cursor.execute("PRAGMA mmap_size=268435456") # 256MB memory mapped + cursor.execute("PRAGMA wal_autocheckpoint=500") # More frequent checkpoints + cursor.execute("PRAGMA optimize") # Enable automatic index optimization cursor.close() app = Flask(__name__) app.config['SQLALCHEMY_DATABASE_URI'] = config.database_uri app.config['SECRET_KEY'] = open("instance/flask_secret_key", "r").read().strip() +app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = False + +# Enhanced connection pool configuration for better concurrency +app.config['SQLALCHEMY_ENGINE_OPTIONS'] = { + 'pool_size': 10, # Maintain 10 connections in pool + 'pool_recycle': 3600, # Recycle connections every hour + 'pool_pre_ping': True, # Test connections before use + 'pool_timeout': 5, # Wait up to 5 seconds for connection + 'max_overflow': 20 # Allow up to 20 additional connections +} # Configure secure session settings from config app.config['SESSION_COOKIE_SECURE'] = config.get('security.session_cookie_secure', False) @@ -88,18 +136,10 @@ csrf.exempt('get_top_quotes') csrf.exempt('search_quotes') csrf.exempt('get_stats') -# Initialize rate limiter with custom IP detection -limiter = Limiter( - key_func=get_real_ip, - app=app -) +# Remove rate limiting - immediate response for all requests db = SQLAlchemy(app) -# Apply ProxyFix middleware for Cloudflare + nginx setup -# x_for=2: nginx (1) + Cloudflare (1) = 2 proxies in X-Forwarded-For chain -app.wsgi_app = ProxyFix(app.wsgi_app, x_for=2, x_proto=1, x_host=1, x_port=1, x_prefix=1) - # Initialize Argon2 password hasher ph = PasswordHasher() @@ -136,12 +176,18 @@ class Quote(db.Model): id = db.Column(db.Integer, primary_key=True) text = db.Column(db.Text, nullable=False) votes = db.Column(db.Integer, default=0) - date = db.Column(db.DateTime, default=datetime.datetime.utcnow) - status = db.Column(db.Integer, default=0) # 0 = pending, 1 = approved, 2 = rejected + date = db.Column(db.DateTime, nullable=True) # Legacy field for old quotes + status = db.Column(db.Integer, default=0, index=True) # 0 = pending, 1 = approved, 2 = rejected ip_address = db.Column(db.String(45)) # Store IPv4 and IPv6 addresses user_agent = db.Column(db.String(255)) # Store user-agent strings - submitted_at = db.Column(db.DateTime, default=datetime.datetime.utcnow) - flag_count = db.Column(db.Integer, default=0) # Track how many times quote has been flagged + submitted_at = db.Column(db.DateTime, nullable=True) # New timestamp field for new quotes + flag_count = db.Column(db.Integer, default=0, index=True) # Track how many times quote has been flagged + + # Add composite indexes for common queries + __table_args__ = ( + db.Index('idx_status_id', 'status', 'id'), + db.Index('idx_flag_count_id', 'flag_count', 'id'), + ) # Home route to display quotes @app.route('/') @@ -157,10 +203,11 @@ def index(): # Separate route for submitting quotes @app.route('/submit', methods=['GET', 'POST']) -@limiter.limit(config.get('rate_limiting.endpoints.submit', '5 per minute')) def submit(): if request.method == 'POST': quote_text = request.form.get('quote') + is_preview = 'submit2' in request.form # Preview button is named submit2 + if not quote_text: flash("Oops! Your quote seems to be empty. Please enter some text before submitting.", 'error') return redirect(url_for('submit')) @@ -183,8 +230,18 @@ def submit(): if ' +
@@ -125,7 +126,15 @@ 0 {% endif %} -{{ quote.text|e }}
diff --git a/templates/random.html b/templates/random.html index 4607ad8..e73a7e2 100644 --- a/templates/random.html +++ b/templates/random.html @@ -64,6 +64,14 @@ X C + + + {% if quote.submitted_at %} + {{ quote.submitted_at.strftime('%d/%m/%y %H:%M') }} + {% elif quote.date %} + {{ quote.date.strftime('%d/%m/%y') }} + {% endif %} +{{ quote.text|e }}
{{ quote.text|e }}