From cd27cc8ad98f7e741fe9b9ac56112574e7c758bb Mon Sep 17 00:00:00 2001
From: ComputerTech312
Date: Sun, 21 Sep 2025 19:45:08 +0100
Subject: [PATCH] Major refactor: Fix SQLite concurrency, remove rate limiting,
simplify architecture
- Switch to single Gunicorn worker to eliminate SQLite database locking issues
- Remove Flask-Limiter and all rate limiting complexity
- Remove Cloudflare proxy setup and dependencies
- Simplify configuration and remove unnecessary features
- Update all templates and static files for streamlined operation
- Clean up old files and documentation
- Restore stable database from backup
- System now runs fast and reliably without database locks
---
CLOUDFLARE_SETUP.md | 146 -------
CONFIG_GUIDE.md | 148 ++++++++
DEPLOYMENT.md | 36 +-
app.py | 860 +++++++++++++++++++++++++++++++-----------
config.json | 30 +-
config_manager.py | 84 -----
create_fresh_db.py | 77 ++++
generate_password.py | 8 +-
gunicorn.conf.py | 59 ---
instance/quotes.db | Bin 6447104 -> 6447104 bytes
nginx-ircquotes.conf | 5 +-
production.py | 83 ++++
requirements.txt | 1 -
setup.sh | 12 +-
start_gunicorn.py | 49 ---
static/modapp.js | 189 ++++++++++
static/styles.css | 101 +++--
static/voting.js | 29 +-
templates/browse.html | 10 +-
templates/faq.html | 19 +-
templates/index.html | 4 +-
templates/modapp.html | 11 +-
templates/quote.html | 8 +
templates/random.html | 8 +
templates/search.html | 8 +
templates/submit.html | 18 +-
26 files changed, 1326 insertions(+), 677 deletions(-)
delete mode 100644 CLOUDFLARE_SETUP.md
create mode 100644 CONFIG_GUIDE.md
delete mode 100644 config_manager.py
create mode 100644 create_fresh_db.py
delete mode 100644 gunicorn.conf.py
create mode 100755 production.py
delete mode 100644 start_gunicorn.py
create mode 100644 static/modapp.js
diff --git a/CLOUDFLARE_SETUP.md b/CLOUDFLARE_SETUP.md
deleted file mode 100644
index 9b78dd9..0000000
--- a/CLOUDFLARE_SETUP.md
+++ /dev/null
@@ -1,146 +0,0 @@
-# Cloudflare + nginx Setup Guide for ircquotes
-
-## Overview
-This setup ensures that your ircquotes application can see real client IP addresses even when behind:
-1. **Cloudflare** (CDN/Proxy)
-2. **nginx** (Reverse Proxy)
-3. **Gunicorn** (WSGI Server)
-
-## Architecture
-```
-Client → Cloudflare → nginx → Gunicorn → ircquotes
-```
-
-## Setup Steps
-
-### 1. Cloudflare Configuration
-
-#### Enable Proxy (Orange Cloud)
-- Set your DNS record to "Proxied" (orange cloud icon)
-- This routes traffic through Cloudflare's edge servers
-
-#### Recommended Cloudflare Settings:
-- **SSL/TLS**: Full (Strict) if you have SSL on origin
-- **Security Level**: Medium
-- **Bot Fight Mode**: Enabled
-- **Rate Limiting**: Configure as needed
-- **Page Rules**: Optional caching rules
-
-#### Important Headers:
-Cloudflare automatically adds these headers:
-- `CF-Connecting-IP`: Real client IP address
-- `CF-Ray`: Request identifier
-- `CF-Visitor`: Visitor information
-
-### 2. nginx Configuration
-
-Copy the provided `nginx-ircquotes.conf` to your nginx sites:
-
-```bash
-sudo cp nginx-ircquotes.conf /etc/nginx/sites-available/ircquotes
-sudo ln -s /etc/nginx/sites-available/ircquotes /etc/nginx/sites-enabled/
-sudo nginx -t
-sudo systemctl reload nginx
-```
-
-**Key nginx features:**
-- ✅ Cloudflare IP range restoration
-- ✅ Real IP detection via CF-Connecting-IP
-- ✅ Additional rate limiting layer
-- ✅ Security headers
-- ✅ Gzip compression
-- ✅ Static file optimization
-
-### 3. Application Configuration
-
-The ircquotes app is already configured to:
-- ✅ Use `CF-Connecting-IP` header (Cloudflare's real IP)
-- ✅ Fall back to `X-Forwarded-For` and `X-Real-IP`
-- ✅ Handle 2-proxy setup (Cloudflare + nginx)
-- ✅ Rate limit by real client IP
-
-### 4. Verification
-
-To verify real IPs are being detected:
-
-1. **Check application logs**:
- ```bash
- tail -f /var/log/ircquotes/access.log
- ```
-
-2. **Test from different locations**:
- - Visit your site from different networks
- - Check admin panel for real IPs in quote submissions
- - Verify rate limiting works per real IP
-
-3. **Debug headers** (temporary debug route):
- ```python
- @app.route('/debug-headers')
- def debug_headers():
- return jsonify({
- 'real_ip': get_real_ip(),
- 'cf_connecting_ip': request.headers.get('CF-Connecting-IP'),
- 'x_forwarded_for': request.headers.get('X-Forwarded-For'),
- 'x_real_ip': request.headers.get('X-Real-IP'),
- 'remote_addr': request.remote_addr
- })
- ```
-
-### 5. Security Considerations
-
-#### Cloudflare Settings:
-- Enable **DDoS Protection**
-- Configure **WAF Rules** for your application
-- Set up **Rate Limiting** at Cloudflare level
-- Enable **Bot Management** if available
-
-#### nginx Security:
-- Keep Cloudflare IP ranges updated
-- Monitor for suspicious patterns
-- Implement additional rate limiting
-- Regular security updates
-
-#### Application Security:
-- All security features already implemented
-- Rate limiting per real IP
-- CSRF protection enabled
-- Input validation active
-
-## Troubleshooting
-
-### IPs showing as 127.0.0.1:
-1. Check nginx is passing headers correctly
-2. Verify Cloudflare IP ranges in nginx config
-3. Ensure ProxyFix is configured for 2 proxies
-4. Check `CF-Connecting-IP` header presence
-
-### Rate limiting not working:
-1. Verify real IP detection is working
-2. Check rate limiting configuration
-3. Monitor nginx and application logs
-4. Test with different source IPs
-
-### Performance issues:
-1. Enable nginx caching for static files
-2. Configure Cloudflare caching rules
-3. Monitor Gunicorn worker count
-4. Check database connection pooling
-
-## Monitoring
-
-Recommended monitoring:
-- **Application logs**: Real IP addresses in logs
-- **nginx access logs**: Request patterns
-- **Cloudflare Analytics**: Traffic patterns
-- **Rate limiting metrics**: Blocked vs allowed requests
-
-## Production Checklist
-
-- [ ] Cloudflare proxy enabled (orange cloud)
-- [ ] nginx configuration deployed
-- [ ] Real IP detection working
-- [ ] Rate limiting functional
-- [ ] Security headers present
-- [ ] SSL/TLS configured
-- [ ] Monitoring in place
-- [ ] Backup and recovery tested
\ No newline at end of file
diff --git a/CONFIG_GUIDE.md b/CONFIG_GUIDE.md
new file mode 100644
index 0000000..0aa1cb1
--- /dev/null
+++ b/CONFIG_GUIDE.md
@@ -0,0 +1,148 @@
+# Configuration Guide
+
+This guide explains how to configure the ircquotes application by editing `config.json` manually.
+
+## Configuration File Structure
+
+The `config.json` file is organized into sections:
+
+### App Section
+```json
+"app": {
+ "name": "ircquotes", // Application name
+ "host": "127.0.0.1", // Host to bind to (use 0.0.0.0 for all interfaces)
+ "port": 6969, // Port number to run on
+ "debug": false // Enable debug mode (set to true for development)
+}
+```
+
+### Gunicorn Section (Production Settings)
+```json
+"gunicorn": {
+ "workers": 4, // Number of worker processes
+ "timeout": 30, // Request timeout in seconds
+ "keepalive": 5, // Keep-alive timeout
+ "max_requests": 1000, // Max requests per worker before restart
+ "preload": true // Preload application code
+}
+```
+
+### Database Section
+```json
+"database": {
+ "uri": "sqlite:///quotes.db?timeout=20", // Database connection string
+ "pool_timeout": 20, // Connection pool timeout
+ "pool_recycle": -1, // Connection recycle time (-1 = disabled)
+ "pool_pre_ping": true // Test connections before use
+}
+```
+
+### Security Section
+```json
+"security": {
+ "csrf_enabled": true, // Enable CSRF protection
+ "csrf_time_limit": null, // CSRF token time limit (null = no limit)
+ "session_cookie_secure": false, // Require HTTPS for session cookies
+ "session_cookie_httponly": true, // Prevent JavaScript access to session cookies
+ "session_cookie_samesite": "Lax", // SameSite policy for session cookies
+ "security_headers": {
+ "x_content_type_options": "nosniff",
+ "x_frame_options": "DENY",
+ "x_xss_protection": "1; mode=block",
+ "strict_transport_security": "max-age=31536000; includeSubDomains",
+ "content_security_policy": "default-src 'self'; script-src 'self' 'unsafe-inline'; style-src 'self' 'unsafe-inline'; img-src 'self' data:; connect-src 'self'"
+ }
+}
+```
+
+### Admin Section
+```json
+"admin": {
+ "username": "admin", // Admin username
+ "password_hash": "..." // Argon2 password hash (use generate_password.py)
+}
+```
+
+### Quotes Section
+```json
+"quotes": {
+ "min_length": 1, // Minimum quote length in characters
+ "max_length": 5000, // Maximum quote length in characters
+ "per_page": 25, // Quotes displayed per page
+ "auto_approve": false, // Automatically approve new quotes
+ "allow_html": false // Allow HTML in quotes (not recommended)
+}
+```
+
+### Features Section
+```json
+"features": {
+ "voting_enabled": true, // Enable voting on quotes
+ "flagging_enabled": true, // Enable flagging inappropriate quotes
+ "copy_quotes_enabled": true, // Enable copy-to-clipboard feature
+ "dark_mode_enabled": true, // Enable dark mode toggle
+ "api_enabled": true, // Enable JSON API endpoints
+ "bulk_moderation_enabled": true // Enable bulk moderation actions
+}
+```
+
+### Logging Section
+```json
+"logging": {
+ "level": "DEBUG", // Logging level (DEBUG, INFO, WARNING, ERROR)
+ "format": "%(asctime)s [%(levelname)s] %(message)s" // Log message format
+}
+```
+
+## Common Configuration Tasks
+
+### Change Admin Password
+1. Run: `python generate_password.py`
+2. Edit `config.json` and update `admin.password_hash` with the generated hash
+3. Restart the application
+
+### Change Port
+Edit the `app.port` value in `config.json`:
+```json
+"app": {
+ "port": 8080
+}
+```
+
+### Adjust Quote Limits
+Edit the `quotes` section:
+```json
+"quotes": {
+ "min_length": 10,
+ "max_length": 2000,
+ "per_page": 50
+}
+```
+
+### Disable Features
+Set feature flags to `false`:
+```json
+"features": {
+ "voting_enabled": false,
+ "flagging_enabled": false
+}
+```
+
+### Adjust Rate Limits
+Modify the `rate_limiting.endpoints` section:
+```json
+"rate_limiting": {
+ "endpoints": {
+ "submit": "10 per minute",
+ "vote": "120 per minute"
+ }
+}
+```
+
+## Important Notes
+
+- Always restart the application after making configuration changes
+- Use valid JSON syntax (no trailing commas, proper quotes)
+- Test configuration changes in a development environment first
+- Keep backups of your working configuration
+- Use `python generate_password.py` to create secure password hashes
\ No newline at end of file
diff --git a/DEPLOYMENT.md b/DEPLOYMENT.md
index 3fab514..217c0da 100644
--- a/DEPLOYMENT.md
+++ b/DEPLOYMENT.md
@@ -13,24 +13,30 @@ All application settings are now centralized in `config.json`. You can easily mo
- **Admin credentials**
- **Feature toggles**
-### Viewing Current Configuration
+### Configuration Management
+All configuration is done by editing `config.json` directly. This file contains all application settings organized in sections:
+
+- **app**: Basic application settings (name, host, port, debug)
+- **database**: Database connection settings
+- **security**: Security headers, CSRF, proxy settings
+- **rate_limiting**: Rate limiting configuration for different endpoints
+- **admin**: Admin username and password hash
+- **quotes**: Quote submission settings (length limits, pagination)
+- **features**: Feature toggles (voting, flagging, dark mode, etc.)
+- **logging**: Logging configuration
+
+### Example Configuration Changes
```bash
-python config_manager.py
-```
+# Edit config.json in any text editor
+nano config.json
-### Updating Configuration
-```bash
-# Change port
-python config_manager.py app.port 8080
+# Example changes:
+# - Change port: "port": 8080 in the "app" section
+# - Change quotes per page: "per_page": 50 in the "quotes" section
+# - Disable CSRF: "csrf_enabled": false in the "security" section
+# - Change rate limits: "login": "10 per minute" in rate_limiting.endpoints
-# Change quotes per page
-python config_manager.py quotes.per_page 50
-
-# Disable CSRF (not recommended)
-python config_manager.py security.csrf_enabled false
-
-# Change rate limits
-python config_manager.py rate_limiting.endpoints.login "10 per minute"
+# After making changes, restart the application
```
## Running with Gunicorn (Production)
diff --git a/app.py b/app.py
index 779c7ea..ef7f464 100644
--- a/app.py
+++ b/app.py
@@ -1,7 +1,5 @@
from flask import Flask, render_template, request, redirect, url_for, flash, abort, make_response, session, jsonify
from flask_sqlalchemy import SQLAlchemy
-from flask_limiter import Limiter
-from flask_limiter.util import get_remote_address
from flask_cors import CORS
from flask_wtf.csrf import CSRFProtect
import datetime
@@ -9,59 +7,109 @@ import json
import random
from argon2 import PasswordHasher
from argon2.exceptions import VerifyMismatchError
-from werkzeug.middleware.proxy_fix import ProxyFix # Import ProxyFix
import logging
from sqlalchemy import event
from sqlalchemy.engine import Engine
import sqlite3
+import time
+import ipaddress
from config_loader import config # Import configuration system
-def get_real_ip():
+def db_retry_operation(operation, max_retries=2, delay=0.01):
"""
- Get the real client IP address considering Cloudflare and nginx reverse proxy.
- Checks headers in order of priority:
- 1. CF-Connecting-IP (Cloudflare's real IP header)
- 2. X-Forwarded-For (standard proxy header)
- 3. X-Real-IP (nginx real IP header)
- 4. request.remote_addr (fallback)
+ Retry database operations that might fail due to database locks.
+ Includes session cleanup for better reliability.
+
+ Args:
+ operation: A callable that performs the database operation
+ max_retries: Maximum number of retry attempts
+ delay: Initial delay between retries
+
+ Returns:
+ The result of the operation if successful
+
+ Raises:
+ The last exception if all retries fail
"""
- # Cloudflare provides the real IP in CF-Connecting-IP header
- cf_ip = request.headers.get('CF-Connecting-IP')
- if cf_ip:
- return cf_ip
+ last_exception = None
- # Check X-Forwarded-For (may contain multiple IPs, first is original client)
- forwarded_for = request.headers.get('X-Forwarded-For')
- if forwarded_for:
- # Take the first IP in the chain (original client)
- return forwarded_for.split(',')[0].strip()
+ for attempt in range(max_retries + 1):
+ try:
+ return operation()
+ except Exception as e:
+ last_exception = e
+ error_msg = str(e).lower()
+
+ # Handle specific database errors that benefit from retry
+ if ('database is locked' in error_msg or
+ 'sqlite3.operationalerror' in error_msg or
+ 'transaction has been rolled back' in error_msg):
+
+ try:
+ # Only rollback, don't close session to avoid unbound objects
+ db.session.rollback()
+ except:
+ pass # Ignore cleanup errors
+
+ if attempt < max_retries:
+ logging.warning(f"Database error detected, rollback and retry (attempt {attempt + 1}/{max_retries + 1})")
+ time.sleep(delay)
+ continue
+
+ # For non-database errors or final attempt, re-raise immediately
+ raise
- # Check X-Real-IP (nginx header)
- real_ip = request.headers.get('X-Real-IP')
- if real_ip:
- return real_ip
-
- # Fallback to request.remote_addr or default
- return request.remote_addr or '127.0.0.1'
+ # This should never be reached due to the logic above, but just in case
+ if last_exception:
+ raise last_exception
+ else:
+ raise RuntimeError("Database operation failed for unknown reasons")
-# Configure SQLite for better concurrency
+def validate_ip_address(ip_str):
+ """
+ Validate that an IP address string is a valid IPv4 or IPv6 address.
+ Returns a sanitized IP address string or '127.0.0.1' if invalid.
+ """
+ try:
+ # This will raise ValueError if the IP is invalid
+ ip_obj = ipaddress.ip_address(ip_str)
+ return str(ip_obj)
+ except (ValueError, TypeError):
+ # If IP is invalid, return localhost as fallback
+ app.logger.warning(f"Invalid IP address detected: {ip_str}")
+ return '127.0.0.1'
+
+# Configure SQLite for better concurrency and performance
@event.listens_for(Engine, "connect")
def set_sqlite_pragma(dbapi_connection, connection_record):
if isinstance(dbapi_connection, sqlite3.Connection):
cursor = dbapi_connection.cursor()
# Set WAL mode for better concurrency
cursor.execute("PRAGMA journal_mode=WAL")
- # Set timeout for locked database
- cursor.execute("PRAGMA busy_timeout=30000") # 30 seconds
+ # Reduce timeout for faster failures instead of long waits
+ cursor.execute("PRAGMA busy_timeout=1000") # 1 second - faster failure
# Optimize for performance
cursor.execute("PRAGMA synchronous=NORMAL")
- cursor.execute("PRAGMA cache_size=1000")
+ cursor.execute("PRAGMA cache_size=20000") # Larger cache
cursor.execute("PRAGMA temp_store=memory")
+ cursor.execute("PRAGMA mmap_size=268435456") # 256MB memory mapped
+ cursor.execute("PRAGMA wal_autocheckpoint=500") # More frequent checkpoints
+ cursor.execute("PRAGMA optimize") # Enable automatic index optimization
cursor.close()
app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = config.database_uri
app.config['SECRET_KEY'] = open("instance/flask_secret_key", "r").read().strip()
+app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = False
+
+# Enhanced connection pool configuration for better concurrency
+app.config['SQLALCHEMY_ENGINE_OPTIONS'] = {
+ 'pool_size': 10, # Maintain 10 connections in pool
+ 'pool_recycle': 3600, # Recycle connections every hour
+ 'pool_pre_ping': True, # Test connections before use
+ 'pool_timeout': 5, # Wait up to 5 seconds for connection
+ 'max_overflow': 20 # Allow up to 20 additional connections
+}
# Configure secure session settings from config
app.config['SESSION_COOKIE_SECURE'] = config.get('security.session_cookie_secure', False)
@@ -88,18 +136,10 @@ csrf.exempt('get_top_quotes')
csrf.exempt('search_quotes')
csrf.exempt('get_stats')
-# Initialize rate limiter with custom IP detection
-limiter = Limiter(
- key_func=get_real_ip,
- app=app
-)
+# Remove rate limiting - immediate response for all requests
db = SQLAlchemy(app)
-# Apply ProxyFix middleware for Cloudflare + nginx setup
-# x_for=2: nginx (1) + Cloudflare (1) = 2 proxies in X-Forwarded-For chain
-app.wsgi_app = ProxyFix(app.wsgi_app, x_for=2, x_proto=1, x_host=1, x_port=1, x_prefix=1)
-
# Initialize Argon2 password hasher
ph = PasswordHasher()
@@ -136,12 +176,18 @@ class Quote(db.Model):
id = db.Column(db.Integer, primary_key=True)
text = db.Column(db.Text, nullable=False)
votes = db.Column(db.Integer, default=0)
- date = db.Column(db.DateTime, default=datetime.datetime.utcnow)
- status = db.Column(db.Integer, default=0) # 0 = pending, 1 = approved, 2 = rejected
+ date = db.Column(db.DateTime, nullable=True) # Legacy field for old quotes
+ status = db.Column(db.Integer, default=0, index=True) # 0 = pending, 1 = approved, 2 = rejected
ip_address = db.Column(db.String(45)) # Store IPv4 and IPv6 addresses
user_agent = db.Column(db.String(255)) # Store user-agent strings
- submitted_at = db.Column(db.DateTime, default=datetime.datetime.utcnow)
- flag_count = db.Column(db.Integer, default=0) # Track how many times quote has been flagged
+ submitted_at = db.Column(db.DateTime, nullable=True) # New timestamp field for new quotes
+ flag_count = db.Column(db.Integer, default=0, index=True) # Track how many times quote has been flagged
+
+ # Add composite indexes for common queries
+ __table_args__ = (
+ db.Index('idx_status_id', 'status', 'id'),
+ db.Index('idx_flag_count_id', 'flag_count', 'id'),
+ )
# Home route to display quotes
@app.route('/')
@@ -157,10 +203,11 @@ def index():
# Separate route for submitting quotes
@app.route('/submit', methods=['GET', 'POST'])
-@limiter.limit(config.get('rate_limiting.endpoints.submit', '5 per minute'))
def submit():
if request.method == 'POST':
quote_text = request.form.get('quote')
+ is_preview = 'submit2' in request.form # Preview button is named submit2
+
if not quote_text:
flash("Oops! Your quote seems to be empty. Please enter some text before submitting.", 'error')
return redirect(url_for('submit'))
@@ -183,8 +230,18 @@ def submit():
if '
+
@@ -125,7 +126,15 @@
0
{% endif %}
- | {{ quote.submitted_at.strftime('%Y-%m-%d %H:%M:%S') if quote.submitted_at else 'N/A' }} |
+
+ {% if quote.submitted_at %}
+ {{ quote.submitted_at.strftime('%d/%m/%y %H:%M:%S') }}
+ {% elif quote.date %}
+ {{ quote.date.strftime('%d/%m/%y') }} (legacy)
+ {% else %}
+ No date
+ {% endif %}
+ |
{{ quote.ip_address|e }} |
{{ quote.user_agent|e|truncate(50) }} |
diff --git a/templates/quote.html b/templates/quote.html
index 2c9b089..2f044c7 100644
--- a/templates/quote.html
+++ b/templates/quote.html
@@ -68,6 +68,14 @@
X
C
+
+
+ {% if quote.submitted_at %}
+ {{ quote.submitted_at.strftime('%d/%m/%y %H:%M') }}
+ {% elif quote.date %}
+ {{ quote.date.strftime('%d/%m/%y') }}
+ {% endif %}
+
{{ quote.text|e }}
diff --git a/templates/random.html b/templates/random.html
index 4607ad8..e73a7e2 100644
--- a/templates/random.html
+++ b/templates/random.html
@@ -64,6 +64,14 @@
X
C
+
+
+ {% if quote.submitted_at %}
+ {{ quote.submitted_at.strftime('%d/%m/%y %H:%M') }}
+ {% elif quote.date %}
+ {{ quote.date.strftime('%d/%m/%y') }}
+ {% endif %}
+
{{ quote.text|e }}
|
diff --git a/templates/search.html b/templates/search.html
index 4860a21..a90e1df 100644
--- a/templates/search.html
+++ b/templates/search.html
@@ -94,6 +94,14 @@
X
C
+
+
+ {% if quote.submitted_at %}
+ {{ quote.submitted_at.strftime('%d/%m/%y %H:%M') }}
+ {% elif quote.date %}
+ {{ quote.date.strftime('%d/%m/%y') }}
+ {% endif %}
+
{{ quote.text|e }}
diff --git a/templates/submit.html b/templates/submit.html
index 9d6ed31..6e008fb 100644
--- a/templates/submit.html
+++ b/templates/submit.html
@@ -17,7 +17,7 @@
-
+