⚡ VaultS3 — Multi-S3 Cloud Manager & Loss-Free Auto-Backup System

VaultS3 is an ultra-fast, developer-friendly, serverless-ready multi-S3 account cloud storage management system. Connect and manage multiple S3 storage providers (AWS S3, Cloudflare R2, Wasabi, MinIO, DigitalOcean Spaces, Backblaze B2) under a unified interface with Loss-Free Auto-Backup Redundancy, Exact Storage Analytics, Capacity Capping, and Non-Blocking Stream Uploads/Downloads.

🌟 Highlights & Key Features

📦 1. Unified Multi-S3 Storage Aggregation

Multi-Provider Support: Connect Amazon S3, Cloudflare R2, MinIO, Wasabi, DigitalOcean Spaces, or any custom S3-compliant API.
Dynamic Account Management: Add, edit, toggle active/inactive status, and run real-time S3 connection diagnostics with instant UI feedback.
S3 Connection Pooling & Socket Cache: Reuses TCP connection pools to prevent socket leaks and connection errors over extended upload sessions.

🛡️ 2. Loss-Free Guarantee & Auto-Backup System

Automated Upload Replication: Automatically marks new uploads for cross-S3 redundancy when auto_backup_enabled is set.
Failover Download Fallback: If a file's primary S3 provider becomes inactive or unavailable, VaultS3 automatically retrieves the file from an active backup S3 account with 0% data loss.
One-Click Bulk Replication: System-wide POST /api/files/backup-all trigger to replicate all indexed files across redundant secondary S3 storage in seconds.

📊 3. Capacity Capping & Storage Analytics

Exact Storage Metrics: Live calculation of Usable Upload Space, Backup Reserve Space, Used Storage, Free Storage, and Total Indexed Files.
Per-Account Storage Allocation: Set hard storage capacity limits (e.g., 50 GB) and custom backup allocation percentages per S3 account.
Hard Cap Enforcement: Prevents bucket overflow by validating real-time usage before issuing presigned upload signatures.

⚡ 4. High-Performance Stream Transfer Engine

Non-Blocking Transfer Progress: Real-time progress tracking with accurate transfer speeds (MB/s / KB/s) and batch upload queue tracking (File X of Y).
Direct-to-S3 Presigned Transfers: Direct browser-to-bucket uploads and downloads bypass server bottlenecks for maximum bandwidth.
Error Diagnostics: Human-readable XML error extraction for S3 bucket policy and CORS troubleshooting.

🌐 5. Dual Runtime & Edge Ready

Local Dev Server: Fast Express + SQLite (sql.js) development environment running on Node.js / tsx.
Production Edge Serverless: Cloudflare Pages & Functions (functions/api/[[path]].ts) with Cloudflare D1 Serverless Database binding.
PWA Installable: Offline-capable Progressive Web App with Service Worker auto-update and iOS Safari setup guides.

🏗️ Architecture Overview

               ┌───────────────────────────────────────────────┐
               │              VaultS3 Frontend                 │
               │   React 19 + Vite + Tailwind CSS + PWA UI    │
               └───────────────────────┬───────────────────────┘
                                       │
                      ┌────────────────┴────────────────┐
                      │  Presigned Direct S3 Transfers  │
                      └────────────────┬────────────────┘
                                       │
      ┌────────────────────────────────┼────────────────────────────────┐
      │                                │                                │
┌─────▼───────────────┐      ┌─────────▼─────────────┐      ┌───────────▼───────────┐
│   AWS S3 Bucket     │      │ Cloudflare R2 Bucket  │      │ Wasabi / MinIO / DO   │
│  Primary Storage    │      │ Secondary Backup      │      │ Custom S3 Endpoints   │
└─────────────────────┘      └───────────────────────┘      └───────────────────────┘
                                       │
                      ┌────────────────┴────────────────┐
                      │    VaultS3 Serverless Engine    │
                      │ Express (Dev) / Pages (Prod)    │
                      └────────────────┬────────────────┘
                                       │
                      ┌────────────────┴────────────────┐
                      │   Metadata & Indexing Database  │
                      │  SQLite (Dev) / Cloudflare D1   │
                      └─────────────────────────────────┘

⚡ Quick Start

Prerequisites

Node.js: v18.0.0 or higher
npm or bun or yarn

1. Clone the Repository

git clone https://github.com/your-username/vaults3.git
cd vaults3

2. Install Dependencies

npm install

3. Setup Environment Variables

Copy the example environment file and configure your JWT secret and database path:

cp .env.example .env

.env file configuration:

# Server Configuration
PORT=3000
JWT_SECRET=your_super_secret_jwt_key_here_change_in_production

# S3 Vault Development Database
DEV_DB_PATH=s3vault_dev.sqlite

4. Run Development Server

npm run dev

Open http://localhost:3000 in your browser. Default login credentials will be generated or created on first sign-up.

🚀 Deployment

A. Deploy to Cloudflare Pages & D1 Database (Complete A-Z Guide)

VaultS3 is natively designed for zero-latency Cloudflare Pages Functions deployment coupled with a Cloudflare D1 Serverless SQLite Database. Follow this complete step-by-step guide to build, configure, run locally, and deploy to production.

📋 Step 1: Prerequisites & Cloudflare Authentication

Ensure you have Node.js (v18+) installed, then log in to your Cloudflare account via Wrangler CLI:

# Verify Node.js version
node -v

# Login to Cloudflare via Wrangler
npx wrangler login

🗄️ Step 2: Provision Cloudflare D1 Database

Create a new Cloudflare D1 Database instance using Wrangler:

npx wrangler d1 create s3vault-d1

Output Example:

✅ Created database 's3vault-d1' with ID 'a1b2c3d4-e5f6-7890-abcd-1234567890ef'

[[d1_databases]]
binding = "DB"
database_name = "s3vault-d1"
database_id = "a1b2c3d4-e5f6-7890-abcd-1234567890ef"

⚙️ Step 3: Configure `wrangler.jsonc`

VaultS3 includes a lightweight wrangler.jsonc for Cloudflare Pages:

{
  "$schema": "node_modules/wrangler/config-schema.json",
  "name": "s3vault-personal-storage",
  "pages_build_output_dir": "./dist",
  "compatibility_date": "2026-03-01",
  "compatibility_flags": [
    "nodejs_compat"
  ]
}

⚠️ CRITICAL: Connecting D1 Database (Choose One of Two Methods):

When wrangler.jsonc exists in your repository, Cloudflare Pages operates in "Config as Code" mode and locks the dashboard bindings UI (displaying: "Bindings for this project are being managed through wrangler.toml").

Choose the method that fits your workflow:

Method 1: Connect via wrangler.jsonc (Recommended for Git-based configuration)

In Cloudflare Dashboard, go to Storage & Databases ➔ D1 SQL Database ➔ Click s3vault-d1.

Copy the Database ID (e.g. a1b2c3d4-e5f6-7890-abcd-1234567890ef).

Add the d1_databases array to wrangler.jsonc with your real ID:
{
  "$schema": "node_modules/wrangler/config-schema.json",
  "name": "s3vault-personal-storage",
  "pages_build_output_dir": "./dist",
  "compatibility_date": "2026-03-01",
  "compatibility_flags": [
    "nodejs_compat"
  ],
  "d1_databases": [
    {
      "binding": "DB",
      "database_name": "s3vault-d1",
      "database_id": "PASTE_YOUR_COPIED_D1_DATABASE_ID_HERE"
    }
  ]
}
Push to GitHub. Cloudflare Pages binds your database automatically.

Method 2: Connect via Cloudflare Dashboard Settings UI (Without editing git)

Delete wrangler.jsonc from the repository root.

Push to GitHub.

Cloudflare Pages will immediately unlock the Dashboard bindings UI under Settings → Functions → D1 database bindings.

Click Add binding ➔ Variable name: DB ➔ D1 database: select s3vault-d1.

Add compatibility flag nodejs_compat under Settings → Functions → Compatibility flags.

🗃️ Step 4: Database Schema Initialization

VaultS3 includes automatic D1 schema initialization. On the very first request to functions/api/[[path]].ts, the serverless function automatically executes ensureD1Tables(env.DB) to create all 4 core tables:

users: Master admin authentication accounts.
s3_accounts: S3 provider credentials, buckets, and capacity allocations.
file_metadata: Indexed file records, folder paths, and backup status.
s3_settings: Global storage rules and chunk size configurations.

(Optional) If you wish to manually seed/apply the schema to remote D1:

# Create schema.sql file
cat << 'EOF' > schema.sql
CREATE TABLE IF NOT EXISTS users (
  id TEXT PRIMARY KEY,
  email TEXT UNIQUE NOT NULL,
  password_hash TEXT NOT NULL,
  role TEXT NOT NULL DEFAULT 'admin',
  created_at TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS s3_accounts (
  id TEXT PRIMARY KEY,
  user_id TEXT NOT NULL,
  provider TEXT NOT NULL,
  name TEXT NOT NULL,
  access_key_id TEXT NOT NULL,
  secret_access_key TEXT NOT NULL,
  region TEXT NOT NULL,
  endpoint TEXT NOT NULL,
  bucket TEXT NOT NULL,
  total_capacity_gb REAL NOT NULL DEFAULT 10.0,
  used_capacity_bytes INTEGER NOT NULL DEFAULT 0,
  backup_allocation_pct INTEGER NOT NULL DEFAULT 20,
  is_active INTEGER NOT NULL DEFAULT 1,
  created_at TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS file_metadata (
  id TEXT PRIMARY KEY,
  user_id TEXT NOT NULL,
  s3_account_id TEXT NOT NULL,
  key TEXT NOT NULL,
  name TEXT NOT NULL,
  folder TEXT NOT NULL DEFAULT '/',
  mime_type TEXT NOT NULL,
  size_bytes INTEGER NOT NULL,
  etag TEXT,
  last_modified TEXT NOT NULL,
  storage_class TEXT DEFAULT 'STANDARD',
  is_backed_up INTEGER NOT NULL DEFAULT 0,
  created_at TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS s3_settings (
  id TEXT PRIMARY KEY,
  user_id TEXT UNIQUE NOT NULL,
  upload_usage_pct INTEGER NOT NULL DEFAULT 80,
  backup_usage_pct INTEGER NOT NULL DEFAULT 20,
  chunk_size_mb INTEGER NOT NULL DEFAULT 5,
  auto_backup_enabled INTEGER NOT NULL DEFAULT 1,
  encryption_enabled INTEGER NOT NULL DEFAULT 1
);
EOF

# Execute schema on remote D1 Database
npx wrangler d1 execute s3vault-d1 --file=./schema.sql --remote

🧪 Step 5: Test Locally with Cloudflare Wrangler Simulator

You can simulate full Cloudflare Pages Functions and D1 locally before deploying:

# 1. Build the Vite production bundle
npm run build

# 2. Run local Cloudflare Pages server with D1 emulator
npx wrangler pages dev dist --d1=DB

Now access http://localhost:8788 to test local serverless functions and D1 database queries in real-time.

🚀 Step 6: Deploy to Cloudflare Pages

Method A: Deploy via Wrangler CLI (Fastest)

# Build production bundle
npm run build

# Deploy directly to Cloudflare Pages
npx wrangler pages deploy dist --project-name=vaults3

Method B: Deploy via GitHub / Cloudflare Dashboard

Push your code to your GitHub / GitLab repository.
Log in to Cloudflare Dashboard -> Workers & Pages -> Create Application -> Pages -> Connect to Git.
Select your repository and configure build settings:
- Project Name: s3vault-personal-storage (or your choice)
- Production Branch: main
- Framework Preset: None (or Vite)
- Build Command: npm run build
- Build Output Directory: dist
Click Save and Deploy.

💡 Notice: Cloudflare triggers the initial build immediately. Because wrangler.jsonc does not include placeholder invalid database IDs, this initial deployment will succeed and publish your assets cleanly.
Bind D1 Database After Initial Deploy:
- Once the deployment finishes, the Settings tab becomes accessible.
- Go to Project Settings -> Functions -> D1 Database Bindings.
- Click Add binding:
  - Variable Name: DB (Must be uppercase DB)
  - D1 Database: Select your s3vault-d1 database
- Click Save. Any subsequent deployment or retry will automatically connect your D1 database.

🔐 Step 7: Configure Production Environment Variables

In Cloudflare Dashboard -> Pages -> vaults3 -> Settings -> Environment Variables, add:

JWT_SECRET: your_secure_random_production_jwt_key_32_chars_min

🌐 Step 8: S3 Bucket CORS Configuration

To allow direct browser uploads/downloads to your S3 storage from your deployed Cloudflare Pages domain (e.g. https://vaults3.pages.dev), add the following CORS policy in your S3/R2 bucket dashboard:

[
  {
    "AllowedHeaders": ["*"],
    "AllowedMethods": ["GET", "PUT", "DELETE", "HEAD"],
    "AllowedOrigins": [
      "https://vaults3.pages.dev",
      "http://localhost:3000",
      "http://localhost:8788"
    ],
    "ExposeHeaders": ["ETag"],
    "MaxAgeSeconds": 3000
  }
]

B. Deploy with Node.js / Docker / VPS

Build the static Vite frontend and run the Express backend on any Node.js host:

# Build production bundle
npm run build

# Start Node.js production server
npm start

⚙️ Setting Up S3 Storage Providers

VaultS3 supports any S3-compliant cloud storage. When adding a new provider in Settings → S3 Storage Accounts, fill in:

Provider	Endpoint URL Syntax	Region
AWS S3	Leave blank (default AWS API)	e.g. `us-east-1`
Cloudflare R2	`https://<ACCOUNT_ID>.r2.cloudflarestorage.com`	`auto`
Wasabi	`https://s3.wasabisys.com` or region endpoint	e.g. `us-east-1`
MinIO	`https://minio.yourdomain.com`	`us-east-1`
DigitalOcean	`https://<REGION>.digitaloceanspaces.com`	e.g. `nyc3`

Note on CORS: Ensure your S3 bucket has CORS enabled for GET, PUT, DELETE, and HEAD requests from your VaultS3 origin domain.

📡 API Reference Overview

VaultS3 exposes clean RESTful API endpoints for auth, file metadata, S3 accounts, and backups:

Authentication

POST /api/auth/login — Sign in and receive JWT token.
POST /api/auth/register — Initialize master account credentials.

File & Storage Operations

GET /api/files — Fetch file metadata list with search & category filters.
POST /api/files/upload — Request presigned upload URL with S3 capacity cap check.
GET /api/files/download — Get presigned download URL with automatic secondary S3 fallback.
DELETE /api/files/delete — Remove file metadata and delete object from S3.

S3 Multi-Account Management

GET /api/s3/accounts — List connected S3 accounts with dynamic storage usage metrics.
POST /api/s3/accounts — Register new S3 credentials & bucket connection.
PUT /api/s3/accounts — Update existing account caps, backup reserve, or status.
DELETE /api/s3/accounts — Remove S3 provider account and clear connection pool.
POST /api/s3/test — Execute live S3 connection test (HeadBucket).

Storage Rules & Backup Operations

GET /api/s3/settings — Get global user backup rules and allocation limits.
POST /api/s3/settings — Save storage allocation rules & auto-backup toggles.
POST /api/files/backup — Replicate single file to secondary S3 storage.
POST /api/files/backup-all — System-wide backup replication across all files.

🧪 Verification & Quality Control

VaultS3 maintains high code quality standards:

# Type check & lint
npm run lint

# Build verification
npm run build

🤝 Contributing

Contributions, bug reports, and feature suggestions are highly appreciated! Please read our Contributing Guidelines and Code of Conduct before submitting Pull Requests.

Fork the repository
Create your feature branch (git checkout -b feature/AmazingFeature)
Commit your changes (git commit -m 'Add amazing feature')
Push to the branch (git push origin feature/AmazingFeature)
Open a Pull Request

📄 License

Distributed under the MIT License. See LICENSE for details.

Designed & Built with ❤️ for Developers, Power Users, and Self-Hosters.

Name		Name	Last commit message	Last commit date
functions/api		functions/api
public		public
src		src
.cursorrules		.cursorrules
.env.example		.env.example
.gitignore		.gitignore
AI_INSTRUCTIONS.md		AI_INSTRUCTIONS.md
CHANGELOG.md		CHANGELOG.md
CODE_OF_CONDUCT.md		CODE_OF_CONDUCT.md
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
MEMORY.md		MEMORY.md
README.md		README.md
SECURITY.md		SECURITY.md
index.html		index.html
metadata.json		metadata.json
package.json		package.json
schema.sql		schema.sql
server.ts		server.ts
tsconfig.json		tsconfig.json
vite.config.ts		vite.config.ts
wrangler.jsonc		wrangler.jsonc

Latest commit

History

Folders and files

Repository files navigation