Add enterprise readiness: governance, observability, local LLMs, scaling, deployment

Governance: RBAC access policy, OIDC bearer-token verification (with real
alg-confusion/downgrade-attack tests), a durable SQLite-backed audit log,
versioned Data Contracts for CI gates, and CycloneDX SBOM generation.

Providers: Azure OpenAI, Bedrock Anthropic, and a generic openai_compatible
client, plus first-class local-LLM providers (ollama, lmstudio, vllm,
llamacpp) with a network_policy=local structural guarantee that calls can
never leave a loopback/private address. Secrets resolve through a swappable
backend (env, Vault, AWS/GCP/Azure secret managers) and datasets read/write
cloud storage (s3/gs/az) directly.

Observability: OpenTelemetry spans and Prometheus metrics per pipeline
stage, with a node_exporter textfile-collector sidecar for batch runs.

Scaling: dataset sharding for external-orchestrator parallelism
(Airflow/Dagster/k8s examples) plus a Ray-based in-process alternative,
both verified against real local runtimes.

Deployment: a Dockerfile, Helm chart, and Terraform module for AWS,
clearly labeled where they could not be executed in this environment.

Also: the accuracy-gated generation pipeline, plugin registry, cloud
dataset formats, a 5-dataset x 3-scale real-world recovery benchmark
(+8.14 accuracy points average, full evidence in benchmarks/), a
gemini-3.5-flash-lite vs gemini-3.7-flash judge-model comparison, and a
docs/ reference layer covering all of the above.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Author: voidstackloop

.env.example

@@ -1,8 +1,13 @@
-# Gemini API Key (Required for Gemini API calls)
-GEMINI_API_KEY=your_gemini_api_key_here
-
-# Optional Default Configurations
-BUFFDATA_DEFAULT_MODEL=gemini-3.7-flash
-BUFFDATA_EMBEDDING_MODEL=text-embedding-004
-BUFFDATA_CONCURRENCY=10
-BUFFDATA_MAX_RPM=60
+# Gemini API Key (Required for Gemini API calls)
+GEMINI_API_KEY=your_gemini_api_key_here
+OPENAI_API_KEY=your_openai_api_key_here
+ANTHROPIC_API_KEY=your_anthropic_api_key_here
+# Optional: enables `buffdata productivity docs-*` live documentation lookups.
+CONTEXT7_API_KEY=your_context7_api_key_here
+
+# Optional Default Configurations
+BUFFDATA_DEFAULT_MODEL=gemini-3.7-flash
+BUFFDATA_PROVIDER=gemini
+BUFFDATA_EMBEDDING_MODEL=text-embedding-004
+BUFFDATA_CONCURRENCY=10
+BUFFDATA_MAX_RPM=60

.gitignore

@@ -1,17 +1,18 @@
-.venv/
-__pycache__/
-*.pyc
-*.pyo
-*.pyd
-.pytest_cache/
-.coverage
-htmlcov/
-dist/
-build/
-*.egg-info/
-.env
-checkpoints/
-outputs/
-*.parquet
-*.jsonl
-!examples/*.jsonl
+.venv/
+__pycache__/
+*.pyc
+*.pyo
+*.pyd
+.pytest_cache/
+.coverage
+htmlcov/
+dist/
+build/
+*.egg-info/
+.env
+checkpoints/
+outputs/
+*.parquet
+*.jsonl
+!examples/*.jsonl
+graphify-out/

.graphifyignore

@@ -0,0 +1,6 @@
+.git/
+.venv/
+__pycache__/
+benchmarks/results/
+benchmarks/results-large/
+graphify-out/