diff --git a/.claude/settings.json b/.claude/settings.json new file mode 100644 index 0000000..8e47ae2 --- /dev/null +++ b/.claude/settings.json @@ -0,0 +1,28 @@ +{ + "permissions": { + "allow": [ + "Bash(git:*)", + "Bash(python:*)", + "Bash(python3:*)", + "Bash(pip:*)", + "Bash(poetry:*)", + "Bash(ls:*)", + "Bash(find:*)", + "Bash(grep:*)", + "Bash(jq:*)", + "Bash(echo:*)", + "Read(*)" + ], + "deny": [ + "Read(.env)", + "Bash(rm -rf /*)", + "Bash(git reset --hard:*)", + "Bash(git rebase:*)", + "Bash(git merge:*)", + "Bash(git branch -D:*)", + "Bash(gh pr merge:*)", + "Bash(gh pr close:*)", + "Bash(gh pr review:*)" + ] + } +} diff --git a/.claude/skills/scaffold-example/SKILL.md b/.claude/skills/scaffold-example/SKILL.md new file mode 100644 index 0000000..baeea7a --- /dev/null +++ b/.claude/skills/scaffold-example/SKILL.md @@ -0,0 +1,80 @@ +--- +name: scaffold-example +description: Scaffold a brand-new Comet example in this repo from the canonical template under templates/integration-example/. Use whenever the user wants to create, add, or start a new example, demo, or integration on Comet — phrasings like "add a Comet example for X", "scaffold an integration example", "create a new example for ", "start a hello-world for ", "set up a new example project". Do NOT hand-build the folder structure or copy files manually — this skill stamps out the example and renames it correctly. Require a name and a one-line description of what it does; refuse to invent an example from nothing. +--- + +# Scaffold a new Comet example + +This repo has one canonical example shape, a real, runnable template under +[`templates/integration-example/`](../../../templates/integration-example): a `uv` project +(`pyproject.toml`), a single `comet_ml` script (`example_integration.py`), and a README in the house +structure (Title → Documentation → See it → Setup → Run the example). Getting it right by hand is +fiddly and easy to drift on. This skill copies the template and renames it, so you start from +something that already runs. + +## When to refuse + +You need two things: + +- A **name** (kebab-case, e.g. `pytorch-amp-example`, `keras-mnist-dnn`). It becomes the folder, the + Python module, and the Comet project name (`comet-example-`). +- A **one-line description** of what it does. + +If either is missing, ask one short question. Don't invent an example — an empty scaffold with a +guessed purpose wastes the user's time. + +## Step 1 — Generate the example + +Run the bundled generator. It copies the template, renames its sentinel identity +(`example_integration` / `example-integration`) to the new name, and refuses to overwrite an +existing folder. + +```bash +python .claude/skills/scaffold-example/scripts/scaffold.py \ + --description "" \ + --dest +``` + +`--dest` is the parent directory (default `integrations`). New examples live under +`integrations///`, so pass e.g. +`--dest integrations/model-training/pytorch`. Categories in use: `model-training`, +`model-evaluation`, `model-optimization`, `model-deployment`, `workflow-orchestration`, +`reinforcement-learning`, `llm`, `data-management`. The script prints the folder / script / project +names it chose and a next-steps checklist. + +The name is normalised: the folder and Comet project use kebab-case, the Python file uses +snake_case (so it stays importable). The description is written into the `pyproject.toml`. + +## Step 2 — Make it real + +The scaffold runs as-is but is generic. Fill the parts marked `TODO`: + +- **`.py`** — replace the stub loop with the real instrumentation: your framework's training + or inference, plus the `comet_ml` logging that matters (`log_parameters`, `log_metrics`, + `log_model`, …). Keep the `comet_ml.login(...)` + `comet_ml.start()` + `experiment.end()` frame. +- **`pyproject.toml`** — add the framework dependency alongside the pinned `comet_ml`. +- **`README.md`** — fill the title, intro (paste the description), the docs link, and a public + Comet project link if one exists. This is required for every example (see + [AGENTS.md](../../../AGENTS.md)). + +Keep the repo conventions: credentials from env vars only, `# WHY:`-only comments, no emojis. + +## Step 3 — Verify + +From the new example directory: + +```bash +uv sync +uv run python .py # logs a real run (needs COMET_API_KEY) +COMET_MODE=offline uv run python .py # runs without an account, if practical +``` + +To get the example tested, add it to the matrix in +[`.github/workflows/test-examples.yml`](../../../.github/workflows/test-examples.yml) — the +`notebooks` list for a `.ipynb`, or the `example` list for a script. See +[CONTRIBUTING.md](../../../CONTRIBUTING.md#ci). + +## What this skill does NOT do + +- **No git commit / PR.** Leave the new files in the working tree for the user to review. +- **No domain logic.** It scaffolds; you (with the user) write the actual instrumentation, deps, and README. diff --git a/.claude/skills/scaffold-example/scripts/scaffold.py b/.claude/skills/scaffold-example/scripts/scaffold.py new file mode 100644 index 0000000..b1e0bbb --- /dev/null +++ b/.claude/skills/scaffold-example/scripts/scaffold.py @@ -0,0 +1,148 @@ +#!/usr/bin/env python3 +"""Scaffold a new Comet example by copying and renaming the canonical template. + +One template lives under `templates/integration-example/`: a `pyproject.toml`, a single `comet_ml` +script, and a README in the house structure. It is a real, runnable example under a +sentinel identity (`example_integration` / `example-integration`). This script copies it to the +target directory and rewrites those sentinels to the new name — a pure identifier rename, no +template language. Run it, then fill in the TODO stubs. + + python scaffold.py pytorch-amp-example --description "..." --dest integrations/model-training/pytorch +""" + +import argparse +import re +import shutil +import sys +from pathlib import Path + +TEMPLATE_DIR = ("templates", "integration-example") +PKG_SENTINEL = "example_integration" # snake_case: the .py module name +PROJECT_SENTINEL = "example-integration" # kebab-case: folder + comet project name + +EXCLUDE = shutil.ignore_patterns( + ".venv", "uv.lock", "__pycache__", "*.pyc", ".ruff_cache", ".env", "*.log", ".tmp" +) + + +def to_snake(name: str) -> str: + s = re.sub(r"[^0-9a-zA-Z]+", "_", name.strip().lower()).strip("_") + if not s: + raise ValueError(f"cannot derive a name from {name!r}") + if s[0].isdigit(): + s = f"x_{s}" + return s + + +def to_kebab(snake: str) -> str: + return snake.replace("_", "-") + + +def is_text(path: Path) -> bool: + try: + path.read_text(encoding="utf-8") + return True + except (UnicodeDecodeError, OSError): + return False + + +def main() -> int: + parser = argparse.ArgumentParser( + description="Scaffold a new Comet example from templates/integration-example." + ) + parser.add_argument("name", help="Example name, e.g. 'pytorch-amp-example'.") + parser.add_argument( + "--description", + default=None, + help="One-line description (written to pyproject.toml).", + ) + parser.add_argument( + "--dest", + default="integrations", + help="Parent directory for the new example (default: integrations). " + "Relative paths are resolved from the repo root; absolute paths are used as-is.", + ) + parser.add_argument( + "--repo-root", + default=None, + help="Repo root (default: inferred from this script).", + ) + args = parser.parse_args() + + repo_root = ( + Path(args.repo_root).resolve() + if args.repo_root + else Path(__file__).resolve().parents[4] + ) + template = repo_root.joinpath(*TEMPLATE_DIR) + + pkg = to_snake(args.name) + kebab = to_kebab(pkg) + + if not template.is_dir(): + print(f"error: template not found at {template}", file=sys.stderr) + return 1 + + dest_parent = ( + Path(args.dest) if Path(args.dest).is_absolute() else repo_root / args.dest + ) + dest = dest_parent / kebab + if dest.exists(): + print(f"error: {dest} already exists — refusing to overwrite.", file=sys.stderr) + return 1 + + dest.parent.mkdir(parents=True, exist_ok=True) + shutil.copytree(template, dest, ignore=EXCLUDE) + + # Rename the module file (script kind). + mod_file = dest / f"{PKG_SENTINEL}.py" + if mod_file.is_file(): + mod_file.rename(dest / f"{pkg}.py") + + # Rewrite sentinels in every text file. Snake and kebab sentinels don't overlap + # (underscore vs hyphen), so independent replacement is safe. + for path in dest.rglob("*"): + if not path.is_file() or not is_text(path): + continue + text = path.read_text(encoding="utf-8") + new = text.replace(PKG_SENTINEL, pkg).replace(PROJECT_SENTINEL, kebab) + if new != text: + path.write_text(new, encoding="utf-8") + + # Description -> pyproject.toml. + if args.description: + pyproject = dest / "pyproject.toml" + text = pyproject.read_text(encoding="utf-8") + text = re.sub( + r'^description = ".*"$', + f'description = "{args.description}"', + text, + count=1, + flags=re.MULTILINE, + ) + pyproject.write_text(text, encoding="utf-8") + + try: + rel = dest.relative_to(repo_root) + except ValueError: + rel = dest + print(f"Scaffolded {rel}") + print(f" folder: {kebab}/") + print(f" script: {pkg}.py") + print(f" comet project: comet-example-{kebab}") + if args.description: + print(f" description: {args.description}") + print("\nNext steps:") + print(f" 1. cd {rel} && uv sync") + print( + f" 2. Fill in {pkg}.py (real logic), pyproject.toml deps, and the README sections." + ) + print( + f" 3. Run it: uv run python {pkg}.py (or: COMET_MODE=offline uv run python {pkg}.py)" + ) + print(" 4. To test it in CI, add it to .github/workflows/test-examples.yml.") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/.gitignore b/.gitignore index 7c37f9f..b2d6926 100644 --- a/.gitignore +++ b/.gitignore @@ -108,3 +108,8 @@ data/* # pytest .pytest_cache/ MagicMock/ + +# Tooling / agent harness +.ruff_cache/ +.tmp/ +*-workspace/ # skill eval scratch (e.g. skill-creator benchmarks) — never commit diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..fc33d70 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,146 @@ +# Agent instructions for comet-examples + +This repo is a reference library of examples for [Comet](https://www.comet.com/site/), the ML +experiment-tracking, model-management, and observability platform. Examples instrument popular ML +frameworks (PyTorch, Keras, fastai, scikit-learn, XGBoost, Transformers, …) with `comet_ml`. Read +this file before generating or editing any code. + +## Repo structure + +``` +comet-examples/ +├── integrations/ # Add Comet to a framework, grouped by ML task (see below) +├── guides/ # How-to notebooks for Comet workflows +├── panels/ # Custom Comet panel (visualization) examples +├── notebooks/ # General/standalone notebooks +└── templates/ # Starter template (integration-example) +``` + +`integrations/` is grouped by ML task, then by framework, then by example: + +``` +integrations//// +``` + +Categories in use: `model-training`, `model-evaluation`, `model-optimization`, `model-deployment`, +`workflow-orchestration`, `reinforcement-learning`, `llm`, `data-management`. + +**Where does new code belong?** + +| If you are… | Put it in | +|---|---| +| Instrumenting a framework with Comet (the common case) | `integrations////` | +| Writing a how-to notebook for a Comet workflow | `guides/` | +| Building a custom Comet panel | `panels/` | + +New examples go under `integrations/`. The top-level legacy dirs (`pytorch/`, `fastai/`, +`xgboost/`, `keras/`, `tensorflow/`) are historical — **do not add new examples there.** Example +folder names are **kebab-case** (`pytorch-mnist`, `fastai-hello-world`). + +When adding an example, use the [`scaffold-example`](.claude/skills/scaffold-example/SKILL.md) +skill — it stamps [`templates/integration-example/`](templates/integration-example/) into the +target directory and renames it for you. + +## Non-negotiable conventions + +### Credentials — always from environment variables + +```python +import os + +COMET_API_KEY = os.environ.get("COMET_API_KEY") +COMET_WORKSPACE = os.environ.get("COMET_WORKSPACE") +``` + +Never hardcode keys. Never read `.env` files in example code. In CI the workspace is +`cometexamples-tests` and `COMET_API_KEY` comes from a secret. + +### The Comet SDK idiom + +Match the house style used across the repo: + +```python +import comet_ml + +comet_ml.login(project_name="comet-example-") +experiment = comet_ml.start() +# ... training / logging ... +experiment.end() +``` + +`comet_ml.login()` reads `COMET_API_KEY` from the environment. Project names follow +`comet-example--`. + +### Offline mode — the no-account path (recommended, not required) + +Most examples log a real run, so they need an API key. Where it is practical, let a reader run +without an account using Comet's offline mode — no code change required: + +```bash +COMET_MODE=offline uv run python .py +``` + +Note this in the README's run section when it works. Don't force it onto examples that genuinely +need a live run. + +### Comments — only when the WHY is non-obvious + +These are teaching examples: keep them short and readable. Don't add comments that restate what the +code does. Use a one-line `# WHY:` comment only when a behaviour would surprise a reader (a hidden +API constraint, a non-obvious ordering requirement, a known gotcha). No type-hint or docstring +mandate — match the surrounding file. + +### Dependencies — new examples use `uv` + `pyproject.toml` + +New examples are `uv` projects: declare dependencies in `pyproject.toml` (the single source of +truth), install with `uv sync`, and run with `uv run`. Pin `comet_ml>=3.44.0` plus the framework +deps. Don't assume a repo-wide virtualenv is active. + +Existing examples that ship a `requirements.txt` (most of the repo) are **legacy and stay as they +are** — don't migrate them in passing. Only new examples follow the `uv` convention. + +## Coding best practices + +- **Principles:** DRY, KISS, YAGNI. Prefer reusing an existing helper over adding a new one. +- **Match the surrounding file's** style, naming, and comment density. +- **Git / PR safety:** + - Never `git commit` or `git push` on `master`. Cut a feature branch (`/`, e.g. + `fschlz/pytorch-amp-example`), push there, open a PR, and let a human merge. + - Commits follow **Conventional Commits**: `feat:`, `fix:`, `chore:`, `refactor:`, `docs:`. + - Never `gh pr merge`, `gh pr close`, or `gh pr review --approve` — author/reviewer actions only. + Blocked in [`.claude/settings.json`](.claude/settings.json). + - No AI-attribution footers in commit messages or PR bodies. + - **Update READMEs before opening a PR** — the example's own `README.md`, and the root + [`readme.md`](readme.md) if you add a new top-level area. + +## CI + +Tests run via [`.github/workflows/test-examples.yml`](.github/workflows/test-examples.yml), which +holds **explicit matrices** of notebooks and scripts — examples are not auto-discovered. The +workflow installs deps with `pip install -r requirements.txt` (it has not been migrated to `uv`). +To get a new example covered: + +1. Add it to the matrix — the `notebooks` list (run with `ipython`) for a `.ipynb`, or the + `example` list (run with `python