Skip to content

Add client-side W&B logging example for Tinker RL loops - #33

Merged
kailash109 merged 1 commit into
mainfrom
devin/1789616475-wandb-rl-example
Sep 17, 2026
Merged

kailash109 merged 1 commit into
mainfrom
devin/1789616475-wandb-rl-example

Conversation

@micahtyong

Copy link
Copy Markdown
Contributor

Summary

Adds scripts/wandb_rl_example.py: a minimal multi-step Tinker RL loop against a Lilo deployment that logs per-step metrics to Weights & Biases, plus a README pointer. Answers the "how do you use W&B with Tinker code?" question — W&B is entirely client-side; the Lilo server never sees it.

Per step: save_weights_and_get_sampling_client() → sample a group → toy binary reward → forward_backward(datums, "importance_sampling") + optim_step(AdamParams) → wandb.log({reward/*, response_len/mean, time/*, train/<metrics Lilo returns>, optim/*}, step=step).

Two details that matter against the deployed backend: create_lora_training_client(..., train_unembed=False) (matches the deployment's LoRA targets), and target_tokens = prompt[1:] + tokens so it lines up exactly with model_input = prompt + tokens[:-1].

Validated with a 3-step run against the lilo deployment (reward 1.0 each step, ~14 s/step after warm-up): https://wandb.ai/modal-labs/lilo-examples/runs/zg7rnr16. ruff check/ruff format --check pass.

tinker-cookbook users don't need this: set wandb_project/wandb_name on the cookbook Config (that's what the LongRLVR runs in #30 do).

Link to Devin session: https://modal.devinenterprise.com/sessions/8d6804dd317c41329d77eb2c427dda6f
Open in Devin Desktop: https://modal.devinenterprise.com/desktop/session/8d6804dd317c41329d77eb2c427dda6f?variant=devin
Requested by: @micahtyong

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR that start with 'DevinAI' or '@devin'.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

@kailash109

Copy link
Copy Markdown
Contributor

lgtm!

@kailash109
kailash109 merged commit ad5956c into main Sep 17, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants