[Fix] Gate Jupyter Ready on HTTP /lab serving - #306
Open
royischoss wants to merge 4 commits into
Open
Conversation
royischoss
marked this pull request as ready for review
August 4, 2026 13:40
liranbg
reviewed
Aug 4, 2026
Comment on lines
+345
to
+351
| livenessProbe: | ||
| httpGet: | ||
| path: /lab | ||
| port: http | ||
| periodSeconds: 30 | ||
| timeoutSeconds: 5 | ||
| failureThreshold: 6 |
Member
There was a problem hiding this comment.
note that if jupyter doesnt answer within this time - it means it kills the pod completely. perhaps make it more graceful
tal-haim
self-requested a review
August 5, 2026 11:25
GiladShapira94
approved these changes
Aug 6, 2026
| port: http | ||
| periodSeconds: 10 | ||
| timeoutSeconds: 5 | ||
| failureThreshold: 3 |
Collaborator
There was a problem hiding this comment.
one small, maybe worth increasing to (1056) = 300 ~ 5m
Collaborator
|
LGTM |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
📝 Description
On-prem CE installs could report success while mlrun-jupyter was still not serving HTTP.
The pod was marked Ready in ~1s because the CE chart had no probes, even though mlce-start.sh can take minutes to extract basehome.tar before Jupyter binds :8888.
Downstream naipi then hit /lab and got 502 Bad Gateway.
This PR adds an HTTP
readinessProbeonGET /labso Kubernetes Ready andhelm --waitonly succeed once Jupyter is actually serving. AstartupProbe/livenessProbecombo was tried first but dropped in favor ofreadinessProbe-only — readiness alone holds the pod out of rotation during the slow cold-start extract without risking a restart loop if extraction runs long.Complements the ML-12950 jupyter image fix that reduces cold-start extract time
🛠️ Changes Made
charts/mlrun-ce/values.yaml — added configurable
jupyterNotebook.readinessProbedefaults (HTTP GET /lab on port http, periodSeconds 10 / timeoutSeconds 5 / failureThreshold 3). Note:startupProbe/livenessProbevalue blocks are still present in values.yaml but are no longer wired into the deployment (see below) — left as-is, can be removed in a follow-up if not needed.charts/mlrun-ce/templates/jupyter-notebook/deployment.yaml — wired the
readinessProbeinto the jupyter-notebook containercharts/mlrun-ce/Chart.yaml — bumped version 0.12.0-rc.6 → 0.12.0-rc.7
✅ Checklist
charts/mlrun-ce/Chart.yaml.🧪 Testing
helm template mlrun charts/mlrun-ce -f charts/mlrun-ce/values.yaml --show-only templates/jupyter-notebook/deployment.yaml — probe renders with expected path/port/thresholds
./tests/helm-template-test.sh — 82/82 passed
Real-cluster verification (2026-08-06, vmdev137ig4 / 192.168.236.51): packaged the chart locally (
make package,mlrun-ce-0.12.0-rc.7.tgz), scp'd it to the lab, removed the priormlrun-cerelease + PVCs, thenhelm install my-mlrun <tgz> -n mlrun --wait --timeout 15m. Install completed successfully.mlrun-jupyterpod: container started, then took ~2m29s (19 failed readiness probes on/lab,connection refusedwhilemlce-start.shextractedbasehome.tar) before passing readiness and going1/1 Ready— with 0 restarts throughout. Confirms the fix:helm --waitnow blocks until Jupyter is actually serving, and the pod isn't killed while cold-starting.Not yet re-verified on local Docker Desktop.
🔗 References
Ticket link: https://ecliptos.atlassian.net/browse/CEML-730
Related: https://ecliptos.atlassian.net/browse/ML-12950 (jupyter basehome.tar shrink, mlrun#9990)
Related: #304 (TimescaleDB startupProbe precedent)
External links: #336 (progressDeadlineSeconds — exposed this latent gap)
🚨 Breaking Changes?
🔍️ Additional Notes
values.yamlstill declaresjupyterNotebook.startupProbeandjupyterNotebook.livenessProbeblocks that are no longer referenced by the template (removed fromdeployment.yamlin a follow-up commit on this branch). Left in place for now since they're harmless unused config, but worth pruning in a follow-up if we're confident readiness-only is the final shape.