notebooks: (actually its workspaces) add Spark Connect operator and user guides

[vikas-saxena02]

What this does

Adds two guides covering how to run Spark from a Kubeflow Workspace using the Kubeflow SDK, so users can start a Spark Connect session from a notebook without hand-authoring SparkApplication manifests.

Addresses kubeflow/notebooks#1362. Scope confirmed with @christian-heusel on that issue: guides live here, no sample WorkspaceKind, the docs instead explain what to modify on an existing WorkspaceKind and why.

Changes

Three new files, nothing modified or removed.

• workspaces/operator-guides/spark-connect.md — the one WorkspaceKind change an administrator has to make (binding kubeflow-spark-edit via podTemplate.serviceAccount.clusterRoles), why the narrow role rather than kubeflow-edit, how to verify with kubectl auth can-i, and image requirements.
• workspaces/user-guides/spark-connect.md — the SparkClient / connect() flow, version pinning, the three required PodTemplateOverride settings with the reasoning for each, and troubleshooting.
• workspaces/user-guides/_index.md — new section, since Workspaces had Operator and Contributor guides but no User guides.

How the split landed

Worth flagging, because it isn't what I expected going in: the only administrator-side change is the RBAC binding. The driver ServiceAccount, Istio sidecar exclusion, and the executor container name are all supplied per session by the user in the connect() call, not configured on the WorkspaceKind. So the operator guide is short and most of the substance sits in the user guide.

On the Notebooks-side follow-ups asked for in kubeflow/notebooks#1362: the only candidate I found was a sample jupyterlab-spark WorkspaceKind, which @christian-heusel has already ruled out. I don't think anything else is needed in kubeflow/notebooks for this.

Depends on kubeflow/community-distribution#3590

Please hold this until that PR merges. Both guides link to the integration test it adds, so the reference links 404 until then and the Netlify link check will flag them. Every configuration snippet here is taken from that test rather than written from scratch, so it should also merge first for the docs to describe verified behaviour.

Raising now so review can start in parallel. Happy to convert to a draft if you'd rather.

Related

kubeflow/spark-operator#3141 replaces the Jupyter Enterprise Gateway notebooks guide with the same SDK approach, on the Spark Operator docs side. It states the platform-specific requirements inline as a stopgap; I'll swap those for links to these pages once both land.

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign thesuperzapper for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• content/en/docs/components/workspaces/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment