Skip to content

update rapids version for 2606 release#1396

Open
nvliyuan wants to merge 7 commits into
GoogleCloudDataproc:mainfrom
nvliyuan:rapids-v2606
Open

update rapids version for 2606 release#1396
nvliyuan wants to merge 7 commits into
GoogleCloudDataproc:mainfrom
nvliyuan:rapids-v2606

Conversation

@nvliyuan

@nvliyuan nvliyuan commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

Update the spark-rapids jar version after plugin release.

CC @sameerz @viadea @jayadeep-jayaraman

Verification

  • Confirmed com.nvidia:rapids-4-spark_2.12:26.06.0 is published in Maven metadata.
  • Ran bash -n spark-rapids/spark-rapids.sh.

nvliyuan and others added 3 commits May 28, 2026 16:14
@nvliyuan
Fix Spark RAPIDS Ubuntu 22 driver install
02049cd
@nvliyuan @gemini-code-assist
Update spark-rapids/spark-rapids.sh
b21ca04 
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
@nvliyuan
update rapids version for 2606 release
7bcaf1c
@nvliyuan
nvliyuan marked this pull request as ready for review June 24, 2026 02:41
gemini-code-assist[bot]
gemini-code-assist Bot reviewed Jun 24, 2026
View reviewed changes

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the default Spark Rapids version from 26.04.2 to 26.06.0 in the spark-rapids.sh script. There are no review comments, and I have no feedback to provide.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

@nvliyuan

nvliyuan commented Jun 29, 2026

Copy link
Copy Markdown
Contributor Author

CC @cjac @jayadeep-jayaraman please help review thx

@cjac

cjac commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

/gcbrun

@nvliyuan

nvliyuan commented Jul 15, 2026

Copy link
Copy Markdown
Contributor Author

Hi @cjac could you help merge? thx

@cjac

cjac commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

/gcbrun

cjac
cjac approved these changes Jul 15, 2026
View reviewed changes

@cjac cjac left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Simple enough. Running tests now

@cjac

cjac commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Uh oh. Failing presubmit. I'll poke at it today. I also want to finish that patch to make it run faster on -ml or custom images with drivers pre-installed.

@sameerz sameerz mentioned this pull request Jul 17, 2026
Open
@sameerz

sameerz commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Hi @cjac any update on this?

@cjac

cjac commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

I've been meaning to get it working. I'll look now.

@cjac

cjac commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

I checked gs://dataproc-108de5de-43c2-4a4b-979a-adebc15a58a8-us-central1/google-cloud-dataproc-
metainfo/5a49fddd-fee9-4b93-b61c-6a1d5fe2d95f/test-rapids-single-2-2-20260715-102646-td5r-m/dataproc-
initialization-script-0_output .

Here are the key findings from that log file:

  1. Initialization Parameters Recorded:
    • OS Name / Version: ubuntu (22.04)
    • Dataproc Image: 2.2-ubuntu22
    • Kernel Version: 6.8.0-1060-gcp
    • Selected RAPIDS / Driver Defaults: SPARK_RAPIDS_VERSION=26.06.0, CUDA_VERSION=12.4.1,
    NVIDIA_DRIVER_VERSION=550.54.15

  2. Error Trace in Log (Lines 800–825 & 841–870):
    Loading new nvidia-550.54.15 DKMS files...
    Building for 6.8.0-1060-gcp
    Building for architecture x86_64
    Building initial module for 6.8.0-1060-gcp
    ERROR: Cannot create report: [Errno 17] File exists: '/var/crash/nvidia-dkms-550-open.0.crash'
    Error! Bad return status for module build on kernel: 6.8.0-1060-gcp (x86_64)
    Consult /var/lib/dkms/nvidia/550.54.15/build/make.log for more information.
    dpkg: error processing package nvidia-dkms-550-open (--configure):
    installed nvidia-dkms-550-open package post-installation script subprocess returned error exit status 10
    dpkg: dependency problems prevent configuration of nvidia-driver-550-open:
    nvidia-driver-550-open depends on nvidia-dkms-550-open (<= 550.54.15-1); however:
    Package nvidia-dkms-550-open is not configured yet.
    ...
    E: Sub-process /usr/bin/dpkg returned an error code (1)

  3. Conclusion:
    The log confirms that spark-rapids.sh failed during apt-get install -y nvidia-driver-550-open because
    nvidia-550.54.15 open-gpu-kernel-modules failed DKMS compilation against the guest kernel (6.8.0-1060-gcp),
    causing 10 consecutive retry loops to fail and exiting with status 1.

@cjac
spark-rapids: Update default NVIDIA driver to 590.48.01 and CUDA to 1…
b5ab3cb 
…3.1.1

Update default NVIDIA driver version from 550.54.15 to 590.48.01 and default
CUDA version from 12.4.1 to 13.1.1 across spark-rapids initialization actions
and tests.

Key changes:
- Fixes DKMS kernel module compilation failures on Linux kernel 6.8+ (Ubuntu
  22.04) by upgrading to driver 590.48.01.
- Pairs CUDA 13.1.1 with NVIDIA driver 590.48.01 in spark-rapids/spark-rapids.sh
  and spark-rapids/mig.sh to align with NVIDIA's published local installer
  repository URLs.
- Replaces desktop meta-package 'cuda-drivers' with headless server integration
  package 'nvidia-driver-cuda' under Debian/Ubuntu installation paths, resolving
  unmet Wayland/X11 display dependency errors during headless installation.
- Adds 'python3-venv' and 'python3-pip' to download_agent() package setup to
  ensure virtual environment creation succeeds for compute-gpu-monitoring.
- Updates parameterized test in spark-rapids/test_spark_rapids.py to reflect
  the updated driver version 590.48.01.

TAG=agy
CONV=9a445231-182b-46af-b964-261b2b473d6e
@cjac

cjac commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

/gcbrun

@nvliyuan

nvliyuan commented Jul 20, 2026

Copy link
Copy Markdown
Contributor Author

I had previously addressed this Ubuntu 22 / Linux 6.8 NVIDIA DKMS issue in #1394.

That fix makes Ubuntu 22 use the online NVIDIA repository instead of the older driver embedded in the local CUDA repository, and selects GCC/G++ 12 for CUDA 12 driver builds. I validated it on Dataproc 2.2.82-ubuntu22 with kernel 6.8.0-1058-gcp; cluster creation succeeded and nvidia-smi reported driver 550.163.01.

I have now merged #1394 into this PR and reverted b5ab3cb. The RAPIDS version remains 26.06.0.

While checking b5ab3cb, I also reproduced two issues with that commit:

  • nvidia-driver-cuda has no installation candidate in the CUDA 13.1 local Ubuntu repository.
  • The non-default test combination CUDA 12.4.0 + driver 590.48.01 points to a nonexistent NVIDIA installer URL.

@nvliyuan

nvliyuan commented Jul 20, 2026

Copy link
Copy Markdown
Contributor Author

verified worked
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cjac cjac cjac approved these changes
+1 more reviewer
@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@nvliyuan @cjac @sameerz