FEAT: sparkeks script_uri and entry_point on the SQL wrapper

[Nitin Bhakar (nitinbhakar)]

Summary

sparkeks can submit a PySpark job on the existing SQL wrapper path. The job names code already on S3; the plugin does not upload query.sql for that case.

• parameters.script_uri: skip the query.sql upload and pass this object through as query_uri (.py, .zip, .tar.gz, or any other S3 object).
• parameters.entry_point: on the SQL wrapper, prepended as the next argv after the usual app query_uri user result prefix (path inside an archive). JAR jobs still use entry_point as the main class.
• Command-level image override so a command can pin a Spark image other than the cluster default.
• Required spark.sql.extensions are merged after job properties so they cannot be cleared at submit time.

SQL jobs with no script_uri are unchanged: the plugin still uploads query.sql.

Job examples

Same sparkeks command as SQL (Python wrapper). The wrapper inspects query_uri.

1. Single file on S3

POST /api/v1/job
{
  "name": "pyspark-script-job",
  "version": "1.0.0",
  "command_criteria": ["type:pyspark-eks"],
  "cluster_criteria": ["type:spark"],
  "context": {
    "parameters": {
      "script_uri": "s3://example-bucket/pyspark/scripts/job.py"
    }
  }
}

2. Archive on S3

POST /api/v1/job
{
  "name": "pyspark-archive-job",
  "version": "1.0.0",
  "command_criteria": ["type:pyspark-eks"],
  "cluster_criteria": ["type:spark"],
  "context": {
    "parameters": {
      "script_uri": "s3://example-bucket/pyspark/jobs/abc123.zip",
      "entry_point": "src/job.py"
    }
  }
}

Test plan

Submitting a python file

Submitting a bundle

Testing exsisting Spark SQL logic

Verifying the output of pyspark table creation

• go test ./internal/pkg/object/command/sparkeks/...
• Submit a .py via parameters.script_uri (no query.sql upload)
• Submit a zip/tar.gz via script_uri + entry_point
• Confirm SQL jobs with no script_uri still upload query.sql