Repository navigation
Conversation
Signed-off-by: Roxana Nicolescu <rnicolescu@ciq.com>
Just an interface that provides more clarity around ciq headers, but needs refining. Especially around getters and setters. Signed-off-by: Roxana Nicolescu <rnicolescu@ciq.com>
Had to change --sha and --ticket params to --commit and --jira to match the CommitHeader dataclass. Also created CherryPickCommand class that keeps track of the arguments given to this script, in order to avoid long function with too many parameters. The logic did not change at all, except for the fact that cherry_pick functions and their helpers receive a CommitHeader object instead of the multiple params. Signed-off-by: Roxana Nicolescu <rnicolescu@ciq.com>
Here not a huge change. upstream commit should be just one, so I removed the useless for. What's nice that we do not need regex anymore. Signed-off-by: Roxana Nicolescu <rnicolescu@ciq.com>
Signed-off-by: Roxana Nicolescu <rnicolescu@ciq.com>
Signed-off-by: Roxana Nicolescu <rnicolescu@ciq.com>
commit_author and commit field represent the commit in mainline. If the commit is not in mainline yet, commit value can be `-` or left empty. The fields that show the actual source is something else are: 1. commit_source which could be one of these: arm64: git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git centos-stream-9: https://gitlab.com/redhat/centos-stream/src/kernel/centos-stream-9.git centos-stream-10: git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git linux-stable: git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git 2. commit_source_sha: the actual sha from the remotes presented above. 3. commit_source_author (optional) Introduces upstream_sources.yaml file that contains a list of upstream sources other than mainline. List should be extended to other subsystems soon. It also modifies the following scripts to take into account different commit sources: 1. ciq-cherry-pick.py It receives extra arguments with the commit source and it is responsible of fetching the corresponding upstream remote and cherry picking the commit. 2. check_kernel_commits.py If there is no mainline commit, it does not do any extra checks, since this means the commit is probably in review process (taken from lore). NOTE: This should be revisited later, since we may need checks regardless if the commit is in mainline or not. 3. run_interdiff.py If there is another source commit other than mainline, that is the one used to show the diff. Signed-off-by: Roxana Nicolescu <rnicolescu@ciq.com>
Signed-off-by: Roxana Nicolescu <rnicolescu@ciq.com>
…sed bugs
The {rnicolescu}_commit_source refactor replaced the flat ciq-cherry-pick
functions with CherryPickCommand, CommitHeader and UpstreamRemotes, but the
test suite still targeted the deleted API (28 failures) and the new code
itself was untested.
Test changes:
- Rewrite tests/test_ciq_cherry_pick.py against the CherryPickCommand API:
_manage_commit_message (header tags, ordering, upstream-diff on conflict,
# Conflicts: stripping, commit-source tags), _cherry_pick (mainline,
cve-bf, source-sha pick via extract_upstream_name_and_sha, conflict,
author resolution), _check_fixes, _update_jira_success/failure,
_cherry_pick_fixes (skip guard for source-only picks, cve -> cve-bf swap)
and from_argsparse_to_commit_header (tag parsing, sha expansion, remote
fetch only when both --commit-source and --commit-source-sha are given).
- Expand tests/kt/ktlib/test_commit_header.py: parsing of source-only and
mainline bodies, to_str() round trip, cve helpers,
extract_upstream_name_and_sha, and a regression test proving the
commit-source / commit-source-sha tags can never match the
'^commit <40hex>' anchor check_kernel_commits.py validates against.
- Add tests/kt/ktlib/test_git_remote.py for UpstreamRemotes/GitRemoteInfo:
from_dict, auto-add + fetch, idempotency, unknown remote name,
URL-mismatch warning.
- Adapt tests/kt/ktlib/test_ciq_helpers.py to the commit_header signature.
Fixes the new tests exposed:
- commit_header: to_str() returned a list despite '-> str', leaking a
Python list repr into every standardized commit message; return the tag
lines joined with newlines.
- ciq-cherry-pick: map hyphenated tag names (cve-pre, cve-bf) to the
underscored CommitHeader fields, which from_dict otherwise drops
silently; tolerate multi-word and comma-separated --ciq-tag values
(split with maxsplit); skip the Fixes: search when there is no mainline
sha (source-only pick); let CherryPickCommand construction errors
propagate instead of crashing later on an uninitialized command.
- git_remote: warn when fetching an existing remote whose url differs from
the configured one instead of silently fetching from the old url.
The commit_source refactor renamed the target-sha flag --sha to --commit, which breaks existing callers - notably kernel-tools' cve_remediation_automation.py, which still invokes 'ciq-cherry-pick.py --sha <sha> --ticket ... --ciq-tag ...' and would fail with 'unrecognized arguments: --sha'. Accept both flags naming the same internal parameter: - --sha is parsed into its own dest and folded into args.commit by resolve_sha_alias() right after parse_args() - passing both with conflicting values is an argparse error - passing both with the same value is accepted --help marks --sha as a deprecated alias.
The commit-source tag is a provenance label and can now be either an upstream_sources.yaml remote name (e.g. centos-stream-9) or a stable branch name (e.g. linux-6.12.y). Both consumers of that tag treated it strictly as a remote-name key: - ciq-cherry-pick.py fetch_remote() would KeyError on a branch name - run_interdiff.py likewise, flagging every branch-provenance PR commit as an ERROR during CI validation Add UpstreamRemotes.fetch_for_commit_source(commit_source, sha, repo_dir): 1. sha already resolvable locally -> no-op (no network; the CI fast path when the workflow pre-fetched the branch) 2. commit_source is a yaml key -> fetch that remote (unchanged behavior) 3. commit_source is a branch -> ls-remote the configured remotes (ref query only, no objects) and fetch the first one carrying refs/heads/<branch> 4. otherwise raise RuntimeError with actionable guidance Both call sites switch to the helper. run_interdiff additionally only fetches when both source name and sha are present.
….org centos-stream-10 pointed at the arm64 kernel tree (copy-paste bug); point it at the CentOS Stream 10 kernel gitlab instead. kernel.org entries used git://, an unauthenticated transport whose port 9418 is commonly blocked by firewalls; switch to https.
The commit_source refactor renamed --ticket to --jira, breaking existing callers - kernel-tools' cve_remediation_automation.py still passes --ticket VULN-xxx and died with 'unrecognized arguments' (invisible to its logs, which discard stderr). Same pattern as the --sha alias: separate dest folded by resolve_ticket_alias() right after parse_args(); conflicting values are an argparse error; same value is fine.
There was a problem hiding this comment.
🟡 Changes recommended
Credential exposure, incorrect follow-up source selection, and metadata-handling regressions remain unresolved.
15 open issues
What changed in this PR
Expands ciq-cherry-pick to support alternate upstream sources and shares commit-header handling across backporting and PR-validation tools.
Changes:
- Adds source provenance tags, remote discovery, and automatic fetching.
- Refactors cherry-picking into
CherryPickCommand, retaining legacy CLI aliases. - Migrates validation tools to shared headers and expands regression coverage.
| File | Description |
|---|---|
upstream_sources.yaml |
Defines available upstream repositories. |
tests/test_ciq_cherry_pick.py |
Updates command tests and covers source parameters and aliases. |
tests/kt/ktlib/test_git_remote.py |
Tests remote fetching and source resolution. |
tests/kt/ktlib/test_commit_header.py |
Tests header parsing, serialization, and source selection. |
tests/kt/ktlib/test_ciq_helpers.py |
Updates message-standardization coverage. |
run_interdiff.py |
Selects and fetches alternate comparison sources. |
kt/ktlib/git_remote.py |
Adds shared remote discovery and fetching. |
kt/ktlib/commit_header.py |
Introduces the shared commit-header model. |
kt/ktlib/ciq_tag.py |
Recognizes source provenance tags. |
kt/ktlib/ciq_helpers.py |
Uses structured headers for commit messages. |
jira_pr_check.py |
Migrates Jira validation to shared headers. |
ciq-tag.py |
Imports the packaged tag library. |
ciq-cherry-pick.py |
Adds alternate-source handling and legacy aliases. |
check_kernel_commits.py |
Uses shared headers for upstream and CVE checks. |
Review effort: Balanced
| except (CherryPickException, RuntimeError) as e: | ||
| # TODO would add some extra information to jira if the cve-bf deps are not applied | ||
| raise e | ||
| print(args) |
Comment on lines
+129
to
+133
| short_uhash = commit_header.commit[:12] | ||
| uhash = commit_header.commit | ||
| if not uhash or uhash == "-": | ||
| print(f"[NOTE]: Mainline commit not present for {pr_commit_desc}, no extra checks for this commit") | ||
| continue |
Comment on lines
+72
to
+73
| if not commit_header.is_cve(): | ||
| return commit_header |
| # Continue with the existing tag | ||
| return commit_header | ||
|
|
||
| commit_header.set_cve(cve_number=vuln_cve_number) |
| author and the original commit full sha. | ||
| # Cherry pick the fixed-by dependencies | ||
| try: | ||
| self._cherry_pick_fixes(original_commit_header=updated_commit_header) |
| tag_value = value[0][1]._value | ||
| msg_dict[tag_name] = tag_value | ||
|
|
||
| return cls(**msg_dict) |
|
|
||
| @classmethod | ||
| def from_commit_body(cls, commit_body: str) -> CommitHeader: | ||
| ciq_msg = CiqMsg(commit_body) |
|
|
||
| def to_str(self) -> str: | ||
| commit_header_dict = asdict(self) | ||
| return "\n".join(f"{key.replace('_', '-')} {value}" for key, value in commit_header_dict.items() if value) |
| f"remote '{self.name}' exists with url {remote.url}, " | ||
| f"different from configured {self.url}; fetching from the existing url" | ||
| ) | ||
| remote.fetch() |
|
|
||
| # 3. Branch-style provenance: find a configured remote carrying the branch. | ||
| for git_remote_info in self.remotes.values(): | ||
| ls = Repo(repo_dir).git.ls_remote(git_remote_info.url, f"refs/heads/{commit_source}") |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adding in additional features from Roxana, there was a push to make more shared resources between kt libraries and this tool
I used GLM to update the tests and add legacy switches to do the same thing
--ticket/--jirado the same thing,--commit/--shado the same thing.Coverage Report