Fix: support xcresult node types

[artemii-yanushevskyi]

xcresulttool test-results schema 0.2.0 (default on Xcode 27) adds XCTest node types Expected Failure and Skip Message. Neither was supported in XcTestNodeType, so parsing failed when such a node was present in a *.xcresult bundle (e.g. from XCTExpectFailure / XCTSkip):

ValueError: 'Expected Failure' is not a valid XcTestNodeType

Summary

• Accept Expected Failure and Skip Message XCTest node types from xcresulttool schema 0.2.0+
• Keep skip reasons in JUnit when Apple emits Skip Message children (still support Xcode 16 Failure Message + result=Skipped); dedupe identical messages from both sources
• Exclude expected-failure messages from JUnit <error> when a test case also has a real failure
• Emit XCTExpectFailure reason text in JUnit <system-out> for expected failures, including failed cases that also have expected failures (not only when status="Expected Failure")

Out of scope

Additive schema fields (runtimeWarnings, sourceLocation, nodeIdentifierURL) are ignored by our from_dict paths and are not modeled in this PR.
Codemagic builder/UI does not yet map @status="Expected Failure" / <system-out> (expected failures still show as passes in the overview until a follow-up).

QA

• Existing Xcode 16-style skip via Failure Message + result=Skipped still works (unit test)
• Skip messages from both schema shapes are deduped (unit test)
• Expected-failure noise is excluded from _get_test_case_error on mixed fail cases (unit test)
• Pure / mixed expected failure → status + <system-out> as intended (unit + least)
• Skipped cases keep a non-empty <skipped message="...">

Xcode app tests

//
//  ExpectedFailureProbeTests.swift
//  leastTests
//
//  Demo cases for xcresult node types Expected Failure and Skip Message (CM-1343).
//

import XCTest

final class ExpectedFailureProbeTests: XCTestCase {
    func testPass() {
        XCTAssertTrue(true)
    }

    func testExpectedFailure() {
        XCTExpectFailure("known bad assertion")
        XCTAssertEqual(1, 2, "should be expected failure")
    }

    func testExpectedFailureThenRealFailure() {
        XCTExpectFailure("known bad assertion") {
            XCTAssertEqual(1, 2, "should be expected failure")
        }
        XCTAssertEqual(3, 4, "should be real failure")
    }

    func testSkip() throws {
        throw XCTSkip("skipping on purpose")
    }
}

JUnit from Xcode app tests

Build: 6ac49e8084481fd34b0c621c — convert succeeded; build failed intentionally on mixed probe (exit 65). Artifact: test-reports/test-results.xml.

Test	@status	Children
testExpectedFailure()	Expected Failure	<system-out>known bad assertion</system-out>
testExpectedFailureThenRealFailure()	Failed	<error message="XCTAssertEqual failed: (&quot;3&quot;) is not equal to (&quot;4&quot;) - should be real failure" type="Failure"/> + <system-out>known bad assertion</system-out>
testPass()	Passed	(none)
testSkip()	Skipped	<skipped message="Test skipped - skipping on purpose"/>

<testcase name="testExpectedFailure()" classname="ExpectedFailureProbeTests" time="0.52" status="Expected Failure">
  <system-out>known bad assertion</system-out>
</testcase>
<testcase name="testExpectedFailureThenRealFailure()" classname="ExpectedFailureProbeTests" time="0.0062" status="Failed">
  <error message="XCTAssertEqual failed: (&quot;3&quot;) is not equal to (&quot;4&quot;) - should be real failure" type="Failure"/>
  <system-out>known bad assertion</system-out>
</testcase>
<testcase name="testPass()" classname="ExpectedFailureProbeTests" time="0.0011" status="Passed"/>
<testcase name="testSkip()" classname="ExpectedFailureProbeTests" time="0.018" status="Skipped">
  <skipped message="Test skipped - skipping on purpose"/>
</testcase>

Details about schema changes in 0.2.0+

$ xcrun xcresulttool version
xcresulttool version 25115, schema version: 0.4.0 (legacy commands format version: 3.60)

$ xcrun xcresulttool get test-results tests --schema --schema-version 0.1.0 | jq .schemas.TestNodeType
{
  "type": "string",
  "enum": [
    "Test Plan",
    "Unit test bundle",
    "UI test bundle",
    "Test Suite",
    "Test Case",
    "Device",
    "Test Plan Configuration",
    "Arguments",
    "Repetition",
    "Test Case Run",
    "Failure Message",
    "Source Code Reference",
    "Attachment",
    "Expression",
    "Test Value",
    "Runtime Warning"
  ]
}

$ xcrun xcresulttool get test-results tests --schema --schema-version 0.2.0 | jq .schemas.TestNodeType
{
  "type": "string",
  "enum": [
    "Test Plan",
    "Unit test bundle",
    "UI test bundle",
    "Test Suite",
    "Test Case",
    "Device",
    "Test Plan Configuration",
    "Arguments",
    "Repetition",
    "Test Case Run",
    "Failure Message",
    "Source Code Reference",
    "Attachment",
    "Expression",
    "Test Value",
    "Runtime Warning",
    "Skip Message",
    "Expected Failure"
  ]
}

[priitlatt]

Outputs of xcrun xcresulttool get test-results summary --schema on Xcode 26.2 and Xcode 27.0:

• xcresult_summary_schema_xcode_26_2.json
• xcresult_summary_schema_xcode_27_0.json

And their diff

57c57,66
<           "$ref": "#/schemas/TestFailure"
---
>           "type": "array",
>           "items": {
>             "$ref": "#/schemas/TestFailure"
>           }
>         },
>         "runtimeWarnings": {
>           "type": "array",
>           "items": {
>             "$ref": "#/schemas/Issue"
>           }
72c81,82
<         "testFailures"
---
>         "testFailures",
>         "runtimeWarnings"
229a240,263
>     },
>     "Issue": {
>       "type": "object",
>       "properties": {
>         "issueType": {
>           "type": "string"
>         },
>         "message": {
>           "type": "string"
>         },
>         "targetName": {
>           "type": "string"
>         },
>         "sourceURL": {
>           "type": "string"
>         },
>         "className": {
>           "type": "string"
>         }
>       },
>       "required": [
>         "issueType",
>         "message"
>       ]

Similarly, outputs of xcrun xcresulttool get test-results tests --schema on Xcode 26.2 and Xcode 27.0:

• xcresult_test_schema_xcode_26_2.json
• xcresult_test_schema_xcode_27_0.json

And their diff

84a85,87
>         "nodeIdentifierURL": {
>           "type": "string"
>         },
111a115,117
>         "sourceLocation": {
>           "$ref": "#/schemas/SourceLocation"
>         },
152c158,160
<         "Runtime Warning"
---
>         "Runtime Warning",
>         "Skip Message",
>         "Expected Failure"
153a162,176
>     },
>     "SourceLocation": {
>       "type": "object",
>       "properties": {
>         "filePath": {
>           "type": "string"
>         },
>         "lineNumber": {
>           "type": "integer"
>         }
>       },
>       "required": [
>         "filePath",
>         "lineNumber"
>       ]

Note that schemas are identical on Xcode versions 27.0, 27.1 and 27.2.

---

From the above diffs we can see that in addition to the crash patch there are some missing parts that we're currently not modeling in any way:

1. Summary.runtimeWarnings: [Issue] is missing from XcSummary
2. TestNode.sourceLocation is missing from XcTestNode. Various Junit XML implementations support both file and line refs, see nextest-rs/nextest#2420 for example.
3. TestNode.nodeIdentifierURL is missing from XcTestNode.

What are the plans with missing data that is present in the new schema?

[priitlatt]

There are now different sources for skipped messages. Is it possible that we can capture duplicate messages? I'd add this to filter out anything that we don't need:

unique_skipped_messages = dict.fromkeys(skipped_messages)

[priitlatt]

Now that there is support for expected failures I think this has an effect on our _get_test_case_error function. It might be that expected-failure messages can leak into the Error of a failed test. I'm not sure if this is indeed the case, but if so, we should filter out the expected-failures from failure_messages. That would have to be verified with an actual test which contains both expected failures and errors.

If true, then the error building needs to be made "smarter" to exclude the expected-failures.

Possible patch

class Xcode16XcResultConverter(XcResultConverter):
    def _iter_nodes(
        cls,
        root_node: XcTestNode,
        node_type: XcTestNodeType,
        skip_subtree_node_type: Optional[XcTestNodeType] = None,
    ) -> Iterator[XcTestNode]:
        if root_node.node_type is node_type:
            yield root_node
        elif root_node.node_type is not skip_subtree_node_type:
            for child in root_node.children:
                yield from cls._iter_nodes(child, node_type, skip_subtree_node_type)

    @classmethod
    def _get_test_case_error(cls, xc_test_case: XcTestNode) -> Optional[Error]:
        if xc_test_case.result is not XcTestResult.FAILED:
            return None

        # Exclude expected failures which can accompany actual failures in the same test case.
        failure_messages_nodes = cls._iter_nodes(
            xc_test_case,
            XcTestNodeType.FAILURE_MESSAGE,
            skip_subtree_node_type=XcTestNodeType.EXPECTED_FAILURE,
        )
        unexpected_failure_message_nodes = (
            node for node in failure_messages_nodes if node.result is not XcTestResult.EXPECTED_FAILURE
        )
        failure_messages = [node.name for node in unexpected_failure_message_nodes if node.name]
        return Error(
            message=failure_messages[0] if failure_messages else "",
            type="Error" if any("caught error" in m for m in failure_messages) else "Failure",
            error_description="\n".join(failure_messages) if len(failure_messages) > 1 else None,
        )

[priitlatt]

This PR is a great reminder that we should get rid of LegacyXcResultConverter by now.

[artemii-yanushevskyi]

Thanks for sharing schema notes diff.

What are the plans with missing data that is present in the new schema?

I propose to keep the new fields out of the scope of this PR. It seems like they don't break parsing. Happy to follow up separately.

[priitlatt]

Should be fine now.