Add format-based negative test generation

CHANGELOG.md

@@ -5,10 +5,24 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## [Unreleased]
+## [0.3.1]
 
 ### Added
 - Retry logic for fetching `/openapi.json`: the plugin now retries up to 3 times (4 total attempts) before failing, with a warning printed on each failed attempt.
+- **Format-based negative test generation** for string fields annotated with
+  `format: email`, `uri`, `ipv4`, `ipv6`, `hostname`, `idn-hostname`, or
+  `uuid`. When the OpenAPI spec documents a `400` response for an endpoint,
+  the plugin automatically generates one request body per required
+  format-constrained field that contains a deliberately invalid value. The
+  server is expected to return 400 or 422; the plugin additionally requires
+  the response body to contain at least one non-whitespace string (a
+  human-readable error message) — an empty `{}` body causes the test to fail.
+- Eight new mock servers and corresponding integration tests covering all
+  supported formats plus URL port-range edge cases (port 0 and ports > 65535
+  are invalid; ports 1–65535 are valid).
+- OpenAPI compatibility table added to `README.md` showing 3.0.x vs 3.1.x
+  feature support (as of v0.3.0).
+- Format validation documented in `docs/how-it-works/test-cases.md`.
 
 ## [0.3.0] - 2026-03-26
 

README.md

@@ -136,6 +136,7 @@ Each OpenAPI test appears as an individual pytest test item.
 ✔️ Compares live responses against examples
 ✔️ Produces a readable test report
 ✔️ Supports OpenAPI 3.0 and **3.1.x** (nullable types, `const`, `$ref` siblings, `allOf`)
+✔️ Generates **format-based negative tests** for `email`, `uri`, `ipv4`, `ipv6`, `hostname`, `uuid` fields — expects 400/422 with a descriptive error message
 
 
 # ▶️ Detailed Example
@@ -218,12 +219,52 @@ See here an example server - `email-server`: [tests/test_servers/email_server/se
 
 [tests/test_servers/email_server/email_test_output.txt](tests/test_servers/email_server/email_test_output.txt)
 
+## Format-Based Negative Test Generation
+
+When a string property in an OpenAPI request body declares a `format`, pytest-openapi automatically generates an invalid value and sends it to the server. The server **must** return 400 or 422 **and** include a human-readable error message in the body (an empty `{}` body causes the test to fail).
+
+| Format | Tested? | Example invalid value |
+|---|---|---|
+| `email` | ✔️ | `not-an-email` |
+| `uri` / `url` | ✔️ | `not-a-uri` |
+| `ipv4` / `ip` | ✔️ | `999.999.999.999` |
+| `ipv6` | ✔️ | `gggg::1` |
+| `hostname` / `idn-hostname` | ✔️ | `-invalid.start.com` |
+| `uuid` | ✔️ | `not-a-uuid` |
+| `date` / `date-time` / `time` | ❌ (out of scope) | — |
+
+## OpenAPI Version Compatibility
+
+As of **v0.3.0**, pytest-openapi supports both OpenAPI 3.0.x and OpenAPI 3.1.x.
+
+|pytest-openapi|OpenAPI|
+|--------------|-------|
+|0.3.x         |3.0.x  |
+|0.3.x         |3.1.x  |
+
+The plan is to keep this backwards compatible for now.
+
+
+Here is a detailed breakdown.
+
+| Feature | OpenAPI 3.0.x | OpenAPI 3.1.x |
+|---|---|---|
+| Basic types (`string`, `integer`, `boolean`, `array`, `object`) | ✔️ | ✔️ |
+| `nullable: true` | ✔️ | ✔️ (via `type: ["string", "null"]`) |
+| `type: ["string", "null"]` array syntax | ❌ | ✔️ |
+| `$ref` resolution (`components/schemas`) | ✔️ | ✔️ |
+| `$ref` siblings (keywords next to `$ref`) | ❌ | ✔️ |
+| `allOf` composition | ✔️ | ✔️ |
+| `oneOf` / `anyOf` | ✔️ | ✔️ |
+| `const` keyword | ✔️ | ✔️ |
+| `enum` validation & negative tests | ✔️ | ✔️ |
+| Format-based negative tests (`email`, `uri`, …) | ✔️ | ✔️ |
+
 # Future Plans / TODO
 
 This is a work in progress.
 - [ ] A check that the example matches the schema
 - [ ] Ask that 400 responses be in the documentation.
-- [ ] A check for regexp and email formats.
 
 ## Issues? Feedback?
 

docs/how-it-works/test-cases.md

@@ -21,6 +21,27 @@ Before generating test cases, schemas are fully resolved:
 
 For object schemas, pytest-openapi generates test cases by taking the Cartesian product of valid values for each property. Only valid enum values are included in these combinations to avoid generating request bodies that would be rejected by the server.
 
+In addition to the valid combinations, one extra object is generated for every string property annotated with a recognised format (see below). That extra object replaces only the format-constrained field with a deliberately invalid value while keeping all other fields valid. This produces a negative test case that must trigger a 400 or 422 from the server.
+
+### Format-Based Negative Tests
+
+When an OpenAPI schema marks a string property with one of the following formats, pytest-openapi generates an invalid value and expects the server to return **400 Bad Request** or **422 Unprocessable Entity** with a **non-empty, human-readable error message** in the response body.
+
+| Format | Example valid value | Example invalid value used |
+|---|---|---|
+| `email` | `test@example.com` | `not-an-email` |
+| `uri` / `url` | `https://example.com/path` | `not-a-uri` |
+| `ipv4` / `ip` | `192.168.1.1` | `999.999.999.999` |
+| `ipv6` | `2001:db8::1` | `gggg::1` |
+| `hostname` / `idn-hostname` | `api.example.com` | `-invalid.start.com` |
+| `uuid` | `550e8400-e29b-41d4-a716-446655440000` | `not-a-uuid` |
+
+Formats intentionally excluded: `date`, `date-time`, `time` (server-side validation for these is inconsistent and out of scope).
+
+#### Error message requirement
+
+When the server returns 400 or 422 for an invalid-format value, the response body must contain at least one non-whitespace string. An empty body (`{}`) causes the test to **fail**. The plugin does not check for any specific word — it only verifies that a human-readable explanation is present.
+
 ## Validation Strategy
 
 pytest-openapi uses different validation approaches depending on the test case origin:

src/pytest_openapi/__init__.py

@@ -1 +1 @@
-__version__ = "0.3.0"
+__version__ = "0.3.1"

src/pytest_openapi/case_generator.py

@@ -5,6 +5,22 @@
 
 from .schema import primary_type, resolve_schema
 
+_INVALID_FORMAT_VALUES = {
+    "email": [
+        "user name@domain.com",
+        "user@@double.com",
+        "user;name@domain.com",
+    ],
+    "uri": ["not-a-uri", "://missing-host"],
+    "url": ["not-a-url", "://missing-host"],
+    "ipv4": ["999.999.999.999", "1.2.3"],
+    "ip": ["999.999.999.999", "1.2.3"],
+    "ipv6": ["gggg::1", "not-ipv6"],
+    "hostname": ["-invalid.start.com", "has space.com"],
+    "idn-hostname": ["-invalid.start.com", "has space.com"],
+    "uuid": ["not-a-uuid", "xxxxxxxx-not-a-valid-uuid"],
+}
+
 
 def generate_invalid_enum_value(schema):
     """Generate an invalid value for an enum field.
@@ -178,7 +194,16 @@ def generate_string_test_cases(schema, valid_only=False):
             target_length = min_length + 5
         filtered.append("a" * target_length)
 
-    return filtered if filtered else ["test-string"]
+    result = filtered if filtered else ["test-string"]
+
+    # Append invalid-format values for negative tests (not subject to length
+    # constraints — they must be recognisably broken to trigger server-side 400)
+    if not valid_only and "format" in schema:
+        fmt = schema["format"]
+        if fmt in _INVALID_FORMAT_VALUES:
+            result = result + _INVALID_FORMAT_VALUES[fmt]
+
+    return result
 
 
 def generate_integer_test_cases(schema, field_name="field", valid_only=False):
@@ -460,6 +485,7 @@ def generate_object_test_cases(schema, field_name="field", spec=None):
         tuple: (list of test objects, list of warnings)
     """
     properties = schema.get("properties", {})
+    required_props = set(schema.get("required", []))
 
     warnings = []
 
@@ -495,6 +521,21 @@ def generate_object_test_cases(schema, field_name="field", spec=None):
     if not combos:
         combos = [{}]
 
+    # Append one format-invalid object per REQUIRED format-constrained property.
+    # Required fields must always be present and are the most likely to be
+    # server-validated; non-required format fields are skipped to keep the
+    # test-case count predictable for servers that don't validate format.
+    base = combos[0]
+    for prop_name, prop_schema in properties.items():
+        if prop_name not in required_props:
+            continue
+        resolved = resolve_schema(spec, prop_schema)
+        fmt = resolved.get("format")
+        if fmt and fmt in _INVALID_FORMAT_VALUES:
+            invalid_obj = dict(base)
+            invalid_obj[prop_name] = _INVALID_FORMAT_VALUES[fmt][0]
+            combos.append(invalid_obj)
+
     return combos, warnings