Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 13 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ uv run pytest

## Tests

> Tool names are given for geocontext 0.10.x (`GEOCONTEXT_DEV=1`). With 0.9.x, `gpf_*` tools are named `gpf_wfs_*` and counts rely on `gpf_wfs_get_features` (see [config/constants.py](config/constants.py)).
> Tool names are given for geocontext 0.10.x (`GEOCONTEXT_DEV=1`). With 0.9.x, `gpf_*` tools are named `gpf_wfs_*`, counts rely on `gpf_wfs_get_features` and tests using `distance` are skipped (see [config/constants.py](config/constants.py)).

### Single tool tests

Expand All @@ -102,16 +102,18 @@ uv run pytest

### Chaining tests (multiple tools)

| File | Tool chain | Description |
| ---------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | ------------------------------------------------------------------- |
| [test_chaining_geocode_altitude.py](test_chaining_geocode_altitude.py) | `geocode` → `altitude` | Altitude of Chamonix town hall (900-1200 m) |
| [test_chaining_geocode_adminexpress.py](test_chaining_geocode_adminexpress.py) | `geocode` → `adminexpress` | 1 rue de Rivoli → commune and department |
| [test_chaining_geocode_assiette_sup.py](test_chaining_geocode_assiette_sup.py) | `geocode` → `assiette_sup` | 10 place Bellecour, Lyon → public utility easements |
| [test_chaining_cadastre_urbanisme.py](test_chaining_cadastre_urbanisme.py) | `geocode` → `cadastre` → `urbanisme` | Address → parcel → urban planning rules |
| [test_chaining_discovery.py](test_chaining_discovery.py) | `gpf_search_types` → `gpf_describe_type` → `gpf_get_features` | Full discovery: watercourse near the Eiffel Tower (Seine) |
| [test_count_batiment_saint_mande.py](test_count_batiment_saint_mande.py) | `geocode` → `gpf_search_types` → `gpf_count_features` | Number of buildings in Saint-Mandé (1700-1800) |
| [test_count_batiment_30m_angouleme.py](test_count_batiment_30m_angouleme.py) | `geocode` → `gpf_search_types` → `gpf_describe_type` → `gpf_count_features` | Number of buildings higher than 30 m in Angoulême (19) |
| [test_count_lycees_2km_chateau_vincennes.py](test_count_lycees_2km_chateau_vincennes.py) | `geocode` → `gpf_search_types` → `gpf_describe_type` → `gpf_count_features` | Number of high schools within 2 km of the Château de Vincennes (14) |
| File | Tool chain | Description |
| ---------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- |
| [test_chaining_geocode_altitude.py](test_chaining_geocode_altitude.py) | `geocode` → `altitude` | Altitude of Chamonix town hall (900-1200 m) |
| [test_chaining_geocode_adminexpress.py](test_chaining_geocode_adminexpress.py) | `geocode` → `adminexpress` | 1 rue de Rivoli → commune and department |
| [test_chaining_geocode_assiette_sup.py](test_chaining_geocode_assiette_sup.py) | `geocode` → `assiette_sup` | 10 place Bellecour, Lyon → public utility easements |
| [test_chaining_cadastre_urbanisme.py](test_chaining_cadastre_urbanisme.py) | `geocode` → `cadastre` → `urbanisme` | Address → parcel → urban planning rules |
| [test_chaining_discovery.py](test_chaining_discovery.py) | `gpf_search_types` → `gpf_describe_type` → `gpf_get_features` | Full discovery: watercourse near the Eiffel Tower (Seine) |
| [test_count_batiment_saint_mande.py](test_count_batiment_saint_mande.py) | `geocode` → `gpf_search_types` → `gpf_count_features` | Number of buildings in Saint-Mandé (1700-1800) |
| [test_count_batiment_30m_angouleme.py](test_count_batiment_30m_angouleme.py) | `geocode` → `gpf_search_types` → `gpf_describe_type` → `gpf_count_features` | Number of buildings higher than 30 m in Angoulême (19) |
| [test_count_lycees_2km_chateau_vincennes.py](test_count_lycees_2km_chateau_vincennes.py) | `geocode` → `gpf_search_types` → `gpf_describe_type` → `gpf_count_features` | Number of high schools within 2 km of the Château de Vincennes (14) |
| [test_chaining_geocode_distance.py](test_chaining_geocode_distance.py) | `geocode` → `distance` | Walking distance between the Gare de Lyon and the Eiffel Tower (5-9 km) |
| [test_chaining_distance_piscine_caen.py](test_chaining_distance_piscine_caen.py) | `geocode` → `gpf_search_types` → `gpf_describe_type` → `gpf_get_features` → `distance` | Walking time from the cinéma LUX (Caen) to the nearest swimming pool: Sivom, Mondeville (25-35 min) |

### Negative tests

Expand Down
2 changes: 2 additions & 0 deletions config/constants.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
TOOL_GPF_COUNT_FEATURES = "gpf_count_features"
TOOL_GPF_GET_FEATURES_LAYER = "gpf_get_features_layer"
TOOL_GPF_GET_FEATURE_BY_ID_LAYER = "gpf_get_feature_by_id_layer"
TOOL_DISTANCE = "distance"


EXPECTED_TOOLS = [
Expand All @@ -39,3 +40,4 @@
EXPECTED_TOOLS.append(TOOL_GPF_COUNT_FEATURES)
EXPECTED_TOOLS.append(TOOL_GPF_GET_FEATURES_LAYER)
EXPECTED_TOOLS.append(TOOL_GPF_GET_FEATURE_BY_ID_LAYER)
EXPECTED_TOOLS.append(TOOL_DISTANCE)
52 changes: 52 additions & 0 deletions test_chaining_distance_piscine_caen.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
import pytest

from config.constants import (
GEOCONTEXT_DEV,
TOOL_DISTANCE,
TOOL_GEOCODE,
TOOL_GPF_SEARCH_TYPES,
TOOL_GPF_DESCRIBE_TYPE,
TOOL_GPF_GET_FEATURES,
)
from helpers import extract_numbers

pytestmark = pytest.mark.skipif(
not GEOCONTEXT_DEV, reason="distance tool requires geocontext 0.10.x (GEOCONTEXT_DEV=1)"
)

# Scenario from https://github.com/ignfab/geocontext/pull/207
USER_INPUT = (
"Quel est le temps de marche exact entre le cinéma LUX, situé au sud-est de Caen, "
"et la piscine la plus proche ?"
)
EXPECTED_RESPONSE_FRAGMENTS = ["Sivom", "Mondeville"]

@pytest.mark.asyncio
async def test_chaining_distance_piscine_caen(mcp_agent, tracker):
result = await mcp_agent.ainvoke(
{"messages": [{"role": "user", "content": USER_INPUT}]},
config={"callbacks": [tracker], "thread_id": __name__},
)

tool_names_called = {c.get("name") for c in tracker.tool_calls if c.get("type") == "start"}

required_tools = [
TOOL_GEOCODE,
TOOL_GPF_SEARCH_TYPES,
TOOL_GPF_DESCRIBE_TYPE,
TOOL_GPF_GET_FEATURES,
TOOL_DISTANCE,
]
for tool_name in required_tools:
assert tool_name in tool_names_called, f"{tool_name} tool was not called"

last_message = result["messages"][-1]
message_text = str(last_message)

for fragment in EXPECTED_RESPONSE_FRAGMENTS:
assert fragment.lower() in message_text.lower(), f"Expected fragment not found in response: {fragment}"

# Walking time between 25 and 35 minutes
numbers = extract_numbers(message_text)
assert any(25 <= n <= 35 for n in numbers), \
f"Expected a walking time between 25 and 35 min in response: {message_text[:200]}"
40 changes: 40 additions & 0 deletions test_chaining_geocode_distance.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
import pytest

from config.constants import GEOCONTEXT_DEV, TOOL_DISTANCE, TOOL_GEOCODE
from helpers import extract_numbers

pytestmark = pytest.mark.skipif(
not GEOCONTEXT_DEV, reason="distance tool requires geocontext 0.10.x (GEOCONTEXT_DEV=1)"
)

USER_INPUT = "Quelle distance à pied entre la gare de Lyon et la tour Eiffel ?"

@pytest.mark.asyncio
async def test_chaining_geocode_distance(mcp_agent, tracker):
result = await mcp_agent.ainvoke(
{"messages": [{"role": "user", "content": USER_INPUT}]},
config={"callbacks": [tracker], "thread_id": __name__},
)

tool_names_called = {c.get("name") for c in tracker.tool_calls if c.get("type") == "start"}

assert TOOL_GEOCODE in tool_names_called, f"{TOOL_GEOCODE} tool was not called"
assert TOOL_DISTANCE in tool_names_called, f"{TOOL_DISTANCE} tool was not called"

# The tracker only records tool names, arguments are read from the AI messages
distance_profiles = [
tool_call.get("args", {}).get("profile")
for message in result["messages"]
for tool_call in getattr(message, "tool_calls", None) or []
if tool_call.get("name") == TOOL_DISTANCE
]
assert "pedestrian" in distance_profiles, \
f"Expected {TOOL_DISTANCE} to be called with profile=pedestrian, got {distance_profiles}"

last_message = result["messages"][-1]
message_text = str(last_message)

# Gare de Lyon → Eiffel Tower on foot: ~7 km
numbers = extract_numbers(message_text)
assert any(5 <= n <= 9 or 5000 <= n <= 9000 for n in numbers), \
f"Expected a walking distance around 7 km in response: {message_text[:200]}"
Loading