geocontext-test

This repository contains an integration test suite designed to validate the correct behavior of the MCP server ignfab/geocontext across different language models.

Purpose

The main goal is to ensure that end-to-end interactions with ignfab/geocontext remain stable and consistent when switching model providers or model versions.

What is tested

• Agent creation using a configured model name
• Real model invocation through integration scenarios
• Functional response checks for expected outputs

Quick Start

See CODING.md for comprehensive setup instructions including:

• Environment configuration and API key setup
• Running tests with different models
• Testing a local version of geocontext
• Debugging and troubleshooting

Reports

Generated with geocontext v0.9.8 :

• reports/claude-haiku-4-5.md : OK
• reports/claude-sonnet-4-6.md : OK
• reports/gemini-3.5-flash.md : OK
• reports/gemini-3.1-flash-lite.md : 1 test KO (gpf_wfs_describe_type tool was not called)
• reports/gemma4-26b-moe.md : 2 tests KO (ERP, gpf_wfs_describe_type tool was not called)
• reports/qwen3-6-35b-moe.md : 2 test KO (ERP and detailed counts for test_count_lycees_2km_chateau_vincennes.py...)
• reports/ministral-14b-latest.md : 3 tests KO (ERP and prompt ignored for test_geocode_address_not_found.py)
• reports/mistral-small-latest.md : 1 test KO (ERP)
• reports/mistral-medium-latest.md : OK
• reports/albert-openai-gpt-oss-120b.md : 2 tests KO (ERP)

About ERP, see gpf-schema-store#48 and gpf-schema-store#56

Running the tests also writes the details of each conversation to reports/<model>/<test name>.txt (not versioned, see CODING.md).

Usage

Testing with reports

export ANTHROPIC_API_KEY=YourKey

# to run with some anthropic model
uv run scripts/run_tests.py config/models-anthropic.yaml
# to rerun with a single model
uv run scripts/run_tests.py config/models-anthropic.yaml --model=claude-haiku-4-5

Testing with anthropic models

export MODEL_NAME="anthropic:claude-sonnet-4-6"
export ANTHROPIC_API_KEY=YourKey

uv run pytest

Testing with google models

export MODEL_NAME="google_genai:gemini-3.5-flash"
export GOOGLE_API_KEY=YourKey

uv run pytest

export MODEL_NAME="google_genai:gemini-3.1-flash-lite-preview"
export GOOGLE_API_KEY=YourKey

uv run pytest

Tests

Tool names are given for geocontext 0.10.x (GEOCONTEXT_DEV=1). With 0.9.x, gpf_* tools are named gpf_wfs_*, counts rely on gpf_wfs_get_features and tests using distance are skipped (see config/constants.py).

Single tool tests

File	Tool	Description
test_france_capital.py	none (LLM only)	Basic test without MCP: checks that the LLM answers "Paris"
test_tools_discovery.py	none (tool listing)	Checks that all the expected tools are exposed by the MCP server
test_geocode.py	geocode	Coordinates of 1 rue de Rivoli, Paris
test_altitude.py	altitude	Altitude at (6.87, 45.92), Chamonix area (900-1200 m)
test_adminexpress.py	adminexpress	Commune and department at (2.35, 48.85): Paris, 75
test_cadastre.py	cadastre	Cadastral parcel at 73 avenue de Paris, Saint-Mandé
test_urbanisme.py	urbanisme	Urban planning rules at 73 avenue de Paris, Saint-Mandé
test_assiette_sup.py	assiette_sup	Public utility easements at (4.83, 45.76), Lyon
test_describe_type.py	gpf_describe_type	Attributes of the BDTOPO_V3:batiment table
test_get_features.py	gpf_get_features	BD TOPO buildings near Chamonix
test_get_feature_by_id.py	gpf_get_feature_by_id	code_insee of a commune given its feature_id (first commune of the WFS)
test_search_batiment.py	gpf_search_types	Search for tables about buildings
test_search_ecoles.py	gpf_search_types	Search for tables about schools

Chaining tests (multiple tools)

File	Tool chain	Description
test_chaining_geocode_altitude.py	geocode → altitude	Altitude of Chamonix town hall (900-1200 m)
test_chaining_geocode_adminexpress.py	geocode → adminexpress	1 rue de Rivoli → commune and department
test_chaining_geocode_assiette_sup.py	geocode → assiette_sup	10 place Bellecour, Lyon → public utility easements
test_chaining_cadastre_urbanisme.py	geocode → cadastre → urbanisme	Address → parcel → urban planning rules
test_chaining_discovery.py	gpf_search_types → gpf_describe_type → gpf_get_features	Full discovery: watercourse near the Eiffel Tower (Seine)
test_count_batiment_saint_mande.py	geocode → gpf_search_types → gpf_count_features	Number of buildings in Saint-Mandé (1700-1800)
test_count_batiment_30m_angouleme.py	geocode → gpf_search_types → gpf_describe_type → gpf_count_features	Number of buildings higher than 30 m in Angoulême (19)
test_count_lycees_2km_chateau_vincennes.py	geocode → gpf_search_types → gpf_describe_type → gpf_count_features	Number of high schools within 2 km of the Château de Vincennes (14)
test_count_lycees_15min_mairie_saint_mande.py	gpf_search_types → gpf_describe_type → gpf_count_features / gpf_get_features (travel_time_filter)	Number of high schools within 15 minutes on foot from the Saint-Mandé town hall, coordinates given (7)
test_area_lac_daumesnil.py	gpf_search_types → gpf_describe_type → gpf_get_features (spatial_extras: area)	Area of the Lac Daumesnil (~101 000 m², ~10.1 ha)
test_nearest_lycee_chateau_vincennes.py	gpf_search_types → gpf_describe_type → gpf_get_features (dwithin_point_filter, spatial_extras: distance_to_filter_center)	Nearest high school to the Château de Vincennes, coordinates, radius (2 km) and limit (20) given (Lycée Professionnel Jean Moulin)
test_hauteur_tour_montparnasse.py	gpf_search_types → gpf_describe_type → gpf_get_features (intersects_point_filter)	Height of the building at a given point, the Tour Montparnasse (207.4 m ± 10 m)
test_count_batiment_bbox_chateau_vincennes.py	gpf_search_types → gpf_count_features (bbox_filter)	Number of buildings in a bounding box around the Château de Vincennes, bbox given (238)
test_highest_batiment_bbox_chateau_vincennes.py	gpf_search_types → gpf_describe_type → gpf_get_features (bbox_filter)	Highest building in a bounding box around the Château de Vincennes, bbox given (the donjon, 50.1 m ± 1 m)
test_map_lycees_2km_chateau_vincennes.py	geocode → gpf_search_types → gpf_describe_type → gpf_get_features_layer → show_map	Map of the high schools within 2 km of the Château de Vincennes: show_map called with the data_url of the layer (14 features)
test_map_commune_saint_mande.py	adminexpress → gpf_get_feature_by_id_layer → show_map	Map of the commune of Saint-Mandé: show_map called with the data_url of the layer (1 commune)
test_count_batiment_vendee.py	gpf_search_types → gpf_get_features → gpf_count_features (intersects_feature_filter)	Number of buildings in the Vendée department (800 000-900 000). Skipped by default until #42 is fixed (SKIP_TEST_COUNT_BATIMENT_VENDEE=0 to run it)
test_chaining_geocode_distance.py	geocode → distance	Walking distance between the Gare de Lyon and the Eiffel Tower (5-9 km)
test_chaining_distance_piscine_caen.py	geocode → gpf_search_types → gpf_describe_type → gpf_get_features → distance	Walking time from the cinéma LUX (Caen) to the nearest swimming pool: Sivom, Mondeville (25-35 min)

Negative tests

File	Description
test_geocode_address_not_found.py	Non-existent address: checks that the agent reports "adresse non trouvée"

MCP server tests (no model)

File	Description
test_mcp_concurrency.py	Concurrent tool calls must not be crossed. Skipped by default until #33 is fixed (SKIP_TEST_MCP_CONCURRENCY=0 to run it)

Fake tools (no MCP)

Tool	Description
show_map	show_map(title, data_url) mimics a map MCP (MCP Carto, ...) and returns "map displayed". It is added to the agent of all the tests (see conftest.py and #43)

Critical cases not covered

Case	Criticality	Description
Ambiguous address	Medium	No test with an address matching several results (e.g. "rue de la République" without a city)
Overseas territories (DOM-TOM)	Medium	No test on overseas territories (Réunion, Guadeloupe)
Network error / tool error recovery	Low	No test of the agent's ability to retry after a tool error

License

MIT