From f9d374c07e0f03925641b37a41fb2f81f8511912 Mon Sep 17 00:00:00 2001 From: Nicolas Joubert Date: Wed, 30 Sep 2026 09:46:34 +0200 Subject: [PATCH] chore(doc) #30 Add missing documentations: REST client reference page, complete RequestTask reference, cookbooks. Harmonize and fix existing documentation. Co-Authored-By: Claude Opus 5.5 (1M context) --- CHANGELOG.md | 1 + docs/cookbooks/api_to_dto.md | 123 ++++++++++++++++++ docs/cookbooks/csv_to_api.md | 107 +++++++++++++++ docs/index.md | 48 ++++++- docs/reference/client.md | 187 +++++++++++++++++++++++++++ docs/reference/tasks/_template.md | 33 ++--- docs/reference/tasks/request_task.md | 174 ++++++++++++++++++------- 7 files changed, 607 insertions(+), 66 deletions(-) create mode 100644 docs/cookbooks/api_to_dto.md create mode 100644 docs/cookbooks/csv_to_api.md create mode 100644 docs/reference/client.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 367e9ef..929995a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,7 @@ Latest ### Changes * [#28](https://github.com/cleverage/rest-process-bundle/issues/28) Update quality stack: use Rector `withComposerBased()` sets (removed `SYMFONY_64` / `PHPUNIT_100` sets), declare used Symfony packages and PHPUnit range in composer.json, apply quality tools fixes +* [#30](https://github.com/cleverage/rest-process-bundle/issues/30) Add missing documentations: REST client reference page, complete RequestTask reference, cookbooks. Harmonize and fix existing documentation. v3.0 ------ diff --git a/docs/cookbooks/api_to_dto.md b/docs/cookbooks/api_to_dto.md new file mode 100644 index 0000000..0a2c13f --- /dev/null +++ b/docs/cookbooks/api_to_dto.md @@ -0,0 +1,123 @@ +Fetch API data into DTOs +======================== + +This recipe calls a REST API with a parameter given on execution, deserializes the JSON response into a list of DTOs, +then exports them to a CSV file. + +It uses the public [API Carto](https://apicarto.ign.fr/api/doc/codes-postaux) of the IGN, which lists the French +municipalities of a postal code, e.g. `GET https://apicarto.ign.fr/api/codes-postaux/communes/46800`: + +```text +[ + {"codePostal": "46800", "codeCommune": "...", "nomCommune": "...", "libelleAcheminement": "..."}, + ... +] +``` + +The DTO: + +```php +/products/AB-001` with the JSON body `{"name": "Blue chair", "price": 49.9}` (JSON because the + method is `PUT` and `sends` defaults to `application/json`). +- On success (`200`, `201` or `204`, see `valid_response_code`), the response body goes to the + [StatCounterTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/stat_counter_task.md), + which logs the number of pushed lines at the end of the process. +- On any other status code, the raw response body is sent to the `error_outputs`: the + [LoggerTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/logger_task.md) logs it as a + warning, and `error_strategy: skip` goes on with the next line. + +Note that the input replaces whole options: an input `headers` key would replace the `Authorization` header configured +on the task. To send the same header to every request of an API, prefer a +[custom client or a scoped HTTP client](../reference/client.md#examples). diff --git a/docs/index.md b/docs/index.md index 1b7effa..ad72d9f 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,6 +1,6 @@ ## Prerequisite -CleverAge/ProcessBundle must be [installed](https://github.com/cleverage/process-bundle/blob/main/docs/01-quick_start.md#installation. +CleverAge/ProcessBundle must be [installed](https://github.com/cleverage/process-bundle/blob/main/docs/01-quick_start.md#installation). ## Installation @@ -13,13 +13,51 @@ Open a command console, enter your project directory and install it using compos composer require cleverage/rest-process-bundle ``` -Remember to add the following line to config/bundles.php (not required if Symfony Flex is used) +Remember to add the following line to `config/bundles.php` (not required if Symfony Flex is used): ```php CleverAge\RestProcessBundle\CleverAgeRestProcessBundle::class => ['all' => true], ``` -## Reference +## Configuration + +The bundle has no configuration of its own. Each API is accessed through a [REST client](reference/client.md): a +service tagged `cleverage.rest.client`, identified by a unique code and holding the base URI of the API. + +```yaml +# config/services.yaml +services: + app.cleverage_rest_process.client.apicarto_ign: + class: CleverAge\RestProcessBundle\Client\Client + arguments: + $httpClient: '@http_client' + $logger: '@logger' + $code: 'apicarto_ign' + $uri: 'https://apicarto.ign.fr/api' + tags: + - { name: cleverage.rest.client } +``` + +The [RequestTask](reference/tasks/request_task.md) then references this client by its code: + +```yaml +# Task configuration level +fetch: + service: '@CleverAge\RestProcessBundle\Task\RequestTask' + options: + client: apicarto_ign + url: '/codes-postaux/communes/{codePostal}' + method: GET + url_parameters: { codePostal: '{{ codePostal }}' } +``` + +## Documentation -- Tasks - - [RequestTask](reference/tasks/request_task.md) +- Cookbooks + - [Fetch API data into DTOs](cookbooks/api_to_dto.md) + - [Push CSV lines to an API](cookbooks/csv_to_api.md) +- Reference + - [REST client](reference/client.md) + - Tasks + - [RequestTask](reference/tasks/request_task.md) +- [CleverAge/ProcessBundle documentation](https://github.com/cleverage/process-bundle/blob/main/docs/index.md) diff --git a/docs/reference/client.md b/docs/reference/client.md new file mode 100644 index 0000000..20186a2 --- /dev/null +++ b/docs/reference/client.md @@ -0,0 +1,187 @@ +REST client +=========== + +A REST client holds the connection to one API (its base URI, the HTTP client to use) and sends the requests built by +the [RequestTask](tasks/request_task.md). Each client is a service tagged `cleverage.rest.client`, identified by a +unique code that the task references with its `client` option. + +Client reference +---------------- + +* **Interface**: `CleverAge\RestProcessBundle\Client\ClientInterface` +* **Default implementation**: `CleverAge\RestProcessBundle\Client\Client`, based on the Symfony + [HttpClient](https://symfony.com/doc/current/http_client.html) +* **Service tag**: `cleverage.rest.client` +* **Registry**: `CleverAge\RestProcessBundle\Registry\ClientRegistry` (service `cleverage_rest_process.registry.client`) + +Constructor arguments +--------------------- + +Arguments of the default `CleverAge\RestProcessBundle\Client\Client`: + +| Code | Type | Required | Default | Description | +|---------------|------------------------------------------------------|:--------:|---------|-----------------------------------------------------------------------------------------| +| `$httpClient` | `Symfony\Contracts\HttpClient\HttpClientInterface` | **X** | | HTTP client used to send the requests (e.g. `@http_client` or a scoped client) | +| `$logger` | `Psr\Log\LoggerInterface` | **X** | | Logger used to log failed requests | +| `$code` | `string` | **X** | | Unique code of the client, used by the `client` option of the task | +| `$uri` | `string` | **X** | | Base URI of the API, without trailing `/` (e.g. `https://domain/api`) | + +Request options +--------------- + +`ClientInterface::call(array $options)` receives the request options built by the [RequestTask](tasks/request_task.md) +(task options merged with the input). The default client resolves them with the following rules: + +| Code | Type | Required | Default | Description | +|------------------|---------------------------|:--------:|--------------------|--------------------------------------------------------------------------| +| `url` | `string` | **X** | | Path of the endpoint, appended to the base URI | +| `method` | `string` | | `GET` | HTTP method (see below) | +| `sends` | `string` | | `application/json` | `Content-Type` header, not sent if empty | +| `expects` | `string` | | `application/json` | `Accept` header, not sent if empty | +| `url_parameters` | `array` | | `[]` | List of `placeholder => value` replaced in the URL | +| `headers` | `array` | | `[]` | HTTP headers, as `name => value` | +| `data` | `array`, `string`, `null` | | `null` | Payload of the request (see below) | + +Any other key throws a `Symfony\Component\OptionsResolver\Exception\UndefinedOptionsException`. + +The request is then built this way: +- **URL**: `/`, the leading `/` of `url` being removed. Then each `{key}` of `url_parameters` is + replaced by its value, encoded with [`rawurlencode()`](https://www.php.net/manual/en/function.rawurlencode.php) + (values must be strings). +- **Method**: must be one of `HEAD`, `GET`, `POST`, `PUT`, `DELETE`, `OPTIONS`, `TRACE`, `PATCH` (case-sensitive), + otherwise a `CleverAge\RestProcessBundle\Exception\RestRequestException` (` is not an HTTP method`) is thrown. +- **Headers**: `headers`, plus `Content-Type: ` and `Accept: ` when these options are not empty + (they replace the same keys of `headers`). +- **Payload**, passed as a Symfony HttpClient + [request option](https://symfony.com/doc/current/http_client.html#making-requests): + +| `method` | `sends` | `data` is sent as | Result | +|-------------------------|--------------------|-------------------|----------------------------------------------------------------| +| `POST`, `PUT`, `PATCH` | `application/json` | `json` | JSON-encoded body | +| `GET` | any | `query` | Query string parameters (`data` must be an `array` or `null`) | +| any other combination | any | `body` | Raw body (`string`), or form-urlencoded body (`array`) | + +If the HTTP client throws while creating the request, the error is logged (`Rest request failed`, with `url` and +`error`) and a `RestRequestException` is thrown. Note that Symfony HttpClient requests are lazy: most transport errors +only occur when the task reads the response. + +Registration +------------ + +Every service tagged `cleverage.rest.client` is added to the registry (a compiler pass calls +`ClientRegistry::addClient()` for each of them). Two clients with the same code throw an `UnexpectedValueException` +(`Client is already defined`) when the registry is instantiated; a task referencing an unknown code throws a +`CleverAge\RestProcessBundle\Exception\MissingClientException` (`No rest client with code : `). + +Implementing a client +--------------------- + +A client must implement `ClientInterface`: + +| Method | Description | +|-------------------------------------------------------|----------------------------------------------------------------------------------------------| +| `getCode(): string` | Code of the client, used by the `client` option of the task. Must be unique | +| `geUri(): string` | Base URI of the API (the method name is misspelled, `geUri`, in the interface) | +| `setUri(string $uri): void` | Change the base URI | +| `call(array $options = []): ResponseInterface` | Send the request described by the request options, return a Symfony HttpClient response | + +The simplest way is to extend the default `Client` and override one of its protected methods: +- `configureOptions(OptionsResolver $resolver)`: request options accepted by `call()` +- `getRequestOptions(array $options)`: Symfony HttpClient options (headers, `json`, `query`, `body`...) +- `getApiUrl()`: base URI used to build the request URL (defaults to `geUri()`) +- `constructUri(array $options)` / `replaceParametersInUri(string $uri, array $options)`: URL construction + +Examples +-------- + +* Default client + - with [autowiring](https://symfony.com/doc/current/service_container/autowiring.html) enabled (default + `config/services.yaml`), `$httpClient` and `$logger` are injected automatically and only `$code` and `$uri` must be + bound + +```yaml +services: + app.cleverage_rest_process.client.apicarto_ign: + class: CleverAge\RestProcessBundle\Client\Client + bind: + $code: 'apicarto_ign' + $uri: 'https://apicarto.ign.fr/api' + tags: + - { name: cleverage.rest.client } +``` + +* Default client without autowiring, using a + [scoped HTTP client](https://symfony.com/doc/current/http_client.html#scoping-client) for timeouts and authentication + +```yaml +# config/packages/framework.yaml +framework: + http_client: + scoped_clients: + domain_sample.http_client: + base_uri: 'https://domain' + timeout: 10 + auth_bearer: '%env(DOMAIN_API_TOKEN)%' +``` + +```yaml +services: + app.cleverage_rest_process.client.domain_sample: + class: CleverAge\RestProcessBundle\Client\Client + arguments: + $httpClient: '@domain_sample.http_client' + $logger: '@logger' + $code: 'domain_sample' + $uri: 'https://domain/api' + tags: + - { name: cleverage.rest.client } +``` + +* Custom client adding an API key header to every request + +```php +apiKey; + + return $requestOptions; + } +} +``` + +```yaml +services: + app.cleverage_rest_process.client.domain_sample: + class: App\Rest\ApiKeyClient + arguments: + $httpClient: '@http_client' + $logger: '@logger' + $code: 'domain_sample' + $uri: 'https://domain/api' + $apiKey: '%env(DOMAIN_API_KEY)%' + tags: + - { name: cleverage.rest.client } +``` diff --git a/docs/reference/tasks/_template.md b/docs/reference/tasks/_template.md index ed1d4a5..919390a 100644 --- a/docs/reference/tasks/_template.md +++ b/docs/reference/tasks/_template.md @@ -1,44 +1,47 @@ TaskName ======== -_Describe main goal an use cases of the task_ +_Describe the main goal and use cases of the task._ Task reference -------------- -* **Service**: `ClassName` +* **Service**: `Fully\Qualified\ClassName` +* **Iterable task** _(only if it implements `IterableTaskInterface`)_ +* **Blocking task** _(only if it implements `BlockingTaskInterface`)_ +* **Flushable task** _(only if it implements `FlushableTaskInterface`)_ Accepted inputs --------------- -_Description of allowed types_ +_Description of allowed types, or "Input is ignored"._ Possible outputs ---------------- -_Description of possible types_ +_Description of possible types._ Options ------- -| Code | Type | Required | Default | Description | -| ---- | ---- | :------: | ------- | ----------- | -| `code` | `type` | **X** _or nothing_ | `default value` _if available_ | _description_ | +| Code | Type | Required | Default | Description | +|--------|--------|:--------:|-----------------|---------------| +| `code` | `type` | **X** | `default value` | _description_ | + +_If the task has no option, replace the table with "This task has no option."._ Examples -------- -_YAML samples and explanations_ - * Example 1 - details - - details - + ```yaml # Task configuration level code: - service: '@service_ref' - options: - a: 1 - b: 2 + service: '@Fully\Qualified\ClassName' + options: + a: 1 + b: 2 + outputs: [next_task] ``` diff --git a/docs/reference/tasks/request_task.md b/docs/reference/tasks/request_task.md index 590ee7c..d3a1644 100644 --- a/docs/reference/tasks/request_task.md +++ b/docs/reference/tasks/request_task.md @@ -1,90 +1,172 @@ RequestTask -=============== +=========== -Call a Rest Request and get result. +Sends an HTTP request through a registered [REST client](../client.md) and outputs the raw response body. + +The request is built from the task options, optionally overridden by the input: this allows calling the same endpoint +for each item of a flow (e.g. one `PUT` per CSV line) with a different URL, parameters or payload. Task reference -------------- -* **Client Service Interface**: `CleverAge\RestProcessBundle\Client\ClientInterface` -* **Task Service**: `CleverAge\RestProcessBundle\Task\RequestTask` +* **Service**: `CleverAge\RestProcessBundle\Task\RequestTask` Accepted inputs --------------- -`array`: inputs are merged with task defined options. +`array` or empty value (`null`, `[]`...): request options overriding the task options (shallow merge, the input wins). +Allowed keys are `url`, `method`, `headers`, `url_parameters`, `sends`, `expects` and `data`: any other key is passed to +the client, which rejects it (the default client throws an `UndefinedOptionsException`). Possible outputs ---------------- -`string`: the result content of the rest call. +`string`: the body of the response (empty string for a `204 No Content`), as returned by +`Symfony\Contracts\HttpClient\ResponseInterface::getContent()`. It is not decoded: chain a +[DeserializerTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/deserializer_task.md) or a +[TransformerTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/transformer_task.md) to +decode it. + +When the status code is not in `valid_response_code`, the raw response body is sent to the `error_outputs` (see +[Notes](#notes)). Options ------- -### For Client - -| Code | Type | Required | Default | Description | -|--------|----------|:--------:|---------|------------------------------------------------| -| `code` | `string` | **X** | | Service identifier, used by Task client option | -| `uri` | `string` | **X** | | Base uri, concatenated with Task `url` | - -### For Task - -| Code | Type | Required | Default | Description | -|-----------------------|-----------------------------|:--------:|--------------------|------------------------------------------------------------------------------------------| -| `client` | `string` | **X** | | `ClientInterface` service identifier | -| `url` | `string` | **X** | | Relative url to call | -| `method` | `string` | **X** | | HTTP method from `['HEAD', 'GET', 'POST', 'PUT', 'DELETE', 'OPTIONS', 'TRACE', 'PATCH']` | -| `headers` | `array` | | `[]` | | -| `url_parameters` | `array` | | `[]` | Search/Replace data on `url` | -| `data` | `array`, `string` or `null` | | `null` | Treated as `body`, `query` or `json` on HttpClient, depending on `method` and `sends` | -| `sends` | `string` | | `application/json` | `Content-Type` header, if value is not empty | -| `expects` | `string` | | `application/json` | `Accept` header, if value is not empty | -| `valid_response_code` | `array` | | `[200, 201, 204]` | One or more [HTTP status code](https://en.wikipedia.org/wiki/List_of_HTTP_status_codes) | -| `log_response` | `bool` | | `false` | | +| Code | Type | Required | Default | Description | +|-----------------------|---------------------------|:--------:|--------------------|------------------------------------------------------------------------------------------------------------------------------------------------| +| `client` | `string` | **X** | | Code of the [REST client](../client.md) to use (value returned by its `getCode()`) | +| `url` | `string` | **X** | | Path of the endpoint, appended to the client base URI (a leading `/` is optional). May contain `{placeholders}` replaced by `url_parameters` | +| `method` | `string` | **X** | | HTTP method, in uppercase, among `HEAD`, `GET`, `POST`, `PUT`, `DELETE`, `OPTIONS`, `TRACE`, `PATCH` (checked by the default client) | +| `headers` | `array` | | `[]` | HTTP headers, as `name => value` | +| `url_parameters` | `array` | | `[]` | List of `placeholder => value`: each `{placeholder}` of the URL is replaced by the URL-encoded value (values must be strings) | +| `data` | `array`, `string`, `null` | | `null` | Payload of the request, sent as JSON body, query string or raw body depending on `method` and `sends` (see [REST client](../client.md#request-options)) | +| `sends` | `string` | | `application/json` | Value of the `Content-Type` header (not sent if empty) | +| `expects` | `string` | | `application/json` | Value of the `Accept` header (not sent if empty) | +| `valid_response_code` | `array` | | `[200, 201, 204]` | List of the [HTTP status codes](https://en.wikipedia.org/wiki/List_of_HTTP_status_codes) considered as a success | +| `log_response` | `bool` | | `false` | Log the request options and the response object (`debug` level) once the response is received | + +Options are resolved once per process execution: [contextual values](https://github.com/cleverage/process-bundle/blob/main/docs/01-quick_start.md#contextual-values) +like `'{{ code }}'` (passed with `-c code:"'value'"`) are allowed in any option. Use the input to change the request for +each item. Examples -------- -### Client +* `GET` with a URL parameter taken from the process context + - run with `bin/console cleverage:process:execute -c codePostal:"'46800'"` + - requests `https://apicarto.ign.fr/api/codes-postaux/communes/46800` with the client below ```yaml services: app.cleverage_rest_process.client.apicarto_ign: class: CleverAge\RestProcessBundle\Client\Client - bind: - $code: 'domain_sample' - $uri: 'https://domain/api' + arguments: + $httpClient: '@http_client' + $logger: '@logger' + $code: 'apicarto_ign' + $uri: 'https://apicarto.ign.fr/api' tags: - { name: cleverage.rest.client } -``` +``` -### Task +```yaml +# Task configuration level +entry: + service: '@CleverAge\RestProcessBundle\Task\RequestTask' + error_strategy: stop + options: + client: apicarto_ign + url: '/codes-postaux/communes/{codePostal}' + method: GET + url_parameters: { codePostal: '{{ codePostal }}' } + outputs: [deserialize] +``` + +* `GET` with query string parameters + - requests `https://domain/api/books?page=2&limit=50` ```yaml # Task configuration level -code: +list_books: service: '@CleverAge\RestProcessBundle\Task\RequestTask' - error_strategy: 'stop' options: client: domain_sample - url: '/sample/{parameter}' - method: 'GET' - url_parameters: { parameter: '{{ parameter }}' } + url: '/books' + method: GET + data: + page: 2 + limit: 50 + outputs: [deserialize] ``` +* `POST` with a JSON body + - `data` is JSON-encoded because the method is `POST` and `sends` is `application/json` + ```yaml # Task configuration level -code: +entry: service: '@CleverAge\RestProcessBundle\Task\RequestTask' - error_strategy: 'stop' + error_strategy: stop + options: + client: apicarto_ign + url: '/aoc/appellation-viticole' + method: POST + data: + geom: + type: Point + coordinates: [-1.691634, 48.104237] + outputs: [deserialize] +``` + +* `POST` of a form, with an authentication header + - `data` is sent as an `application/x-www-form-urlencoded` body + +```yaml +# Task configuration level +get_token: + service: '@CleverAge\RestProcessBundle\Task\RequestTask' + options: + client: domain_sample + url: '/oauth/token' + method: POST + sends: 'application/x-www-form-urlencoded' + headers: + Authorization: 'Basic {{ credentials }}' + data: + grant_type: client_credentials + outputs: [deserialize] +``` + +* Request overridden by the input + - the previous task outputs `{url_parameters: {id: '42'}, data: {title: 'New title'}}` + - requests `PUT https://domain/api/books/42` with the JSON body `{"title": "New title"}` + - a `404 Not Found` is logged and the item is skipped, the process goes on with the next one + +```yaml +# Task configuration level +update_book: + service: '@CleverAge\RestProcessBundle\Task\RequestTask' + error_strategy: skip options: client: domain_sample - url: '/sample' - method: 'POST' - data: # May be a json string or an array - parameter_1: - parameter_11: "eleven" - array: [-1, 666] + url: '/books/{id}' + method: PUT + error_outputs: [log_error] # Receives the raw response body ``` + +Notes +----- + +* The input is merged with `array_merge()`: an input key replaces the whole option (e.g. an input `url_parameters` + replaces the configured `url_parameters`, it is not merged with it). `client`, `valid_response_code` and + `log_response` cannot be overridden by the input. +* When the status code is not in `valid_response_code`, the task sets the raw response body as error output, then + fails with an `Invalid response code` exception, logged with the response headers and body. The process then follows + the task `error_strategy`: `skip` goes on with the next item, `stop` stops the process. +* The response content is read with `getContent()`, which throws for `3xx`, `4xx` and `5xx` status codes: adding + such a code to `valid_response_code` only prevents the error log, the task still fails. Redirections are followed by + the HTTP client, so a `3xx` status is only received when redirections are disabled or exceeded. +* A transport error (DNS failure, timeout...) or an unknown `method` makes the task fail as well. +* A `client` that is not registered throws a `CleverAge\RestProcessBundle\Exception\MissingClientException` + (`No rest client with code : `) on the first execution of the task.