diff --git a/CHANGELOG.md b/CHANGELOG.md
index 367e9ef..929995a 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -3,6 +3,7 @@ Latest
### Changes
* [#28](https://github.com/cleverage/rest-process-bundle/issues/28) Update quality stack: use Rector `withComposerBased()` sets (removed `SYMFONY_64` / `PHPUNIT_100` sets), declare used Symfony packages and PHPUnit range in composer.json, apply quality tools fixes
+* [#30](https://github.com/cleverage/rest-process-bundle/issues/30) Add missing documentations: REST client reference page, complete RequestTask reference, cookbooks. Harmonize and fix existing documentation.
v3.0
------
diff --git a/docs/cookbooks/api_to_dto.md b/docs/cookbooks/api_to_dto.md
new file mode 100644
index 0000000..0a2c13f
--- /dev/null
+++ b/docs/cookbooks/api_to_dto.md
@@ -0,0 +1,123 @@
+Fetch API data into DTOs
+========================
+
+This recipe calls a REST API with a parameter given on execution, deserializes the JSON response into a list of DTOs,
+then exports them to a CSV file.
+
+It uses the public [API Carto](https://apicarto.ign.fr/api/doc/codes-postaux) of the IGN, which lists the French
+municipalities of a postal code, e.g. `GET https://apicarto.ign.fr/api/codes-postaux/communes/46800`:
+
+```text
+[
+ {"codePostal": "46800", "codeCommune": "...", "nomCommune": "...", "libelleAcheminement": "..."},
+ ...
+]
+```
+
+The DTO:
+
+```php
+/products/AB-001` with the JSON body `{"name": "Blue chair", "price": 49.9}` (JSON because the
+ method is `PUT` and `sends` defaults to `application/json`).
+- On success (`200`, `201` or `204`, see `valid_response_code`), the response body goes to the
+ [StatCounterTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/stat_counter_task.md),
+ which logs the number of pushed lines at the end of the process.
+- On any other status code, the raw response body is sent to the `error_outputs`: the
+ [LoggerTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/logger_task.md) logs it as a
+ warning, and `error_strategy: skip` goes on with the next line.
+
+Note that the input replaces whole options: an input `headers` key would replace the `Authorization` header configured
+on the task. To send the same header to every request of an API, prefer a
+[custom client or a scoped HTTP client](../reference/client.md#examples).
diff --git a/docs/index.md b/docs/index.md
index 1b7effa..ad72d9f 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -1,6 +1,6 @@
## Prerequisite
-CleverAge/ProcessBundle must be [installed](https://github.com/cleverage/process-bundle/blob/main/docs/01-quick_start.md#installation.
+CleverAge/ProcessBundle must be [installed](https://github.com/cleverage/process-bundle/blob/main/docs/01-quick_start.md#installation).
## Installation
@@ -13,13 +13,51 @@ Open a command console, enter your project directory and install it using compos
composer require cleverage/rest-process-bundle
```
-Remember to add the following line to config/bundles.php (not required if Symfony Flex is used)
+Remember to add the following line to `config/bundles.php` (not required if Symfony Flex is used):
```php
CleverAge\RestProcessBundle\CleverAgeRestProcessBundle::class => ['all' => true],
```
-## Reference
+## Configuration
+
+The bundle has no configuration of its own. Each API is accessed through a [REST client](reference/client.md): a
+service tagged `cleverage.rest.client`, identified by a unique code and holding the base URI of the API.
+
+```yaml
+# config/services.yaml
+services:
+ app.cleverage_rest_process.client.apicarto_ign:
+ class: CleverAge\RestProcessBundle\Client\Client
+ arguments:
+ $httpClient: '@http_client'
+ $logger: '@logger'
+ $code: 'apicarto_ign'
+ $uri: 'https://apicarto.ign.fr/api'
+ tags:
+ - { name: cleverage.rest.client }
+```
+
+The [RequestTask](reference/tasks/request_task.md) then references this client by its code:
+
+```yaml
+# Task configuration level
+fetch:
+ service: '@CleverAge\RestProcessBundle\Task\RequestTask'
+ options:
+ client: apicarto_ign
+ url: '/codes-postaux/communes/{codePostal}'
+ method: GET
+ url_parameters: { codePostal: '{{ codePostal }}' }
+```
+
+## Documentation
-- Tasks
- - [RequestTask](reference/tasks/request_task.md)
+- Cookbooks
+ - [Fetch API data into DTOs](cookbooks/api_to_dto.md)
+ - [Push CSV lines to an API](cookbooks/csv_to_api.md)
+- Reference
+ - [REST client](reference/client.md)
+ - Tasks
+ - [RequestTask](reference/tasks/request_task.md)
+- [CleverAge/ProcessBundle documentation](https://github.com/cleverage/process-bundle/blob/main/docs/index.md)
diff --git a/docs/reference/client.md b/docs/reference/client.md
new file mode 100644
index 0000000..20186a2
--- /dev/null
+++ b/docs/reference/client.md
@@ -0,0 +1,187 @@
+REST client
+===========
+
+A REST client holds the connection to one API (its base URI, the HTTP client to use) and sends the requests built by
+the [RequestTask](tasks/request_task.md). Each client is a service tagged `cleverage.rest.client`, identified by a
+unique code that the task references with its `client` option.
+
+Client reference
+----------------
+
+* **Interface**: `CleverAge\RestProcessBundle\Client\ClientInterface`
+* **Default implementation**: `CleverAge\RestProcessBundle\Client\Client`, based on the Symfony
+ [HttpClient](https://symfony.com/doc/current/http_client.html)
+* **Service tag**: `cleverage.rest.client`
+* **Registry**: `CleverAge\RestProcessBundle\Registry\ClientRegistry` (service `cleverage_rest_process.registry.client`)
+
+Constructor arguments
+---------------------
+
+Arguments of the default `CleverAge\RestProcessBundle\Client\Client`:
+
+| Code | Type | Required | Default | Description |
+|---------------|------------------------------------------------------|:--------:|---------|-----------------------------------------------------------------------------------------|
+| `$httpClient` | `Symfony\Contracts\HttpClient\HttpClientInterface` | **X** | | HTTP client used to send the requests (e.g. `@http_client` or a scoped client) |
+| `$logger` | `Psr\Log\LoggerInterface` | **X** | | Logger used to log failed requests |
+| `$code` | `string` | **X** | | Unique code of the client, used by the `client` option of the task |
+| `$uri` | `string` | **X** | | Base URI of the API, without trailing `/` (e.g. `https://domain/api`) |
+
+Request options
+---------------
+
+`ClientInterface::call(array $options)` receives the request options built by the [RequestTask](tasks/request_task.md)
+(task options merged with the input). The default client resolves them with the following rules:
+
+| Code | Type | Required | Default | Description |
+|------------------|---------------------------|:--------:|--------------------|--------------------------------------------------------------------------|
+| `url` | `string` | **X** | | Path of the endpoint, appended to the base URI |
+| `method` | `string` | | `GET` | HTTP method (see below) |
+| `sends` | `string` | | `application/json` | `Content-Type` header, not sent if empty |
+| `expects` | `string` | | `application/json` | `Accept` header, not sent if empty |
+| `url_parameters` | `array` | | `[]` | List of `placeholder => value` replaced in the URL |
+| `headers` | `array` | | `[]` | HTTP headers, as `name => value` |
+| `data` | `array`, `string`, `null` | | `null` | Payload of the request (see below) |
+
+Any other key throws a `Symfony\Component\OptionsResolver\Exception\UndefinedOptionsException`.
+
+The request is then built this way:
+- **URL**: `/`, the leading `/` of `url` being removed. Then each `{key}` of `url_parameters` is
+ replaced by its value, encoded with [`rawurlencode()`](https://www.php.net/manual/en/function.rawurlencode.php)
+ (values must be strings).
+- **Method**: must be one of `HEAD`, `GET`, `POST`, `PUT`, `DELETE`, `OPTIONS`, `TRACE`, `PATCH` (case-sensitive),
+ otherwise a `CleverAge\RestProcessBundle\Exception\RestRequestException` (` is not an HTTP method`) is thrown.
+- **Headers**: `headers`, plus `Content-Type: ` and `Accept: ` when these options are not empty
+ (they replace the same keys of `headers`).
+- **Payload**, passed as a Symfony HttpClient
+ [request option](https://symfony.com/doc/current/http_client.html#making-requests):
+
+| `method` | `sends` | `data` is sent as | Result |
+|-------------------------|--------------------|-------------------|----------------------------------------------------------------|
+| `POST`, `PUT`, `PATCH` | `application/json` | `json` | JSON-encoded body |
+| `GET` | any | `query` | Query string parameters (`data` must be an `array` or `null`) |
+| any other combination | any | `body` | Raw body (`string`), or form-urlencoded body (`array`) |
+
+If the HTTP client throws while creating the request, the error is logged (`Rest request failed`, with `url` and
+`error`) and a `RestRequestException` is thrown. Note that Symfony HttpClient requests are lazy: most transport errors
+only occur when the task reads the response.
+
+Registration
+------------
+
+Every service tagged `cleverage.rest.client` is added to the registry (a compiler pass calls
+`ClientRegistry::addClient()` for each of them). Two clients with the same code throw an `UnexpectedValueException`
+(`Client is already defined`) when the registry is instantiated; a task referencing an unknown code throws a
+`CleverAge\RestProcessBundle\Exception\MissingClientException` (`No rest client with code : `).
+
+Implementing a client
+---------------------
+
+A client must implement `ClientInterface`:
+
+| Method | Description |
+|-------------------------------------------------------|----------------------------------------------------------------------------------------------|
+| `getCode(): string` | Code of the client, used by the `client` option of the task. Must be unique |
+| `geUri(): string` | Base URI of the API (the method name is misspelled, `geUri`, in the interface) |
+| `setUri(string $uri): void` | Change the base URI |
+| `call(array $options = []): ResponseInterface` | Send the request described by the request options, return a Symfony HttpClient response |
+
+The simplest way is to extend the default `Client` and override one of its protected methods:
+- `configureOptions(OptionsResolver $resolver)`: request options accepted by `call()`
+- `getRequestOptions(array $options)`: Symfony HttpClient options (headers, `json`, `query`, `body`...)
+- `getApiUrl()`: base URI used to build the request URL (defaults to `geUri()`)
+- `constructUri(array $options)` / `replaceParametersInUri(string $uri, array $options)`: URL construction
+
+Examples
+--------
+
+* Default client
+ - with [autowiring](https://symfony.com/doc/current/service_container/autowiring.html) enabled (default
+ `config/services.yaml`), `$httpClient` and `$logger` are injected automatically and only `$code` and `$uri` must be
+ bound
+
+```yaml
+services:
+ app.cleverage_rest_process.client.apicarto_ign:
+ class: CleverAge\RestProcessBundle\Client\Client
+ bind:
+ $code: 'apicarto_ign'
+ $uri: 'https://apicarto.ign.fr/api'
+ tags:
+ - { name: cleverage.rest.client }
+```
+
+* Default client without autowiring, using a
+ [scoped HTTP client](https://symfony.com/doc/current/http_client.html#scoping-client) for timeouts and authentication
+
+```yaml
+# config/packages/framework.yaml
+framework:
+ http_client:
+ scoped_clients:
+ domain_sample.http_client:
+ base_uri: 'https://domain'
+ timeout: 10
+ auth_bearer: '%env(DOMAIN_API_TOKEN)%'
+```
+
+```yaml
+services:
+ app.cleverage_rest_process.client.domain_sample:
+ class: CleverAge\RestProcessBundle\Client\Client
+ arguments:
+ $httpClient: '@domain_sample.http_client'
+ $logger: '@logger'
+ $code: 'domain_sample'
+ $uri: 'https://domain/api'
+ tags:
+ - { name: cleverage.rest.client }
+```
+
+* Custom client adding an API key header to every request
+
+```php
+apiKey;
+
+ return $requestOptions;
+ }
+}
+```
+
+```yaml
+services:
+ app.cleverage_rest_process.client.domain_sample:
+ class: App\Rest\ApiKeyClient
+ arguments:
+ $httpClient: '@http_client'
+ $logger: '@logger'
+ $code: 'domain_sample'
+ $uri: 'https://domain/api'
+ $apiKey: '%env(DOMAIN_API_KEY)%'
+ tags:
+ - { name: cleverage.rest.client }
+```
diff --git a/docs/reference/tasks/_template.md b/docs/reference/tasks/_template.md
index ed1d4a5..919390a 100644
--- a/docs/reference/tasks/_template.md
+++ b/docs/reference/tasks/_template.md
@@ -1,44 +1,47 @@
TaskName
========
-_Describe main goal an use cases of the task_
+_Describe the main goal and use cases of the task._
Task reference
--------------
-* **Service**: `ClassName`
+* **Service**: `Fully\Qualified\ClassName`
+* **Iterable task** _(only if it implements `IterableTaskInterface`)_
+* **Blocking task** _(only if it implements `BlockingTaskInterface`)_
+* **Flushable task** _(only if it implements `FlushableTaskInterface`)_
Accepted inputs
---------------
-_Description of allowed types_
+_Description of allowed types, or "Input is ignored"._
Possible outputs
----------------
-_Description of possible types_
+_Description of possible types._
Options
-------
-| Code | Type | Required | Default | Description |
-| ---- | ---- | :------: | ------- | ----------- |
-| `code` | `type` | **X** _or nothing_ | `default value` _if available_ | _description_ |
+| Code | Type | Required | Default | Description |
+|--------|--------|:--------:|-----------------|---------------|
+| `code` | `type` | **X** | `default value` | _description_ |
+
+_If the task has no option, replace the table with "This task has no option."._
Examples
--------
-_YAML samples and explanations_
-
* Example 1
- details
- - details
-
+
```yaml
# Task configuration level
code:
- service: '@service_ref'
- options:
- a: 1
- b: 2
+ service: '@Fully\Qualified\ClassName'
+ options:
+ a: 1
+ b: 2
+ outputs: [next_task]
```
diff --git a/docs/reference/tasks/request_task.md b/docs/reference/tasks/request_task.md
index 590ee7c..d3a1644 100644
--- a/docs/reference/tasks/request_task.md
+++ b/docs/reference/tasks/request_task.md
@@ -1,90 +1,172 @@
RequestTask
-===============
+===========
-Call a Rest Request and get result.
+Sends an HTTP request through a registered [REST client](../client.md) and outputs the raw response body.
+
+The request is built from the task options, optionally overridden by the input: this allows calling the same endpoint
+for each item of a flow (e.g. one `PUT` per CSV line) with a different URL, parameters or payload.
Task reference
--------------
-* **Client Service Interface**: `CleverAge\RestProcessBundle\Client\ClientInterface`
-* **Task Service**: `CleverAge\RestProcessBundle\Task\RequestTask`
+* **Service**: `CleverAge\RestProcessBundle\Task\RequestTask`
Accepted inputs
---------------
-`array`: inputs are merged with task defined options.
+`array` or empty value (`null`, `[]`...): request options overriding the task options (shallow merge, the input wins).
+Allowed keys are `url`, `method`, `headers`, `url_parameters`, `sends`, `expects` and `data`: any other key is passed to
+the client, which rejects it (the default client throws an `UndefinedOptionsException`).
Possible outputs
----------------
-`string`: the result content of the rest call.
+`string`: the body of the response (empty string for a `204 No Content`), as returned by
+`Symfony\Contracts\HttpClient\ResponseInterface::getContent()`. It is not decoded: chain a
+[DeserializerTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/deserializer_task.md) or a
+[TransformerTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/transformer_task.md) to
+decode it.
+
+When the status code is not in `valid_response_code`, the raw response body is sent to the `error_outputs` (see
+[Notes](#notes)).
Options
-------
-### For Client
-
-| Code | Type | Required | Default | Description |
-|--------|----------|:--------:|---------|------------------------------------------------|
-| `code` | `string` | **X** | | Service identifier, used by Task client option |
-| `uri` | `string` | **X** | | Base uri, concatenated with Task `url` |
-
-### For Task
-
-| Code | Type | Required | Default | Description |
-|-----------------------|-----------------------------|:--------:|--------------------|------------------------------------------------------------------------------------------|
-| `client` | `string` | **X** | | `ClientInterface` service identifier |
-| `url` | `string` | **X** | | Relative url to call |
-| `method` | `string` | **X** | | HTTP method from `['HEAD', 'GET', 'POST', 'PUT', 'DELETE', 'OPTIONS', 'TRACE', 'PATCH']` |
-| `headers` | `array` | | `[]` | |
-| `url_parameters` | `array` | | `[]` | Search/Replace data on `url` |
-| `data` | `array`, `string` or `null` | | `null` | Treated as `body`, `query` or `json` on HttpClient, depending on `method` and `sends` |
-| `sends` | `string` | | `application/json` | `Content-Type` header, if value is not empty |
-| `expects` | `string` | | `application/json` | `Accept` header, if value is not empty |
-| `valid_response_code` | `array` | | `[200, 201, 204]` | One or more [HTTP status code](https://en.wikipedia.org/wiki/List_of_HTTP_status_codes) |
-| `log_response` | `bool` | | `false` | |
+| Code | Type | Required | Default | Description |
+|-----------------------|---------------------------|:--------:|--------------------|------------------------------------------------------------------------------------------------------------------------------------------------|
+| `client` | `string` | **X** | | Code of the [REST client](../client.md) to use (value returned by its `getCode()`) |
+| `url` | `string` | **X** | | Path of the endpoint, appended to the client base URI (a leading `/` is optional). May contain `{placeholders}` replaced by `url_parameters` |
+| `method` | `string` | **X** | | HTTP method, in uppercase, among `HEAD`, `GET`, `POST`, `PUT`, `DELETE`, `OPTIONS`, `TRACE`, `PATCH` (checked by the default client) |
+| `headers` | `array` | | `[]` | HTTP headers, as `name => value` |
+| `url_parameters` | `array` | | `[]` | List of `placeholder => value`: each `{placeholder}` of the URL is replaced by the URL-encoded value (values must be strings) |
+| `data` | `array`, `string`, `null` | | `null` | Payload of the request, sent as JSON body, query string or raw body depending on `method` and `sends` (see [REST client](../client.md#request-options)) |
+| `sends` | `string` | | `application/json` | Value of the `Content-Type` header (not sent if empty) |
+| `expects` | `string` | | `application/json` | Value of the `Accept` header (not sent if empty) |
+| `valid_response_code` | `array` | | `[200, 201, 204]` | List of the [HTTP status codes](https://en.wikipedia.org/wiki/List_of_HTTP_status_codes) considered as a success |
+| `log_response` | `bool` | | `false` | Log the request options and the response object (`debug` level) once the response is received |
+
+Options are resolved once per process execution: [contextual values](https://github.com/cleverage/process-bundle/blob/main/docs/01-quick_start.md#contextual-values)
+like `'{{ code }}'` (passed with `-c code:"'value'"`) are allowed in any option. Use the input to change the request for
+each item.
Examples
--------
-### Client
+* `GET` with a URL parameter taken from the process context
+ - run with `bin/console cleverage:process:execute -c codePostal:"'46800'"`
+ - requests `https://apicarto.ign.fr/api/codes-postaux/communes/46800` with the client below
```yaml
services:
app.cleverage_rest_process.client.apicarto_ign:
class: CleverAge\RestProcessBundle\Client\Client
- bind:
- $code: 'domain_sample'
- $uri: 'https://domain/api'
+ arguments:
+ $httpClient: '@http_client'
+ $logger: '@logger'
+ $code: 'apicarto_ign'
+ $uri: 'https://apicarto.ign.fr/api'
tags:
- { name: cleverage.rest.client }
-```
+```
-### Task
+```yaml
+# Task configuration level
+entry:
+ service: '@CleverAge\RestProcessBundle\Task\RequestTask'
+ error_strategy: stop
+ options:
+ client: apicarto_ign
+ url: '/codes-postaux/communes/{codePostal}'
+ method: GET
+ url_parameters: { codePostal: '{{ codePostal }}' }
+ outputs: [deserialize]
+```
+
+* `GET` with query string parameters
+ - requests `https://domain/api/books?page=2&limit=50`
```yaml
# Task configuration level
-code:
+list_books:
service: '@CleverAge\RestProcessBundle\Task\RequestTask'
- error_strategy: 'stop'
options:
client: domain_sample
- url: '/sample/{parameter}'
- method: 'GET'
- url_parameters: { parameter: '{{ parameter }}' }
+ url: '/books'
+ method: GET
+ data:
+ page: 2
+ limit: 50
+ outputs: [deserialize]
```
+* `POST` with a JSON body
+ - `data` is JSON-encoded because the method is `POST` and `sends` is `application/json`
+
```yaml
# Task configuration level
-code:
+entry:
service: '@CleverAge\RestProcessBundle\Task\RequestTask'
- error_strategy: 'stop'
+ error_strategy: stop
+ options:
+ client: apicarto_ign
+ url: '/aoc/appellation-viticole'
+ method: POST
+ data:
+ geom:
+ type: Point
+ coordinates: [-1.691634, 48.104237]
+ outputs: [deserialize]
+```
+
+* `POST` of a form, with an authentication header
+ - `data` is sent as an `application/x-www-form-urlencoded` body
+
+```yaml
+# Task configuration level
+get_token:
+ service: '@CleverAge\RestProcessBundle\Task\RequestTask'
+ options:
+ client: domain_sample
+ url: '/oauth/token'
+ method: POST
+ sends: 'application/x-www-form-urlencoded'
+ headers:
+ Authorization: 'Basic {{ credentials }}'
+ data:
+ grant_type: client_credentials
+ outputs: [deserialize]
+```
+
+* Request overridden by the input
+ - the previous task outputs `{url_parameters: {id: '42'}, data: {title: 'New title'}}`
+ - requests `PUT https://domain/api/books/42` with the JSON body `{"title": "New title"}`
+ - a `404 Not Found` is logged and the item is skipped, the process goes on with the next one
+
+```yaml
+# Task configuration level
+update_book:
+ service: '@CleverAge\RestProcessBundle\Task\RequestTask'
+ error_strategy: skip
options:
client: domain_sample
- url: '/sample'
- method: 'POST'
- data: # May be a json string or an array
- parameter_1:
- parameter_11: "eleven"
- array: [-1, 666]
+ url: '/books/{id}'
+ method: PUT
+ error_outputs: [log_error] # Receives the raw response body
```
+
+Notes
+-----
+
+* The input is merged with `array_merge()`: an input key replaces the whole option (e.g. an input `url_parameters`
+ replaces the configured `url_parameters`, it is not merged with it). `client`, `valid_response_code` and
+ `log_response` cannot be overridden by the input.
+* When the status code is not in `valid_response_code`, the task sets the raw response body as error output, then
+ fails with an `Invalid response code` exception, logged with the response headers and body. The process then follows
+ the task `error_strategy`: `skip` goes on with the next item, `stop` stops the process.
+* The response content is read with `getContent()`, which throws for `3xx`, `4xx` and `5xx` status codes: adding
+ such a code to `valid_response_code` only prevents the error log, the task still fails. Redirections are followed by
+ the HTTP client, so a `3xx` status is only received when redirections are disabled or exceeded.
+* A transport error (DNS failure, timeout...) or an unknown `method` makes the task fail as well.
+* A `client` that is not registered throws a `CleverAge\RestProcessBundle\Exception\MissingClientException`
+ (`No rest client with code : `) on the first execution of the task.