Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ Latest

### Changes
* [#28](https://github.com/cleverage/rest-process-bundle/issues/28) Update quality stack: use Rector `withComposerBased()` sets (removed `SYMFONY_64` / `PHPUNIT_100` sets), declare used Symfony packages and PHPUnit range in composer.json, apply quality tools fixes
* [#30](https://github.com/cleverage/rest-process-bundle/issues/30) Add missing documentations: REST client reference page, complete RequestTask reference, cookbooks. Harmonize and fix existing documentation.

v3.0
------
Expand Down
123 changes: 123 additions & 0 deletions docs/cookbooks/api_to_dto.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,123 @@
Fetch API data into DTOs
========================

This recipe calls a REST API with a parameter given on execution, deserializes the JSON response into a list of DTOs,
then exports them to a CSV file.

It uses the public [API Carto](https://apicarto.ign.fr/api/doc/codes-postaux) of the IGN, which lists the French
municipalities of a postal code, e.g. `GET https://apicarto.ign.fr/api/codes-postaux/communes/46800`:

```text
[
{"codePostal": "46800", "codeCommune": "...", "nomCommune": "...", "libelleAcheminement": "..."},
...
]
```

The DTO:

```php
<?php

declare(strict_types=1);

namespace App\Dto;

class Commune
{
public string $codePostal;
public string $codeCommune;
public string $nomCommune;
public string $libelleAcheminement;
}
```

The REST client (with autowiring enabled, see [REST client](../reference/client.md)):

```yaml
# config/services.yaml
services:
app.cleverage_rest_process.client.apicarto_ign:
class: CleverAge\RestProcessBundle\Client\Client
bind:
$code: 'apicarto_ign'
$uri: 'https://apicarto.ign.fr/api'
tags:
- { name: cleverage.rest.client }
```

The process, executed with `bin/console cleverage:process:execute app.communes_export -c codePostal:"'46800'"`:

```yaml
clever_age_process:
configurations:
app.communes_export:
description: 'Export the municipalities of a postal code'
help: "bin/console cleverage:process:execute app.communes_export -c codePostal:\"'46800'\""
tasks:
fetch:
service: '@CleverAge\RestProcessBundle\Task\RequestTask'
error_strategy: stop
options:
client: apicarto_ign
url: '/codes-postaux/communes/{codePostal}'
method: GET
url_parameters: { codePostal: '{{ codePostal }}' }
outputs: [deserialize]

deserialize:
service: '@CleverAge\ProcessBundle\Task\Serialization\DeserializerTask'
options:
type: 'App\Dto\Commune[]'
format: json
outputs: [iterate]

iterate:
service: '@CleverAge\ProcessBundle\Task\InputIteratorTask'
outputs: [normalize]

normalize:
service: '@CleverAge\ProcessBundle\Task\Serialization\NormalizerTask'
options:
format: csv
outputs: [write]

write:
service: '@CleverAge\ProcessBundle\Task\File\Csv\CsvWriterTask'
options:
file_path: '%kernel.project_dir%/var/exports/communes_{date}.csv'
```

How it works:
- The [RequestTask](../reference/tasks/request_task.md) replaces `{codePostal}` in the URL by the `codePostal` context
value, then sends `GET https://apicarto.ign.fr/api/codes-postaux/communes/46800` through the `apicarto_ign`
[client](../reference/client.md). It outputs the raw JSON body as a string. With `error_strategy: stop`, an invalid
status code (e.g. `404` for an unknown postal code) stops the process.
- The [DeserializerTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/deserializer_task.md)
decodes the JSON and denormalizes it into an array of `App\Dto\Commune` (the `[]` suffix denormalizes a list).
- The [InputIteratorTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/input_iterator_task.md)
outputs each DTO one by one.
- The [NormalizerTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/normalizer_task.md)
converts each DTO back into an array, written as a line by the
[CsvWriterTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/csv_writer_task.md) (the
headers are the keys of the first line).

To keep the decoded data as arrays instead of DTOs, replace the `deserialize` task by a
[TransformerTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/transformer_task.md) using
the [callback](https://github.com/cleverage/process-bundle/blob/main/docs/reference/transformers/callback_transformer.md)
transformer:

```yaml
# Task configuration level
json_decode:
service: '@CleverAge\ProcessBundle\Task\TransformerTask'
options:
transformers:
callback:
callback: json_decode
right_parameters: [true] # Decode JSON objects as associative arrays
outputs: [iterate]
```

To send a JSON payload instead, use the `POST` method with `data` (see the
[RequestTask examples](../reference/tasks/request_task.md#examples)).
107 changes: 107 additions & 0 deletions docs/cookbooks/csv_to_api.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
Push CSV lines to an API
========================

This recipe reads a CSV file and updates one API resource per line, with a `PUT` request whose URL and JSON payload
are built from the line. A line rejected by the API is logged and skipped, without stopping the process.

The CSV file `var/data/products.csv`:

```csv
sku;name;price
AB-001;Blue chair;49.90
AB-002;Red table;129.00
```

The REST client (with autowiring enabled, see [REST client](../reference/client.md)):

```yaml
# config/services.yaml
services:
app.cleverage_rest_process.client.catalog:
class: CleverAge\RestProcessBundle\Client\Client
bind:
$code: 'catalog'
$uri: '%env(CATALOG_API_URI)%' # e.g. https://catalog.example.com/api
tags:
- { name: cleverage.rest.client }
```

The process:

```yaml
clever_age_process:
configurations:
app.products_push:
description: 'Push the products CSV file to the catalog API'
tasks:
read:
service: '@CleverAge\ProcessBundle\Task\File\Csv\CsvReaderTask'
options:
file_path: '%kernel.project_dir%/var/data/products.csv'
outputs: [build_request]

build_request:
service: '@CleverAge\ProcessBundle\Task\TransformerTask'
options:
transformers:
mapping:
mapping:
url_parameters:
code:
sku: '[sku]'
data:
code: '.'
transformers:
mapping:
mapping:
name:
code: '[name]'
price:
code: '[price]'
transformers:
cast:
type: float
outputs: [push]

push:
service: '@CleverAge\RestProcessBundle\Task\RequestTask'
error_strategy: skip # A rejected line does not stop the process
options:
client: catalog
url: '/products/{sku}'
method: PUT
headers:
Authorization: 'Bearer %env(CATALOG_API_TOKEN)%'
outputs: [count_pushed]
error_outputs: [log_rejected]

count_pushed:
service: '@CleverAge\ProcessBundle\Task\Reporting\StatCounterTask'

log_rejected:
service: '@CleverAge\ProcessBundle\Task\Reporting\LoggerTask'
options:
level: warning
message: 'Product rejected by the catalog API'
```

How it works:
- [CsvReaderTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/csv_reader_task.md) is
iterable: each line goes through the following tasks before the next one is read.
- The [TransformerTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/transformer_task.md)
builds the request options of the line with the
[mapping](https://github.com/cleverage/process-bundle/blob/main/docs/reference/transformers/mapping_transformer.md)
transformer, e.g. `{url_parameters: {sku: 'AB-001'}, data: {name: 'Blue chair', price: 49.9}}`.
- The [RequestTask](../reference/tasks/request_task.md) merges this input with its options: it sends
`PUT <CATALOG_API_URI>/products/AB-001` with the JSON body `{"name": "Blue chair", "price": 49.9}` (JSON because the
method is `PUT` and `sends` defaults to `application/json`).
- On success (`200`, `201` or `204`, see `valid_response_code`), the response body goes to the
[StatCounterTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/stat_counter_task.md),
which logs the number of pushed lines at the end of the process.
- On any other status code, the raw response body is sent to the `error_outputs`: the
[LoggerTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/logger_task.md) logs it as a
warning, and `error_strategy: skip` goes on with the next line.

Note that the input replaces whole options: an input `headers` key would replace the `Authorization` header configured
on the task. To send the same header to every request of an API, prefer a
[custom client or a scoped HTTP client](../reference/client.md#examples).
48 changes: 43 additions & 5 deletions docs/index.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
## Prerequisite

CleverAge/ProcessBundle must be [installed](https://github.com/cleverage/process-bundle/blob/main/docs/01-quick_start.md#installation.
CleverAge/ProcessBundle must be [installed](https://github.com/cleverage/process-bundle/blob/main/docs/01-quick_start.md#installation).

## Installation

Expand All @@ -13,13 +13,51 @@ Open a command console, enter your project directory and install it using compos
composer require cleverage/rest-process-bundle
```

Remember to add the following line to config/bundles.php (not required if Symfony Flex is used)
Remember to add the following line to `config/bundles.php` (not required if Symfony Flex is used):

```php
CleverAge\RestProcessBundle\CleverAgeRestProcessBundle::class => ['all' => true],
```

## Reference
## Configuration

The bundle has no configuration of its own. Each API is accessed through a [REST client](reference/client.md): a
service tagged `cleverage.rest.client`, identified by a unique code and holding the base URI of the API.

```yaml
# config/services.yaml
services:
app.cleverage_rest_process.client.apicarto_ign:
class: CleverAge\RestProcessBundle\Client\Client
arguments:
$httpClient: '@http_client'
$logger: '@logger'
$code: 'apicarto_ign'
$uri: 'https://apicarto.ign.fr/api'
tags:
- { name: cleverage.rest.client }
```

The [RequestTask](reference/tasks/request_task.md) then references this client by its code:

```yaml
# Task configuration level
fetch:
service: '@CleverAge\RestProcessBundle\Task\RequestTask'
options:
client: apicarto_ign
url: '/codes-postaux/communes/{codePostal}'
method: GET
url_parameters: { codePostal: '{{ codePostal }}' }
```

## Documentation

- Tasks
- [RequestTask](reference/tasks/request_task.md)
- Cookbooks
- [Fetch API data into DTOs](cookbooks/api_to_dto.md)
- [Push CSV lines to an API](cookbooks/csv_to_api.md)
- Reference
- [REST client](reference/client.md)
- Tasks
- [RequestTask](reference/tasks/request_task.md)
- [CleverAge/ProcessBundle documentation](https://github.com/cleverage/process-bundle/blob/main/docs/index.md)
Loading
Loading