chore(doc) #32 Add missing documentations

CHANGELOG.md

@@ -3,6 +3,7 @@ Latest
 
 ### Changes
 * [#30](https://github.com/cleverage/flysystem-process-bundle/issues/30) Update quality stack: use Rector `withComposerBased()` sets (removed `SYMFONY_64` / `PHPUNIT_100` sets), declare used Symfony packages and PHPUnit range in composer.json, apply quality tools fixes
+* [#32](https://github.com/cleverage/flysystem-process-bundle/issues/32) Add missing documentations: harmonize and complete reference pages for every Task (renamed to snake_case), add SFTP import, SFTP export and remote cleanup cookbooks. Harmonize and fix existing documentation.
 
 v3.0
 ------

docs/cookbooks/remote_cleanup.md

@@ -0,0 +1,66 @@
+Remote cleanup
+==============
+
+This recipe describes how to purge files from a remote storage, e.g. the archives of the [SFTP import](sftp_import.md)
+cookbook, while logging each deleted file.
+
+```yaml
+clever_age_process:
+    configurations:
+        app.remote_cleanup:
+            description: 'Purge the archived CSV files of a given year'
+            help: "bin/console cleverage:process:execute app.remote_cleanup -c year:2025"
+            tasks:
+                list:
+                    service: '@CleverAge\FlysystemProcessBundle\Task\ListContentTask'
+                    options:
+                        filesystem: 'remote.storage.archive'
+                        file_pattern: '/^catalog_{{ year }}.*\.csv$/'
+                    outputs: [only_files]
+
+                only_files:
+                    service: '@CleverAge\ProcessBundle\Task\FilterTask'
+                    options:
+                        match:
+                            type: 'file'
+                    outputs: [get_path]
+
+                get_path:
+                    service: '@CleverAge\ProcessBundle\Task\PropertyGetterTask'
+                    options:
+                        property: 'path'
+                    outputs: [remove]
+
+                remove:
+                    service: '@CleverAge\FlysystemProcessBundle\Task\RemoveFileTask'
+                    options:
+                        filesystem: 'remote.storage.archive'
+```
+
+How it works:
+- The [ListContentTask](../reference/tasks/list_content_task.md) iterates over the items at the root of
+  `remote.storage.archive` whose path matches `file_pattern`; the `{{ year }}` placeholder is replaced by the `year`
+  context value given on the command line. Each item is a `League\Flysystem\StorageAttributes` object.
+- `ListContentTask` also lists directories: the
+  [FilterTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/filter_task.md) keeps only the
+  items whose `type` is `file`.
+- The [PropertyGetterTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/property_getter_task.md)
+  extracts the `path` of the file, relative to the storage root.
+- The [RemoveFileTask](../reference/tasks/remove_file_task.md) deletes the file given as input, and logs it. A deletion
+  failure is logged as a warning and does not stop the process.
+
+When you do not need to act on each file (log, filter on other attributes, etc.), a single
+[RemoveFileTask](../reference/tasks/remove_file_task.md) with a `file_pattern` option does the same (it only
+considers files):
+
+```yaml
+clever_age_process:
+    configurations:
+        app.remote_cleanup_pattern:
+            tasks:
+                remove:
+                    service: '@CleverAge\FlysystemProcessBundle\Task\RemoveFileTask'
+                    options:
+                        filesystem: 'remote.storage.archive'
+                        file_pattern: '/^catalog_{{ year }}.*\.csv$/'
+```

docs/cookbooks/sftp_export.md

@@ -0,0 +1,70 @@
+SFTP export
+===========
+
+This recipe describes a typical export flow to a remote SFTP server: write a CSV file locally, then upload it to the
+remote server and remove the local file.
+
+It uses the `local.storage` storage (whose directory is the `local.storage.dir` parameter) described in the
+[SFTP import](sftp_import.md) cookbook, and a `remote.storage.outgoing` SFTP storage.
+
+```yaml
+clever_age_process:
+    configurations:
+        app.sftp_export:
+            description: 'Export the catalog to the SFTP server'
+            tasks:
+                data:
+                    service: '@CleverAge\ProcessBundle\Task\ConstantIterableOutputTask' # Replace with your own reader
+                    options:
+                        output:
+                            - { sku: 'SKU-001', name: 'Product 1', price: '12.50' }
+                            - { sku: 'SKU-002', name: 'Product 2', price: '8.90' }
+                    outputs: [write]
+
+                write:
+                    service: '@CleverAge\ProcessBundle\Task\File\Csv\CsvWriterTask'
+                    options:
+                        file_path: '%local.storage.dir%/catalog_{date_time}.csv'
+                        headers: [sku, name, price]
+                    outputs: [relative_path]
+
+                relative_path:
+                    service: '@CleverAge\ProcessBundle\Task\TransformerTask'
+                    options:
+                        transformers:
+                            callback:
+                                callback: basename
+                    outputs: [upload]
+
+                upload:
+                    service: '@CleverAge\FlysystemProcessBundle\Task\FileFetchTask'
+                    options:
+                        source_filesystem: 'local.storage'
+                        destination_filesystem: 'remote.storage.outgoing'
+                        remove_source: true
+                    outputs: [log_upload]
+
+                log_upload:
+                    service: '@CleverAge\ProcessBundle\Task\Reporting\LoggerTask'
+                    options:
+                        level: info
+                        message: 'File uploaded'
+```
+
+How it works:
+- The [ConstantIterableOutputTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/constant_iterable_output_task.md)
+  stands for your own reader (database, API...): it outputs the lines one at a time.
+- The [CsvWriterTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/csv_writer_task.md)
+  is blocking: it writes every line in a file located in the `local.storage` directory, then outputs the absolute path
+  of this file once all lines have been written.
+- [FileFetchTask](../reference/tasks/file_fetch_task.md) expects a path relative to the root of its source storage:
+  the [TransformerTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/transformer_task.md)
+  keeps only the file name with the
+  [callback](https://github.com/cleverage/process-bundle/blob/main/docs/reference/transformers/callback_transformer.md)
+  transformer (`basename`).
+- The [FileFetchTask](../reference/tasks/file_fetch_task.md) copies the file from `local.storage` to
+  `remote.storage.outgoing`, then deletes the local file (`remove_source: true`), and outputs the file name to the
+  [LoggerTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/logger_task.md).
+
+The same flow can target any Flysystem adapter (FTP, Amazon S3, Azure...): only the `remote.storage.outgoing` storage
+configuration changes.

docs/cookbooks/sftp_import.md

@@ -0,0 +1,102 @@
+SFTP import
+===========
+
+This recipe describes a typical import flow from a remote SFTP server: download every CSV file of an `incoming`
+directory to a local storage, read each file line by line, then archive the remote file into an `archive` directory
+and delete the local copy.
+
+It uses the following storages:
+
+```yaml
+# config/packages/flysystem.yaml
+parameters:
+    local.storage.dir: '%kernel.project_dir%/var/storage/local'
+
+flysystem:
+    storages:
+        local.storage:
+            adapter: 'local'
+            options:
+                directory: '%local.storage.dir%'
+
+        remote.storage.incoming:
+            adapter: 'sftp'
+            options:
+                host: '%env(string:SFTP_HOST)%'
+                username: '%env(string:SFTP_USERNAME)%'
+                password: '%env(string:SFTP_PASSWORD)%'
+                root: '%env(string:SFTP_ROOT)%/incoming'
+
+        remote.storage.archive:
+            adapter: 'sftp'
+            options:
+                host: '%env(string:SFTP_HOST)%'
+                username: '%env(string:SFTP_USERNAME)%'
+                password: '%env(string:SFTP_PASSWORD)%'
+                root: '%env(string:SFTP_ROOT)%/archive'
+```
+
+```yaml
+clever_age_process:
+    configurations:
+        app.sftp_import:
+            description: 'Import the CSV files dropped on the SFTP server'
+            tasks:
+                fetch:
+                    service: '@CleverAge\FlysystemProcessBundle\Task\FileFetchTask'
+                    options:
+                        source_filesystem: 'remote.storage.incoming'
+                        destination_filesystem: 'local.storage'
+                        file_pattern: '/\.csv$/'
+                    outputs: [read, archive] # "archive" is executed once "read" has read the whole file
+
+                read:
+                    service: '@CleverAge\ProcessBundle\Task\File\Csv\InputCsvReaderTask'
+                    options:
+                        base_path: '%local.storage.dir%'
+                        delimiter: ';'
+                    outputs: [import_line]
+
+                import_line:
+                    service: '@CleverAge\ProcessBundle\Task\Reporting\LoggerTask' # Replace with your own transformation and writer tasks
+                    options:
+                        level: info
+                        message: 'Imported line'
+
+                archive:
+                    service: '@CleverAge\FlysystemProcessBundle\Task\FileFetchTask'
+                    options:
+                        source_filesystem: 'remote.storage.incoming'
+                        destination_filesystem: 'remote.storage.archive'
+                        remove_source: true
+                    outputs: [cleanup_local]
+
+                cleanup_local:
+                    service: '@CleverAge\FlysystemProcessBundle\Task\RemoveFileTask'
+                    options:
+                        filesystem: 'local.storage'
+```
+
+How it works:
+- The first [FileFetchTask](../reference/tasks/file_fetch_task.md) (`fetch`) is iterable: it copies the files matching
+  `file_pattern` one at a time from `remote.storage.incoming` to `local.storage`, and outputs the path of each file
+  (e.g. `products.csv`) relative to the storage root. Each file goes through all the following tasks before the next
+  one is downloaded. As `ignore_missing` defaults to `true`, the process ends without error when there is no file to
+  import; set it to `false` to make the process fail instead.
+- The [InputCsvReaderTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/input_csv_reader_task.md)
+  reads the local copy: its `base_path` option is the directory of `local.storage`, so that the relative path becomes a
+  local path. It iterates over the lines of the file, sent to the
+  [LoggerTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/logger_task.md): replace it
+  with your own tasks (e.g. a
+  [TransformerTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/transformer_task.md)
+  and a writer).
+- Outputs are executed depth-first, in order: once the whole file has been read, the second
+  [FileFetchTask](../reference/tasks/file_fetch_task.md) (`archive`) receives the same path as input and moves the
+  remote file from `remote.storage.incoming` to `remote.storage.archive` (`remove_source: true`). Using two storages
+  with different roots on the same server allows moving files between remote directories.
+- The [RemoveFileTask](../reference/tasks/remove_file_task.md) finally deletes the local copy, whose path is the
+  output of `archive`.
+
+Note that if a line cannot be imported with the default `stop` error strategy, the process stops before the file is
+archived: it will be imported again on the next execution. If your import takes a long time, the SFTP server may
+close the idle connection used by `fetch`: see [SFTP stale connection](../troubleshooting.md).

docs/index.md

@@ -1,6 +1,6 @@
 ## Prerequisite
 
-CleverAge/ProcessBundle must be [installed](https://github.com/cleverage/process-bundle/blob/main/docs/01-quick_start.md#installation.
+CleverAge/ProcessBundle must be [installed](https://github.com/cleverage/process-bundle/blob/main/docs/01-quick_start.md#installation).
 
 ## Installation
 
@@ -19,28 +19,54 @@ Remember to add the following line to config/bundles.php (not required if Symfon
 CleverAge\FlysystemProcessBundle\CleverAgeFlysystemProcessBundle::class => ['all' => true],
 ```
 
-Configure at least one flysytem/storage into `config/packages/flysytem.yaml`
+This bundle relies on [league/flysystem-bundle](https://github.com/thephpleague/flysystem-bundle), installed as a
+dependency: the bundle `League\FlysystemBundle\FlysystemBundle` must be enabled as well. Adapters other than `local`
+require their own package, e.g. `composer require league/flysystem-sftp-v3` for SFTP.
+
+## Configuration
+
+The tasks of this bundle work with the Flysystem **storages** configured in `config/packages/flysystem.yaml`: their
+`filesystem`, `source_filesystem` and `destination_filesystem` options take the name of a storage (the key under
+`flysystem.storages`). Configure at least one storage:
 
 ```yaml
-#config/packages/flysytem.yaml
+# config/packages/flysystem.yaml
 flysystem:
-  storages:
-    storage.source: # This is the identifier of flysytem/storage
-    adapter: 'local'
-    options:
-      directory: '%kernel.project_dir%/var/storage/source'
-```
-
-See https://github.com/thephpleague/flysystem-bundle?tab=readme-ov-file for more sample configuration (sftp, ftp, amazon s3 ...)
+    storages:
+        local.storage: # Name of the storage, used in the task options
+            adapter: 'local'
+            options:
+                directory: '%kernel.project_dir%/var/storage/local'
 
+        remote.storage:
+            adapter: 'sftp'
+            options:
+                host: '%env(string:SFTP_HOST)%'
+                port: 22
+                username: '%env(string:SFTP_USERNAME)%'
+                password: '%env(string:SFTP_PASSWORD)%'
+                root: '%env(string:SFTP_ROOT)%'
+```
 
-## Reference
+Paths used and returned by the tasks are relative to the root of the storage (`directory` or `root` option).
+Listings (`file_pattern` options, [ListContentTask](reference/tasks/list_content_task.md)) are not recursive: to work
+in a sub-directory, configure a dedicated storage whose root is this directory.
 
-- Tasks
-  - [FileFetchTask](reference/tasks/01-FileFetchTask.md)
-  - [ListContentTask](reference/tasks/02-ListContentTask.md)
-  - [RemoveFileTask](reference/tasks/03-RemoveFileTask.md)
+See the [flysystem-bundle documentation](https://github.com/thephpleague/flysystem-bundle?tab=readme-ov-file)
+for the configuration of other adapters (FTP, Amazon S3, Azure, Google Cloud Storage...).
 
-## Troubleshooting
+## Documentation
 
-- [SFTP stale connection on long-running processes](troubleshooting.md) (`Got packet type` / `Connection closed prematurely`)
+- Cookbooks
+    - [SFTP import](cookbooks/sftp_import.md)
+    - [SFTP export](cookbooks/sftp_export.md)
+    - [Remote cleanup](cookbooks/remote_cleanup.md)
+- Reference
+    - Tasks
+        - [FileFetchTask](reference/tasks/file_fetch_task.md)
+        - [ListContentTask](reference/tasks/list_content_task.md)
+        - [RemoveFileTask](reference/tasks/remove_file_task.md)
+- [Troubleshooting](troubleshooting.md)
+    - [SFTP stale connection on long-running processes](troubleshooting.md#sftp-long-running-process-fails-with-got-packet-type--connection-closed-prematurely)
+      (`Got packet type` / `Connection closed prematurely`)
+- [CleverAge/ProcessBundle documentation](https://github.com/cleverage/process-bundle/blob/main/docs/index.md)