diff --git a/CHANGELOG.md b/CHANGELOG.md index b5d1202..4e4c7de 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,7 @@ Latest ### Changes * [#30](https://github.com/cleverage/flysystem-process-bundle/issues/30) Update quality stack: use Rector `withComposerBased()` sets (removed `SYMFONY_64` / `PHPUNIT_100` sets), declare used Symfony packages and PHPUnit range in composer.json, apply quality tools fixes +* [#32](https://github.com/cleverage/flysystem-process-bundle/issues/32) Add missing documentations: harmonize and complete reference pages for every Task (renamed to snake_case), add SFTP import, SFTP export and remote cleanup cookbooks. Harmonize and fix existing documentation. v3.0 ------ diff --git a/docs/cookbooks/remote_cleanup.md b/docs/cookbooks/remote_cleanup.md new file mode 100644 index 0000000..3d79682 --- /dev/null +++ b/docs/cookbooks/remote_cleanup.md @@ -0,0 +1,66 @@ +Remote cleanup +============== + +This recipe describes how to purge files from a remote storage, e.g. the archives of the [SFTP import](sftp_import.md) +cookbook, while logging each deleted file. + +```yaml +clever_age_process: + configurations: + app.remote_cleanup: + description: 'Purge the archived CSV files of a given year' + help: "bin/console cleverage:process:execute app.remote_cleanup -c year:2025" + tasks: + list: + service: '@CleverAge\FlysystemProcessBundle\Task\ListContentTask' + options: + filesystem: 'remote.storage.archive' + file_pattern: '/^catalog_{{ year }}.*\.csv$/' + outputs: [only_files] + + only_files: + service: '@CleverAge\ProcessBundle\Task\FilterTask' + options: + match: + type: 'file' + outputs: [get_path] + + get_path: + service: '@CleverAge\ProcessBundle\Task\PropertyGetterTask' + options: + property: 'path' + outputs: [remove] + + remove: + service: '@CleverAge\FlysystemProcessBundle\Task\RemoveFileTask' + options: + filesystem: 'remote.storage.archive' +``` + +How it works: +- The [ListContentTask](../reference/tasks/list_content_task.md) iterates over the items at the root of + `remote.storage.archive` whose path matches `file_pattern`; the `{{ year }}` placeholder is replaced by the `year` + context value given on the command line. Each item is a `League\Flysystem\StorageAttributes` object. +- `ListContentTask` also lists directories: the + [FilterTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/filter_task.md) keeps only the + items whose `type` is `file`. +- The [PropertyGetterTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/property_getter_task.md) + extracts the `path` of the file, relative to the storage root. +- The [RemoveFileTask](../reference/tasks/remove_file_task.md) deletes the file given as input, and logs it. A deletion + failure is logged as a warning and does not stop the process. + +When you do not need to act on each file (log, filter on other attributes, etc.), a single +[RemoveFileTask](../reference/tasks/remove_file_task.md) with a `file_pattern` option does the same (it only +considers files): + +```yaml +clever_age_process: + configurations: + app.remote_cleanup_pattern: + tasks: + remove: + service: '@CleverAge\FlysystemProcessBundle\Task\RemoveFileTask' + options: + filesystem: 'remote.storage.archive' + file_pattern: '/^catalog_{{ year }}.*\.csv$/' +``` diff --git a/docs/cookbooks/sftp_export.md b/docs/cookbooks/sftp_export.md new file mode 100644 index 0000000..17a03d0 --- /dev/null +++ b/docs/cookbooks/sftp_export.md @@ -0,0 +1,70 @@ +SFTP export +=========== + +This recipe describes a typical export flow to a remote SFTP server: write a CSV file locally, then upload it to the +remote server and remove the local file. + +It uses the `local.storage` storage (whose directory is the `local.storage.dir` parameter) described in the +[SFTP import](sftp_import.md) cookbook, and a `remote.storage.outgoing` SFTP storage. + +```yaml +clever_age_process: + configurations: + app.sftp_export: + description: 'Export the catalog to the SFTP server' + tasks: + data: + service: '@CleverAge\ProcessBundle\Task\ConstantIterableOutputTask' # Replace with your own reader + options: + output: + - { sku: 'SKU-001', name: 'Product 1', price: '12.50' } + - { sku: 'SKU-002', name: 'Product 2', price: '8.90' } + outputs: [write] + + write: + service: '@CleverAge\ProcessBundle\Task\File\Csv\CsvWriterTask' + options: + file_path: '%local.storage.dir%/catalog_{date_time}.csv' + headers: [sku, name, price] + outputs: [relative_path] + + relative_path: + service: '@CleverAge\ProcessBundle\Task\TransformerTask' + options: + transformers: + callback: + callback: basename + outputs: [upload] + + upload: + service: '@CleverAge\FlysystemProcessBundle\Task\FileFetchTask' + options: + source_filesystem: 'local.storage' + destination_filesystem: 'remote.storage.outgoing' + remove_source: true + outputs: [log_upload] + + log_upload: + service: '@CleverAge\ProcessBundle\Task\Reporting\LoggerTask' + options: + level: info + message: 'File uploaded' +``` + +How it works: +- The [ConstantIterableOutputTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/constant_iterable_output_task.md) + stands for your own reader (database, API...): it outputs the lines one at a time. +- The [CsvWriterTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/csv_writer_task.md) + is blocking: it writes every line in a file located in the `local.storage` directory, then outputs the absolute path + of this file once all lines have been written. +- [FileFetchTask](../reference/tasks/file_fetch_task.md) expects a path relative to the root of its source storage: + the [TransformerTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/transformer_task.md) + keeps only the file name with the + [callback](https://github.com/cleverage/process-bundle/blob/main/docs/reference/transformers/callback_transformer.md) + transformer (`basename`). +- The [FileFetchTask](../reference/tasks/file_fetch_task.md) copies the file from `local.storage` to + `remote.storage.outgoing`, then deletes the local file (`remove_source: true`), and outputs the file name to the + [LoggerTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/logger_task.md). + +The same flow can target any Flysystem adapter (FTP, Amazon S3, Azure...): only the `remote.storage.outgoing` storage +configuration changes. diff --git a/docs/cookbooks/sftp_import.md b/docs/cookbooks/sftp_import.md new file mode 100644 index 0000000..8c508a5 --- /dev/null +++ b/docs/cookbooks/sftp_import.md @@ -0,0 +1,102 @@ +SFTP import +=========== + +This recipe describes a typical import flow from a remote SFTP server: download every CSV file of an `incoming` +directory to a local storage, read each file line by line, then archive the remote file into an `archive` directory +and delete the local copy. + +It uses the following storages: + +```yaml +# config/packages/flysystem.yaml +parameters: + local.storage.dir: '%kernel.project_dir%/var/storage/local' + +flysystem: + storages: + local.storage: + adapter: 'local' + options: + directory: '%local.storage.dir%' + + remote.storage.incoming: + adapter: 'sftp' + options: + host: '%env(string:SFTP_HOST)%' + username: '%env(string:SFTP_USERNAME)%' + password: '%env(string:SFTP_PASSWORD)%' + root: '%env(string:SFTP_ROOT)%/incoming' + + remote.storage.archive: + adapter: 'sftp' + options: + host: '%env(string:SFTP_HOST)%' + username: '%env(string:SFTP_USERNAME)%' + password: '%env(string:SFTP_PASSWORD)%' + root: '%env(string:SFTP_ROOT)%/archive' +``` + +```yaml +clever_age_process: + configurations: + app.sftp_import: + description: 'Import the CSV files dropped on the SFTP server' + tasks: + fetch: + service: '@CleverAge\FlysystemProcessBundle\Task\FileFetchTask' + options: + source_filesystem: 'remote.storage.incoming' + destination_filesystem: 'local.storage' + file_pattern: '/\.csv$/' + outputs: [read, archive] # "archive" is executed once "read" has read the whole file + + read: + service: '@CleverAge\ProcessBundle\Task\File\Csv\InputCsvReaderTask' + options: + base_path: '%local.storage.dir%' + delimiter: ';' + outputs: [import_line] + + import_line: + service: '@CleverAge\ProcessBundle\Task\Reporting\LoggerTask' # Replace with your own transformation and writer tasks + options: + level: info + message: 'Imported line' + + archive: + service: '@CleverAge\FlysystemProcessBundle\Task\FileFetchTask' + options: + source_filesystem: 'remote.storage.incoming' + destination_filesystem: 'remote.storage.archive' + remove_source: true + outputs: [cleanup_local] + + cleanup_local: + service: '@CleverAge\FlysystemProcessBundle\Task\RemoveFileTask' + options: + filesystem: 'local.storage' +``` + +How it works: +- The first [FileFetchTask](../reference/tasks/file_fetch_task.md) (`fetch`) is iterable: it copies the files matching + `file_pattern` one at a time from `remote.storage.incoming` to `local.storage`, and outputs the path of each file + (e.g. `products.csv`) relative to the storage root. Each file goes through all the following tasks before the next + one is downloaded. As `ignore_missing` defaults to `true`, the process ends without error when there is no file to + import; set it to `false` to make the process fail instead. +- The [InputCsvReaderTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/input_csv_reader_task.md) + reads the local copy: its `base_path` option is the directory of `local.storage`, so that the relative path becomes a + local path. It iterates over the lines of the file, sent to the + [LoggerTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/logger_task.md): replace it + with your own tasks (e.g. a + [TransformerTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/transformer_task.md) + and a writer). +- Outputs are executed depth-first, in order: once the whole file has been read, the second + [FileFetchTask](../reference/tasks/file_fetch_task.md) (`archive`) receives the same path as input and moves the + remote file from `remote.storage.incoming` to `remote.storage.archive` (`remove_source: true`). Using two storages + with different roots on the same server allows moving files between remote directories. +- The [RemoveFileTask](../reference/tasks/remove_file_task.md) finally deletes the local copy, whose path is the + output of `archive`. + +Note that if a line cannot be imported with the default `stop` error strategy, the process stops before the file is +archived: it will be imported again on the next execution. If your import takes a long time, the SFTP server may +close the idle connection used by `fetch`: see [SFTP stale connection](../troubleshooting.md). diff --git a/docs/index.md b/docs/index.md index 0c25277..180eb8b 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,6 +1,6 @@ ## Prerequisite -CleverAge/ProcessBundle must be [installed](https://github.com/cleverage/process-bundle/blob/main/docs/01-quick_start.md#installation. +CleverAge/ProcessBundle must be [installed](https://github.com/cleverage/process-bundle/blob/main/docs/01-quick_start.md#installation). ## Installation @@ -19,28 +19,54 @@ Remember to add the following line to config/bundles.php (not required if Symfon CleverAge\FlysystemProcessBundle\CleverAgeFlysystemProcessBundle::class => ['all' => true], ``` -Configure at least one flysytem/storage into `config/packages/flysytem.yaml` +This bundle relies on [league/flysystem-bundle](https://github.com/thephpleague/flysystem-bundle), installed as a +dependency: the bundle `League\FlysystemBundle\FlysystemBundle` must be enabled as well. Adapters other than `local` +require their own package, e.g. `composer require league/flysystem-sftp-v3` for SFTP. + +## Configuration + +The tasks of this bundle work with the Flysystem **storages** configured in `config/packages/flysystem.yaml`: their +`filesystem`, `source_filesystem` and `destination_filesystem` options take the name of a storage (the key under +`flysystem.storages`). Configure at least one storage: ```yaml -#config/packages/flysytem.yaml +# config/packages/flysystem.yaml flysystem: - storages: - storage.source: # This is the identifier of flysytem/storage - adapter: 'local' - options: - directory: '%kernel.project_dir%/var/storage/source' -``` - -See https://github.com/thephpleague/flysystem-bundle?tab=readme-ov-file for more sample configuration (sftp, ftp, amazon s3 ...) + storages: + local.storage: # Name of the storage, used in the task options + adapter: 'local' + options: + directory: '%kernel.project_dir%/var/storage/local' + remote.storage: + adapter: 'sftp' + options: + host: '%env(string:SFTP_HOST)%' + port: 22 + username: '%env(string:SFTP_USERNAME)%' + password: '%env(string:SFTP_PASSWORD)%' + root: '%env(string:SFTP_ROOT)%' +``` -## Reference +Paths used and returned by the tasks are relative to the root of the storage (`directory` or `root` option). +Listings (`file_pattern` options, [ListContentTask](reference/tasks/list_content_task.md)) are not recursive: to work +in a sub-directory, configure a dedicated storage whose root is this directory. -- Tasks - - [FileFetchTask](reference/tasks/01-FileFetchTask.md) - - [ListContentTask](reference/tasks/02-ListContentTask.md) - - [RemoveFileTask](reference/tasks/03-RemoveFileTask.md) +See the [flysystem-bundle documentation](https://github.com/thephpleague/flysystem-bundle?tab=readme-ov-file) +for the configuration of other adapters (FTP, Amazon S3, Azure, Google Cloud Storage...). -## Troubleshooting +## Documentation -- [SFTP stale connection on long-running processes](troubleshooting.md) (`Got packet type` / `Connection closed prematurely`) +- Cookbooks + - [SFTP import](cookbooks/sftp_import.md) + - [SFTP export](cookbooks/sftp_export.md) + - [Remote cleanup](cookbooks/remote_cleanup.md) +- Reference + - Tasks + - [FileFetchTask](reference/tasks/file_fetch_task.md) + - [ListContentTask](reference/tasks/list_content_task.md) + - [RemoveFileTask](reference/tasks/remove_file_task.md) +- [Troubleshooting](troubleshooting.md) + - [SFTP stale connection on long-running processes](troubleshooting.md#sftp-long-running-process-fails-with-got-packet-type--connection-closed-prematurely) + (`Got packet type` / `Connection closed prematurely`) +- [CleverAge/ProcessBundle documentation](https://github.com/cleverage/process-bundle/blob/main/docs/index.md) diff --git a/docs/reference/tasks/01-FileFetchTask.md b/docs/reference/tasks/01-FileFetchTask.md deleted file mode 100644 index 4ad65e3..0000000 --- a/docs/reference/tasks/01-FileFetchTask.md +++ /dev/null @@ -1,76 +0,0 @@ -FileFetchTask -======== - -Perform copy between 2 flysystems storage - -Task reference --------------- - -* **Service**: [`CleverAge\FlysystemProcessBundle\Task\FileFetchTask`](../src/Task/FileFetchTask.php) - -Accepted inputs ---------------- - -The filename or filenames to copy from. - -If the option `file_pattern` is not set the input is used as strict filename(s) to match. - -If input is set but not corresponding as any file into `source_filesystem` task failed with UnableToReadFile exception. - -If FileFetchTask is the first task of you process and you want to use input, don't forgive to set the `entry_point` task name at process level - -Possible outputs ----------------- - -Filename of copied file. - -Options -------- - -| Code | Type | Required | Default | Description | -|--------------------------|:--------:|:--------:|:-------:|----------------------------------------------------------------------------------------------------------------------------------------------| -| `source_filesystem` | `string` | **X** | | The source flysystem/storage.
See config/packages/flysystem.yaml to see configured flysystem/storages. | -| `destination_filesystem` | `string` | **X** | | The source flysystem/storage.
See config/packages/flysystem.yaml to see configured flysystem/storages. | -| `file_pattern` | `string` | | null | The file_pattern used in preg_match to match into `source_filesystem` list of files. If not set try to use input as strict filename to match | -| `remove_source` | `bool` | | false | If true delete source file after copy | -| `ignore_missing` | `bool` | | true | Ignore property accessor errors for this source | - - -Examples --------- - -* Simple fetch task configuration - - See config/packages/flysystem.yaml to see configured flysystems/storages. - - copy all .csv files from 'storage.source' to 'storage.destination' - - remove .csv from 'storage.source' after copy - - output will be filename of copied files - - throw Exception when file(s) not found -```yaml -# Task configuration level -code: - service: '@CleverAge\FlysystemProcessBundle\Task\FileFetchTask' - options: - source_filesystem: 'storage.source' - destination_filesystem: 'storage.destination' - file_pattern: '/.csv$/' - remove_source: true - ignore_missing: false -``` - -* Simple fetch process configuration to cipy a specific file from --input option via
```bin/console cleverage:process:execute my_custom_process --input=foobar.csv -vv``` - - See config/packages/flysystem.yaml to see configured flysystems/storages. - - copy input file from 'storage.source' to 'storage.destination' - - remove .csv from 'storage.source' after copy - - output will be filename of copied file -```yaml -# Full process configuration to use input as filename with the following call -my_custom_process: - entry_point: copy_from_input - tasks: - copy_from_input: - service: '@CleverAge\FlysystemProcessBundle\Task\FileFetchTask' - options: - source_filesystem: 'storage.source' - destination_filesystem: 'storage.destination' - remove_source: true -``` diff --git a/docs/reference/tasks/02-ListContentTask.md b/docs/reference/tasks/02-ListContentTask.md deleted file mode 100644 index 036c0c3..0000000 --- a/docs/reference/tasks/02-ListContentTask.md +++ /dev/null @@ -1,42 +0,0 @@ -ListContentTask -======== - -List files of a flysystem storage - -Task reference --------------- - -* **Service**: [`CleverAge\FlysystemProcessBundle\Task\ListContentTask`](../src/Task/ListContentTask.php) - -Accepted inputs ---------------- - -Input is ignored - -Possible outputs ----------------- - -League\Flysystem\StorageAttributes - -Options -------- - -| Code | Type | Required | Default | Description | -|----------------|:----------:|:---------:|:---------:|------------------------------------------------------------------------------------------------------------| -| `filesystem` | `string` | **X** | | The source flysystem/storage.
See config/packages/flysystem.yaml to see configured flysystem/storages. | -| `file_pattern` | `string` | | | The file_parttern used in preg_match to match into `filesystem` | - -Examples --------- -* Simple list task configuration from a filesystem - - see config/packages/flysystem.yaml to see configured flysystems/storages. - - list all .csv files from 'storage.source' - - output will be League\Flysystem\StorageAttributes representation of copied files -```yaml -# Task configuration level -code: - service: '@CleverAge\FlysystemProcessBundle\Task\ListContentTask' - options: - filesystem: 'storage.source' - file_pattern: '/.csv$/' -``` diff --git a/docs/reference/tasks/03-RemoveFileTask.md b/docs/reference/tasks/03-RemoveFileTask.md deleted file mode 100644 index 8e1b6c4..0000000 --- a/docs/reference/tasks/03-RemoveFileTask.md +++ /dev/null @@ -1,64 +0,0 @@ -RemoveFileTask -======== - -Remove a file from a flysystem storage - -Task reference --------------- - -* **Service**: [`CleverAge\FlysystemProcessBundle\Task\RemoveFileTask`](../src/Task/RemoveFileTask.php) - -Accepted inputs ---------------- - -The filename of the file to remove on `filesystem`. - -If the option `file_pattern` is not set the input is used as strict filename(s) to match. - -When filename is deleted add a info log. - -If filename not found or cannot be deleted on `filesystem` add a warning log. - -Possible outputs ----------------- - -None - -Options -------- - -| Code | Type | Required | Default | Description | -|----------------|:----------:|:--------:|:---------:|----------------------------------------------------------------------------------------------------------------------------------------------| -| `filesystem` | `string` | **X** | | The source flysystem/storage.
See config/packages/flysystem.yaml to see configured flysystem/storages. | -| `file_pattern` | `string` | | null | The file_pattern used in preg_match to match into `source_filesystem` list of files. If not set try to use input as strict filename to match | - - -Examples --------- -* Simple process to remove a file on 'storage.source' via
```bin/console cleverage:process:execute input_process --input=foobar.csv -vv``` - - see config/packages/flysystem.yaml to see configured flysystems/storages. - - remove file with name passed as input -```yaml -# -input_process: - entry_point: remove_from_input - tasks: - remove_from_input: - service: '@CleverAge\FlysystemProcessBundle\Task\RemoveFileTask' - options: - filesystem: 'storage.source' -``` - -* Simple process to remove a file on 'storage.source' via
```bin/console cleverage:process:execute pattern_process -vv``` - - see config/packages/flysystem.yaml to see configured flysystems/storages. - - remove files filtered by file_pattern -```yaml -# -pattern_process: - tasks: - remove_from_file_pattern: - service: '@CleverAge\FlysystemProcessBundle\Task\RemoveFileTask' - options: - filesystem: 'storage.source' - file_pattern: '/.csv$/' -``` diff --git a/docs/reference/tasks/_template.md b/docs/reference/tasks/_template.md index ed1d4a5..919390a 100644 --- a/docs/reference/tasks/_template.md +++ b/docs/reference/tasks/_template.md @@ -1,44 +1,47 @@ TaskName ======== -_Describe main goal an use cases of the task_ +_Describe the main goal and use cases of the task._ Task reference -------------- -* **Service**: `ClassName` +* **Service**: `Fully\Qualified\ClassName` +* **Iterable task** _(only if it implements `IterableTaskInterface`)_ +* **Blocking task** _(only if it implements `BlockingTaskInterface`)_ +* **Flushable task** _(only if it implements `FlushableTaskInterface`)_ Accepted inputs --------------- -_Description of allowed types_ +_Description of allowed types, or "Input is ignored"._ Possible outputs ---------------- -_Description of possible types_ +_Description of possible types._ Options ------- -| Code | Type | Required | Default | Description | -| ---- | ---- | :------: | ------- | ----------- | -| `code` | `type` | **X** _or nothing_ | `default value` _if available_ | _description_ | +| Code | Type | Required | Default | Description | +|--------|--------|:--------:|-----------------|---------------| +| `code` | `type` | **X** | `default value` | _description_ | + +_If the task has no option, replace the table with "This task has no option."._ Examples -------- -_YAML samples and explanations_ - * Example 1 - details - - details - + ```yaml # Task configuration level code: - service: '@service_ref' - options: - a: 1 - b: 2 + service: '@Fully\Qualified\ClassName' + options: + a: 1 + b: 2 + outputs: [next_task] ``` diff --git a/docs/reference/tasks/file_fetch_task.md b/docs/reference/tasks/file_fetch_task.md new file mode 100644 index 0000000..504d45c --- /dev/null +++ b/docs/reference/tasks/file_fetch_task.md @@ -0,0 +1,115 @@ +FileFetchTask +============= + +Copies (or moves) files from one Flysystem storage to another, and iterates over the copied files, outputting each +file path. Files are either selected with a regular expression on the source storage, or given as input. + +Typical use cases: download files from a remote SFTP/FTP/S3 storage to a local storage before reading them, upload +generated files to a remote storage, or archive processed files into another storage. + +Task reference +-------------- + +* **Service**: `CleverAge\FlysystemProcessBundle\Task\FileFetchTask` +* **Iterable task** + +Accepted inputs +--------------- + +* When `file_pattern` is set, the input is ignored. +* Otherwise, `string|array`: path, or list of paths, of the file(s) to copy, relative to the root of the + `source_filesystem` storage. An empty input throws an `\UnexpectedValueException` + (`No pattern neither input provided for the Task`). + +If the task is the first task of the process and takes its files from the input, set the process `entry_point` to +this task (e.g. `bin/console cleverage:process:execute my_process --input=foobar.csv`). + +Possible outputs +---------------- + +`string`: for each copied file, its path relative to the storage root (the same path is used in the +`source_filesystem` and in the `destination_filesystem`). + +When no file matches `file_pattern` (and `ignore_missing` is `true`), or when every received file has already been +copied during this process execution, no output is produced and the task is skipped. + +Options +------- + +| Code | Type | Required | Default | Description | +|--------------------------|----------------|:--------:|---------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `source_filesystem` | `string` | **X** | | Name of the Flysystem storage to read files from, as configured under `flysystem.storages` (see [configuration](../../index.md#configuration)) | +| `destination_filesystem` | `string` | **X** | | Name of the Flysystem storage to write files to, as configured under `flysystem.storages` | +| `file_pattern` | `string\|null` | | `null` | Regular expression (see [preg_match](https://www.php.net/manual/en/function.preg-match.php)) tested on the path of each file at the root of `source_filesystem`. If `null` (or empty), the file path(s) are taken from the input | +| `remove_source` | `bool` | | `false` | Delete the file from `source_filesystem` after the copy (move instead of copy) | +| `ignore_missing` | `bool` | | `true` | Only used with `file_pattern`: if `false`, throw an `\UnexpectedValueException` (`File(s) not found in source filesystem`) when no file matches the pattern. If `true`, the task is skipped | + +Examples +-------- + +* Move all CSV files from a remote storage to a local one + - each file is copied then deleted from `remote.storage` + - the output is the path of each copied file, e.g. `products.csv` + - an exception is thrown if no CSV file is found + +```yaml +# Task configuration level +fetch: + service: '@CleverAge\FlysystemProcessBundle\Task\FileFetchTask' + options: + source_filesystem: 'remote.storage' + destination_filesystem: 'local.storage' + file_pattern: '/\.csv$/' + remove_source: true + ignore_missing: false + outputs: [read] +``` + +* Copy a file given as input, with contextualized storages + - `bin/console cleverage:process:execute my_process --input=foobar.csv -c source:remote.storage -c destination:local.storage` + +```yaml +# Process configuration level +my_process: + entry_point: copy_from_input + tasks: + copy_from_input: + service: '@CleverAge\FlysystemProcessBundle\Task\FileFetchTask' + options: + source_filesystem: '{{ source }}' + destination_filesystem: '{{ destination }}' +``` + +* Archive a processed file into another storage + - the input is the path of a file previously fetched from `remote.storage.incoming` + +```yaml +# Task configuration level +archive: + service: '@CleverAge\FlysystemProcessBundle\Task\FileFetchTask' + options: + source_filesystem: 'remote.storage.incoming' + destination_filesystem: 'remote.storage.archive' + remove_source: true +``` + +Notes +----- + +* Both storages are resolved when the task is initialized, at the start of the process: an unknown storage name makes + the process fail before any task is executed. +* `file_pattern` is only tested on the files located at the root of `source_filesystem` (the listing is not + recursive and directories are ignored). The pattern is tested on the path relative to the storage root: to target + a sub-directory, configure a dedicated storage whose root is this directory. +* The source storage is listed again before each iteration: files added to the source during the iteration are also + copied. With an SFTP storage and long-running downstream tasks, see + [SFTP stale connection](../../troubleshooting.md). +* A given file is copied only once per process execution: already copied paths are remembered, so receiving the same + path again (or a new input while using `file_pattern`) skips the task. +* The file is written to `destination_filesystem` with the same path, overwriting any existing file. A file given as + input that does not exist in `source_filesystem` throws a `League\Flysystem\UnableToReadFile` exception, whatever + the value of `ignore_missing`. +* To read a copied file with a core task (e.g. + [InputCsvReaderTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/input_csv_reader_task.md)), + prefix the output with the directory of the local destination storage (e.g. with the `base_path` option). +* See the [SFTP import](../../cookbooks/sftp_import.md) and [SFTP export](../../cookbooks/sftp_export.md) cookbooks. diff --git a/docs/reference/tasks/list_content_task.md b/docs/reference/tasks/list_content_task.md new file mode 100644 index 0000000..1db9737 --- /dev/null +++ b/docs/reference/tasks/list_content_task.md @@ -0,0 +1,86 @@ +ListContentTask +=============== + +Lists the content (files and directories) at the root of a Flysystem storage and iterates over it, outputting each +item as a `League\Flysystem\StorageAttributes` object. Items can be filtered with a regular expression on their path. + +Task reference +-------------- + +* **Service**: `CleverAge\FlysystemProcessBundle\Task\ListContentTask` +* **Iterable task** + +Accepted inputs +--------------- + +Input is ignored + +Possible outputs +---------------- + +`League\Flysystem\StorageAttributes`: for each listed item, a `League\Flysystem\FileAttributes` (file) or a +`League\Flysystem\DirectoryAttributes` (directory) object. Useful properties are `path` (relative to the storage +root), `type` (`file` or `dir`), `lastModified`, and for files `fileSize` and `mimeType` +(see [Flysystem directory listings](https://flysystem.thephpleague.com/docs/usage/directory-listings/)). + +Use [PropertyGetterTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/property_getter_task.md) +with `property: path` to get the path, as expected by [FileFetchTask](file_fetch_task.md) or +[RemoveFileTask](remove_file_task.md). + +When the storage is empty, or when no item matches `file_pattern`, no output is produced and the task is skipped. + +Options +------- + +| Code | Type | Required | Default | Description | +|----------------|----------------|:--------:|---------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `filesystem` | `string` | **X** | | Name of the Flysystem storage to list, as configured under `flysystem.storages` (see [configuration](../../index.md#configuration)) | +| `file_pattern` | `string\|null` | | `null` | Regular expression (see [preg_match](https://www.php.net/manual/en/function.preg-match.php)) tested on the path of each item. If `null`, every item (files and directories) is output | + +Examples +-------- + +* List CSV files of a storage and output their path + +```yaml +# Task configuration level +list: + service: '@CleverAge\FlysystemProcessBundle\Task\ListContentTask' + options: + filesystem: 'remote.storage' + file_pattern: '/\.csv$/' + outputs: [get_path] +get_path: + service: '@CleverAge\ProcessBundle\Task\PropertyGetterTask' + options: + property: 'path' + outputs: [log] +``` + +* Keep only files (exclude directories) with a + [FilterTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/filter_task.md) + +```yaml +# Task configuration level +list: + service: '@CleverAge\FlysystemProcessBundle\Task\ListContentTask' + options: + filesystem: 'remote.storage' + outputs: [only_files] +only_files: + service: '@CleverAge\ProcessBundle\Task\FilterTask' + options: + match: + type: 'file' + outputs: [get_path] +``` + +Notes +----- + +* The listing is not recursive: only the items located at the root of the storage are listed. To list a + sub-directory, configure a dedicated storage whose root is this directory. +* Unlike [FileFetchTask](file_fetch_task.md) and [RemoveFileTask](remove_file_task.md), directories are not excluded: + make `file_pattern` specific enough (e.g. on the extension) or filter on `type`. +* The storage is listed once, when the iteration starts; the listing is done again on the next execution of the task, + once the previous iteration is over. diff --git a/docs/reference/tasks/remove_file_task.md b/docs/reference/tasks/remove_file_task.md new file mode 100644 index 0000000..c57aedb --- /dev/null +++ b/docs/reference/tasks/remove_file_task.md @@ -0,0 +1,72 @@ +RemoveFileTask +============== + +Deletes files from a Flysystem storage: either every file matching a regular expression, or the file given as input. + +Task reference +-------------- + +* **Service**: `CleverAge\FlysystemProcessBundle\Task\RemoveFileTask` + +Accepted inputs +--------------- + +* When `file_pattern` is set, the input is ignored (but the deletion is run again each time the task is executed). +* Otherwise, `string`: path of the file to delete, relative to the root of the `filesystem` storage. An empty input + throws an `\UnexpectedValueException` (`No pattern neither input provided for the Task`). A list of paths is not + supported: iterate over it first (e.g. with + [InputIteratorTask](https://github.com/cleverage/process-bundle/blob/main/docs/reference/tasks/input_iterator_task.md)). + A `StorageAttributes` output of [ListContentTask](list_content_task.md) must be converted to its `path` first. + +Possible outputs +---------------- + +No output is set. + +Options +------- + +| Code | Type | Required | Default | Description | +|----------------|----------------|:--------:|---------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `filesystem` | `string` | **X** | | Name of the Flysystem storage to delete files from, as configured under `flysystem.storages` (see [configuration](../../index.md#configuration)) | +| `file_pattern` | `string\|null` | | `null` | Regular expression (see [preg_match](https://www.php.net/manual/en/function.preg-match.php)) tested on the path of each file at the root of `filesystem`: every matching file is deleted. If `null` (or empty), the file path is taken from the input | + +Examples +-------- + +* Delete the file given as input + - `bin/console cleverage:process:execute my_process --input=foobar.csv` + +```yaml +# Process configuration level +my_process: + entry_point: remove_from_input + tasks: + remove_from_input: + service: '@CleverAge\FlysystemProcessBundle\Task\RemoveFileTask' + options: + filesystem: 'remote.storage' +``` + +* Delete every CSV file of a storage + +```yaml +# Task configuration level +purge: + service: '@CleverAge\FlysystemProcessBundle\Task\RemoveFileTask' + options: + filesystem: 'remote.storage' + file_pattern: '/\.csv$/' +``` + +Notes +----- + +* `file_pattern` is only tested on the files located at the root of the storage (the listing is not recursive and + directories are ignored). +* Deletion errors do not stop the process: each deleted file is logged with the `info` level (`Deleted input file`), + and a deletion failure (`League\Flysystem\FilesystemException`) is logged with the `warning` level, with the file + path in the log context. Deleting a file that does not exist is not an error for most adapters (e.g. `local`, + `sftp`): it is logged as deleted. +* The storage is resolved on each execution: an unknown storage name makes the task fail when it is executed. +* See the [Remote cleanup](../../cookbooks/remote_cleanup.md) cookbook. diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index b60c821..4f8ad85 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -4,8 +4,8 @@ ### Symptom -A long-running process that uses an SFTP storage (e.g. `FileFetchTask`, `ListContentTask`, -`RemoveFileTask`) fails on a **later** task with one of: +A long-running process that uses an SFTP storage (e.g. [FileFetchTask](reference/tasks/file_fetch_task.md), +[ListContentTask](reference/tasks/list_content_task.md), [RemoveFileTask](reference/tasks/remove_file_task.md)) fails on a **later** task with one of: ``` Expected NET_SFTP_HANDLE or NET_SFTP_STATUS. Got packet type: