fix(task) #37 #38 #39 #40 #41 FileFetchTask and RemoveFileTask fixes, #33 add missing tests

[njoubert-cleverage]

Description

Fixes #33.
Fixes #37.
Fixes #38.
Fixes #39.
Fixes #40.
Fixes #41.

Grouped in one PR because the fixes change the same tasks, tests, reference pages and the CHANGELOG Latest section.

FileFetchTask

• FileFetchTask lists the source storage again before each file #37: the files are listed once per input, when its iteration starts (listContents() ran before each file: N + 1 listings for N files, which also triggered the stale SFTP connection of SFTP connection timeout causing archiving step failure #24).
• FileFetchTask skips the files already handled during the process execution #38: the list of files is reset when the iteration of an input ends (next()), so each input is processed: with file_pattern, only the first input copied the files, and a path received twice was skipped.
• FileFetchTask ignore_missing has no effect on the files given as input #39: the files given as input are checked with fileExists(): skipped with ignore_missing: true (default), \UnexpectedValueException (File <path> not found in source filesystem) otherwise. A missing file threw UnableToReadFile, whatever ignore_missing.
• FileFetchTask and RemoveFileTask handle a file named 0 as no file #40: strict checks instead of falsy tests: a file named 0 is copied (it threw No pattern neither input provided for the Task, or ended the iteration with file_pattern).

RemoveFileTask

• RemoveFileTask logs a missing file as deleted and does not accept a list of paths #41: a file given as input that does not exist (or is a directory) is logged as Input file not found (warning) instead of Deleted input file (the local / sftp adapters do not fail on a missing file); a list of paths is accepted (as for FileFetchTask, it triggered a TypeError); the failure message is fixed (Failed to delete input file).
• FileFetchTask and RemoveFileTask handle a file named 0 as no file #40: a file named 0 is deleted.

Tests (#33)

• StorageTestCase (new): the tasks run on real local storages in a temporary directory (with a listing counter), iterating as the process manager does; symfony/filesystem added to require-dev for the cleanup.
• FileFetchTaskStorageTest, ListContentTaskTest, RemoveFileTaskTest (new): pattern / input modes, one listing per input, several inputs, paths received twice, file 0, remove_source, overwriting, ignore_missing both ways, errors, options, logs.
• CleverAgeFlysystemProcessExtensionTest, CleverAgeFlysystemProcessBundleTest (new): non-shared services with their storage locator and public aliases, bundle path.
• FileFetchTaskTest: the source mock answers fileExists().

Coverage: 115/115 lines (35 tests). Without the fix, 9 of the new tests fail.

Documentation: file_fetch_task.md (ignore_missing, outputs, notes on listing / inputs / missing files), remove_file_task.md (inputs, logs), troubleshooting.md.

Checked with process-bundle-demo (see the demo PR):

Process	Before	After
demo.flysystem.fetch_multiple_inputs (a.txt, b.txt, a.txt, 0)	a.txt, b.txt, then No pattern neither input provided for the Task	a.txt, b.txt, a.txt, 0
demo.flysystem.fetch_pattern_multiple_inputs (2 inputs)	a.txt, b.txt	a.txt, b.txt, a.txt, b.txt
demo.flysystem.fetch_missing_input (missing.txt, a.txt)	UnableToReadFile	a.txt
demo.flysystem.remove_missing_file (a.txt, missing.txt, dir)	missing.txt logged as deleted, Failed to deleted input file for dir	Input file not found for missing.txt and dir
demo.flysystem.remove_files_input (['a.txt', 'b.txt'])	TypeError	both deleted
demo.flysystem.fetch_write_failure	fails, source kept	unchanged

Not changed: ListContentTask still lists the directories (audit point, documented and intended: it outputs StorageAttributes with their type).

Requirements

• Documentation updates
  • Reference
  • Changelog
• Unit tests

Breaking changes

• FileFetchTask lists the source storage again before each file #37: files added to the source storage while FileFetchTask iterates are copied by the next execution of the task, not during the current iteration.
• FileFetchTask skips the files already handled during the process execution #38: each input is processed: a path received twice is copied twice, each input of a file_pattern task copies the matching files.
• FileFetchTask ignore_missing has no effect on the files given as input #39: with the default ignore_missing: true, a missing file given as input is skipped (the process failed).
• RemoveFileTask logs a missing file as deleted and does not accept a list of paths #41: a missing file given to RemoveFileTask is logged with the warning level.
• FileFetchTask::$matchingFiles is now ?list<string> and findMatchingFiles() returns the files (protected members).

🤖 Generated with Claude Code