Repository navigation
Conversation
Run the hasher in an executor thread. This should make Bob more responsive when hashing a large workspace.
The hash sum of directory entries holds the file modes. Use a defined endianess (little endian) to make the result system independent. Previously the hash sum of an identical directory tree would have differed between little and big endian systems.
The calls to hashDirectory() and hashPath() should calculate the same size.
The "index" file for a hashed directory tree is just an optimization. If it cannot be read, it's better to just ignore the cache instead of failing the build. Likewise, if writing the cache fails, keep going on. Nothing bad happens. It's just that the next hashing of the same tree will be slower.
File time stamps have a limited granularity. A file that is modified again in the same file system clock tick after it was hashed would keep its stat data. Such "racily clean" entries must not be trusted. Any file whose change-time is not older than the current file system time is thus not recorded in the index.
The hash index implementation kept entries of deleted files. These may accumulate over time. __match() now tracks whether the current entry has already matched a file. If it has to skip an entry that never matched, it marks the index as changed and starts the new index right away, while the copy position still stops before the stale entry. Related to that there was a potential bug in the handling of the last index entry. When a new file sorted after the last index entry, __readEntry() hit end-of-file without moving __inPosOld. The copied prefix then stopped one entry short, so the last matching entry was silently left out. Fortunately that only cost a cache miss on the next run, keeping the result still correct.
On fast NVMe disks, hashing large directory trees is limited by the CPU. Doing that in parallel will greatly reduce the time that is needed for that. The rust python will spawn as many threads as CPUs are available and do the hashing in parallel. On the other hand, re-scanning a clean tree is bound by the file system meta data throughput. A parallel approach saves significant time here as well. This discovery phase is done in a thread pool too. This cuts the scan of a clean Linux source tree (91k files, 2.6 GB) with hot page cache and index from >250ms to ~60ms. If the tree is not in the page cache, hashing without index is 6x faster (17.8s -> 3.0s). See BobBuildTool#281.
Add additional common tests for both the Python and Rust implementation. Also add tests that compare the result and index file of both implementations for a given tree. They should behave the same.
When using Bob from sources, changes to the rust directory hasher should lead to a rebuild of the Python plugin. Do that before executing any commands or tests.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #723 +/- ##
==========================================
- Coverage 89.57% 89.52% -0.06%
==========================================
Files 50 50
Lines 16993 17081 +88
==========================================
+ Hits 15222 15292 +70
- Misses 1771 1789 +18 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds a cpython plugin that implements the directory hashing. It's written in Rust and uses multiple threads without holding the GIL while hashing the tree. This significantly improves the throughput with cold and hot caches.
Fixes #281.