Skip to content

Adding locking to prevent race condition - #2975

Closed
fnachabe wants to merge 2 commits into
devfrom
2974-close-database-engines-before-cloud-sync
Closed

fnachabe wants to merge 2 commits into
devfrom
2974-close-database-engines-before-cloud-sync

Conversation

@fnachabe

@fnachabe fnachabe commented Sep 3, 2026

Copy link
Copy Markdown

Description
Fixes a cloud synchronization issue where SEMOSS could upload a database engine’s local H2 audit file while H2 was modifying or compacting it.
External databases can return false from holdsFileLocks() because their primary database is remote, even though they still maintain a local audit_log_database.mv.db. Uploading this live file could cause the Azure/Netty chunked uploader to enter a zero-progress CPU loop and stall other cloud-storage requests.

Changes Made
• Close all DATABASE engines before synchronizing their engine folders.
• Preserve the existing holdsFileLocks() behavior for other engine types.
• Reopen database engines after successful synchronization.
• Reopen database engines when synchronization fails.
• Ensure the engine synchronization lock is always released.
• Add unit tests covering:
o External databases without primary file locks.
o Non-database engines without file locks.
o Engine recovery and lock release after transfer failure.
• Update central cloud storage documentation to describe local audit-file handling.

How To Test
From the SEMOSS repository root, run:
mvn -Dtest=prerna.cluster.util.clients.CentralCloudStorageUnitTests test
Verify that the test output ends with:
Failures: 0, Errors: 0
BUILD SUCCESS
For an integration test in a clustered environment:

  1. Perform an insert or update against an external database.
  2. Confirm the database engine is closed before cloud synchronization.
  3. Confirm the engine is reopened after synchronization.
  4. Verify the application can continue reading from and writing to the database.
  5. Capture two JVM thread dumps approximately 10 seconds apart.
  6. Confirm no Reactor thread is continuously consuming a full CPU core in ChunkedNioFile or ChunkedWriteHandler.
  7. Confirm the Azure upload completes and other cloud-storage operations remain responsive.

@fnachabe
fnachabe requested a review from a team as a code owner September 3, 2026 14:19
@fnachabe fnachabe linked an issue Sep 3, 2026 that may be closed by this pull request
1 task done
@snyk-io

snyk-io Bot commented Sep 3, 2026 •

Copy link
Copy Markdown
Contributor

✅ Snyk checks have passed. No issues have been found so far.

Status Scan Engine Critical High Medium Low Total (0)
✅ Open Source Security 0 0 0 0 0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

@fnachabe

Copy link
Copy Markdown
Author

Superseded by #3172, which removed the per-engine AuditDatabase and stopped pushing external database engines after writes. The mutable local H2 file that motivated this PR is no longer present in the synchronized engine folder, so closing every database during pushEngine is unnecessary and could disrupt active database operations.

@fnachabe fnachabe closed this Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Close Database Engines Before Cloud Sync

1 participant