Repository navigation
Conversation
OpenShell becomes a fourth selectable SANDBOX_MODE alongside FAKECHROOT / NAMESPACE / SECCOMP_ONLY. Nothing is retired: the existing modes and their launch paths are untouched, so a deployment opts in by setting one property and reverts the same way. - Constants: SANDBOX_MODE_OPENSHELL = "OPENSHELL", documented as additive. - OpenShellTransport: creates a named sandbox running the gaas worker as its canonical main process, forwards the worker port to a free local port over the gateway, and tears the sandbox down on close. Deterministic sandbox naming from a session key (SHA-256, DNS-1123 safe) so no registry lookup is needed. A sandbox that already exists is success, not an error. - ClientProcessWrapper: one new branch in the existing dispatch, before the NAMESPACE branch. Sets no udsPath, so the existing socket client takes its normal TCP path to 127.0.0.1:<forwarded port> -- the socket layer is unchanged. Fails closed: never falls back to an in-Tomcat worker when the deployment asked for an external boundary. - shutdown(): closes the forward and deletes the sandbox, since the worker cannot outlive its sandbox. Verified: mvn compile clean. Existing modes behaviourally unchanged.
OPENSHELL_IMAGE (used when no OPENSHELL_TEMPLATE is set) and OPENSHELL_POLICY (a filesystem/egress policy YAML applied at creation). The policy has to be set at creation because Landlock settings are locked at sandbox startup and cannot be widened afterwards -- that immutability is the control, not an inconvenience. The stock OpenShell base image carries no python3, so an executor image is required in practice; this makes it a deployment property rather than a hard-coded default.
…t harness Three fixes found by running the transport outside Tomcat: 1. FAIL LOUDLY on missing config. create() added --from/--policy only when the property was present, so an unset OPENSHELL_IMAGE silently fell back to the stock OpenShell base image -- which has no python3. The sandbox is still created and reaches Ready; only the workload fails to spawn, so the caller sees an unrelated null response several layers away. Both inputs now throw at construction. Same defect class as the AGENT_SANDBOX_ENABLE/ENFORCE mismatch: a control that is configured but never reaches enforcement, failing open. 2. Sandbox name must fit the gateway's 19-character limit (was 25). MAX_SANDBOX_NAME_LENGTH is now a named constant with its reason recorded; the digest is truncated to fit. 3. Wait for the port forward to actually accept before returning the port. The CLI binds lazily, so returning early produced a connection refused that surfaced as a null response. The readiness probe is a TCP connect, never a protocol exchange, because the worker is single-connection. Also adds OpenShellTransportHarness: the same classpath with a console attached, which is what made all three visible. It bootstraps the DIHelper from RDF_Map.prop because Utility.getDIHelperProperty reads that, not -D system properties -- a distinction that cost three debugging iterations. Known open issue recorded in the wiki: the worker exits ~0.5s after start when it is the sandbox's canonical main process, while the identical command run as a backgrounded child stays up and listens. Under investigation; the transport itself is correct up to and including the forward.
awaitForwardReady() connected to the forwarded port. The forward proxies straight into the worker, the worker is single-connection (--max_count 1), so the probe consumed its only slot and its disconnect called stop_request() -- ending the worker and taking the sandbox to Completed. The symptom was indistinguishable from "the workload never started", which is why image, policy, argv, HOME, I/O pattern and environment breadth were all eliminated first. Readiness now comes from the CLI's own output: forward service prints a line containing "Forwarding" once its listener is bound. The method documents why it must not be "simplified" back into a socket connect. Also pins HOME for CLI children (OPENSHELL_CLI_HOME). The CLI resolves its gateway registration and mTLS client bundle from $HOME/.local/state/openshell/, so a caller with a thinner environment (Tomcat: 13 vars vs a shell's 124) could silently reach a different or missing gateway. Found by running the CLI with HOME unset, which reports "Or register one with: openshell gateway add". Adds OPENSHELL_KEEP_SANDBOX so a failed run leaves the sandbox in place for inspection -- deleting it destroyed the evidence every previous time. VERIFIED: a sandbox created by this class served a working persistent REPL through the protocol client (x=1 persists, mutates to 42, import math survives).
The forward's exit reason was being discarded: awaitForwardReady returned false without recording why, and close() then deleted the sandbox, so the evidence went with it. Both paths now log the CLI output and the process exit code before cleanup. Also records two measured facts in comments so they are not re-derived: - The forwarded port IS listening once start() returns (verified), so a connection failure is not a readiness problem. - The worker's insight/log.txt captures every request it receives, which is the only reliable way to tell "never arrived" from "arrived and failed" -- the sandbox phase and the supervisor log both say nothing about the workload. No behavioural change to the transport path.
…harness run() read the CLI child with a blocking readLine() loop and no timeout. Because close() also calls run() to delete the sandbox, a wedged child parked the Tomcat request thread inside the failure path itself: the sandbox was never cleaned up and the call never returned. BufferedReader.readLine -> run(:459) -> close(:399) Now the read happens on a daemon thread while the caller waits on waitFor(timeout); on timeout the child is destroyed forcibly and the reader interrupted. A cleanup path must not be able to block indefinitely on the object it is cleaning up. Adds OpenShellNativeClientHarness, which wires the transport to NativePySocketClient -- the class that actually ships -- with no protocol code of its own. Verified: transport up on 38331, NativePySocketClient connected, ready=true. Debugging through a hand-rolled protocol harness had been mistaking a failing test client for a failing transport.
…ient The harness previously only proved the handshake. It now sends OPERATION.PYTHON and reads the response, which is what PyTranslator does, so the chain is verified against shipping code rather than test scaffolding. Also corrects the PayloadStruct import (prerna.tcp, not prerna.om) and replaces a port probe with a plain sleep, since probing the forwarded port consumes the single-connection worker's only slot.
… binding The forward binds in ~60ms but the worker needs ~5s to reach serve_forever(). A client connecting in that window completes the TCP handshake and then consumes the single-connection worker only slot, wedging it. Measured: connect at +0s -> no reply; connect at +5s -> response=true. Readiness now waits for the supervisor NET:OPEN 127.0.0.1:<port>/tcp line, which needs no connection to the worker. Removes the temporary diagnostics.
1. Readiness is taken from the worker own log line, never by connecting to the port: the worker is single-connection (--max_count 1), so a probe consumes its only slot and its disconnect wedges it. The forward Forwarding line binds in ~60ms and is not readiness. 2. The worker stdout/stderr goes to a file instead of the supervisor pipe, which made startup erratic (sometimes never binding while the sandbox still read Ready). 3. run() now redirects stdin from /dev/null: openshell sandbox exec reads stdin, so with ProcessBuilder inherited open pipe the CLI blocked until the 30s timeout, and three of those produced the 180s pixel stall. The same command returns instantly from a shell, which is why this only showed up under Tomcat. Verified: Py("print("LIVE_OK")") returns LIVE_OK at 1599ms; repeats 1011/4/4ms; os.getpid()=60 vs Tomcat JVM 4059545.
createProcessAndClient is shared by the user REPL and engine-owned workers. Branching on SANDBOX_MODE alone sandboxed the model engine too, so AskRoom failed with ModuleNotFoundError: No module named openai several layers from the cause: AskRoomReactor -> AbstractModelEngine.askRoom -> AbstractPythonModelEngine.startServer -> PyTranslator. That read as a missing package but was a scope error. engineOwned already distinguishes them: User.pythonCPW and Project.cpw are user-authored and untrusted and get the sandbox; model engines, vector databases, guardrails and LocalPythonFunctionEngine are our own code calling controlled SDKs and stay on the Tomcat side. Verified: AskRoom now reaches the OpenAI API (401 on the engine own key, no import error, no sandbox) and Py() still returns pid 60 against a Tomcat JVM pid of 21788 with exactly one sandbox created.
Replaces the !engineOwned guard in ClientProcessWrapper with the branch the other jails already use. The jail has always been chosen by which caller supplies the SymlinkHelper: User.startPythonSocketServerAndClient passes getUserSymlinkHelper() for user-authored python, while engine-owned workers and Project pass null. OPENSHELL is now simply the third branch of that chain, alongside NAMESPACE and FAKECHROOT, and SymlinkHelper resolves SANDBOX_MODE once into a SandboxMode enum. ClientProcessWrapper no longer re-reads the mode at all -- it is a property of the user jail, so it belongs to the object that only exists for a user worker. Verified: AskRoom on an Ollama engine answers (Hello, hi, hey, 1028ms, no sandbox) and Py() still runs at pid 60 with exactly one sandbox.
…lper Every jail hangs off SymlinkHelper, because all three answer the same question: this host path must be usable by the worker. FAKECHROOT builds a chroot tree (symlinks on this host) NAMESPACE adds the path to the jail via SandboxInjector (control socket) OPENSHELL must COPY the tree, because a sandbox has its own filesystem Making a path visible is not making it present. SandboxTransporter is the OPENSHELL counterpart of SandboxInjector: wired by ClientProcessWrapper at the same point, called through a transportOrEnqueue dispatcher from symlinkFolder/symlinkProject so every invocation site is covered without editing any of them. activeTransports makes it once per sandbox, and setTransport replays paths registered before the worker existed, mirroring setInjector's replay of activeInjects. Two bugs found while verifying: 1. `sandbox upload <src> <dst>` nests the basename, so passing dst == src produced app_root/app_root/version/... and the project never landed where the worker looks. Destination must be the parent. 2. The transporter needs the same isNamespaceInjectablePath guard the injector has. MCPUtility also symlinks git's own directories, so /usr/bin, /etc/ssl and /usr/lib/git-core were being uploaded -- several 30s CLI timeouts and a 77s pixel. Now 8.4s. Verified: RunMCPTool now finds and executes the project's mcp_driver.py. It fails on the project's own `import yfinance`, which is absent from both the host venv and py/install_config/pyproject.toml and would fail under NAMESPACE too -- a missing declared dependency, not a visibility problem. Write-back (FETCH_BACK) is NOT implemented: files written inside the sandbox are lost at teardown. That asymmetry is worse than no transport at all, since the sandbox can now read project files while silently discarding writes.
…aper Two additions, both in their own package prerna.util.openshell. No existing class is modified. OpenShellSessionLifecycleHarness — proves the lifecycle the product relies on, without booting Tomcat: no sandbox exists before first use; the first user-authored python request creates it on demand (13.4s cold start); Py() executes inside it through the shipping NativePySocketClient (pid 60, python 3.12.14); the exact logout call -- ClientProcessWrapper.shutdown(true), as issued by UserSessionLoader.sessionDestroyed -- deletes it (385ms); and a later session starts a fresh one. 11/11 checks. OpenShellSandboxReaper — the backstop for sessions that never log out (JVM killed, container restart, abandoned session). Lists only sandboxes carrying the executor label AND the pyw- prefix, deletes only those older than a threshold, skips unparseable timestamps rather than guessing. Verified against a labelled throwaway: dry-run listed it, the real run deleted it, and an unlabelled sandbox on the same gateway was untouched.
Found live, in the UI path: if the named sandbox already exists, start() logged the create error and carried on. The worker inside allows one connection and the forward that reached it first still holds that slot, so the second forward leaves the real client queued -- Py() never returned and the logout thread sat in clearInsightGlobals until the sandbox was deleted by hand. A sandbox this transport is not driving belongs to a dead session, so the correct reading of 'already exists' is delete and recreate, not attach.
CmdExecUtil ran bash with commons-exec in a chroot on Tomcat's machine and never
consulted SANDBOX_MODE, so with SANDBOX_MODE=OPENSHELL the terminal, git
operations and the agent tool handlers all stayed on the host. Measured before
this change: Command("hostname") returned the host identity while Py() returned
the sandbox's.
OpenShellShellRunner runs the same command with 'openshell sandbox exec', which
starts a process inside the sandbox. Deliberately not the worker socket: the
worker serves one connection and a shell command must not compete with the REPL
for it. The sandbox is started on demand, so a shell command can be the first
thing a session does.
One delegation point in CmdExecUtil, taken only when the user's SymlinkHelper
reports an external sandbox; every other mode keeps the existing path untouched.
After: hostname 936b273a023c, uid 1000, ls of the sandbox tree -- host is
kunal0137-vnet.
Before this, the sandbox was write-only: a file created by Py() or a shell command lived in the sandbox copy and died with it, so the asset browser, git and MCP generation never saw it. Measured: to_csv into the project assets -> visible to 'ls' in the terminal, absent from GetAppAssets, gone at teardown. Now, after the terminal response of an executeCommand, the wrapper reconciles the session's transported roots back onto the host: - OpenShellTransport.fetchBack streams the tree out with 'sandbox exec -- tar -cf -' (sandbox download is workspace-scoped and refuses project paths), validates the archive (relative paths only, size cap), extracts to a temp dir and reconciles per file: content must differ AND the sandbox copy must be newer, so a stale sandbox cannot clobber a host edit. - SymlinkHelper exposes the roots it transported; ClientProcessWrapper filters to project/room/user roots and throttles to one reconcile per 1.5s. - NativePySocketClient calls it once per completed request. Verified: Py writes iris_fb.csv in the sandbox -> host file appears (16 bytes) -> GetAppAssets returns its contents. Single pod only: the push to central storage for the other pods is the next step.
…e teardown Three gaps found by testing, all in the fetch-back added in 863c3f443d: - The 1.5s throttle SKIPPED the reconcile instead of deferring it. Three Py blocks in one session left two files stranded in the sandbox until some later call. Skipping is wrong for a durable-write path; the throttle is gone. - Shell commands do not travel over the python socket, so "cat > file" or a git operation in the terminal never hit the hook. OpenShellShellRunner now reconciles after the command runs. - The last block before teardown could still be lost; shutdown() now reconciles once before deleting the sandbox. Also stopped excluding .git from the fetched archive. It costs bytes and writes nothing today -- the probe image has no git (git: not found) so the sandbox copy never changes -- but when the production image ships git, a commit made in the shell is exactly the work that must come back. Concurrent host/sandbox commits still need the revision guard. Verified: three Py writes plus one shell write in a single session all appear on the host.
Write-back used to tar every transported root at every execution boundary:
~190 ms per root, so `LoadApp(a); LoadApp(b); LoadApp(c)` paid ~750 ms per
block whether or not anything changed. The worker now knows which paths
changed and the host fetches those.
Worker side (py/):
- new gaas_change_watcher.py: ctypes inotify (no dependency), coalesced
per-path events, 2k cap, IN_Q_OVERFLOW flag, and a baseline flag. A new
directory is armed *and scanned*, because `mkdir x; write x/f` inside one
block otherwise reports the directory and loses the file.
- handler gains a CHANGES operation returning
{changes[], overflow, baseline, watched, events, roots}.
- the watcher is a module-level singleton created lazily on the first
CHANGES call: the TCP handler is built per connection, and at setup() the
session's project tree has not been materialised yet, so arming there
yields zero roots and pins the host to the slow path forever.
Host side (Java):
- PayloadStruct: CHANGES operation.
- NativePySocketClient.getFileChanges(); the after-execution hook now fires
only for PYTHON and CMD, so PollNotifications and other internal polls no
longer tar the tree.
- OpenShellTransport.fetchBackPaths(): one tar of the named paths, with the
same traversal and size checks as the whole-root fetch.
- ClientProcessWrapper.fetchBackChangedPaths(): groups paths by transported
root, deletes reported removals, and falls back to the full reconcile on
baseline, overflow, or no answer.
- OpenShellShellRunner takes the same path for shell commands.
Verified on a 240-file project: three blocks, three unique files, all on the
host; journal shows "Fetch-back wrote 1 file(s) from 1 changed path(s)";
watcher counters went watched:173 baseline:true -> changes:[x.txt] events:2
with root inodes unchanged from arm time. Cost 0.20 s/block vs ~0.30 s for the
full reconcile; the first block after materialise still does one full
reconcile, by design.
…a host edit Four correctness defects found by external review and a seven-model council, all in the change-set fetch-back path: 1. reconcile() trusted mtime. A sandbox file whose content differed but whose mtime was older than the host copy was silently dropped (os.utime, cp -p, tar extraction). Content is now the authority; mtime only decides whether the host copy must be preserved first. 2. Reported deletes were applied unconditionally, destroying a concurrent host edit. Deletes now quarantine the host copy first, and are suppressed entirely if that copy cannot be made. 3. A full-root fallback carried survivors but no tombstones, so sandbox deletions were silently resurrected. Full captures now quarantine host files absent from the capture; directories the capture excludes (node_modules, __pycache__, .venv) are skipped, since "not in the archive" there means "never captured", not "deleted". 4. inotify cannot see a persistent mmap write. The watcher now also compares one stat tuple (mtime_ns, size, ino, ctime_ns) per file at each checkpoint, reported as "stat_scan". Measured: ~6 ms for 240 files, silent on a quiet block, catches mmap writes and deletions with no events at all. Verified live in a sandbox: backdated rewrite, mmap write, and delete all reached the host; the overwritten host version and the deleted file were both preserved under /opt/semosshome/writeback-quarantine/<stamp>/.
… folders
Watch descriptors follow inodes, so renaming a watched directory left every
descendant path stale: events arrived under the old name and were written back
to the wrong place. The watcher now treats IN_MOVED_FROM, IN_MOVE_SELF,
IN_IGNORED and IN_UNMOUNT as lost coverage, reports `coverage: false` with
`baseline: true`, drops the tuple index and re-arms from the roots. The host
falls back to a full capture for that one block and resumes the change-set path
afterwards.
Java also treats `coverage: false` as a fallback trigger on its own, so a future
worker that reports it without `baseline` cannot leave the host trusting a
partial list.
Deleting a file now also prunes directories it left empty, up to but never
including the transported root - an empty folder left behind after a rename
reads as corruption.
Verified live: rename block took the full path ("wrote 2 file(s)", "moved 1
deleted file(s)"), the next block returned to the change set ("1 changed
path"), and the post-rename write reached the host.
The stat-tuple scan ran on every CHANGES call, so an idle block on a 100k-file tree paid ~0.57 s for a scan that could not have found anything. Measured: 240 files 1-2 ms, 10k 54 ms, 50k 285 ms, 100k 570 ms - linear at ~5.7 us/file. The scan is now due on the first call, after a lost baseline or coverage, and otherwise at most once per interval. The interval is max(2 s, 20 x the last measured duration), so the backstop holds roughly a 5% duty cycle at any size: 2 s at 240 files, ~10 s at 100k. Skipping never drops a change - the index is untouched, so the next scan reports everything accumulated; normal writes are unaffected entirely because they arrive as events. Measured at 100k: idle block 0.01 ms (was ~0.49 s), scan 0.486 s when due, interval self-tuned to 9.7 s. Payload reports stat_scan and scan_skipped.
Measured on a real project: .git was 112 directories and 137 of 256 files - over half the watch set and over half of every scan - for data that cannot change, since the sandbox image has no git binary. node_modules was only excluded from the scan and the tar, so it was still watched: one npm install adds thousands of directories, exhausts fs.inotify.max_user_watches and floods the queue, turning a cheap block into a full capture. One ignore set now covers arming, adoption, scanning and pull-back: node_modules, __pycache__, .venv, .git. Directories created mid-block are adopted only if they are not in that set, so an install inside a block cannot smuggle the tree back in. Override with SEMOSS_SYNC_IGNORE. Synthetic tree shaped like the real one (20 assets, 112-dir .git, 200-dir node_modules): watches 318 -> 4, indexed files 332 -> 20, asset writes still detected, node_modules writes unreported by design.
The sandbox runs outside the platform's permission checks, so any member who could open a project could write a file with Py() and have fetch-back publish it into the project folder - a read-only member silently gaining write access. ClientProcessWrapper now binds the owning User (set at client creation in User.java) and checks it before any root is pulled back: the project id is taken from the root folder (`<name>__<uuid>`, or a bare uuid), and SecurityProjectUtils.getActualUserProjectPermission must answer OWNER or EDIT. Anything else is skipped with "Refusing write-back into ... has READ_ONLY access to project ...", and an unresolvable permission fails closed. The check runs in both paths - the change-set pull and the full fallback - so a baseline reconcile cannot bypass it either. Verified live: with kunal0137 downgraded to READ_ONLY on workbench_1, a Py block wrote 29 bytes inside the sandbox, the host file did not appear and the refusal was logged; restoring OWNER made the same write land on the host. Rooms and user assets are NOT gated: this build has no room permission model, and asset roots are the user's own. If rooms are meant to inherit project access, that rule needs stating before it can be enforced.
…ll-through
Write-back now decides every root by its own space instead of special-casing
rooms and letting everything else through:
- user assets: the folder is Asset__<id> where the id must equal one of the
caller's own asset project ids (User.getAssetProjectId(provider), backed by
ASSETENGINE keyed on userid) - anything else under user/ is somebody
else's and is refused;
- rooms: ROOM.USER_ID in the model-inference-logs database via
ModelInferenceLogsUtils.doCheckRoomExistsForUser;
- projects: OWNER/EDIT as before;
- any unrecognised space: refused, where the old code returned true.
Every refusal is logged with the path and the reason, and each check fails
closed on an exception.
Verified: with the deployed class carrying isOwnUserAsset, a Py block writing
29 bytes into an owned project root still reached the host ("Fetch-back wrote
1 file(s)"). The asset/room denial paths are identity-keyed - this instance has
one human user per asset project and only kunal0137 owns rooms, so a foreign
root cannot be transported into a session to exercise the refusal.
…from py/ The executor image was assembled outside this repo: executor-probe/py was a hand copy of py/, and executor-proto just layered on top of it. That copy is how the image kept running month-old python while the repo looked fixed. docker/Dockerfile.executor replaces the two-image dance with one stage whose build context is py/, so the image carries exactly the committed worker. It keeps what matters from both predecessors: python:3.12-slim, git with a system identity (local repo work only - egress is still per-binary in the OpenShell policy), pandas + jsonpickle for the boot-time lazy imports, the insight/grants layout, uid 1000 ownership of /opt/semosshome, and the writable-workspace USER the sandbox requires. docker/build-executor-image.sh is the one command: it labels the tag with a hash of the python tree (org.semoss.py-sha) so `docker inspect` answers "which source is this", then smoke-tests the result inside the image. Built as semoss-executor:0.1 from py sha 1701eb4f4006: git 2.47.3, imports ok, worker present, watcher ignore set intact, id 1000. The worker boots as the main process and listens on 7100. RDF_Map.prop still points at semoss-executor-proto:0.2; switching is OPENSHELL_IMAGE + restart.
docs/deployment/executor_sandbox.md covers the opt-in SANDBOX_MODE=OPENSHELL path: the process model and session lifecycle, the per-block write-back pipeline (CHANGES verb, targeted tar, content-authoritative reconcile, quarantine, tombstones), the two change detectors and why the stat-tuple scan exists at all, per-space write permissions, the repo-owned image build, configuration keys, a verification recipe and the known limits (2000-path cap, racing mmap writers, no host->sandbox refresh). Linked from the Deployment and operations section of docs/README.md.
The first full build came out at 22.4 GB. Cause: flattening pyproject.toml to a requirements list dropped [tool.uv.sources], which is what maps torch/torchvision/ torchaudio to the pytorch-cpu index. PyPI then resolved CUDA builds - 3.3 GB of nvidia plus 815 MB of triton inside an image that never sees a GPU. export_requirements.py now skips the torch trio by default (--keep-torch to opt back in) so the Dockerfile can install it from the right index first, and it emits [tool.uv] override-dependencies as ordinary requirements so the sandbox keeps the platform's security pins (marshmallow, protobuf, numpy, pyasn1, lxml). Dockerfile.executor-full installs torch from download.pytorch.org/whl/cpu for the cpu extra and /cu126 for gpu, then the remaining declared dependencies from PyPI.
Shows the per-session shape — Tomcat, gateway, supervisor container, workload container with network=none, PID 1 sandbox runtime, the python worker reached through openshell forward — so a reader can see why a worker connection travels four hops instead of being a local socket.
The script that shipped with the first Dockerfile only ever built the slim image; the MODE=full path (staging pyproject.toml + export_requirements.py into the build context, passing EXTRAS) was written but never committed, and was lost to a checkout. Restored and committed so the full-image build is reproducible from the repo, not from shell history.
… venv The sandbox's PATH belongs to OpenShell, not the image. Measured inside a live sandbox: PATH=/usr/local/bin:/usr/bin:/bin, so an image's ENV PATH=/opt/venv/bin is ignored, `python3` resolves to the base interpreter, and the worker died at boot with no pandas/jsonpickle. A symlink /usr/local/bin/python3 -> /opt/venv/bin/python does not help either: CPython looks for pyvenv.cfg relative to the invoked path and silently runs as the base interpreter. Dependencies now install with `uv pip install --system` and the final stage copies /usr/local, so whatever `python3` the sandbox resolves to already has them. No venv, no PATH dependency, and the image is smaller (3.7 GB vs 6.9 GB) because the venv duplicated the interpreter. The smoke test now runs the parity check under `env -i PATH=/usr/local/bin:/usr/bin:/bin` - the sandbox's real PATH - instead of docker run's defaults, which is why the previous build reported success while failing at worker boot. Verified in a live OpenShell sandbox: python3 -> /usr/local/bin/python3, and numpy/boto3/pandas all import.
…constraint Records that OpenShell sets the workload's PATH (/usr/local/bin:/usr/bin:/bin), so an image's ENV PATH is ignored and a venv is invisible - hence the full build installs into /usr/local - and that smoke tests must run under env -i with that PATH.
Three changes that belong together, because the sandbox runtime - not the image - decides the environment its processes get. 1. Dockerfile.executor-full now builds on ubuntu:24.04, matching docker/Dockerfile.python: same distro, glibc and native-library expectations for document/OCR stacks. python3.12 comes from apt (plus python3.12-dev, without which annoy cannot compile its C extension), and the declared dependencies install into /opt/venv with uv, as upstream does. 2. OpenShellTransport launches the worker as /opt/venv/bin/python3 when it exists and falls back to `command -v python3` otherwise. Invoking `python3` by name was the actual defect: OpenShell builds the environment for spawned processes, so an image's ENV PATH never applies and the bare name resolved to the base interpreter with none of the packages. 3. Sandbox creation passes --env PATH=/opt/venv/bin:... so user code that shells out to python3 resolves to the same venv the worker uses. Verified so far: the deployed class carries both the resolver and the --env flag, and the fallback branch works against an image without a venv (sys.executable = /usr/local/bin/python3, worker boots, Py() returns).
…ch it by path
Third and final blocker in the same family: the policy allowlists /usr, /lib, /etc, /bin,
/sbin, /proc (ro) plus /sandbox, /tmp, /opt/semosshome (rw). A venv at /opt/venv is outside
every one of those, so the worker died with
PermissionError: [Errno 13] Permission denied: '/opt/venv/pyvenv.cfg'
before python could start.
The venv now lives at /usr/lib/python/semossvenv - the path docker/Dockerfile.python already
uses, which sits inside the allowed /usr - and OpenShellTransport resolves it explicitly
(/usr/lib/python/semossvenv/bin/python3, falling back to `command -v python3` for images
without a venv), with `--env PATH=/usr/lib/python/semossvenv/bin:...` for user code.
Also drops the chown -R over the venv: it is read-only to the sandbox, so root-owned 755 is
correct and it removes minutes from the build.
Verified live through SEMOSS: sys.executable = /usr/lib/python/semossvenv/bin/python3,
python 3.12.3, boto3 1.43.118, numpy 2.5.2 - imports that all failed before. That path is
the proof the interpreter was resolved by name rather than by luck.
…captures - Infer deletions only for paths the sandbox is known to have held. A full capture used to remove every host file it did not contain, which deleted gitignored files (never uploaded) and files added on the host mid-session. - Owe a full capture when a drained change set cannot be applied, instead of losing those paths. - Serialise fetch-back per process: python blocks, shell commands and teardown no longer interleave reconciles. - Quarantine a differing host file whenever it changed since this session last left it, and skip the write if the copy fails. Sandbox mtimes no longer decide. - Refuse host writes and deletes that leave the root or cross a symlink. - Write back only git's data files (objects, refs, logs, HEAD, index): config, hooks and info are executed or obeyed by host-side git. - Stream captures to disk under the cap instead of buffering them in the heap. - Check archive paths per segment and require a clean tar listing; normalise permissions after extraction so mode-000 directories cannot leak temp dirs. - Use userCanEditProject for write-back, the check that grants the read-write upload. - Shell runner: capture output to a file (a full pipe deadlocked past 64 KB), cap it, and only change directory on a clean cd. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Use poll() instead of select(): select raises on any fd >= 1024 and the watcher thread exited silently while still reporting full coverage. - Report coverage loss when the loop thread is dead, when a watch cannot be added (ENOSPC, EACCES), or when a directory cannot be listed. - A scan that cannot list part of the tree no longer reports the unlisted files as deleted (a mode-000 directory read as "everything removed"). - Drain queued events inside snapshot(), so a write finished just before CHANGES belongs to that block. - Re-evaluate roots on every CHANGES and force a baseline when one is added: room/ and user/ appear after the first call and were never watched. - Prune ignored directories when adopting a new one (node_modules, .git). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Full captures run with --ignore-failed-read so one unreadable directory no longer fails the root on every block. Anything tar reports on stderr marks the capture partial: files are written back, deletions are not inferred. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The slim build stages py/ into the context, so `COPY .` landed the worker at /opt/semosshome/py/py/. Copy py/ like the full image does. The smoke test now fails the build unless the image's worker tree hashes to the repo's and the worker's interpreter imports the change watcher. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…x doc - Remove the transport and native-client harnesses (main() classes in src/, shipped in the production jar). Move the session-lifecycle harness to test/ and require -DRDF_MAP instead of defaulting to a host path. - Doc: describe write-back as it now behaves (owed full captures, stamped quarantine, held-path deletions, partial captures, symlink containment, .git data allowlist), the permission check actually used, every OPENSHELL_* key, and how the image build proves its contents. Drop the claim that a reaper runs and the host-specific commands and timings. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
shutdown() reconciled after stopping the socket client. The worker is the sandbox's main process, so by then the sandbox had completed and every teardown capture failed with "sandbox is not ready (phase: Completed)". Reconcile first, then stop the worker and delete the sandbox. Also log each full capture with its size and whether it was partial, so the fallback path is visible in the server log. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…vers A user's python now runs in one persistent worker in their own sandbox. Any session, cookie-less API call or server instance attaches to it; it is deleted when the user's last live session ends, not when one session or one server goes away. - worker: --persist keeps serving after a client leaves, accepts several clients, and exits by itself after --timeout minutes with nobody connected. Callbacks to Java use the connection whose request is running. - OpenShellUserSandboxes: owner -> sandbox registry and per-session leases (Redis when configured, in-memory otherwise), a per-user create lock, one helper and wrapper per user on each instance, and a sweeper for sandboxes whose owner has no live session or that stopped running. - OpenShellTransport: attach/detach next to start/close; owner label; phase lookup and labelled listing for the sweeper. - ClientProcessWrapper: acquire through the registry, detach on shutdown, full capture on reattach. User: shared helper and wrapper in OPENSHELL mode. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…sandbox Measured: a Tomcat restart fires sessionDestroyed for every session, which released every lease and deleted the user's sandbox. Only a logout or a real timeout now gives up a lease; a session leaving with its server keeps it, so the restarted or another instance reattaches. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The shell runner derived the sandbox name from this instance's chroot folder, which no longer names a user-scoped sandbox; every Command() failed with 'sandbox not found'. Ask the user's wrapper for the sandbox its worker runs in. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ettings Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A sandbox used to keep the copy of a project it got at first use, and its next full capture put that stale copy back over newer host files (another user's write-back, a cluster pull, a UI upload), leaving the newer version only in quarantine. Measured live before this change. - write-back: a file goes back only if the sandbox changed it since the last sync (both sides carry the same size and mtime after one; uploads and captures keep mtime). When both changed, the last write wins and the loser is kept in quarantine. A delete never removes a host file that changed since the sandbox had it. - refresh: before each python block or shell command, host files whose stamp moved are copied into the sandbox in one upload, and host deletions are applied there. - cluster: write-back into a project is pushed to cluster storage and announced, at most every ten seconds per root and on teardown, so other servers pull it and their sandboxes refresh in turn. - the user's running sandbox is recovered by its owner label when the registry has no entry (in-memory registry after a restart, Redis flushed). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Correctness:
- phaseOf throws when the gateway cannot be asked instead of answering
"gone", and acquire retries rather than deleting a live user's sandbox
over a hiccup; one failed attach is retried before the sandbox is replaced
- a per-user lock serialises session bookkeeping with the release of the
shared wrapper, and User retries if a release won the race to the wrapper
- every request records its session here, not only lease refreshes
- the worker's live-handler set is guarded by the server monitor
- a whole-tree refresh still applies host deletions, in batches
- fetch-back counts files again, not roots
Redis registry follows the room-coordination layout: openshell:{owner}:*
hash-tag keys, SET NX PX lock with a random token, compare-and-delete
scripts for forget/unlock/drop.
Cleanup: TransportedRoots replaces four copies of the user-space check and
three of the id parsing; SandboxInfo record; one stamp helper; imports
instead of fully qualified names; constants for the timing values;
appliesTo() keeps users without a primary login on the per-session path;
fetchBackPaths gets its javadoc back; stale single-connection docs updated.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
Replace the slim/full executor Dockerfiles and export_requirements.py with docker/Dockerfile.sandbox: ubuntu 24.04 plus /usr/lib/python copied from quay.io/semoss/python-builder:3.14-cpu (same as the Tomcat images), git, a uid 1000 user and py/ as the worker. Add sandbox-builder.yml to build amd64/arm64 on py/ changes and after Python Builder runs, smoke-test each digest, then tag quay.io/semoss/semoss-sandbox:<branch>-3.14-cpu. The local build script and CI share docker/sandbox-smoke-test.sh.
setup() replaced a class-level thread_local on every new connection, so on a persistent worker a client connecting mid-request wiped the payload of a Py() still running on another connection; its reply went out with a random epoc and no insightId and the Java caller hung. Each handler now owns its own thread_local. Also log why the change watcher failed instead of silently forcing a full reconcile.
Logging out of the last provider drops its token before the session is destroyed. sessionDestroyed then threw in getSingleLogginName, which skipped every cleanup step, and even without the throw the owner could no longer be derived from the user, so the lease was never released. - getSingleLogginName returns "" when the primary login's token is gone - sessionEnded takes the owner; ownerForSession reads it from the session (recorded by the filter) and falls back to the user's login
…emosshome
Files cross into a sandbox at their host path, which is what keeps the
python SEMOSS generates (MCP loaders, PySource, the pandas JSON hand-off)
working, since it embeds absolute host paths. A Mac's BaseFolder is
somewhere like /Users/me/Documents/semoss, which no image has. The
sandbox can only create it under a root that exists at start and that
the policy grants: uploads run as the sandbox user inside Landlock.
- Dockerfile.sandbox makes /Users and /home, owned by uid 1000; at
create time the policy grants the top folder of BaseFolder's path, and
the transport makes the home inside the sandbox. One generic image
serves every developer; nothing is built per machine.
- The policy is rendered from a template bundled in the jar (or
OPENSHELL_POLICY), with ${SEMOSS_HOME_ROOT} and ${PYTHON} filled in.
OPENSHELL_POLICY is no longer required, so a checkout needs no policy
file from outside the repo. Egress is pypi only.
- Dockerfile.sandbox moves uv's CPython to /usr/lib/python/cpython, so
the egress rule names /usr/lib/python/cpython/bin/python3.14 on every
patch version and architecture; the smoke test checks it.
- Sandbox creation fails with the cause in the message when the home
cannot be made, instead of the bare upload error.
- OPENSHELL_IMAGE defaults to the published
quay.io/semoss/semoss-sandbox:dev-3.14-cpu, so a deployment that
tracks dev sets neither the image nor the policy; build-sandbox-image.sh
is gone, and the doc keeps the one docker build line for trying a py/
change before it is published
- OPENSHELL_EGRESS: host[:port] list rendered into the bundled policy's network_policies for the sandbox interpreter; default pypi, "none" for no egress; entries are validated as plain hostnames - OPENSHELL_CPU / OPENSHELL_MEMORY: per-sandbox caps passed to sandbox create, default 2 CPUs and 4Gi, "none" for no cap; caps not reservations, so idle sandboxes (~130 MB worker) cost little - the policy grants /dev/shm: multiprocessing, joblib and torch data loaders failed with PermissionError on SemLock without it Measured through the docker driver: caps land on the container (NanoCpus/Memory), an allocation past the cap is killed, custom egress allows only what is listed.
…rt guard /dev/shm makes it work for processes in the sandbox; inside Py() the worker's InsightGlobalStore guard removes os.fork/os.kill and bans subprocess, so Pool fails there (measured on the live instance).
…roviders A project declares SANDBOX_SECRETS (name + hosts) in its .smss and sets SANDBOX_SECRET_<NAME>. When a user's sandbox loads the project, each secret becomes an OpenShell profile + provider attached to that sandbox, so code gets a placeholder (smss_secrets.secret(name)) that OpenShell's supervisor swaps for the value only in requests to the declared hosts. The .smss never enters the sandbox, so the value stays on the host. - credential names carry the project key, so two apps may share a name - the value reaches the CLI through the environment, never argv - an attach is waited out, and the placeholder file is rewritten after every attach or update once the new generation appears; old generations stop resolving (attach) or keep the old value (update) - an unchanged value is not re-applied, so later loads cost ~0.5 s Measured against a real sandbox with a live OpenWeather key: 200 through secret(), refused at another app's host, rotation reaches running code.
…P example The old slim image pip-installed it as a stopgap for the example; the python-builder venv does not declare it, so Stock MCP failed with ModuleNotFoundError after the 3.14 switch. APP_PACKAGES keeps the list explicit until apps can declare their own dependencies.
…Stock MCP example" Skip yfinance as an image package for now; the Stock MCP example is parked. This reverts commit 3076363.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Changes Made
How to Test
Notes