Conversation
A TorQ application overlay that captures data partitioned by date AND instrument, and serves it through kdb-x virtual tables. No RDB, no HDB, no sort process and no gateway: the writer writes where the readers read, and nothing moves at end of day. The instrument becomes a directory rather than a column, so a selective lookup is a directory lookup instead of a scan. That needs no p# attribute, which means the live day is as fast as history and there is nothing to rebuild overnight. Contents code/wdb/vtwrite.q writer overrides (5 of them) code/processes/vtidb.q the reader code/processes/vtcompress.q the weekend compression job appconfig/ process list, settings, compression tiers testfiles/ 16 assertion tests + 9 evidence scripts docs/ architecture document and status report Verification ./regress.sh 16 assertion tests, 16/16 ./selftest.sh end-to-end through the live stack, 7/7 Results checked against the same bytes in a stock kdb+ database: 16 of 19 queries identical, 3 raise an error, none silently different. Known limitation: a single virtual table cannot span both this format and existing date-partitioned history, because the column list is taken from the first directory only. Raised with KX; see section 10 of the doc. Parts 2 (split capture stacks) and 3 (chained tickerplant) are not included. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
jonnypress
reviewed
Sep 1, 2026
| @@ -0,0 +1,26 @@ | |||
| discovery:pass | |||
| feed:pass | |||
| gateway:pass | |||
Contributor
There was a problem hiding this comment.
Don't think we need all of these in here given we only have a few processes
| // them - on the reference partition (doc §7.3). | ||
| // set 0 to compress everything, as stock TorQ does | ||
|
|
||
| // the hdbstructure override that teaches compression to see this layout is NOT here, even |
Contributor
There was a problem hiding this comment.
What is all this nonesense in the comments?
Can you compress these to something sensible.
| @@ -0,0 +1,2 @@ | |||
| table,minage,column,calgo,cblocksize,clevel | |||
| default,1,default,2,16,9 | |||
Contributor
There was a problem hiding this comment.
if this is a test file we shouldn't have it in the main code base, it should be in a test directory somewhere
| inputcsv:@[value;`inputcsv;.proc.getconfigfile["compressionconfig.csv"]]; | ||
| hdbpath:@[value;`hdbpath;`:hdb]; | ||
| maxage:@[value;`maxage;365]; | ||
| dryrun:@[{[x] `dryrun in key .proc.params};::;0b]; |
Contributor
There was a problem hiding this comment.
This seems an odd way to do it. Should this not just be "`dryrun in key .proc.params"?
The pack hand-wrote its start lines in start.sh, duplicating what TorQ's own
process manager already builds from appconfig/process.csv. Follow the FSP
instead: it ships no Linux wrapper at all, and its README directs Linux users
to "torq.sh in the bin directory with the command line argument start all".
- start.sh and stop.sh removed. torq.sh is the interface:
export SETENV=$PWD/setenv.sh
$TORQHOME/torq.sh start all | stop all | summary | print all | debug <p>
- vt-env.sh renamed to setenv.sh. This is required, not cosmetic:
installtorqapp.sh copies the application's setenv.sh into deploy/bin by
that exact name and rewrites TORQHOME, TORQAPPHOME and TORQDATAHOME in it.
- installlatest.sh added, modelled on the FSP's. It can be fetched on its own
into an empty directory, resolves the latest TorQ release, and hands off to
installtorqapp.sh to produce a runnable deploy/ tree. Unlike the FSP it has
no release of its own to download, so the default installs TorQ and points
at the checkout; --deploy builds the full tree.
- loadtest.sh brings its stack up with "torq.sh start discovery1 stp1 wdb1
idb1"; naming the processes is what leaves the demo feed out.
- compress.sh keeps its direct q invocation, with a comment saying why: it is
a foreground one-shot that tees its output, where torq.sh start backgrounds.
This also fixes a real defect. $KDBSTACKID was referenced in four places and
set by nothing, so processes started without -stackid - which is exactly what
torq.sh matches on to find them. They were invisible to "torq.sh summary" and
could not be stopped by "torq.sh stop". torq.sh supplies -stackid $KDBBASEPORT
itself, so the variable is gone.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
.vtidb.symbytes summed the result of "each" over the domain files found at each root. On a database with no sym file yet that list is empty, and sum of an empty list is () rather than 0. symchanged then evaluates symsize<>(), which is () , and $[();...] throws 'type - so vtidb.q failed to load outright and the reader never started. Nothing else in the stack notices: every other process comes up healthy and only the reader is missing, so the failure looks like a configuration problem rather than a load error. This never showed up because the pack has only ever run in place against a var/db that already held data. A fresh install is empty by definition, so it reproduces every time - which is how installlatest.sh surfaced it. Seeding the sum with 0 keeps the result numeric when there is nothing to add. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Four review comments, plus a pass over the comments throughout.
- appconfig/passwords/accesslist.txt trimmed from 26 entries to 7. Only the
proctypes appconfig/process.csv actually starts, plus admin for qcon; the
rest were inherited boilerplate for processes this pack does not run
(gateway, rdb, hdb, dqc, dqe, ...). compression was missing despite cmp1
existing, so it is added. No comments in the file: -U is parsed by kdb+
rather than TorQ and the format has no comment syntax, and the FSP's own
list is plain user:pass. The explanation is in the README instead.
- compressionconfig-test.csv moved from appconfig/ to testfiles/, with
compress.sh, testfiles/vt-compress-ab.q, the README and the architecture
doc updated to match.
- .cmp.dryrun simplified to "`dryrun in key .proc.params" as suggested. The
trap it replaces guarded against .proc.params not existing, but
vtcompress.q is only ever loaded through torq.q, which sets it during init.
- Comments condensed across the runtime code and config. Banner rules
dropped, measurements left to the architecture document, and restatement of
what the code plainly does removed: vtidb.q 177 -> 132 comment lines,
vtwrite.q 88 -> 49, settings/default.q 19 -> 7.
Kept deliberately: the traps that are silent when undone - settings files
loading before code/common/compress.q (so an hdbstructure override there
compresses nothing), the replay carry-over in vtsavetodisk, why the writer
overrides must be installed before the log replay, why mutable is
vectorised, why the catalogue drop must precede current moving, and why the
feed resolves its handle on every publish.
No behaviour changes beyond the dryrun line.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
docs/status-report.md was a point-in-time progress summary organised around ticket numbers - epic summary, VT-1 through VT-21, status at a glance. That is tracker state, not documentation of the code: the VT numbers mean nothing outside the tracker, and the file was already stale. Its Risks table was the exception, and is the one thing in it not recorded elsewhere: a consolidated register of every known failure mode and whether detection exists for it. The detail lives in the sections it cites, but nothing else gathered them in one place. It is now section 13 of the architecture document, with the twelve retired rows dropped and the fourteen live ones kept. The "notes for reviewers" list of testfiles is dropped as duplicate - the README already indexes them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A TorQ application overlay that captures data partitioned by date AND instrument, and serves it through kdb-x virtual tables. No RDB, no HDB, no sort process and no gateway: the writer writes where the readers read, and nothing moves at end of day.
The instrument becomes a directory rather than a column, so a selective lookup is a directory lookup instead of a scan. That needs no
p#attribute, which means the live day is as fast as history and there is nothing to rebuild overnight.Contents
Verification
Notes for review
appconfig/passwords/is the stock TorQ demo:passlist, same as the Finance Starter Pack ships. No real credentials.mkPviause`kx.pq.t, anduseis a kdb-x keyword.