Generate standalone Ruby frontends from Ruby sources

.gitignore

@@ -7,3 +7,4 @@
 /pkg/
 /spec/reports/
 /tmp/
+/*.md

AGENTS.md

@@ -3,15 +3,18 @@
 ## Project status
 
 `arpaka` is a Ruby language parser frontend and an output backend for Lrama.
-Keep implementation decisions aligned with both roles. The frontend is exposed
-as `Arpaka`; the generic Ruby parser generator remains under `Lrama::Ruby`.
+Keep implementation decisions aligned with both roles. Arpaka generates a
+standalone Ruby frontend from a user-supplied Ruby source
+tree; the generic Ruby parser generator remains under `Lrama::Ruby`.
 
 The initial backend generates standalone Ruby parsers from Lrama's LALR/IELR
 tables. `Lrama::Ruby.generate` returns source; `Lrama::Ruby.compile` returns a
 parser class loaded in its own Ruby Box. See README for supported grammar features.
 Ruby >= 4.0.0 is required. CI runs Ruby 4.0 and head.
 Generation uses Ruby Box to isolate Ruby-specific extensions to Lrama; start Ruby
-with `RUBY_BOX=1`. Prism handles Ruby action tokens and generated syntax validation.
+with `RUBY_BOX=1`. A Ruby action scanner handles action tokens. Generated frontend
+code is validated
+in an isolated Ruby Box.
 Whether to provide a fallback for Ruby
 3.4 and earlier will be decided during implementation; compatibility with those
 versions is not currently promised.
@@ -23,8 +26,10 @@ versions is not currently promised.
 - `lib/lrama/ruby/parser.rb.erb`: standalone parser template.
 - `lib/lrama/ruby/backend.rb` and `action_code.rb`: internal files loaded in the generator's box.
 - `examples/calculator.y`: executable example grammar.
-- `lib/lrama/ruby/languages/ruby/`: bundled grammar, AST, builder and runtime metadata.
-- `tool/ruby/`: grammar adapter, action definitions and original rule inventory.
+- `lib/arpaka/`: frontend generator, CLI, source profile and Action mappings.
+- `lib/arpaka/runtime/`: AST, builder, lexer and frontend source templates.
+- `exe/arpaka`: installed generation command.
+- `tool/ruby/`: development entry point for frontend generation and checks.
 - `vendor/ruby/`: pinned upstream grammar, preprocessing tools and license files.
 - `sig/lrama/ruby.rbs`: RBS declarations.
 - `test/lrama/*_test.rb`: Test::Unit tests; `test/test_helper.rb` loads the library.
@@ -36,17 +41,18 @@ versions is not currently promised.
 
 - `bin/setup`: install dependencies with Bundler.
 - `RUBY_BOX=1 bundle exec rake test`: run the test suite.
-- `RUBY_BOX=1 bundle exec rake`: verify the bundled grammar and run the test suite.
-- `bundle exec rake ruby:generate`: regenerate bundled grammar and metadata after action changes.
-- `bundle exec rake ruby:check`: verify source checksums, rule mappings and generated artifacts without editing them.
+- `RUBY_BOX=1 bundle exec rake`: verify generation, tests and installed packaging.
+- `RUBY_BOX=1 OUTPUT=/path/to/frontend.rb bundle exec rake ruby:generate`: generate a standalone frontend.
+- `RUBY_BOX=1 bundle exec rake ruby:check`: verify source checksums, rule mappings and frontend generation without editing artifacts.
 - `RUBY_BOX=1 bundle exec ruby -Itest test/lrama/ruby_test.rb`: run one test file.
 - `RUBY_BOX=1 bin/console`: open an interactive Ruby session with the library loaded.
 
 ## Conventions
 
 Follow the existing Ruby style: two-space indentation, double-quoted strings,
 snake_case filenames and methods, and `# frozen_string_literal: true` in Ruby
-source files. Keep library code under `Lrama::Ruby`. Add relevant Test::Unit
+source files. Keep generic backend code under `Lrama::Ruby` and frontend
+generation under `Arpaka`. Add relevant Test::Unit
 coverage for behavior changes and maintain RBS declarations when changing the
 public API. No formatter or linter is configured.
 
@@ -55,10 +61,13 @@ public API. No formatter or linter is configured.
 Keep `Gemfile.lock` untracked; it is ignored intentionally.
 The gemspec packages Git-tracked files, so review package contents when adding
 repository documentation or tooling files.
-Runtime language grammars and licenses must be packaged; `vendor/`, `tool/`
-and local design/plan documents are excluded. Parsing must work without these
-development files and without filesystem writes. The Ruby grammar is generated
-and cached in memory on first use, with a fresh builder per parse.
+The gem packages frontend generation templates, Action mappings, source profiles
+and notices, but no Ruby parse.y or generated Ruby frontend. `vendor/`, `tool/`
+and local design/plan documents are excluded. The generated single Ruby file must
+parse without Arpaka/Lrama gems, Ruby Box, auxiliary files or filesystem writes.
+Embed required metadata and license notices; create fresh state per parse.
+The user supplies Ruby sources to `Arpaka.generate` or `Arpaka.compile`; the
+returned frontend exposes `.parse(source, filename:)` to obtain its AST.
 
 `bundle exec rake release` creates tags, pushes to Git, and publishes a gem.
 Run it only when a release is explicitly requested.

README.md

@@ -1,9 +1,21 @@
 # Arpaka
 
-A Ruby language parser frontend and Ruby output backend for
-[Lrama](https://github.com/ruby/lrama). `Arpaka` parses Ruby tokens into an AST,
-while `Lrama::Ruby` generates standalone Ruby parsers with Ruby semantic actions.
-The Ruby frontend is experimental and partial.
+<p align="center">
+  Generate Ruby parsers with Lrama, and use their ASTs from Ruby.
+</p>
+
+Arpaka provides a Ruby output backend for
+[Lrama](https://github.com/ruby/lrama) and a tool for generating a Ruby language
+frontend. Its goal is to make Ruby syntax accessible as an AST from Ruby code.
+
+`Lrama::Ruby` turns Yacc-style grammars with Ruby semantic actions into
+standalone Ruby parsers using Lrama's LALR/IELR tables. Arpaka applies its Ruby
+Action mappings to a Ruby source tree supplied by the user and emits a single
+Ruby file containing the parser, lexer, AST and builder. Users choose the output
+location and regenerate when updating the grammar or Actions.
+
+The Ruby frontend is experimental and partial. The gem contains generation
+support, not a bundled `parse.y` or a pre-generated Ruby frontend.
 
 ## Requirements
 
@@ -111,68 +123,77 @@ the tree's shape. Parentheses affect grouping but are omitted from the AST by
 the action `$$ = $2`. Evaluation or compilation of the resulting tree belongs
 to the application; even `1 / 0` produces a tree without performing division.
 
-### Using the bundled Ruby grammar
+### Generating a Ruby frontend
 
-```ruby
-require "arpaka"
+Provide a Ruby source tree containing `parse.y`, `tool/id2token.rb`,
+`defs/id.def`, `COPYING` and `BSDL`. Arpaka does not download Ruby sources.
+The first supported source profile is ruby/ruby revision
+`37d60dd3241d5fdb10ca43f36077f5d556ae1b8d`; other revisions are not automatically
+supported. Input checksums and expanded rule mappings are checked before
+applying Actions. Changed input files are reported and generation stops.
+Even comment-only changes currently require updating the source profile.
 
-tree = Arpaka.parse(
-  [[:tIDENTIFIER, :a], ["=", nil], [:tINTEGER, 1]]
-)
-# Arpaka::AST::Program containing LocalWrite(:a, Literal(1))
+```sh
+RUBY_BOX=1 bundle exec arpaka generate \
+  --ruby-source /path/to/ruby \
+  --class-name MyRubyParser \
+  --output lib/my_library/ruby_parser.rb
 ```
 
-Run with `RUBY_BOX=1`. The first call generates and compiles the bundled grammar
-in memory; later calls reuse the class. Each call has its own parser and local
-variable table. No build command, network access or writable working directory
-is needed. `language:` is required; currently only the symbol `:ruby` is supported.
+The output directory must already exist. `--class-name` defaults to
+`RubyParser` and must be a single Ruby constant name. Existing output files are
+preserved unless `--force` is supplied. A failed generation leaves the previous
+file intact. The output includes its source profile, generator version and
+copyright/license notices.
+
+The consumer only needs the generated file and Ruby 4.0 or later:
 
-Supply token pairs from your own lexer. `tINTEGER` values are Integers, `tFLOAT`
-values are Floats, and `tIDENTIFIER` values are Symbols or Strings. Operators and
-keywords may carry `nil`. Parentheses use Ruby's context-dependent token names
-(for example `tLPAREN` followed by `")"`), not a source string tokenizer.
+```ruby
+require_relative "lib/my_library/ruby_parser"
+
+tree = MyRubyParser.parse("a = 1\na + 2", filename: "example.rb")
+# MyRubyParser::AST::Program containing the assignment and addition
+```
+
+`parse` takes Ruby source and returns an AST. The parser, lexer and builder are
+internal; callers do not need to supply tokens. Each call has fresh parsing and
+local-variable state. No Arpaka/Lrama gem, Ruby Box, grammar files, network access
+or filesystem writes are needed at runtime.
 
-For source input, use `Arpaka.parse_source`:
+For generation from Ruby, `Arpaka.generate` returns that same standalone source.
+To try a frontend in memory, use `Arpaka.compile`; both require `RUBY_BOX=1`:
 
 ```ruby
-tree = Arpaka.parse_source("a = 1\na + 2", filename: "example.rb")
+require "arpaka"
+
+frontend = Arpaka.compile(ruby_source: "/path/to/ruby", class_name: "MyRubyParser")
+tree = frontend.parse("a = 1")
+
+code = Arpaka.generate(ruby_source: "/path/to/ruby", class_name: "MyRubyParser")
+File.write("ruby_parser.rb", code)
 ```
 
-The source lexer is byte-oriented and context-sensitive. It supports the same
-frontend slice as the token API, including numbers, variables, calls, arrays,
-hashes, control flow, definitions, strings, regular expressions, percent
-literals and static heredocs. Lexer failures include filename, line and byte
-column. Ruby interpolation and syntax whose value cannot be represented by the
-current AST raises an explicit lexer error.
-
-The supported slice includes numeric and nil/boolean literals, binary `+ - * /`,
-unary `+ -`, single-expression parentheses, simple local assignments, and
-statements separated by semicolons or newlines. Every successful parse returns
-an `AST::Program`. Its frozen `statements` array contains immutable Data nodes:
-`Literal(value)`, `Binary(operator, left, right)`, `Unary(operator, operand)`,
-`LocalRead(name)`, `LocalWrite(name, value)`, and `BareCall(name)`.
-An unassigned bare identifier is a `BareCall`; assignment registers a local
-before reading its right-hand side, including in `a = a`.
-
-Explicit receiver calls without blocks produce
-`ReceiverCall(receiver, operator, name, arguments)`. They preserve chained calls,
-ordered positional arguments, the `.` / `::` / `&.` operator, and the implicit
-`call` method in `handler.(message)`. Receiverless calls use `Call(name, arguments)`.
-
-Double quoted strings and backtick strings may contain simple interpolation and
-produce `InterpolatedString(parts)`, where each part is a string or an AST node.
-
-This is not yet a complete Ruby frontend. Percent literal interpolation, blocks,
-method argument scopes, qualified constant paths, and other constructs are still incomplete; a successful
-parse does not guarantee that all source semantics appear in the AST. Nodes have
-no source locations. Token input must already reflect the lexical decisions
-Ruby's parser and lexer normally make together; use `parse_source` for source text.
-
-`Arpaka::ParseError` exposes `token` and `state`. `Arpaka::UnsupportedSyntax`
-exposes the original `rule` and upstream `line`. Both inherit from
-`Lrama::Ruby::Error`. Unported rules raise
-instead of returning a partial AST. Lexer exceptions and invalid token values
-propagate to the caller.
+The former `Arpaka.parse`, `Arpaka.parse_source` and `Lrama::Ruby.parse`
+APIs have been replaced by explicitly generating or compiling a frontend.
+The generic `Lrama::Ruby.generate` and `Lrama::Ruby.compile` APIs are unchanged.
+
+This is not a complete Ruby frontend. Existing support includes literals,
+assignments, calls, collections and portions of control flow and definitions.
+Some constructs parse without preserving all their semantics in the current
+AST, and nodes do not have source locations. Known unported rules raise
+`MyRubyParser::UnsupportedSyntax`, exposing `rule` and upstream `line`.
+Syntax errors raise `MyRubyParser::ParseError`, exposing `token` and `state`.
+Lexer failures raise `MyRubyParser::LexerError` with filename, line and byte
+column. These inherit from the generated `MyRubyParser::Error`.
+
+The generated frontend has also been exercised against real-world source. On
+Ruby 4.0.6, it parsed all 3,436 Ruby files in a local Rails checkout. This is
+the practical compatibility signal for the current frontend. Ruby's Trick
+contest programs are kept as a separate language-compatibility challenge: the
+frontend currently parses 12 of 23 Ruby files in the `trick18` checkout, while
+the remaining files use deliberately adversarial syntax that is not yet
+represented by the current Action mappings. These are smoke-test results, not
+a claim of complete Ruby language coverage.
 
 ## Current scope
 
@@ -186,7 +207,7 @@ Locations, typed values/`%union`, `%code`, prologues, epilogues, parse/lex param
 initial actions, hooks, printers, destructors and error recovery are not implemented.
 The generator rejects these features rather than dropping their behavior.
 Use plain Ruby values in actions; C actions are not translated into Ruby.
-Prism tokenizes Ruby actions and checks generated syntax; Bison is not required.
+A Ruby action scanner handles semantic action tokens; Bison is not required.
 
 ### Inspecting a C grammar
 
@@ -219,26 +240,58 @@ ruby /path/to/ruby/tool/id2token.rb /path/to/ruby/parse.y > parse.preprocessed.y
 ## Development
 
 ```sh
+RUBY_BOX=1 bundle exec rake
 RUBY_BOX=1 bundle exec rake test
-RUBY_BOX=1 bin/console
+RUBY_BOX=1 bundle exec rake ruby:check
+RUBY_BOX=1 OUTPUT=/path/to/frontend.rb bundle exec rake ruby:generate
+RUBY_BOX=1 bundle exec rake benchmark:parse
 ```
 
-Tests cover generated parsers, Ruby actions, parse errors and Box isolation,
-including execution of a generated file without gems. CI uses Ruby 4.0.6.
-`Gemfile.lock` stays local and is not tracked.
-
-The default task also checks the bundled Ruby grammar against the pinned
-upstream source in `vendor/ruby`. `tool/ruby/actions.rb` defines the ported
-actions; `tool/ruby/upstream_rules.json` records the original expanded rules.
-Run `bundle exec rake ruby:generate` after changing action definitions, and
-`bundle exec rake ruby:check` to check for stale artifacts without writing files.
-The check compares every production, token ID and precedence, including empty
-productions representing midrule actions. Update the pinned revision and rule
-inventory together when deliberately upgrading upstream.
-
-Runtime grammar and rule metadata are packaged; upstream source and developer
-tools are not. Ruby-derived artifacts retain the upstream license notices in
-`lib/lrama/ruby/languages/ruby/COPYING` and `BSDL`.
+Development commands default to the pinned Ruby source fixture in `vendor/ruby`.
+In an uninstalled checkout, invoke the CLI with
+`RUBY_BOX=1 bundle exec ruby -Ilib exe/arpaka generate ...`.
+Set `RUBY_SOURCE` to select another source directory. Normal CLI/API usage always
+requires an explicit Ruby source tree. `ruby:check` verifies the source profile,
+Action inventory, grammar transformation and generated frontend without editing
+artifacts. `ruby:generate` writes the standalone frontend to `OUTPUT`.
+
+`lib/arpaka/actions.rb` holds the Ruby Action mappings;
+`lib/arpaka/upstream_rules.json` records the expected expanded upstream rules.
+Update these and the source checksums together when adding support for a changed
+Ruby grammar. Runtime templates live under `lib/arpaka/runtime/`.
+
+Tests exercise the generated frontend and the generic backend. Package checks
+install the gem, generate from external Ruby sources, and execute the resulting
+file with gems and Ruby Box disabled. `Gemfile.lock` stays local and untracked.
+Developer fixtures and tools are excluded from the gem.
+
+### Benchmark
+
+`benchmark:parse` measures frontend generation separately from loading the
+generated file and parsing in a fresh process with Ruby Box disabled.
+Set `ITERATIONS` to change the measured parsing iterations (default: 100).
+Each warm measurement follows ten warmup calls. File reading/class definition
+is measured separately; process startup and `require "prism"` are not timed.
+
+The RubyVM and Prism measurements use the same source but return different AST
+representations. They are reference measurements, not equivalent-work guarantees.
+
+Example results on Ruby 4.0.6, x86_64 Linux, YJIT disabled, Prism 1.9.0
+(10 measured iterations per input, ten warmup calls):
+
+| Input | Arpaka (warm) | RubyVM AST | Prism |
+| ---: | ---: | ---: | ---: |
+| 10 bytes | 0.054 ms | 0.003 ms | 0.003 ms |
+| 1,697 bytes | 5.901 ms | 0.060 ms | 0.128 ms |
+| 18,800 bytes | 61.475 ms | 0.850 ms | 1.859 ms |
+
+Frontend generation took 2.190s and loading the generated file took 20.571ms
+in the same run. The first Arpaka parse took 0.130ms, 6.707ms and 73.381ms
+for the small, medium and large inputs respectively.
+
+Generation is a one-time build step per regeneration, not a cost of parsing an
+input file. The generated frontend is loaded and measured with Ruby Box disabled.
+Results vary with input and environment.
 
 ## License
 

Rakefile

@@ -10,22 +10,30 @@ Rake::TestTask.new(:test) do |t|
 end
 
 namespace :ruby do
-  desc "Regenerate the bundled Ruby grammar and action metadata"
+  desc "Generate a standalone Ruby frontend (requires OUTPUT; optional RUBY_SOURCE)"
   task :generate do
-    ruby "tool/ruby/grammar.rb"
+    abort "Set OUTPUT to the destination Ruby file" unless ENV["OUTPUT"]
+    ruby "tool/ruby/grammar.rb", "--output", ENV.fetch("OUTPUT")
   end
 
-  desc "Verify the bundled Ruby grammar against the pinned upstream source"
+  desc "Verify the Ruby source profile, Action mappings and generated frontend"
   task :check do
     ruby "tool/ruby/grammar.rb", "--check"
   end
 end
 
 namespace :package do
-  desc "Build and install a local gem and verify its bundled language runtime"
+  desc "Verify installed frontend generation and standalone execution"
   task :check do
     ruby "tool/check_package.rb"
   end
 end
 
 task default: ["ruby:check", :test, "package:check"]
+
+namespace :benchmark do
+  desc "Benchmark Ruby source and token parsing"
+  task :parse do
+    ruby "benchmark/parse.rb"
+  end
+end

arpaka.gemspec

@@ -19,7 +19,7 @@ Gem::Specification.new do |spec|
   spec.files = IO.popen(%w[git ls-files -z], chdir: __dir__, err: IO::NULL) do |ls|
     ls.readlines("\x0", chomp: true).reject do |f|
       (f == gemspec) ||
-        f.start_with?(*%w[bin/ Gemfile .gitignore test/ .github/ tool/ vendor/]) ||
+        f.start_with?(*%w[bin/ benchmark/ Gemfile .gitignore test/ .github/ .local/ tool/ vendor/]) ||
         %w[AGENTS.md DESIGN.md plan.md].include?(f)
     end
   end