Skip to content

Match sid-sdk snippet behavior and improve performance - #1

Merged
natirniger merged 5 commits into
mainfrom
sdk-snippet-parity-and-tests
Oct 8, 2026
Merged

natirniger merged 5 commits into
mainfrom
sdk-snippet-parity-and-tests

Conversation

@natirniger

Copy link
Copy Markdown
Collaborator

Summary

  • match Alyze tokenization and Unicode lowercasing
  • speed up snippet selection and window scoring
  • add parity fixtures and the SID-2 reference example
  • move tests into the external sid_test package

Testing

  • go test ./...

- Put API tests under tests/
- Use the external sid_test package
- Add Unicode 17 word-like mappings
- Tokenize content once and render by byte offset
- Keep snippet output in sync with sid-sdk
- Port the documented Python flow to Go
- Leave search backends as TODOs
- Sweep sorted events once per feature
- Skip terms missing from the document
- Add Unicode 17 lowercase mappings
- Add sid-sdk parity fixtures
@natirniger
natirniger merged commit bd02551 into main Oct 8, 2026
2 checks passed
@natirniger
natirniger deleted the sdk-snippet-parity-and-tests branch October 8, 2026 22:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant