Testing

Testing

Three layers check MacroSIP: Swift Testing suites for MacroSIPCore, an end-to-end loopback call against pjsua, and demo launches of the app with a mock engine. All three run in CI on every push and pull request. A milestone is done only after swift build is clean, the tests are green, scripts/selftest-call.sh passes and a real call was verified (ADR 0006, docs/DEVELOPMENT.md).

Unit and engine tests: scripts/test.sh

scripts/test.sh                                  # all suites
scripts/test.sh --filter DialStringTests         # extra arguments go to swift test

Never run bare swift test: the Command Line Tools ship the Testing framework, but SwiftPM does not find its macro plugin on its own. test.sh passes the plugin path explicitly (-Xswiftc -plugin-path -Xswiftc .../usr/lib/swift/host/plugins/testing). XCTest is not available without Xcode, so every suite uses import Testing.

The suites in Tests/MacroSIPCoreTests hold about 210 tests:

File	Covers
DialStringTests	the parsing table (sip:, tel:, plain numbers, user@host)
JSONStoreTests, StoreTests, SettingsCompatTests	atomic writes, corrupt-file recovery, store round trips, old settings.json and call-log files decoding with the defaults, calls/timed DND/recordings
KeychainTests	the wrapper with a test service name
ContactTests	number normalisation, matching, sorting, photos, legacy fields
AutoAnswerPolicyTests	the rule order (DND, another call, answer-after, mode)
QuickKeyTests	key model, pickup URI, the presence registry
RecordingConverterTests	WAV to M4A, stereo split pairs, header repair
EngineTests	start/stop, accounts, loopback calls (one engine calling itself), hold, mute, DTMF, transfer, TLS listeners
EngineMilestone5Tests	conference, mono and stereo recording, presence mapping against a PIDF stub, answer-after trust
EngineAudioTests	the MOS table, per-call and ringtone volume, Opus FEC/DTX, the voice-processing policy, device-event debounce
EngineCallOptionsTests	single-call mode, call-waiting tone, DTMF methods, Do Not Disturb
EngineNetworkTests	registration backoff, network rounds, re-INVITE outcomes, the media watchdog, sleep and wake, with NetworkStub (a UDP registrar/callee)
Engine262Tests	RTP port window, rport, DNS SRV with an in-process DNS stub, deny rules, the additional codecs
DirectoryParsingTests, DirectoryServiceTests, DirectoryLDAPTests	CSV/JSON feeds and number matching, the service with a stub provider, LDAP against an in-process NWListener server

Engine tests run with null audio, bound to loopback, and with Engine(monitorNetwork: false) so they do not depend on the host's network; makeEngine() stops an engine a failed test left behind, so one failure cannot cascade. TestSupport.swift provides temporary directories.

End-to-end call: scripts/selftest-call.sh

scripts/selftest-call.sh                          # native
SELFTEST_ARCH=x86_64 scripts/selftest-call.sh     # the x86_64 slice under Rosetta 2
SELFTEST_LOG=/tmp/selftest.log SELFTEST_LOG_LEVEL=5 scripts/selftest-call.sh

The script starts two pjsua phones from third_party/pjsip/bin/pjsua on ports 5080 and 5081 (null audio, auto-answer, a looping 600 Hz tone, telnet CLIs on 5088 and 5089), builds macrosip-cli and runs macrosip-cli --selftest. The self-test (Sources/macrosip-cli/SelfTest*.swift) checks, against an independent PJSIP implementation: two-way RTP, hold and resume (also back to back), DTMF, mono recording (WAV with the tone audible, then M4A), a stereo split recording with a muted middle second, presence driven through the pjsua CLI (available, away, busy), a two-leg conference with merge and split and a recording that moves between legs, incoming calls (answer, reject with 486), blind and attended transfer between the two phones, DTMF over SIP INFO and a registered account against SIPStub (a digest-challenging UDP registrar inside macrosip-cli; every authorised INVITE must be above 1300 bytes without a transport-switch error), simulated network changes during a call and with the registered account, call quality (must be good), single-call mode, Do Not Disturb, the deny rules and a GSM-only call. With SELFTEST_ARCH=x86_64 the CLI is built universal and the CLI and both phones run through arch -x86_64; pjsua needs a few seconds longer to come up under Rosetta, so the script polls its CLI for up to about 15 s. SELFTEST_CLI=<path> uses a prebuilt CLI.

Demo launches and the mock engine

A debug app (scripts/make-app.sh debug) accepts launch options from Sources/MacroSIPApp/Model/LaunchOptions.swift; a release app ignores them.

Option	Effect
--mock-engine	MockEngine instead of PJSIP: registration succeeds after 1 s (a password wrong or a domain containing fail gives 403), an outgoing call connects after 2 s
--demo-state <name>	seeds sample accounts, contacts and log entries, implies --mock-engine and a throw-away data directory; the Keychain is never touched
--data-dir <path>	JSON stores in <path> instead of Application Support
--snapshot <prefix>	writes <prefix>-<window>.png of every window after --snapshot-after seconds (default 6), then quits; works without a display
--appearance light|dark	forces the theme
-AppleLanguages "(ru)"	the standard macOS argument; runs the app in Russian
--log-level 0-6	PJSIP verbosity (all builds)

Demo states by area (the full list is in the header of Model/DemoState.swift):

• Main window and calls: first-launch, dialpad, contacts, calllog, connected, calls, conference, recording, recording-long, incoming, incoming-hidden, autoAnswer, suggest, keypad, transfer-banner, account-popup, account-popup-live, many-accounts.
• Audio and network: quality-good, quality-fair, quality-poor, quality-poor-calls, call-volume (also writes <prefix>-popover.png), headset, headset-back, network-offline, reconnecting.
• Calls and DND: timed-dnd (writes the menu bar items to <prefix>-menu.txt), call-log-playback, busy-mode.
• Settings: settings-accounts, account-editor, account-editor-basic, account-editor-open, editor-first-launch, settings-general, settings-general-advanced, settings-audio, settings-codecs, settings-network, settings-recording, settings-autoanswer, settings-calls, settings-directory, many-accounts-settings, deny-incoming.
• Quick keys: quick-keys, quick-keys-bottom (writes -main.png and -settings.png).
• Contact card: contact-card, contact-card-new (writes -card.png).
• Directory (MockDirectory): directory, directory-done, directory-incoming, directory-sheet, directory-sheet-url, directory-sheet-google, directory-sheet-macos, directory-import, directory-selftest.
• Housekeeping: selftest, quit (checks that shutdown does not hang).

Demo self-test. --demo-state selftest drives the model the way the views do, through the mock engine, and records a PASS or FAIL line per check: registration states (one account registered, one failed with 403), calling and connected states, the caller's name resolved from a contact, mute, hold and resume, two calls with hold switching, attended transfer, recording playback, account tests, the status summary, and more. The result goes to stdout, or to <prefix>-selftest.txt with --snapshot, and the app quits. --demo-state directory-selftest runs only the directory part: suggestions in the number field, caller lookup, import and merge.

timeout 60 dist/MacroSIP.app/Contents/MacOS/MacroSIP --demo-state selftest
timeout 60 dist/MacroSIP.app/Contents/MacOS/MacroSIP --demo-state directory-selftest

Strings: scripts/check-strings.sh

Checks that every key used in Model/Strings.swift exists in both en.lproj and ru.lproj (Localizable.strings or Localizable.stringsdict), that no file has keys the code does not use, that no key is in both files, that plural categories are complete (one/other for English, one/few/many/other for Russian) and that the Russian value uses no placeholders the English one lacks. Run it after every string change.

Screenshots

The images in docs/screenshots/{light,dark} and docs/screenshots/ru/{light,dark} come from demo launches with --snapshot; there is no separate capture script. A typical capture of the Russian dark variant:

timeout 30 dist/MacroSIP.app/Contents/MacOS/MacroSIP --demo-state contacts \
  --appearance dark -AppleLanguages "(ru)" --snapshot /tmp/shots/contacts --snapshot-after 3

States with a ringing call or an open sheet never quit by themselves, hence timeout. Russian text is about 25 % longer, so check the status bar, the call panel buttons and the account pop-up after adding strings. Google Contacts in the mock answers after 4 s; use --snapshot-after 3 to catch the "Searching..." row.

CI

.github/workflows/ci.yml runs on pushes and pull requests to main, with one in-progress run per ref.

Job	Runner	Steps
build	macos-15 (arm64)	restore the cache of third_party/pjsip and third_party/deps (key: hash of build-pjsip.sh and config_site.h), build-pjsip.sh, swift build, test.sh, selftest-call.sh natively and with SELFTEST_ARCH=x86_64 under Rosetta 2 (installed on demand), make-app.sh release, lipo -info on the binary; uploads the app (14 days) and the universal PJSIP build (3 days) as artifacts, and the build logs on failure
intel	macos-15-intel (x86_64), after build	downloads the universal libraries and the app (an Intel host cannot run build-pjsip.sh), verifies the signature, launches the app for 15 s with --data-dir and MACROSIP_LOG_DIR, then swift build, test.sh and selftest-call.sh natively

release.yml repeats the arm64 steps before building the DMG (see Releasing).

See also Building and Contributing.