Dynamic qualification/setup content, improved device check, and P.804 improvements - #97
Open
babaknaderi wants to merge 111 commits into
Open
Dynamic qualification/setup content, improved device check, and P.804 improvements#97babaknaderi wants to merge 111 commits into
babaknaderi wants to merge 111 commits into
Conversation
- Add generate_math_questions.py to create math audio clips with Azure TTS, random UUID filenames, ASL normalization, and SHA-256 hashes - Add client-side math answer verification via SHA-256 hash in all 11 HTML templates (no raw answers exposed in source) - Move math answers from hardcoded cfg [math] sections to per-project CSV columns (math_ans, math_hash) flowing through general.csv - Update result_parser.py to use math_ans from input data with fallback to legacy config for backward compatibility - Add --general_assets argument to master_script.py for custom asset CSVs - Add general_assets_internal.csv template for internally generated clips - Pass math_ans and math_hash through create_input.py to per-project CSVs - Update documentation for new workflow and arguments Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Normalize every gold source clip to -26 dBov active speech before applying degradations. Replace the relative +/-25 dB loudness change with absolute active-speech targets: about -10 dBov for the too-loud case and -45 dBov for the too-quiet case. Adds active_speech_level_dbov() and normalize_active_speech_level() helpers. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
New src/utils/detect_silence_vad.py computes per-clip active speech with Silero VAD and flags clips with little or no speech. Supports a 'prescreen' mode (report silent clips in a clip-list CSV) and a 'crosscheck' mode (compare VAD against the crowd is_silent_percentage). Wire an optional --check_silence flag into master_script.py, mirroring --check_urls, and add torch and silero-vad to requirements. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep src/assets_master_script/general_assets_internal.csv out of version control; it is for internal usage only. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add a 'The clip is silent - no voice to rate' checkbox to each trial in both the training and rating sections of P808_multi.html. When ticked (after the clip has played once) it sets all seven scales to 1 via the real radios, records an always-submitted is_silent_q{n}/is_silent_t{n} flag, and lets the trial complete. A detailed note is added to the Detailed Instructions section warning that misuse leads to rejection.
result_parser.py drops silent cases (identified by the is_silent flag) before computing the MOS, guards clips left with no valid votes, and reports per-clip n_silent and is_silent_percentage columns.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The April 2023 note in the P808_multi.html ratings panel is outdated and no longer needed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add a show_qualification option to [hit_app_html] (default true), injected by master_script into P808_multi.html as showQualification. When false, the main HIT skips the qualification section entirely - useful when qualification is run as a separate study.
Update Qualification.html into a renderable template: the hearing-test number clips become {{cfg.num1_url}}..{{cfg.num5_url}} and the P.804 bandwidth/quality-discrimination test is added. Add a --qualification_only master_script mode (create_qualification_only) that renders the standalone qualification page from the general assets, skipping clip/session generation.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The rating-section gold/trapping correctness check ran in the browser and could block the HIT (the 'QA system recognized... answers known to us' message). Its correct answers were recoverable client-side - gold answers were base64 (reversible via atob), trapping answers were plain text - so the check could not be made tamper-proof for a 1-5 answer space. Blank the embedded gold/trapping answers and clip identifiers (goldClip/goldClip2 url+answers, knownQuestionUrl, knownQuestionAns), remove the answer example from the config comment, and neutralize the gold-failure warning/block at init. Gold/trapping/math validation is already performed server-side in result_parser.py. Training per-dimension feedback (training_gold_clips, knownQuestionInTraining*) is unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Make the qualification page generic (not study-specific), add the participant-group / 30-day expiry wording, switch the mother-tongue question to 10 multi-select language checkboxes, and fix the broken loudspeaker device image (use the working blob asset). In create_qualification_only: render the hearing-test clips as per-row placeholders, generate N qualification instances (--n_samples) in a single answers CSV (q_*/ans_* columns, plus the Q3-8 criteria) for server-side validation, and support --create_local_test. --method and --cfg are no longer required with --qualification_only. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Apply the same fix as P808_multi.html (commit 04e4ce7) to the other HIT templates: stop embedding the correct gold/trapping answers and clip identifiers in the served page, and neutralize the gold-failure warning/block where present. The embedded answers were recoverable client-side (gold base64, trapping plain text), so the front-end check could not be made tamper-proof; gold/trapping validation is done server-side in result_parser.py. Per template: blank knownQuestionUrl and the gold answer/identity keys (goldClipURL/goldClipAns or the goldClip/goldClip2 url+*_ans fields). DCR/CCR/P831_DCR keep their structural constant knownQuestionAns (5/0, not a secret) but blank knownQuestionUrl. bw_check.html and P835_personalized_template3.html also had the init warning/block neutralized and comment-block placeholders cleaned. Training feedback (knownQuestionInTraining*) is unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Make the 'use a headset, not the loudspeaker' rule red in the instructions/rules of every HIT template and the standalone qualification page, to reduce loudspeaker use and rejections. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add src/utils/create_bandwidth_check_clips.py to generate the P.80x qualification bandwidth-discrimination clips from clean references. Each reference yields five 'A | beep | B' clips: q1-q3 add band-limited noise (3.5/8.5/15-22 kHz at +13 dB over active speech level) to the second half (answer dq), q4/q5 carry no audible change (answer sq). Both halves come from the same source; every segment gets an independent inaudible dither so halves and the two 'same' clips are never bit-identical. Output names are anonymized UUIDs with a bandwidth_check_clips.csv manifest (ref_clip, q1-q5, ans_q1-q5). Add docs/bandwidth_check_clips.md and link it from docs/preparation.md. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add a --no_anonymize flag to create_bandwidth_check_clips.py to emit
descriptive <stem>_q{n}.wav names for listening review; the default remains
random UUID names. Document the option in docs/bandwidth_check_clips.md.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Update the create-study and analyze-results agent runbooks to always emit a batch file so the requester can re-run without re-deriving the command: - create-study writes regenerate_study.bat (re-runs master_script.py) - analyze-results writes rerun_result_parser.bat (re-runs result_parser.py) Both .bat files use %BASE% for input paths and an absolute repo path for the script. Update each agent's description accordingly and add the .bat to the create-study handoff checklist. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
result_parser.py: - Add optional --payment_per_session (reward per session/HIT, e.g. 2.10) so payment-per-hour stats can be computed for Prolific studies, which do not include the reward in their export. Takes precedence over the pre-existing (now deprecated-alias) --rewards. - Guard calc_stats against missing Answer.* columns (Answer.2_birth_year, Answer.Math, Answer.t1_ovrl, Answer.t1) so it no longer raises KeyError on Prolific batches that omit in-form demographics/sections. analyze-results agent: collect payment_per_session when the study was run on Prolific and pass --payment_per_session in the run command and the generated rerun_result_parser.bat. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…lific result_parser.py: read an optional [general] platform (mturk/prolific) from the config; when absent, infer it (prolific if a Prolific export is passed, else mturk). Skip quantity/quality bonus report generation when the platform is Prolific, since bonuses are handled on Prolific rather than via this report. master_script.py + templates: render a platform entry into every generated *_result_parser.cfg, sourced from [create_input] platform (default prolific). Agents: create-study records platform in the study cfg; analyze-results notes the bonus report is MTurk-only. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The Prolific/HIT App merge dropped Prolific rows with an empty study URL and no 32-char completion code (abandoned/returned submissions). This also dropped 'TIMED-OUT' submissions where the worker actually completed the task on the HIT App server, so their HIT App row found no match and was wrongly listed in *_not_found_in_prolific.csv. Now a Prolific row is kept when its submission id matches a completed HIT App assignment (browser_info present), and the HIT App assignment id is stripped as well as lower-cased so matching is whitespace-insensitive on both sides. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The per-run log file (*_logs.txt) used the default append-mode FileHandler, so successive runs stacked in one file and stale lines from earlier runs (e.g. an old 'not found in the Prolific data' count) lingered and misled. Open it in write mode so each run starts fresh, consistent with the other per-run outputs which are overwritten. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add a save_csv helper that writes a dataframe only when it has at least one row and removes any stale file left at the same path by a previous run. Route the report/subset outputs through it (incomplete_submissions, not_found_in_prolific, accept/reject/block/extend lists, accept_reject_gui, quantity/quality bonus) so empty files such as Batch_369_incomplete_submissions.csv are no longer created. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The flat 'Rejection reasons' list over-counts because a submission can fail several checks. Add a per-submission breakdown that reports, for rejected submissions: the marginal count per reason, how many were rejected by a single reason (e.g. removed only because of performance), and the most common reason combinations. Performance and max-hits are attributed only when the submission would otherwise have been accepted, so 'only performance' is meaningful. Writes two files: *_rejection_reason_matrix.csv (reason co-occurrence, with total and only-this-reason columns) and *_rejection_reason_combinations.csv. Store per-submission accept_failures / rejected_by_performance / rejected_by_max_hits, and make write_dict_as_csv robust to per-row key differences (union of keys + restval). Document the new outputs in the analyze-results agent. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Guard against a KeyError in evaluate_rater_performance when a batch has no usable submissions: the old check tested for 'not_used_count' but the body referenced 'used_count', which is absent in the all-not-used case. Now both count columns are ensured before computing acceptance_rate. - Look up a worker's row with an exact match instead of worker_id.str.contains(...), which does regex/substring matching and could match the wrong worker (or error on regex-special ids). Behavior on normal data is unchanged (verified: r1 still 149 performance / 271 total rejections). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…_use evaluate_rater_performance computed the per-worker rate from accept_and_use for both passes, so the reject pass (which decides payment) gated on the 'usable' rate instead of the content-QC pass rate. Workers who passed data cleansing but produced few aggregatable ratings were rejected/not paid. Now the reject pass groups by 'accept' and the use pass keeps 'accept_and_use', so QC-passing workers are paid but still excluded from the MOS when their usable rate is low. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
send_reviews_for_study now fetches each submission's current status once (via the study's submissions list, paginated) and acts accordingly: - Approvals: bulk-approve AWAITING REVIEW; individually approve accepted RETURNED/TIMED-OUT submissions (their work is used, so pay them); skip already-approved; export a CSV of any that still need manual payment. - Reject / ask-return: only submissions still AWAITING REVIEW, skipping ones already returned/rejected/approved. Adds approve_submission helper and an approved/actioned/skipped summary; falls back to per-submission status checks if the bulk fetch fails. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
For P.804 the detailed gold report previously wrote one wide row per submission holding both gold clips side by side. Explode it into one row per gold question (single gold_url per line) with flat per-dimension columns (correct/given/wrong), so a submission with two gold clips contributes two rows. Other methods keep their existing format. Update the analyze-results agent to group by gold_url. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Aggregate the P.804 per-gold rows into gold_summary.csv with one row per gold clip: url, n_submission, and the percentage of submissions that got each scale (noise/col/loud/disc/reverb/sig/ovrl) wrong. Makes bad or too-hard gold clips easy to spot. Document it in the analyze-results agent. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Include <scale>_expected next to <scale>_wrong_pct for each gold clip, so the expected (correct) answer per scale sits alongside the wrong rate. Blank when a scale is not targeted by the clip. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add a max_wrong_pct column (the worst per-scale wrong rate for each gold clip, surfaced right after n_submission) and, per scale, <scale>_mean (mean rating participants gave). Together with <scale>_expected this makes mis-keyed or too-hard gold clips obvious (expected vs mean vs wrong rate). Update the agent. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
detailed_gold_question_performance.csv and gold_summary.csv were written with bare filenames, so they landed in the current working directory instead of next to the --answers file like every other output. Prefix them with the answers base name (e.g. Batch_XXX_detailed_gold_question_performance.csv, Batch_XXX_gold_summary.csv) so their location no longer depends on the working directory. Update the analyze-results agent references. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add an optional --gold_overrides CSV (url + per-scale <scale> corrected answer and optional <scale>_var). For P.804 gold checks, listed clips use the override values and variance; clips not listed keep the answers encoded in the answers CSV. Document the workflow (review gold_summary, then override) in the agent. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Change the override semantics so that when a gold clip URL is present in the --gold_overrides file, that row fully defines its gold answers: filled scales are checked with the given value/variance, and blank scales are skipped (not checked) instead of falling back to the encoded answer. Clips not listed keep the encoded answers. Update the help text and agent doc. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… question (5 templates) Apply the same qualification updates to ACR, DCR, CCR, P835 and P835_personalized: - question 3 "mother tongue" -> Qualification.html-style .mts_group checkbox multi-select (10 languages), validated against config.qualification.mother_tongue; makeCheckboxBeRequired now handles .4_lds and .mts_group independently; - removed the "directly involved ... telephone systems / speech coding" question and its 7_working_area screen (dropped working_area_checked from the pass condition); - renumbered the remaining questions to 1..8 (these templates had a pre-existing 5,6 gap; P835_personalized followed the 1..9 layout like P808_multi). Verified by rendering each with a mock cfg and esprima-checking the scripts; CCR's remaining flag is the pre-existing DoubleClip ES2022 class-field, unrelated. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2ba2b049-11a4-4bfd-bbfa-c19035b09848
…everb/sig)
- New [hit_app_html] flag allow_cannot_rate (default false). When true, master_script sets
config.allowCannotRate and the P808_multi rating/training scales for Coloration,
Discontinuity, Reverb and Signal quality show an extra "Cannot rate it" radio (value 0),
gated by {% if cfg.allow_cannot_rate == "true" %} inside the scale templates (so it
propagates to both training and rating via the t->q clone).
- result_parser: value-0 ("cannot rate") votes are dropped from the per-scale MOS (like the
silent case) and reported per clip as n_cannot_rate / cannot_rate_percentage.
- Documented in master.cfg and docs/conf_master.md.
Only P808_multi (p804) is affected, as requested.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2ba2b049-11a4-4bfd-bbfa-c19035b09848
…tion - Relabel the per-scale option to "Cannot tell". - Add a "(?)" info marker next to it: hovering shows the explanation (title tooltip) and clicking (or Enter/Space when focused) toggles an inline help line. Wiring is a single delegated handler so it works for the dynamically generated per-trial scale blocks. - Explanation: "Select this only if other distortions (for example strong background noise) make the speech so hard to hear that you cannot judge this specific dimension." Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2ba2b049-11a4-4bfd-bbfa-c19035b09848
…archer The coloration degradation (apply_coloration) randomly applies one of three styles (muffled/resonant/telephone) and labels the clip col_ans=1 (sig_ans=1, ovrl_ans=1). The v5 study showed the perceived coloration strength varies by style/content and is not always rated as "very colored" (~13% of votes off, some clips ~60% agreement; studio P501 refs easier than real-world clips). Add a REVIEW NOTE to apply_coloration and the coloration config entry, and print a "[REVIEW REQUIRED]" reminder after generation asking the researcher to listen to each colored clip and confirm/adjust the suggested ratings before using them as gold. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2ba2b049-11a4-4bfd-bbfa-c19035b09848
Listeners perceive coloration as milder on signal quality / overall than a "1" (v5 study: sig/ovrl on colored clips were rated well above 1). Set the coloration gold type default to col_ans=1, sig_ans=2, ovrl_ans=2 so the suggested labels match perception better (raises the >=80%-agreement pass rate on the v5 colored clips from 2/33 to 17/33). Still suggested defaults - the apply_coloration review note keeps asking the researcher to confirm/adjust or drop clips whose coloration is too subtle. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2ba2b049-11a4-4bfd-bbfa-c19035b09848
Screen-out completion codes (screenout_code_qualification, screenout_code_setup) are issued per study by the crowd platform, so they must not be copied between studies. Update the create-study agent to ask for them explicitly and to exclude them from prior-study reuse: description, inputs-to-confirm list, reuse-exclusion note, config-template comment, and key rules. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2ba2b049-11a4-4bfd-bbfa-c19035b09848
The HIT app server compiles the uploaded HIT app as a jinja2 template whose
variable delimiters are ${ } (used to fill ${CMP1_A}, ${math}, screen-out
codes, etc.). A comment added for the screen-out placeholder logic contained the
literal text ${...}, which the server's jinja2 parsed as the expression '...'
and raised TemplateSyntaxError: unexpected '.', breaking project creation.
Reword the comment to drop the literal ${...} in all six templates (ACR, DCR,
CCR, P835, P835_personalized, P808_multi). The runtime token build
("$" + "{" + codeVar + "}") is unaffected since it has no adjacent ${.
Verified: the regenerated P.804 HIT app compiles cleanly under
jinja2.Environment(variable_start_string='${', variable_end_string='}').
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2ba2b049-11a4-4bfd-bbfa-c19035b09848
write_dict_as_csv opened the output file without an explicit encoding, so on Windows it used the cp1252 default and raised UnicodeEncodeError when worker data (e.g. mother-tongue or free-text fields) contained non-cp1252 characters, aborting the whole result-parser run. Pass encoding='utf-8' so reports are written as UTF-8 on every platform. Verified end-to-end on a 2018-submission P.804 batch under the default cp1252 locale (no PYTHONUTF8 workaround needed). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2ba2b049-11a4-4bfd-bbfa-c19035b09848
The "N submissions are not found in the HITAPP server" line now also reports how many of those were legitimately SCREENED OUT on Prolific, e.g. "263 ... From them 255 were SCREENED OUT on Prolific." The count uses the same status normalization as the later separation step, so the two log lines stay consistent. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2ba2b049-11a4-4bfd-bbfa-c19035b09848
len(screened_out) counts rows of not_in_hitapp (deduped by prolific_submission_id), i.e. submissions, not unique participants. Reword the log line accordingly so it is consistent with the "N submissions ... From them X were SCREENED OUT" line. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2ba2b049-11a4-4bfd-bbfa-c19035b09848
Set rater_min_acceptance_rate_current_test to 0 in the [acceptance_criteria] section of the p804 result-parser template, so a low current-test acceptance rate alone no longer fails (rejects) a submission by default. This matches the pp835 template (already 0) and the code fallback default (result_parser.py); acr and dcr_ccr templates omit the key and already fall back to 0. The [accept_and_use] threshold (50) is unchanged, so low-acceptance raters are still excluded from MOS. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2ba2b049-11a4-4bfd-bbfa-c19035b09848
Previously the _setup cookie was created as soon as a participant typed a math answer (userAnsweringToSetupQuestions), and passDeviceCheck / unlockAfterQualification unlocked the detailed-instruction, training and rating sections outright. So a wrong math (or CMP) answer did not block progress: the participant could complete and submit the HIT, and only result_parser rejected it offline. Now the setup is a real gate: - add unlockAfterSetup() (unlocks detailed instruction, training, rating); - passDeviceCheck / unlockAfterQualification unlock only #setup, and unlock the rest only when the _setup cookie already exists (setup already passed within its window); - the setup pass branch (validateCMPSectionAfterMath) calls unlockAfterSetup(); - userAnsweringToSetupQuestions no longer creates the _setup cookie. Combined with the existing two-attempt screen-out (setupCheckFailed), a participant can only continue with a correct setup answer, and a second wrong answer screens them out. Applied to the six device-check study templates (ACR, DCR, CCR, P835, P835_personalized, P808_multi). Verified: esprima syntax clean (the CCR ES2022 class-field warning is pre-existing), no undefined is_test references, and the regenerated P.804 HIT app both contains the new gating and compiles as a server-side (\) Jinja2 template. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2ba2b049-11a4-4bfd-bbfa-c19035b09848
is_test was hardcoded to false and only OR'd into three cookie checks in the load init (details-instruction, training, setup), so it never changed behavior. Drop the definition and the "is_test ||" guards; the checks now read the cookies directly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2ba2b049-11a4-4bfd-bbfa-c19035b09848
transform() computes n_cannot_rate / cannot_rate_percentage for each clip (the P.804 "Cannot tell" value-0 votes, dropped from the MOS), but create_headers_for_per_file_report did not list these columns, so write_dict_as_csv (extrasaction='ignore') silently dropped them from every votes_per_clip_<scale>.csv. Add both columns next to the existing n_silent / is_silent_percentage for p835, echo_impairment_test, p835_personalized and p804. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2ba2b049-11a4-4bfd-bbfa-c19035b09848
…-scales Drop the redundant n_silent / n_cannot_rate count columns from the per-clip reports (n already gives the valid-vote count) and keep only the percentages. Reorder so the per-clip silent share sits next to n and the per-scale cannot-rate share sits next to that scale's MOS. Carry the per-scale cannot_rate_percentage into votes_per_clip_all-scales.csv (renamed cannot_rate_percentage_<scale>, next to each MOS), with the per-clip silent share kept once next to n. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2ba2b049-11a4-4bfd-bbfa-c19035b09848
master_script.py's --general_assets selects the setup-section reference material (math, comparison pairs, hearing test, bandwidth-check clips): the internal set (general_assets_internal.csv) or the default general/public set (general.csv, used when the flag is omitted). Update the create-study agent to prompt for this choice: extend the description, add it as input 11 in "Inputs the agent must confirm", and note how to pass --general_assets in the master_script invocation step. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2ba2b049-11a4-4bfd-bbfa-c19035b09848
Screened-out participants cannot submit the form. Reword both paid blocks (qualification and setup) across all six device-check templates. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2ba2b049-11a4-4bfd-bbfa-c19035b09848
Some high-gain loudness clips can be loud; advise a comfortable, moderate volume. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2ba2b049-11a4-4bfd-bbfa-c19035b09848
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2ba2b049-11a4-4bfd-bbfa-c19035b09848
Reject rows with no Prolific status (result_parser's "no completed HIT App task" rows) were skipped. Resolve status via API when unknown, and always ask-to-return the no-task ones (even under --force_reject). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2ba2b049-11a4-4bfd-bbfa-c19035b09848
- conf_master: "Cannot rate it" -> "Cannot tell"; drop removed n_cannot_rate column - gold_clips: coloration gold defaults sig/ovrl=2; add review + --gold_overrides note - prep_p804: fix --training_gold_clips filename and _p804_ result-parser cfg name - prep_p835: fix _p835_ result-parser cfg name - results: add gold/rejection/screened-out outputs, P.804 per-scale note, echo method - README: fix setup typo Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2ba2b049-11a4-4bfd-bbfa-c19035b09848
Add device_check_dispute_max_net_db (default 30): a mismatch may only be disputed when net_db is near the threshold; a confident detection cannot be disputed away (switch device + retry, or screen out). Wired through master_script + all 6 device- check templates; log dispute_max to webrtc_raw. Docs updated. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2ba2b049-11a4-4bfd-bbfa-c19035b09848
There was a problem hiding this comment.
Pull request overview
This PR introduces a broad set of improvements to the P.808 toolkit’s study generation, templates, and parsing utilities—especially around dynamic qualification/setup assets, objective listening-device checks, and P.804/P.835 workflow updates.
Changes:
- Adds multiple new utility CLIs for qualification/setup asset generation (bandwidth-check clips, JND setup pairs, math questions) and audio validation (Silero VAD silence detection).
- Updates HIT templates and generation logic to support hashed client-side verification and more configurable screening/setup behavior.
- Enhances project tooling/docs (parser templates, master cfg options, docs, and agent runbooks) and introduces new runtime dependencies (
torch,silero-vad).
Reviewed changes
Copilot reviewed 47 out of 50 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| src/utils/scale_result_parser.py | New Scale API result parsing utility for ACR/P.835/echo result exports |
| src/utils/scale_audio_rating.py | Makes src/ importable when running the util directly |
| src/utils/merge_jnd_sources.py | New tool to merge split speech parts with silence gaps for JND sources |
| src/utils/generate_math_questions.py | New Azure TTS-based generator for stereo math-question clips + manifests |
| src/utils/detect_silence_vad.py | New VAD-based silence/no-speech detection + crosscheck tooling |
| src/utils/create_jnd_check_clips.py | New generator for anonymized JND setup A/B pairs + explicit answers |
| src/utils/create_bandwidth_check_clips.py | New generator for bandwidth-check (A|beep|B) qualification clips |
| src/requirements.txt | Adds dependencies for VAD (torch, silero-vad) |
| src/prolific_utils.py | Removes old Prolific utility at src/ root (relocation implied) |
| src/P808Template/Qualification.html | Updates qualification copy and adds bandwidth-check section + dynamic placeholders |
| src/P808Template/P835_template_one_audio.html | Disables embedded TP/gold placeholders; adds math hash verification; rule copy tweaks |
| src/P808Template/P831_DCR_template.html | Adds math hash verification and CMP answer hashing updates |
| src/P808Template/P831_ACR_template.html | Adds math hash verification and CMP answer hashing updates |
| src/P808Template/echo_impairment_test_template.html | Adds math hash verification and CMP answer hashing updates |
| src/P808Template/echo_impairment_test_fest_template.html | Adds math hash verification and CMP answer hashing updates |
| src/P808Template/bw_check.html | Disables front-end gold QC gating; clears embedded TP/gold placeholders; rule copy tweak |
| src/master_script.py | Adds stripping author lines, dynamic assets options, screen-out codes, device-check config, qualification-only mode |
| src/create_input.py | Adds support for math_ans/math_hash, bandwidth-check set sampling, and CMP offline answer columns |
| src/create_gold_clips.py | Refines P.804 gold generation logic (loudness split high/low, ASL normalization, delayed effects, updated defaults) |
| src/configurations/master.cfg | Documents new HIT-app options (screen-out codes, online eval toggles, device check, cannot-rate for P.804) |
| src/assets_master_script/pp835_result_parser_template.cfg | Adds platform + updates math guidance |
| src/assets_master_script/p804_result_parser_template.cfg | Adds platform, updates math guidance, adjusts acceptance-rate threshold |
| src/assets_master_script/general.csv | Extends general assets with math answers/hashes + bandwidth-check clips/answers/hashes |
| src/assets_master_script/dcr_ccr_result_parser_template.cfg | Adds platform + updates math guidance |
| src/assets_master_script/acr_result_parser_template.cfg | Adds platform + updates math guidance |
| README.md | Fixes wording/typos in setup text |
| docs/setup_jnd_clips.md | New documentation for setup JND clips workflow |
| docs/running_test_mturk.md | Updates paths to moved utils scripts |
| docs/results.md | Expands/updates outputs and platform-specific notes; updates utils paths |
| docs/preparation.md | Adds links to new utility-doc pages |
| docs/prep_p835.md | Documents --general_assets and corrects output cfg filename |
| docs/prep_p804.md | Fixes file naming and documents --general_assets and correct output cfg filename |
| docs/prep_dcr_ccr.md | Documents --general_assets |
| docs/prep_acr.md | Documents --general_assets |
| docs/gold_clips.md | Updates P.804 gold-type table and notes to match new generation logic |
| docs/general_res.md | Updates general assets guidance for math answers/hashes |
| docs/conf_master.md | Adds CLI args and documents new master-script config flags |
| docs/bandwidth_check_clips.md | New documentation for bandwidth-check clip generation |
| .gitignore | Ignores internal-only general_assets_internal.csv |
| .github/agents/create-study.agent.md | Updates agent runbook to prompt for screen-out codes and internal-vs-general assets choice |
| .github/agents/analyze-results.agent.md | Updates agent runbook to include Prolific payment and rerunnable batch script guidance |
Comment on lines
513
to
517
| correct_ans = 0 ; | ||
| for (i = 0; i < 4; i++) { | ||
| if (cmp_answers[i] == "o") | ||
| continue; | ||
| url = $('span[id="CMP{0}_{1}"]'.f(i+1, cmp_answers[i].toUpperCase())).attr("data-src"); |
Comment on lines
473
to
477
| correct_ans = 0 ; | ||
| for (i = 0; i < 4; i++) { | ||
| if (cmp_answers[i] == "o") | ||
| continue; | ||
| url = $('span[id="CMP{0}_{1}"]'.f(i+1, cmp_answers[i].toUpperCase())).attr("data-src"); |
Comment on lines
521
to
525
| correct_ans = 0 ; | ||
| for (i = 0; i < 4; i++) { | ||
| if (cmp_answers[i] == "o") | ||
| continue; | ||
| url = $('span[id="CMP{0}_{1}"]'.f(i+1, cmp_answers[i].toUpperCase())).attr("data-src"); |
Comment on lines
521
to
525
| correct_ans = 0 ; | ||
| for (i = 0; i < 4; i++) { | ||
| if (cmp_answers[i] == "o") | ||
| continue; | ||
| url = $('span[id="CMP{0}_{1}"]'.f(i+1, cmp_answers[i].toUpperCase())).attr("data-src"); |
…odes Qualification "Check answers" now asks to complete all questions before grading. Merge screenout_code_qualification/_setup into one screenout_code. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2ba2b049-11a4-4bfd-bbfa-c19035b09848
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Large set of improvements to the crowdsourcing study templates, master script, and result parser.
Qualification & setup (dynamic content)
master_scriptprefers explicitpair_ansover the legacy SNR-in-filename heuristic.screenout_code_qualificationandscreenout_code_setup.Objective listening-device check
net_db(probe rise minus control-band rise); beep-count audibility check catches muted/too-quiet devices; gray-zone dispute policy.required_playback_device= headset | loudspeaker | any (anyskips the check). Configurable threshold, probe gain, dispute band.result_parser.P.804
result_parser
Tooling & structure
create_bandwidth_check_clips,create_jnd_check_clips,merge_jnd_sources,generate_math_questions,detect_silence_vad.src/utils/(history preserved).Co-authored-by: Copilot