Develop - #235
Merged
Merged
Conversation
Aedna/projects
There was a problem hiding this comment.
Pull request overview
This pull request expands the Neotoma API with dataset-linked aeDNA assay/library metadata and project/participant metadata endpoints, while also enriching existing site/sequence payloads and strengthening automated post-deploy test execution.
Changes:
- Added new dataset endpoints for aeDNA assays (
/v2.0/data/datasets/{datasetid}/assays) and projects (/v2.0/data/datasets/{datasetid}/projects) with new SQL + helper handlers. - Enhanced existing outputs by adding
recdatecreatedto site queries/schemas and addingasvto aeDNA sequence queries. - Improved CI/test automation via a new GitHub Actions workflow plus updates to test-generation/execution scripts and environment targeting.
Reviewed changes
Copilot reviewed 22 out of 24 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| v2.0/routes/data.js | Wires new dataset routes for assays/projects into the v2.0 router. |
| v2.0/handlers/data_handlers.js | Adds handler entry points delegating to new helpers. |
| v2.0/helpers/assays/assays.js | Implements dataset assays handler with fallback for missing aeDNA tables. |
| v2.0/helpers/assays/assaysbydataset.sql | Adds SQL to return assay + library metadata for a dataset. |
| v2.0/helpers/projects/projects.js | Implements dataset projects handler returning projects + participants. |
| v2.0/helpers/projects/projectsbydataset.sql | Adds SQL to return project + participant metadata for a dataset. |
| v2.0/helpers/aedna/sequences.sql | Adds asv field to dataset-grouped aeDNA sequences output. |
| v2.0/helpers/aedna/sequencesbytaxon.sql | Adds asv field to sequences-by-taxon output. |
| v2.0/helpers/sites/sitequeryfaster.sql | Adds recdatecreated to site query output. |
| v2.0/helpers/sites/sitebyid.sql | Adds recdatecreated to site-by-id output. |
| v2.0/helpers/sites/sitebygpid.sql | Adds recdatecreated into JSON site payload. |
| v2.0/helpers/sites/sitebydsid.sql | Adds recdatecreated to site-by-dataset output and normalizes whitespace. |
| v2.0/helpers/sites/sitebyctid.sql | Adds recdatecreated into JSON site payload. |
| openapi/paths/v20/data.yml | Documents new assays/projects endpoints in the v2.0 OpenAPI source paths. |
| openapi/components/schemas/schemas.yml | Adds recdatecreated to the sitesv2 schema. |
| openapi.yaml | Updates the consolidated OpenAPI spec with the new paths and recdatecreated. |
| v2.0/helpers/validation/newlogin.sql | Extends session lifetime and returns expiresat alongside sessionuuid. |
| v2.0/helpers/validation/validateuser.js | Updates module header documentation describing ORCID token → session flow. |
| .github/workflows/test.yml | Adds a post-deploy (and manual) workflow to generate tests and run them against dev/prod. |
| package.json | Makes yarn test honor APIPATH (with a default) for environment targeting. |
| runmochabatch.sh | Adds -l for localhost dev targeting and updates test execution commentary. |
| genoatt.sh | Adds scheme selection for generated tests to handle http→https redirects on remote hosts. |
Suppressed comments (2)
openapi/paths/v20/data.yml:362
- The OpenAPI response schema for this endpoint is currently just
type: object, but the handler returns a{status, data, message}wrapper withdata.datasetidanddata.projects. Please describe the response shape so clients and generated tests can validate it.
schema:
type: object
openapi.yaml:3376
- Like openapi/paths/v20/data.yml, this generated OpenAPI spec defines the response schema as just
type: object. The endpoint returns{status, data, message}withdata.datasetidanddata.projects, so the schema should reflect that contract.
schema:
type: object
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+345
to
+346
| schema: | ||
| type: object |
Comment on lines
66
to
72
| json_build_object( | ||
| 'sequenceid', sd.sequenceid, | ||
| 'sequence', sd.sequence, | ||
| 'asv', sd.asv, | ||
| 'model', sd.model, | ||
| 'primername', sd.primername, | ||
| 'publicationdoi', sd.publicationdoi |
Comment on lines
1
to
7
| SELECT | ||
| sq.sequenceid, | ||
| sq.sequence, | ||
| sq.asv, | ||
| am.model, | ||
| sq.primername, | ||
| pb.doi AS publicationdoi |
Comment on lines
+5
to
+10
| // If ORCID recognises it, we record a login in ap.orcidlogins, which mints an | ||
| // opaque session UUID (see newlogin.sql), and send that UUID back to the caller. | ||
| // The session UUID is the credential for every authenticated request afterwards, | ||
| // passed as `Authorization: Bearer <sessionuuid>` and checked by sessionauth.js. | ||
| // Session lifetime is set in newlogin.sql — currently 1 week — and its expiry is | ||
| // returned alongside the UUID so clients don't have to guess. |
Comment on lines
+13
to
15
| -l run the tests against the LOCAL DEV server at localhost:3005 (NODE_ENV=development) | ||
| -d run the tests against the remove development server at api-dev.neotomadb.org | ||
| -p run the tests against the remove production server at api.neotomadb.org |
Comment on lines
+3359
to
+3360
| schema: | ||
| type: object |
Comment on lines
24
to
+28
|
|
||
| run_oatt() { | ||
|
|
||
| rm ./test/v*.js | ||
| oatts generate --host $remote -s ./openapi.yaml -w test | ||
| oatts generate --host $remote -m $scheme -s ./openapi.yaml -w test |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces new endpoints to the Neotoma API for retrieving project and aeDNA assay metadata linked to datasets, improves test automation, and enhances the site and sequence data models with additional fields. It also includes various fixes and improvements to the test scripts and SQL queries.
New API Endpoints and Handlers:
/v2.0/data/datasets/{datasetid}/assaysendpoint and supporting handler to return aeDNA assay and library metadata for a dataset. [1] [2] [3] [4] [5]/v2.0/data/datasets/{datasetid}/projectsendpoint and supporting handler to return project and participant metadata for a dataset. [1] [2] [3] [4] [5]Test Automation Improvements:
.github/workflows/test.ymlto automatically test the API after deployments to thedevelopbranch and on manual dispatch.genoatt.sh,runmochabatch.sh,package.json) to support multiple environments, correct protocol usage, and more flexible test running. [1] [2] [3] [4] [5] [6]Data Model and Query Enhancements:
recdatecreatedfield to site-related schemas and queries, making creation dates available in API responses. [1] [2] [3] [4] [5] [6] [7]asv(amplicon sequence variant) field in aeDNA sequence queries and responses. [1] [2] [3]Other Fixes and Improvements:
These changes add important new dataset metadata endpoints, improve test coverage and reliability, and enhance the richness of site and sequence data returned by the API.