perf(api): eliminate N+1 queries in product and asset list endpoints - #16037
Open
Jaimin2687 wants to merge 3 commits into
Open
Jaimin2687 wants to merge 3 commits into
Jaimin2687 wants to merge 3 commits into
Conversation
ProductViewSet and AssetViewSet.get_queryset() returned a bare queryset with zero prefetching, so every serialized relation was lazy-loaded per product: tags, product_meta, authorized_users, regulations, and the active-finding count — roughly 6 extra queries per product in the response. Apply the same optimization strategy the Product UI views already use: - select_related for the platform/lifecycle/origin FKs (SlugRelatedField reads .value on the related object) - prefetch_related for tags, product_meta, authorized_users, regulations - annotate active_finding_count via a correlated subquery using the project's own build_count_subquery utility — the Product.findings_count cached_property already checks for this attribute before falling back to a per-product count() The only remaining per-product query is open_findings_list(), which returns a variable-length list of finding IDs and cannot be collapsed into a scalar annotation. This is a known limitation (the model method carries a TODO comment) that requires either a PostgreSQL-specific ArrayAgg or deprecating the findings_list field to resolve. Add a regression test (test_api_product_prefetch) that creates 1 vs 5 products with full relation graphs and asserts the query-count growth equals exactly the number of extra products — proving all N+1 sources except the documented open_findings_list are eliminated.
|
This pull request modifies a sensitive authorization file by an author who is not on the allowed list. The finding is classified as low severity and is not blocking.
Configured Sensitive Codepath Modified by Non-Allowed Author in
|
| Vulnerability | Configured Sensitive Codepath Modified by Non-Allowed Author |
|---|---|
| Description | File 'dojo/authorization/query_registrations.py' matches configured sensitive codepath pattern 'dojo/authorization/*.py' and was modified by 'Jaimin2687' (commit 0b3f210) who is not in the allowed authors list. |
Comment to provide feedback on these findings.
Report false positive: @dryrunsecurity fp [FINDING ID] [FEEDBACK]
Report low-impact: @dryrunsecurity nit [FINDING ID] [FEEDBACK]
Example: @dryrunsecurity fp drs_90eda195 This code is not user-facing
All finding details can be found in the DryRun Security Dashboard.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The
/api/v2/products/and/api/v3/assets/list endpoints return a bare queryset fromget_queryset()with zero prefetching. Every serialized relation is lazy-loaded per product in the response: tags, product_meta, authorized_users, regulations, the active-finding count, and the three SlugRelatedField FKs (platform/lifecycle/origin). For a page with N products, this produces roughly 7N extra queries on top of the base query.This PR applies the same optimization strategy that the Product UI views (
dojo/product/ui/views.py:204-256) already use:select_relatedforplatform,lifecycle,origin— these are rendered viaSlugRelatedField(slug_field="value"), which reads.valueon the related object and triggers a lazy FK load withoutselect_related.prefetch_relatedfortags,product_meta,authorized_users,regulations— each fires a single bulkINquery regardless of product count.annotate(active_finding_count=...)using the project's ownbuild_count_subqueryutility (correlated subquery, no multi-table JOIN row-multiplication). TheProduct.findings_count@cached_propertyalready checks for this attribute before falling back to a per-productcount(), so the annotation satisfies it in bulk.The
open_findings_list()method still fires one query per product. It returns a variable-length list of finding IDs that cannot be collapsed into a scalar annotation without either a PostgreSQL-specificArrayAgg(risky 4-table JOIN) or deprecating thefindings_listresponse field. The model method already carries a TODO comment acknowledging this. All other per-product queries are eliminated.Test results
Added
unittests/test_api_product_prefetch.py— a regression test following the exact pattern fromtest_api_notes_nplusone.py(PR #15274):open_findings_list) — proving all other N+1 sources are eliminatedVerified locally against PostgreSQL in Docker. Ruff clean.
Checklist
dev