Send messages to docs.rs after crate actions by LawnGnome · Pull Request #14188 · rust-lang/crates.io

LawnGnome · 2026-07-09T23:10:21Z

This is a sketch of how we might close #14111 by sending messages to docs.rs. It is best reviewed commit-by-commit.¹

I do not expect to merge this in its current form; this draft PR is primarily to gather feedback.

How it works

Very broadly, we need to do the following:

Refactor the existing configuration logic used for CdnLogQueueConfig to be generic so we can reuse it for the docs.rs queue.
Add a docs.rs queue.
Add message sending functionality to SqsQueue.
Change the SyncToGitIndex job to include the event that triggered the sync², which can then be turned into a message.
Actually send the message.

Open questions

I'm not particularly set on this particular implementation. Things I'd like feedback on include:

Should this live in the SyncToGitIndex job at all?
1. If not, should this be a job triggered by SyncToGitIndex, or should this be decoupled from the Git index sync altogether?³
How do we alert on this when sending a message fails, and what happens then?
Are the new SqsQueueImpl tests actually useful?
Similarly, is the smoke test now too big and clunky to be useful in its current form?⁴

Other prerequisites and testing

This depends on rust-lang/simpleinfra#1099. Once that's merged, we could deploy this (or whatever comes out of this) to staging and start testing with a stub consumer to ensure that we get the messages we expect to see.

I'm unsure if the docs.rs team wants to actually publish the docs_rs_crates_io crate, but if they do, then that would also need to happen.

I expect that, in practice, we might want to split the SQS commits into a separate PR since they're technically a separate refactor. ↩
There are still cases where we won't have a trigger that's relevant to docs.rs, most notably manual syncs to the index initiated via crates-admin. ↩
The main motivator behind tying this to the Git index sync is that docs.rs uses the Git index to pull crate metadata, so having the message only be sent after the Git index has already been updated means that docs.rs doesn't need to care about whether the update has occurred. ↩
In particular, I don't love how big that function has now gotten with the usage of MockSqsQueue. ↩

Rather than special casing the CDN log queue configuration, we make the configuration shared so we can reuse it for the docs.rs queue configuration. One of `config::queue` and `sqs` ends up needing to know about the other. Since `config` modules are generally simple, this has been implemented in `sqs`, which means that it is no longer self-contained, but does mean that we follow the same general pattern as other service clients.

As with the other trait methods, this is a thin wrapper around `aws_sdk_sqs`.

I have some mixed feelings on the utility of these tests, since they're really just validating that the very thin `SqsQueueImpl` methods are passing the right things between the caller and `aws_sdk_sqs`, but :shrug:.

This will be used to build the appropriate SQS message to docs.rs when set.

carols10cents · 2026-07-13T15:30:08Z

+        (**self).receive_messages(max_messages).await
+    }
+
+    async fn send_message(&self, message: String) -> anyhow::Result<SendMessageOutput> {


Nit: As of this commit, the SqsQueue trait doesn't have the send_message method on it. That is, if you check out this commit, I don't think it will build.

View changes since the review

yeah, I think this was meant to be introduced two commits later in the "Add SqsQueue::send_message" commit

carols10cents · 2026-07-13T15:40:36Z

 diesel-async = { version = "=0.9.2", features = ["async-connection-wrapper", "deadpool", "postgres"] }
 diesel_full_text_search = "=2.3.0"
 diesel_migrations = { version = "=2.3.2", features = ["postgres"] }
+docs_rs_crates_io = { git = "https://github.com/rust-lang/docs.rs.git", rev = "1604301c72411fbc63082434295fa45036d982b7", version = "0.1.0" }


just pointing out for other reviewers that the docs_rs_crates_io crate and this rev are from rust-lang/docs.rs#3357

View changes since the review

and I assume we won't rely on a git dependency once we merge this, right? 😅

@LawnGnome does this imply that the message schema is fine now? So we can merge / release in docs.rs?

and I assume we won't rely on a git dependency once we merge this, right? 😅

Just to have it written down: yes of course.

carols10cents · 2026-07-13T15:49:16Z

🤷🏻‍♀️ seems fine to me

Turbo87 · 2026-07-14T12:46:09Z

+        (**self).receive_messages(max_messages).await
+    }
+
+    async fn send_message(&self, message: String) -> anyhow::Result<SendMessageOutput> {


yeah, I think this was meant to be introduced two commits later in the "Add SqsQueue::send_message" commit

Turbo87 · 2026-07-14T12:59:27Z

 diesel-async = { version = "=0.9.2", features = ["async-connection-wrapper", "deadpool", "postgres"] }
 diesel_full_text_search = "=2.3.0"
 diesel_migrations = { version = "=2.3.2", features = ["postgres"] }
+docs_rs_crates_io = { git = "https://github.com/rust-lang/docs.rs.git", rev = "1604301c72411fbc63082434295fa45036d982b7", version = "0.1.0" }


and I assume we won't rely on a git dependency once we merge this, right? 😅

Turbo87 · 2026-07-14T13:02:31Z

+    /// The triggering event that led to the sync.
+    ///
+    /// Once the index has been synced, this event will be sent to docs.rs via
+    /// SQS.
+    #[serde(default)]
+    trigger: Option<Trigger>,


unfortunately I think this defeats the purposed of the const DEDUPLICATED: bool = true; below. we explicitly introduced the deduplication to avoid having to sync every update in isolation to the indexes because it was causing performance problems. if we include the triggers then either we don't deduplicate anymore, or we keep deduplicating but docs.rs might miss deduplicated events, or both.

it might also block us from potentially batching index updates in the future for related reasons.

does docs.rs really need this information? we can easily give them the added/updated/deleted information, but the yanked/unyanked/version-deleted would be slightly more complicated since we might combine multiple of these in a single commit.

View changes since the review

Turbo87 · 2026-07-14T13:09:38Z


        let crate_name = self.krate.clone();
        let mut conn = env.deadpool.get().await?;
+        let queue = env.docs_rs_queue.clone();


we should probably make this queue optional, so that it does not break the publishing flow when developing locally without having a queue configured. the default-to-mock behavior for the other queue was okayish in the past because the corresponding background job that used the queue had to be triggered manually anyway, but this one is triggered by the regular publish flow and yank/unyank. this might involve changing QueueConfig::from_env() to return Result<Option<Self>> or something like that.

View changes since the review

Turbo87 · 2026-07-14T13:15:30Z

+            // TODO: is this alertable? How can we notify docs.rs that rebuilds
+            // may be required?
+            error!(
+                krate.name = &self.krate,
+                trigger = ?self.trigger,
+                error = %e,
+                "error notifying docs.rs",
+            );


you could do something like warn!(target: "docs_rs_queue", ...) which makes it easy to filter on in Datadog

I'd also recommend to not use structured logging too much unless you plan on filtering by these fields, and including the fields in the log message too, so that not every log row on Datadog reads "error notifying docs.rs" without any context. something like "Failed to notify docs.rs about {trigger} update on {krate} crate: {error}" is much easier to comprehend at a glance without having to dig into the details of every log event (see https://github.com/rust-lang/crates.io/blob/main/docs/LOGGING.md) :)

View changes since the review

aren't crates.io logs json logs? Datadog makes using them (also making then nicely viewable) quite easy.

ah, sorry, the logging.md explains more

syphar · 2026-07-17T15:37:08Z

+
+        // This is implemented inline for now, but we may want to enqueue
+        // another job to handle sending the actual messages.
+        if let Err(e) = self.notify_docs_rs(queue).await {


do the crates.io jobs have retries?

That would probably help catching the most temporary issues.

View changes since the review

do the crates.io jobs have retries?

yes

But I assume they wouldn't kick in if the error is consumed like it is right now?

Could be a good reason for the separate job for the msg publish later

syphar · 2026-07-17T15:38:22Z

+            // We won't consider this fatal right now, but we do need to know
+            // what failed so we can tell docs.rs.
+            //
+            // TODO: is this alertable? How can we notify docs.rs that rebuilds


generally I like "self-healing" designs,

in this case perhaps a scheduled "repair" job based on the git index, or the crates.io DB?

( or relying less on crates.io data / sync generally).

I'm not sure yet how I would handle the "live" SQS events vs a (less often / hourly) updated git index, and the repair job.

View changes since the review

Could the live SQS event include the commit hash from the get index? "Version 1.2.3 of Crate foo Was published into the git index at commit hash 1243534534". Then it should be possible To write an accurate repair job.

Pause reading real time events.

Check out the index as of the most recently processed real time event.

Check that everything in the index (or all the changes since the last repair) have been correctly ingested.

Go back to reading real time events.

I can dredge up some horror stories of what it's actually like trying to run a service that has both real time ingestion from registries while also having occasional repair tasks from the same registries when there is no way to correlate between the real time events and their snapshots. But I'd rather not have to put them in writing. I'd prefer to hop on a call.

From what I remember, at some point the git index will be updated less often (hourly?) and we will still get the SQS events directly from crates.io.

rustbot · 2026-07-20T07:15:43Z

☔ The latest upstream changes (possibly #14275) made this pull request unmergeable. Please resolve the merge conflicts.

syphar · 2026-08-27T05:56:27Z

 diesel-async = { version = "=0.9.2", features = ["async-connection-wrapper", "deadpool", "postgres"] }
 diesel_full_text_search = "=2.3.0"
 diesel_migrations = { version = "=2.3.2", features = ["postgres"] }
+docs_rs_crates_io = { git = "https://github.com/rust-lang/docs.rs.git", rev = "1604301c72411fbc63082434295fa45036d982b7", version = "0.1.0" }


This is released now:
https://docs.rs/docs_rs_crates_io/latest/docs_rs_crates_io/events/index.html

View changes since the review

LawnGnome added 8 commits July 9, 2026 15:41

Add the docs.rs queue.

c3f6bea

Add SqsQueue::send_message.

b4cc4ff

As with the other trait methods, this is a thin wrapper around `aws_sdk_sqs`.

Add tests for SqsQueueImpl.

81ea3b3

I have some mixed feelings on the utility of these tests, since they're really just validating that the very thin `SqsQueueImpl` methods are passing the right things between the caller and `aws_sdk_sqs`, but :shrug:.

Cargo.toml: depend upon docs_rs_crates_io crate

d9828b1

Change SyncToGitIndex to allow for an optional trigger.

bdc2667

This will be used to build the appropriate SQS message to docs.rs when set.

Update callers of SyncToGitIndex::new to include trigger metadata.

2b70329

Add SQS expectations to the Git index smoke test.

8d1389a

LawnGnome requested a review from Turbo87 July 9, 2026 23:10

LawnGnome added this to crates.io team meetings Jul 9, 2026

rustbot added the A-backend ⚙️ label Jul 9, 2026

Turbo87 moved this to For next meeting in crates.io team meetings Jul 10, 2026

carols10cents reviewed Jul 13, 2026

View reviewed changes

Turbo87 reviewed Jul 14, 2026

View reviewed changes

syphar reviewed Jul 17, 2026

View reviewed changes

syphar reviewed Aug 27, 2026

View reviewed changes

Uh oh!

Uh oh!

Conversation

LawnGnome commented Jul 9, 2026 • edited by carols10cents Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

How it works

Open questions

Other prerequisites and testing

Footnotes

Uh oh!

carols10cents Jul 13, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

carols10cents Jul 13, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

carols10cents commented Jul 13, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Turbo87 Jul 14, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Turbo87 Jul 14, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Turbo87 Jul 14, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

syphar Jul 17, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

syphar Jul 17, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Eh2406 Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rustbot commented Jul 20, 2026

Uh oh!

syphar Aug 27, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

LawnGnome commented Jul 9, 2026 •

edited by carols10cents

Loading

carols10cents Jul 13, 2026 •

edited by rustbot

Loading

carols10cents Jul 13, 2026 •

edited by rustbot

Loading

Turbo87 Jul 14, 2026 •

edited by rustbot

Loading

Turbo87 Jul 14, 2026 •

edited by rustbot

Loading

Turbo87 Jul 14, 2026 •

edited by rustbot

Loading

syphar Jul 17, 2026 •

edited by rustbot

Loading

syphar Jul 17, 2026 •

edited by rustbot

Loading

Eh2406 Jul 17, 2026 •

edited

Loading

syphar Aug 27, 2026 •

edited by rustbot

Loading