Investigating and fixing a nasty clone bug

Summary

Recently I found myself battling with another nasty bug. It took me several hours to understand what is happening, and once I found it, it turned out that the cause of the bug is relevant to the Ergonomic cloning initiative that is currently being discussed a lot. So I thought that it would be a good candidate for a blog post. Context I’m currently trying to finally bring the bors GitHub merge bot to production, so I spent many days in the past weeks fixing various possible race conditions, making its implementation more robust and improving its test suite. The bors test suite is mostly integration-based; it has very few what you could call “unit tests”. In most tests, we spawn the whole bors web application, connect it to a real running instance of Postgres, apply database migrations, send HTTP requests that emulate GitHub webhooks to it, and let it communicate over the network with a fake HTTP server that emulates GitHub endpoints. The mocked HTTP GitHub endpoints are implemented using the cool wiremock crate. A typical bors test looks something like this: #[sqlx::test] async fn try_build_failed_modify_labels(pool: sqlx::PgPool) { let gh = GitHub::default().with_default_config( r#" [labels] try_failed = ["+foo", "+bar", "-baz"] "#, ); run_test((pool, gh), async |ctx: &mut BorsTester| { ctx.post_comment("@bors try").await?; insta::assert_snapshot!(ctx.get_next_comment_text(()).await?, @r" :hourglass: Trying commit pr-1-sha with merge merge-0-pr-1… To cancel the try build, run the command `@bors try cancel`. "); ctx.pr(()).await.expect_added_labels(&[]); ctx.workflow_full_failure(ctx.try_workflow()).await?; ctx.expect_comments((), 1).await; ctx.pr(()) .await .expect_added_labels(&["foo", "bar"]) .expect_removed_labels(&["baz"]); Ok(()) }) .await; } Where’s the body? While doing a particularly big refactoring, a few tests started failing in a very weird way, by panicking in one of the mocked GitHub endpoints. It was a PATCH endpoint that was trying to deserialize a J...