Slow automatic analysis of GitHub PRs are still present

[alexypuli (Alexy Pulivelil)]

Hi,
Still we are experiencing slower automatic analysis. According to this Slowed automatic analysis of GitHub PRs we have monitored the issues and happy to say that the issue was not present last week. But starting today we are experiencing slower analysis.
Hope you could look into this

Thanks,
Alexy P Thomas

[ganncamp (G Ann Campbell)]

Hi Alexy,

Can you put some qualifiers around this, please?

• When did this start?

• How slow is slow? What was the analysis time before, and what is it now?

• Can you characterize - by language? by size? - the projects that are seeing the slowdown?

• Are the projects involved public or private? And if they’re public, can you point us to them?

 
Thx,
Ann

[alexypuli (Alexy Pulivelil)]

Hi @ganncamp ,
Sure, I will list out the details below

• First it occurred yesterday morning, time - 10:00AM IST, but today more developers reported this issue.

• For some repos It took 10 minitues, for some other more than that

• I can’t categorise with respect to language, but the PR’s of the repos include changes in .py, .sql, .tsx etc

• None of our projects are public

Hope I have given more clarity on the issue.

Thanks

[raulmarindev (Raúl Marín)]

We are seeing this behavior too.

• When did this start? Today 10:00 Central European Summer Time
• How slow is slow? What was the analysis time before, and what is it now? Yesterday it was taking a couple of minutes, now is taking more than 10.
• Can you characterize - by language? by size? - the projects that are seeing the slowdown? Python files mostly, 60 files
• Are the projects involved public or private? And if they’re public, can you point us to them? The project is private

[ganncamp (G Ann Campbell)]

Hi guys,

Thanks for the details. GitHub had an incident with PRs this morning that they reported resolved about 30min ago.

Are y’all still seeing a slowdown? Or does it seem resolved?

Thanks for your patience with all my questions.

 
Ann

[alexypuli (Alexy Pulivelil)]

Hi,

Attaching a screenshot below to show the time difference between last commit and last analysis check. So here the analysis happened after 20 minutes of adding a commit.
We will monitor from our end for 24hr and let you know about it.

Thanks

[ganncamp (G Ann Campbell)]

Hi guys,

Thanks for your reports. They helped us find this:

Automatic analysis slowdown processing GitHub PRs

Hi all, A change we made on Tuesday resulted in a slowdown for automatic analysis on GitHub PRs. A fix will be deployed tomorrow morning (28 July). In the meantime, we’ve temporarily increased the number of concurrent workers for this job to clear the processing backlog. We appreciate your patience.   Ann

You should see this start to get better soon, with full resolution by, let’s say mid-morning in Europe.

 
Thx,
Ann

[alexypuli (Alexy Pulivelil)]

Hi,

Thanks, Will monitor from our end.

[Colin]

Automatic analysis slowdown processing GitHub PRs

Hello everyone! Thank you for your patience. The issue should be resolved and the queue cleared of pending tasks. Please let us know if you face any further slowdown.

[alexypuli (Alexy Pulivelil)]

Hi,
Happy to say that the automatic analysis are back on track with full power. Thanks a lot.

Is it possible for you to mention how you have rectified this issue. Or just share some insights.

Thanks
Alexy

[Colin]

Hey @alexypuli

Thanks for the question.

In the process of doing some hardening for Automatic Analysis, we introduced an unexpected delay that went undetected for a few days, and ultimately we had to revert the change.

We will conduct an internal post-mortem soon.