In any case we have to check a lot of different things to find out why the build is not starting. The problem can be caused by some misconfiguration or there can be a performance problem in TeamCity itself. So when we get a complaint that a build does not start, it’s not immediately clear where the problem is. Finally, when all the conditions are met, the queue optimization kicks in and checks whether the build should be started at all or whether an already finished build could be reused instead (if this finished build has the same revision).Įxperience of analyzing issues reported by our users shows that their configuration can be quite tricky and sometimes it is relying on nuanced behavior of some of the TeamCity features. If there are snapshot dependencies, they should be finished too. If the build depends on some shared resources, then these resources should become available. For instance, cloud agents should be launched, fresh commits and settings should be fetched from VCS repositories. The waiting reasonsįor TeamCity to start a build, a lot of conditions should be satisfied. Here is the rundown of how we approached the problem.
#Teamcity slack code#
The build queue bottleneck was so annoying that we’ve decided to dig deeply into the core code of the product. It would not be an exaggeration to say that this issue was driving some of us crazy for quite some time. Hi! I’m waiting for my build to start for 20 minutes, is it intended? During this time I see the agent summary changes, but the configurations don’t start. Why does TeamCity start new builds so slowly? 20 minutes already, agents are idle. īuilds seem to keep piling up in queue and taking longer and longer, are we heading toward a reboot? I see that there are similar problems above with reaching limits but I don’t see such messages, just `No estimate` yet. In this moment, messages from the JetBrains developers were piling up in the teamcity-buildserver Slack channel:įolks, my build is already more than 2 hours in queue. It’s all part of the standard TeamCity functionality, except for that it receives its new features and updates on a daily basis, as we use it for our internal dogfooding.Īt some point, this scale started posing some difficulties for us, as we noticed that during peak hours, newly triggered builds were sitting in the queue for half an hour or more, and basically did not start at all unless you manually moved it to the top. This is all handled by 2000+ build agents (although this number is not static as a lot of them are launched on demand and run builds in hosted environments, such as AWS).Īll of that is handled by our internal CI/CD server called buildserver which is running as a multi-node setup orchestrated by TeamCity.
In total, it amounts to 7,500 projects, 50,000+ build configurations (jobs), and around 65,000 builds per day on average. At JetBrains, we use a single TeamCity installation to build all of our products – including the giants like IntelliJ IDEA, P圜harm, Kotlin, as well as hundreds of plugins, and a lot of internal products and services, such as the website or this blog.
0 kommentar(er)
