Autoscaling has been resolved as has an issue with a special extra large instance not being extra large enough. At this time the migration is complete and stable.
Posted Dec 31, 2020 - 00:19 UTC
Verifying
The last of the issues with the deployment have been ironed out. There are still some issues with the auto-scaling behavior which will cause delays due to limited capacity.
Posted Dec 29, 2020 - 18:22 UTC
Update
Job reconfiguration has completed. We are investigating issues doc jobs are still having issues accessing the ROS documentation host.
Posted Dec 26, 2020 - 16:20 UTC
Update
Reconfiguration of jobs to globally apply new credential is underway. Some test releases have failed and may automatically retry a few more times until reconfiguration has completed.
Posted Dec 25, 2020 - 04:33 UTC
Update
Jenkins performance issues have been resolved. The farm is now stable with two running agents and some test releases are being triggered.
Posted Dec 25, 2020 - 04:22 UTC
Update
There were some issues performing Jenkins operations which may have been lack of JVM memory. The maximum heap size has been increased Jenkins is restarting.
Posted Dec 25, 2020 - 02:59 UTC
Update
Agents have been configured and the first rounds of queued devel jobs are running now.
Posted Dec 24, 2020 - 21:56 UTC
Update
The DNS zone issue has been worked around. Configuration for the build agent pools is underway.
Posted Dec 24, 2020 - 21:16 UTC
Update
Jenkins reconfiguration was successful and several management jobs have turned over. A conflict has been found between public DNS and the private DNS zone used on the build farm internal network and is being investigated.
Posted Dec 24, 2020 - 16:36 UTC
Update
Data transfer completed overnight. The next steps of the migration is verifying and updating Jenkins configuration and preparing the build agent groups.
Posted Dec 24, 2020 - 15:21 UTC
Update
Initial configuration of the new hosts has completed. Data transfer has been initiated.
Sharp-eyed individuals may have briefly seen the new Jenkins or repository host come online and both are stopped now to allow data transfer from the old hosts to occur over the next several hours. Migration will resume in the morning UTC-8.
Posted Dec 24, 2020 - 04:12 UTC
Update
Job cleanup has completed. We are setting up for data transfer from the old hosts to the new.
Posted Dec 24, 2020 - 00:05 UTC
Update
Pre-migration cleanup of the Melodic Debian Stretch jobs is taking longer than expected.
Posted Dec 23, 2020 - 22:11 UTC
In progress
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Posted Dec 23, 2020 - 20:00 UTC
Scheduled
Migration of build.ros.org to the 20.04 based cookbook deployment will begin today.