In T405118#11230813, @Dzahn wrote:

I tried it and I can confirm using mysql+pymysql gets us past the error.

In T405118#11227582, @thcipriani wrote:

Interesting! This looks like it might be something missing in the executor image. My guess is that we copied upstream, but upstream doesn't use mariadb like we do. @dduvall can you take a look at this?

In T404386#11177989, @bd808 wrote:

In T404386#11177958, @dduvall wrote:

@Andrew I don't see any zones listed in the project. Is that normal for a new project?

No, it is not normal for a project to have 0 Designate zones. There should be svc.$PROJECT.eqiad1.wikimedia.cloud., $PROJECT.eqiad1.wmcloud.org., and $PROJECT.wmcloud.org. zones assigned to the project in Designate.

@Andrew I don't see any zones listed in the project. Is that normal for a new project?

In T404150#11173186, @fnegri wrote:

I asked @Andrew about this, and my understanding is that floating IPs are not required to create Octavia load balancers in OpenStack. But I don't have a full understanding of how Magnum works, so I might be wrong! Can you share more details like tofu code, errors you're getting, etc.?

A spike of these errors occurred during wmf.18 group1 promotion today but strangely all instances of the error were from 1.45.0-wmf.17.

In T392526#10871472, @MoritzMuehlenhoff wrote:

In T392526#10871343, @Scott_French wrote:

I'm also unsure how to resolve the difference in package name - e.g., whether there's some suitable override mechanism (create an empty transitional package?) - or whether there are subtle differences in the packaging configuration that make the result incompatible.

One thing to check is whether these debs are actually arch-dependent, usually most vendor debs are just statically linked, we could try one of them in a bullseye and bookworm container to find out. If so, we could have a single update definition and then use it to sync to bullseye and bookworm.

In T392610#11027111, @20after4 wrote:

SpiderPig is hilarious and awesome. Great job everyone!

In T398873#11026091, @dancy wrote:

Yes, but in the meantime scap prep next would continue to work correctly (assuming at least one prior successful branch cut was merged). As it stands, if`MediaWiki branch and publish WMF single-version image` fails, a subsequent scap prep next will fail.

In T398873#11026078, @dancy wrote:

@dduvall I'd like to see MediaWiki branch and publish WMF single-version image changed so that instead of destroying and recreating the wmf/next branch each time it runs, it updates wmf/next if it already exists. This means being able to handle added/dropped extensions.

In T398873#10986500, @dancy wrote:

If CI fails (which happens about 10% of the time), we're left with an unusable wmf/next branch until the next run

@bd808 Kokkuri 2.8.0 will include the digest in the image ref. See if that solves your issue.

@Dzahn the WMF based production images for Zuul and Nodepool have been built and published to our registry. I'll post a summary about how we're managing them in T396245: Build zuul images for production tomorrow, but here are the latest image refs by service:

Looking at the above results, I believe that most of the functionality being served by PipelineLib could potentially be served by docker buildx bake (in conjunction w/ buildkitd and Blubber). Docker bake can build multiple sets of targets/contexts/configs simultaneously and even export the results as generic artifacts (to serve the one case that is using copy).

PipelineLib actions in use, according to codesearch results of 27 Gerrit hosted projects that include a .pipeline/config.yaml file.

I refactored the blubber.yaml that @dancy had written back when we were experimenting with a Zuul setup for GitLab and created a wmf/12.0.0 branch.

Spotted this today as well, following wmf.4 promotion to all wikis.

@thcipriani is this still a blocker or are we good for group1/all wiki promotion today?

In T390251#10759166, @dduvall wrote:

In T390251#10751229, @elukey wrote:

serializes only layers per push. Multiple pushes can still happen simultaneously and IIUC scap does do that. Maybe we could have a flag in scap like e.g. --image-push-concurrency=1 to at least verify/rule out this hypothesis? It would slow down deployment for a couple of weeks but would give a strong signal to guide us better into resolving this.

+1 I like this, @dancy @dduvall what do you think about it?

As a short-term mitigation, it seems reasonable, and serializing the image pushes in build-images.py should be straightforward.

In T390251#10751229, @elukey wrote:

serializes only layers per push. Multiple pushes can still happen simultaneously and IIUC scap does do that. Maybe we could have a flag in scap like e.g. --image-push-concurrency=1 to at least verify/rule out this hypothesis? It would slow down deployment for a couple of weeks but would give a strong signal to guide us better into resolving this.

+1 I like this, @dancy @dduvall what do you think about it?

Looks to have been introduced in:

Removing this task as a blocker as the errors only occurred during a short-ish window, occurred for wmf.24 as well as wmf.25, and only for internal wikis.

Closed as a duplicate that, while ongoing, is not strictly a train blocker.

Other possibly relevant discussions around this issue.

In T390251#10744699, @dduvall wrote:

Also, I wonder if there's a way we can force monolithic uploads?

Also, I wonder if there's a way we can force monolithic uploads?

In T390251#10744640, @dancy wrote:

In the case of uploads, here is the bad sequence:

• The client (e.g. dockerd) issues POST /v2/<repo>/blob/uploads/ to initiate an upload. This returns a new URL for subsequent operations (hereafter called the upload URL)
• The client issues a PATCH to the upload URL to transmit the data.
• The client issues a PUT to the upload URL to finalize the upload. This is where a 404 is sometimes returned by the registry (basically saying that it doesn't know about this upload). A 404 is more likely to be seen if a prior upload was large (i.e, if the replicator is busy). Retrying this PUT does eventually succeed.

In this case it's not the content of the upload that hasn't made it to the replica, but the existence of the upload state itself.

In T390251#10744390, @dancy wrote:

At this point we have two problems.

• Large image pushes are now unreliable (this seems new for mediawiki deployments). No workaround proposed yet.

The Zuul dashboard is available at https://zuul-dev.wmcloud.org/tenants

In T391374#10740144, @hashar wrote:

@dduvall excellent!

The event stream permission is probably good enough, it does not grant any specific access beside the ability to receive events and we have multiple bots on WMCS using that same setup. As long as the user is not granted more permission, it can't do much. Setting up a Gerrit + repos + config might add a bit of a burden, then if you can reuse an existing setup that let gives us a great playground \o/

@hashar FYI I've set up Zuul and friends on zuul-1001.zuul3.eqiad1.wikimedia.cloud using https://opendev.org/zuul/zuul/src/branch/master/doc/source/examples/docker-compose.yaml

In T389499#10676584, @Scott_French wrote:

So, a possibly adequate analogy for the relationship between image "kind" (a new name for an existing concept that did not previously have a name) and image "flavour" (an existing name for an existing concept) would be that between a class definition and the specific set of constructor arguments that produce a concrete instantiation.

In T388769#10644219, @bd808 wrote:

The apt config and implementation is also Debian-based base image specific. Alpine base images would need an apk config and implementation. Red Hat-based base images would need a yum (or dnf?) config and implementation. Arch uses pacman. OpenSUSE uses zypper. I am not aware of any unifying abstraction over the various distro specific package managers that would simplify this readily.

Thanks for pointing out that this isn't a bug, @bd808.

In T387927#10614474, @dancy wrote:

I agree with running a daily timer and trying to spread the knowledge about the availability of scap clean-images to quickly recover space in unusual circumstances.

The scap clean-images implementation has been merged. I plan on doing a release early next week. Sample behavior from train-dev:

@dancy thoughts on the implementation?

Is there something else filling up /srv? It has filled back up and docker system df hasn't changed much.

Even after the deletion, there is still a lot of reclaimable space:

I've removed a bunch of old images for now. I will talk with Tyler and Ahmon tomorrow about long term solutions.

Thank you!

Thanks, Jon!

Just a reminder that all train blockers need to be UBN. If this isn't a blocker, please remove the parent task.

Just a reminder that all train blockers need to be UBN. If this isn't a blocker, please remove the parent task.

What's the status here? Do we need a cherry-pick to wmf.18 to unblock the train?

Should this block today's train promotion?

This error seems to emanate from Blubber's BuildKit frontend, not buildkitd.

Attempting a better summary before closing this:

@hashar Do you know if we can set the Jenkins result as SKIPPED/NOT_BUILT and have Gearman/Zuul honor that? (i.e. Will Zuul fail the pipeline on any non-success status or are there other success-ish statuses?)

Following a botched "safe" restart and subsequent systemctl restart jenkins, the issue seems to be resolved.

Seeing https://github.com/nodejs/build/issues/3754 as well which seems to support your hunch @taavi .

In T386755#10565060, @taavi wrote:

This feels like very familiar to T385553/T377803, and Java was upgraded yesterday on contint1002 per the Apt history log file so that matches too. Maybe let's try restarting the Jenkins service since that fixes the Puppet case of this error?

In T381359#10553905, @thcipriani wrote:

We think we can accomplish this using a slight variation on the documentation on running acceptance tests against your built image, but overriding KOKKURI_REGISTRY_PUBLIC to point to an internal registry (instead of the default docker-registry.wikimedia.org).