Hello, everyone.
TL;DR: I'm thinking of shutting down all gentoo-mirror repositories,
except for gentoo and guru.
Over 10 years ago, I've started the repository mirror & CI project.
What started as a bunch of shell scripts on a user-donated server, has organically grown into a bigger bunch of shell scripts managed by Infra. Nevertheless, it's still a bunch of hacks glued together.
Things don't work well all the time. Sometimes stuff randomly crashes,
and I have to SSH and remove local checkouts to make it work. Sometimes
the git repositories used to transfer logs grow so big they kill infra.
Often some repository starts crashing this or another part and needs to
be disabled.
To be honest, I have no energy to keep maintaining this. I'm really
tired of having to deal with stuff crashing and spamming my mailbox with failure mails. I'm tired of having to go through all the infra hoops
just to disable another repository that can't work for one reason or
another. In fact, I'm even tired that whenever people add new
repositories to api.gentoo.org, I have to go through that idiotic GitHub clickety-click UI to stop receiving notifications for everything that
happens in these repositories.
So what I'm thinking about is winding most of the project down. We'd
stop mirroring third-party repositories, and remove most of gentoo-
mirror organization. What I'd like to leave is mirroring of gentoo
and guru repositories, since these two we have control of, and are very important to Gentoo users.
So, well, unless someone convinces me otherwise, I'm going to disable
all other repositories over the next weekend, and remove their mirrors. Gentoo and GURU will still be mirrored, and CI will keep running
as usual.
What does this mean for the libressl overlay? People use that so please
don't remove it.
Hello, everyone.
TL;DR: I'm thinking of shutting down all gentoo-mirror repositories,
except for gentoo and guru.
Over 10 years ago, I've started the repository mirror & CI project.
What started as a bunch of shell scripts on a user-donated server, has organically grown into a bigger bunch of shell scripts managed by
Infra. Nevertheless, it's still a bunch of hacks glued together.
Things don't work well all the time. Sometimes stuff randomly
crashes, and I have to SSH and remove local checkouts to make it
work. Sometimes the git repositories used to transfer logs grow so
big they kill infra. Often some repository starts crashing this or
another part and needs to be disabled.
To be honest, I have no energy to keep maintaining this. I'm really
tired of having to deal with stuff crashing and spamming my mailbox
with failure mails. I'm tired of having to go through all the infra
hoops just to disable another repository that can't work for one
reason or another. In fact, I'm even tired that whenever people add
new repositories to api.gentoo.org, I have to go through that idiotic
GitHub clickety-click UI to stop receiving notifications for
everything that happens in these repositories.
So what I'm thinking about is winding most of the project down. We'd
stop mirroring third-party repositories, and remove most of gentoo-
mirror organization. What I'd like to leave is mirroring of gentoo
and guru repositories, since these two we have control of, and are
very important to Gentoo users.
So, well, unless someone convinces me otherwise, I'm going to disable
all other repositories over the next weekend, and remove their
mirrors. Gentoo and GURU will still be mirrored, and CI will keep
running as usual.
21.03.2025 14:44, Michał Górny пишет:
On Fri, 2025-03-21 at 14:12 +0000, Alexey Sokolov wrote:
Hi, if you just disable it, it'll break setup of everyone who uses /etc/portage/repos.conf/eselect-repo.conf, right?Sounds about right. Which is probably preferable over the current state
of mirrors silently stopping to update, and people being stuck forever
on old commits.
In that case I think first need to change `eselect repository enable` to
add overlays directly instead of its mirror. And somehow update existing configs, either semi-automatically, or e.g. via news item
On Fri, 2025-03-21 at 14:12 +0000, Alexey Sokolov wrote:
Hi, if you just disable it, it'll break setup of everyone who usesSounds about right. Which is probably preferable over the current state
/etc/portage/repos.conf/eselect-repo.conf, right?
of mirrors silently stopping to update, and people being stuck forever
on old commits.
Hi, if you just disable it, it'll break setup of everyone who uses /etc/portage/repos.conf/eselect-repo.conf, right?
On Fri, 2025-03-21 at 07:12 -0700, orbea wrote:
What does this mean for the libressl overlay? People use that so
please don't remove it.
It means people will have to sync straight from the upstream
repository. Also, we won't be reporting bugs when things break hard.
Hello, everyone.
TL;DR: I'm thinking of shutting down all gentoo-mirror repositories,
except for gentoo and guru.
So what I'm thinking about is winding most of the project down. We'd
stop mirroring third-party repositories, and remove most of gentoo-
mirror organization.
On Fri, 21 Mar 2025 15:44:04 +0100
Michał Górny <mgorny@gentoo.org> wrote:
On Fri, 2025-03-21 at 07:12 -0700, orbea wrote:
What does this mean for the libressl overlay? People use that so
please don't remove it.
It means people will have to sync straight from the upstream
repository. Also, we won't be reporting bugs when things break hard.
Currently when I push changes to the overlay I push to:
git+ssh://git@git.gentoo.org/repo/proj/libressl.git
Is there anything I need to do to make sure I can still push to the
github repo instead? I am not sure I have access to push there directly?
On Fri, 2025-03-21 at 16:50 -0700, orbea wrote:
On Fri, 21 Mar 2025 15:44:04 +0100
Michał Górny <mgorny@gentoo.org> wrote:
On Fri, 2025-03-21 at 07:12 -0700, orbea wrote:
What does this mean for the libressl overlay? People use that so
please don't remove it.
It means people will have to sync straight from the upstream
repository. Also, we won't be reporting bugs when things break
hard.
Currently when I push changes to the overlay I push to:
git+ssh://git@git.gentoo.org/repo/proj/libressl.git
Is there anything I need to do to make sure I can still push to the
github repo instead? I am not sure I have access to push there
directly?
If you're asking about the mirror @
https://github.com/gentoo/libressl, then that's not affected. I'm
talking of https://github.com/gentoo-mirror/libressl.
On 3/21/2025 6:42 PM, Ionen Wolkens wrote:
On Fri, Mar 21, 2025 at 02:32:31PM +0100, Michał Górny wrote:
Hello, everyone.Unfortunate, but just to say that I have nothing to say against
TL;DR: I'm thinking of shutting down all gentoo-mirror repositories,
except for gentoo and guru.
dropping these if it's a maintenance burden.
So what I'm thinking about is winding most of the project down. We'dHad one concern but complete removal (rather than keeping mirrors
stop mirroring third-party repositories, and remove most of gentoo-
mirror organization.
archived) would mostly handle it. Aka so that users can't keep
sync'ing with these dead mirrors by mistake (often caused issues when
we removed overlays but mirror stayed behind, users wouldn't notice a
thing until lacking updates started to break things).
In the OpenStack community, we will put a final commit on HEAD of the
primary branch removing all the content and putting an EOL notice in
the readme (e.g. https://opendev.org/openstack/ironic-lib ). This may
be a good solution in this case as well, potentially?
-JayF
On Fri, Mar 21, 2025 at 02:32:31PM +0100, Michał Górny wrote:
Hello, everyone.Unfortunate, but just to say that I have nothing to say against
TL;DR: I'm thinking of shutting down all gentoo-mirror repositories,
except for gentoo and guru.
dropping these if it's a maintenance burden.
So what I'm thinking about is winding most of the project down. We'dHad one concern but complete removal (rather than keeping mirrors
stop mirroring third-party repositories, and remove most of gentoo-
mirror organization.
archived) would mostly handle it. Aka so that users can't keep
sync'ing with these dead mirrors by mistake (often caused issues when
we removed overlays but mirror stayed behind, users wouldn't notice a
thing until lacking updates started to break things).
Jay Faulkner <jayf@gentoo.org> writes:
On 3/21/2025 6:42 PM, Ionen Wolkens wrote:
On Fri, Mar 21, 2025 at 02:32:31PM +0100, Michał Górny wrote:
Hello, everyone.
TL;DR: I'm thinking of shutting down all gentoo-mirror repositories, except for gentoo and guru.Unfortunate, but just to say that I have nothing to say against
dropping these if it's a maintenance burden.
So what I'm thinking about is winding most of the project down. We'd stop mirroring third-party repositories, and remove most of gentoo- mirror organization.Had one concern but complete removal (rather than keeping mirrors archived) would mostly handle it. Aka so that users can't keep
sync'ing with these dead mirrors by mistake (often caused issues when
we removed overlays but mirror stayed behind, users wouldn't notice a thing until lacking updates started to break things).
In the OpenStack community, we will put a final commit on HEAD of the primary branch removing all the content and putting an EOL notice in
the readme (e.g. https://opendev.org/openstack/ironic-lib ). This may
be a good solution in this case as well, potentially?
We can even inject a news item for the repository.
On 3/21/2025 6:42 PM, Ionen Wolkens wrote:
On Fri, Mar 21, 2025 at 02:32:31PM +0100, Michał Górny wrote:
Hello, everyone.
TL;DR: I'm thinking of shutting down all gentoo-mirror repositories, except for gentoo and guru.Unfortunate, but just to say that I have nothing to say against
dropping these if it's a maintenance burden.
So what I'm thinking about is winding most of the project down. We'd stop mirroring third-party repositories, and remove most of gentoo- mirror organization.Had one concern but complete removal (rather than keeping mirrors
archived) would mostly handle it. Aka so that users can't keep
sync'ing with these dead mirrors by mistake (often caused issues when
we removed overlays but mirror stayed behind, users wouldn't notice a
thing until lacking updates started to break things).
In the OpenStack community, we will put a final commit on HEAD of the primary branch removing all the content and putting an EOL notice in the readme (e.g. https://opendev.org/openstack/ironic-lib ). This may be a
good solution in this case as well, potentially?
On Sat, 2025-03-22 at 08:20 -0700, Jay Faulkner wrote:
In the OpenStack community, we will put a final commit on HEAD of the
primary branch removing all the content and putting an EOL notice in the
readme (e.g. https://opendev.org/openstack/ironic-lib ). This may be a
good solution in this case as well, potentially?
Not sure. It would mean that `emerge --sync` will replace all ebuilds
with a README file that user might not even notice. I suppose
an explicit "repository not found" error may be better here.
Hello, everyone.
TL;DR: I'm thinking of shutting down all gentoo-mirror repositories,
except for gentoo and guru.
Over 10 years ago, I've started the repository mirror & CI project.
What started as a bunch of shell scripts on a user-donated server, has organically grown into a bigger bunch of shell scripts managed by Infra. Nevertheless, it's still a bunch of hacks glued together.
Things don't work well all the time. Sometimes stuff randomly crashes,
and I have to SSH and remove local checkouts to make it work. Sometimes
the git repositories used to transfer logs grow so big they kill infra.
Often some repository starts crashing this or another part and needs to
be disabled.
To be honest, I have no energy to keep maintaining this. I'm really
tired of having to deal with stuff crashing and spamming my mailbox with failure mails. I'm tired of having to go through all the infra hoops
just to disable another repository that can't work for one reason or
another. In fact, I'm even tired that whenever people add new
repositories to api.gentoo.org, I have to go through that idiotic GitHub clickety-click UI to stop receiving notifications for everything that
happens in these repositories.
So what I'm thinking about is winding most of the project down. We'd
stop mirroring third-party repositories, and remove most of gentoo-
mirror organization. What I'd like to leave is mirroring of gentoo
and guru repositories, since these two we have control of, and are very important to Gentoo users.
So, well, unless someone convinces me otherwise, I'm going to disable
all other repositories over the next weekend, and remove their mirrors. Gentoo and GURU will still be mirrored, and CI will keep running
as usual.
Hello, everyone.
TL;DR: I'm thinking of shutting down all gentoo-mirror repositories,
except for gentoo and guru.
Over 10 years ago, I've started the repository mirror & CI project.
What started as a bunch of shell scripts on a user-donated server, has organically grown into a bigger bunch of shell scripts managed by Infra. Nevertheless, it's still a bunch of hacks glued together.
Things don't work well all the time. Sometimes stuff randomly crashes,
and I have to SSH and remove local checkouts to make it work. Sometimes
the git repositories used to transfer logs grow so big they kill infra.
Often some repository starts crashing this or another part and needs to
be disabled.
To be honest, I have no energy to keep maintaining this. I'm really
tired of having to deal with stuff crashing and spamming my mailbox with failure mails. I'm tired of having to go through all the infra hoops
just to disable another repository that can't work for one reason or
another. In fact, I'm even tired that whenever people add new
repositories to api.gentoo.org, I have to go through that idiotic GitHub clickety-click UI to stop receiving notifications for everything that
happens in these repositories.
So what I'm thinking about is winding most of the project down. We'd
stop mirroring third-party repositories, and remove most of gentoo-
mirror organization. What I'd like to leave is mirroring of gentoo
and guru repositories, since these two we have control of, and are very important to Gentoo users.
So, well, unless someone convinces me otherwise, I'm going to disable
all other repositories over the next weekend, and remove their mirrors. Gentoo and GURU will still be mirrored, and CI will keep running
as usual.
Am Freitag, 21. März 2025, 14:32:31 Mitteleuropäische Normalzeit schrieb Michał Górny:
Hello, everyone.
TL;DR: I'm thinking of shutting down all gentoo-mirror repositories,
except for gentoo and guru.
Over 10 years ago, I've started the repository mirror & CI project.
What started as a bunch of shell scripts on a user-donated server, has organically grown into a bigger bunch of shell scripts managed by Infra. Nevertheless, it's still a bunch of hacks glued together.
Things don't work well all the time. Sometimes stuff randomly crashes,
and I have to SSH and remove local checkouts to make it work. Sometimes the git repositories used to transfer logs grow so big they kill infra. Often some repository starts crashing this or another part and needs to
be disabled.
To be honest, I have no energy to keep maintaining this. I'm really
tired of having to deal with stuff crashing and spamming my mailbox with failure mails. I'm tired of having to go through all the infra hoops
just to disable another repository that can't work for one reason or another. In fact, I'm even tired that whenever people add new
repositories to api.gentoo.org, I have to go through that idiotic GitHub clickety-click UI to stop receiving notifications for everything that happens in these repositories.
So what I'm thinking about is winding most of the project down. We'd
stop mirroring third-party repositories, and remove most of gentoo-
mirror organization. What I'd like to leave is mirroring of gentoo
and guru repositories, since these two we have control of, and are very important to Gentoo users.
So, well, unless someone convinces me otherwise, I'm going to disable
all other repositories over the next weekend, and remove their mirrors. Gentoo and GURU will still be mirrored, and CI will keep running
as usual.
First of all, thank you for running it in the first place!
Maybe you like to also continuing mirroring semi-official dev repos like kde and qt.
They are exclusively maintained by Gentoo devs, very large, and beneficial for users who want bleeding edge software.
I would expect that these repos are not the main factor of the maintenance due to their high quality.
And, if I'm right, this reduces sync times per user since the CI's metadata creation (I remember the days, when eix-sync needed extremely long).
Currently, you completely provide the repo mirror infrastructure and also deal with all the problems.
Is it possible to shift this in a large part to the overlay maintainer (in a opt-in approach)?
E.g. something like this (I don't know if the Gentoo git directly provide a CI for users, but maybe Github can do this):
- Per default a overlay is not mirrored, this is an "award" that must be earned.
- Enable a CI controlled by each overlay dev (overlay devs must take care of the CI scripts, they get all the emails).
- (Just) provide CI scripts that do the necessary overlay checking (if I'm not wrong, you already need these scripts for Gentoo and GURU).
- Overlay devs are responsible to run these CI scripts in their overlay, fix errors etc.
- Provide infrastructure that provides a mirror with metadata only for overlays that have enabled the CI, pass it, and the overlay dev asked for being mirrored.
- Before mirroring a commit, wait for the overlay CI to pass to make sure to get no errors on your side.
- If an overlay dev somehow changes the CI scripts in a way that it make the mirroring infrastructure to fail: Remove the overlay permanently.
Best,
Gerion
Hello, everyone.
TL;DR: I'm thinking of shutting down all gentoo-mirror repositories,
except for gentoo and guru.
Over 10 years ago, I've started the repository mirror & CI project.
What started as a bunch of shell scripts on a user-donated server, has organically grown into a bigger bunch of shell scripts managed by Infra. Nevertheless, it's still a bunch of hacks glued together.
Things don't work well all the time. Sometimes stuff randomly crashes,
and I have to SSH and remove local checkouts to make it work. Sometimes
the git repositories used to transfer logs grow so big they kill infra.
Often some repository starts crashing this or another part and needs to
be disabled.
To be honest, I have no energy to keep maintaining this. I'm really
tired of having to deal with stuff crashing and spamming my mailbox with failure mails. I'm tired of having to go through all the infra hoops
just to disable another repository that can't work for one reason or
another. In fact, I'm even tired that whenever people add new
repositories to api.gentoo.org, I have to go through that idiotic GitHub clickety-click UI to stop receiving notifications for everything that
happens in these repositories.
So what I'm thinking about is winding most of the project down. We'd
stop mirroring third-party repositories, and remove most of gentoo-
mirror organization. What I'd like to leave is mirroring of gentoo
and guru repositories, since these two we have control of, and are very important to Gentoo users.
So, well, unless someone convinces me otherwise, I'm going to disable
all other repositories over the next weekend, and remove their mirrors. Gentoo and GURU will still be mirrored, and CI will keep running
as usual.
--
Best regards,
Michał Górny
I've been following the discussion, but I still don't know enough to have
an opinion. Why was this infrastructure created in the first place?
On Mon, 2025-03-24 at 08:46 -0400, Mitchell Dorrell wrote:
I've been following the discussion, but I still don't know enough to haveThere's a number of reasons, and they are still valid today:
an opinion. Why was this infrastructure created in the first place?
On 3/25/2025 1:51 PM, Michał Górny wrote:
On Mon, 2025-03-24 at 08:46 -0400, Mitchell Dorrell wrote:
I've been following the discussion, but I still don't know enough to have an opinion. Why was this infrastructure created in the first place?There's a number of reasons, and they are still valid today:
[snip]
I'm mildly surprised that "get someone else to host this for 'free'"
isn't one of the reasons. I'm assuming there's plenty of headroom on Gentoo's git servers to handle the influx of git requests that would
likely be a side effect of this change?
On Tue, 2025-03-25 at 16:15 -0700, Jay Faulkner wrote:
On 3/25/2025 1:51 PM, Michał Górny wrote:Most of the mirrored repositories aren't on Gentoo git server. This is
On Mon, 2025-03-24 at 08:46 -0400, Mitchell Dorrell wrote:[snip]
I've been following the discussion, but I still don't know enough to have >>>> an opinion. Why was this infrastructure created in the first place?There's a number of reasons, and they are still valid today:
I'm mildly surprised that "get someone else to host this for 'free'"
isn't one of the reasons. I'm assuming there's plenty of headroom on
Gentoo's git servers to handle the influx of git requests that would
likely be a side effect of this change?
part of the problem, because we're already seeing git incompatibilities between Codeberg and GitHub.
Hello, everyone.
TL;DR: I'm thinking of shutting down all gentoo-mirror repositories,
except for gentoo and guru.
I guess i'm a bit late in this discussion but i wanted to let you know
this would also affect my gentoo qa scripts.
(https://gentooqa.levelnine.at). Right now i'm checking the gentoo,
guru, kde, science and pentoo repositories, syncing the repos from >gentoo-mirror for the pre-created metadata.
While the gentoo repository is probably the most valuable, kde,
science and pentoo checks would be non functional and i probably would
have to remove them. (at least until i update my script to create the >metadata myself..)
Hello, everyone.
TL;DR: I'm thinking of shutting down all gentoo-mirror repositories,
except for gentoo and guru.
Over 10 years ago, I've started the repository mirror & CI project.
What started as a bunch of shell scripts on a user-donated server, has organically grown into a bigger bunch of shell scripts managed by
Infra.
Nevertheless, it's still a bunch of hacks glued together.
Things don't work well all the time. Sometimes stuff randomly crashes,
and I have to SSH and remove local checkouts to make it work.
Sometimes
the git repositories used to transfer logs grow so big they kill infra.
Often some repository starts crashing this or another part and needs to
be disabled.
To be honest, I have no energy to keep maintaining this. I'm really
tired of having to deal with stuff crashing and spamming my mailbox
with
failure mails. I'm tired of having to go through all the infra hoops
just to disable another repository that can't work for one reason or
another. In fact, I'm even tired that whenever people add new
repositories to api.gentoo.org, I have to go through that idiotic
GitHub
clickety-click UI to stop receiving notifications for everything that
happens in these repositories.
So what I'm thinking about is winding most of the project down. We'd
stop mirroring third-party repositories, and remove most of gentoo-
mirror organization. What I'd like to leave is mirroring of gentoo
and guru repositories, since these two we have control of, and are very important to Gentoo users.
So, well, unless someone convinces me otherwise, I'm going to disable
all other repositories over the next weekend, and remove their mirrors. Gentoo and GURU will still be mirrored, and CI will keep running
as usual.
| Sysop: | Keyop |
|---|---|
| Location: | Huddersfield, West Yorkshire, UK |
| Users: | 475 |
| Nodes: | 16 (2 / 14) |
| Uptime: | 19:01:27 |
| Calls: | 9,487 |
| Calls today: | 6 |
| Files: | 13,617 |
| Messages: | 6,121,093 |