Are there any useful checks or behaviors of repoman that are missing
from pkgcheck and pkgcommit?
On Wed, Mar 9, 2022 at 4:00 PM Matt Turner <mattst88@gentoo.org> wrote:
Are there any useful checks or behaviors of repoman that are missing
from pkgcheck and pkgcommit?
Would it make sense to package pkgcommit or otherwise stick it
somewhere official? I know there is a copy on mgorny's github
repo/blog, but if this is going to become the recommended way to do
commits we might want to stick it someplace a bit more canonical, and
having it in a package would ensure that if there are any changes they
get distributed.
Obviously there are other ways to set it, but it lacks obtaining the
gpg key to use from make.conf. I'm not sure that is really the best
place to put it in any case. Just figured I'd point it out.
Are there any useful checks or behaviors of repoman that are missing
from pkgcheck and pkgcommit?
Fixing ebuild copyright date is the first one that comes to mind.
I'd like to deprecate and ultimately remove repoman. I believe that dev-util/pkgcheck and pkgcommit (from app-portage/mgorny-dev-scripts)
are far superior replacements, and it makes sense to have people using
the same tool and seeing the same warnings as in the CI.
Are there any useful checks or behaviors of repoman that are missing
from pkgcheck and pkgcommit?
Thanks,
Matt
On 3/9/2022 4:28 PM, Matt Turner wrote:
On Wed, Mar 9, 2022 at 1:27 PM Maciej Barć <xgqt@gentoo.org> wrote:
Are there any useful checks or behaviors of repoman that are missing
from pkgcheck and pkgcommit?
Fixing ebuild copyright date is the first one that comes to mind.
pkgcommit does this.
I highly doubt it as it sits in the github repo https://github.com/mgorny/mgorny-dev-scripts/blob/master/pkgcommit
basically sets up $EDITOR with "cat/pkg:" with an optional message in
front then passes to 'git commit'
Just a quick though:
Looking at the man page of repoman it doesn't look to difficult to
replicate the behavior with pkgcheck. Meaning, we could think of
creating a drop-in replacement for "repoman [full]" (which would just
call pkgcheck) and "repoman commit" which actually does much more than
just prepending the git commit summary line: repoman commit does
- update the manifest
- bail out if files are not correctly "git add"ed
- add the output of [pkgcheck] as a comment to the git commit
description
I'd like to deprecate and ultimately remove repoman. I believe that dev-util/pkgcheck and pkgcommit (from app-portage/mgorny-dev-scripts)
are far superior replacements, and it makes sense to have people using
the same tool and seeing the same warnings as in the CI.
[...]
I wouldn't block anyone from doing this, but it's not something I'm personally interested in pursuing. I see very little value here.
On Wed, Mar 9, 2022, at 15:47 CST, Matt Turner <mattst88@gentoo.org> wrote:
[...]
I wouldn't block anyone from doing this, but it's not something I'm personally interested in pursuing. I see very little value here.
I did not intend to imply that you should do anything.
I just want to point out that we had been educating developers for using "repoman" to commit for a long time and looking at the "git log" history shows that repoman is still used by a sizable number of developers
(around 50%-ish).
So, it would be very nice to have a somewhat official "blessed"
alternative in place.
For example, dev-util/pkgdev was suggested as an alternative, but I
cannot assess whether this is indeed a viable alternative.
I think you just made that number up :)
For example, dev-util/pkgdev was suggested as an alternative, but I
cannot assess whether this is indeed a viable alternative.
Why not?
Give it a try.
On Wed, Mar 9, 2022, at 17:32 CST, Matt Turner <mattst88@gentoo.org> wrote:
I think you just made that number up :)
Nah. My sample was just bad (ago just made a sizable number of commits.
I would estimate the current usage of repoman to be about 20-25%, down
from well over 80-90% back when we just switched to git:
https://tamiko.43-1.org/temp/repoman-usage.pdf
I'd like to deprecate and ultimately remove repoman. I believe that dev-util/pkgcheck and pkgcommit (from app-portage/mgorny-dev-scripts)
are far superior replacements, and it makes sense to have people using
the same tool and seeing the same warnings as in the CI.
Are there any useful checks or behaviors of repoman that are missing
from pkgcheck and pkgcommit?
Thanks,
Matt
On Wed, Mar 9, 2022 at 11:09 PM Joonas Niilola <juippis@gentoo.org> wrote:
On 9.3.2022 23.00, Matt Turner wrote:
I'd like to deprecate and ultimately remove repoman. I believe that dev-util/pkgcheck and pkgcommit (from app-portage/mgorny-dev-scripts)
are far superior replacements, and it makes sense to have people using the same tool and seeing the same warnings as in the CI.
Are there any useful checks or behaviors of repoman that are missing
from pkgcheck and pkgcommit?
Thanks,
Matt
I still fail to see the "why" in here. Repoman is better than pure 'git commit' that I know some people still like to use, and as long as it's
kept maintained.
repoman is inferior to other tooling mentioned. The other tooling is
actually run in CI. Developers should get the same warnings locally as
in CI. Restated another way: I'm tired of telling people to stop using repoman or "pkgcheck would have caught that".
On 9.3.2022 23.00, Matt Turner wrote:
I'd like to deprecate and ultimately remove repoman. I believe that dev-util/pkgcheck and pkgcommit (from app-portage/mgorny-dev-scripts)
are far superior replacements, and it makes sense to have people using
the same tool and seeing the same warnings as in the CI.
Are there any useful checks or behaviors of repoman that are missing
from pkgcheck and pkgcommit?
Thanks,
Matt
I still fail to see the "why" in here. Repoman is better than pure 'git commit' that I know some people still like to use, and as long as it's
kept maintained.
My only worry is: are pkgcheck and pkgdev _really_ maintained anymore?
More than repoman is?
I'd like to deprecate and ultimately remove repoman. I believe that dev-util/pkgcheck and pkgcommit (from app-portage/mgorny-dev-scripts)
are far superior replacements, and it makes sense to have people using
the same tool and seeing the same warnings as in the CI.
Are there any useful checks or behaviors of repoman that are missing
from pkgcheck and pkgcommit?
Thanks,
Matt
On Thu, Mar 10, 2022 at 09:29:59AM -0800, Matt Turner wrote:
On Wed, Mar 9, 2022 at 11:09 PM Joonas Niilola <juippis@gentoo.org> wrote:
On 9.3.2022 23.00, Matt Turner wrote:
I'd like to deprecate and ultimately remove repoman. I believe that dev-util/pkgcheck and pkgcommit (from app-portage/mgorny-dev-scripts) are far superior replacements, and it makes sense to have people using the same tool and seeing the same warnings as in the CI.
Are there any useful checks or behaviors of repoman that are missing from pkgcheck and pkgcommit?
Thanks,
Matt
I still fail to see the "why" in here. Repoman is better than pure 'git commit' that I know some people still like to use, and as long as it's kept maintained.
repoman is inferior to other tooling mentioned. The other tooling is actually run in CI. Developers should get the same warnings locally as
in CI. Restated another way: I'm tired of telling people to stop using repoman or "pkgcheck would have caught that".
I am going to nit-pick here, but pkgcheck pulls in pkgcore still. As far
asI know, pkgcore was meant to be a portage-like package manager, but it isn't maintained. So, can we break that dependency before we make
pkgcheck the official tool for qa checks? I would rather not have
pkgcore landing on everyone's systems unless it is usable. If I am wrong about pkgcore, please correct me and I'll be quiet, but if not let's
make pkgcheck independent from it before we deprecate repoman.
On 3/9/2022 16:00, Matt Turner wrote:
I'd like to deprecate and ultimately remove repoman. I believe that dev-util/pkgcheck and pkgcommit (from app-portage/mgorny-dev-scripts)
are far superior replacements, and it makes sense to have people using
the same tool and seeing the same warnings as in the CI.
Are there any useful checks or behaviors of repoman that are missing
from pkgcheck and pkgcommit?
Thanks,
Matt
repoman has been //the// goto tool for checking in a change since before I was a developer in 2003. It does everything we need, in one simple tool. Your proposal looks to replace repoman's functionality (and snark) with two or more packages, including tools from a package named after a fellow developer.
Additionally, "I believe that <foo> are far superior replacements" is an entirely subjective opinion and, frankly, completely invalid as
justification to replace a tool that has worked fine for the last 20+ years.
What is so fundamentally broken about repoman that cannot be fixed such
that it needs total replacement by multiple independent tools? Please document. the pros and cons here so that we can all make an informed decision.
I'm not opposed to making our tools better, but I think before anything can replace the RepoMan, several more boxes need to be ticked:
- functionality from multiple tools should be packaged into a single tool.
* caveat: at least provide a wrapper that, using args, can invoke the
individual tools if needed.
- app-portage/mgorny-dev-scripts needs a new name. It's fine if it's intended to only be a collection of useful scripts for individual developers on an as-needed basis, but if it's to be the Official Tool(TM), then it
needs to have a proper name. If not all of the scripts contained within it are applicable to the sole function of checking a change into the tree, then only the scripts that deal with change validation and committing should be broken out into a separate package, and ostensibly combined with any other tools/scripts into a single package, and preferably a single tool, to get
the job done.
- all of our developer documentation needs to be updated to reference the
new tool and the new way of doing things, as well as a warning not to use repoman any further after a set date. Additionally, a news item is probably advisable so that developers and proxy maintainers alike can get the message.
- we need the snark. Gentoo is all about personality, and RepoMan has been scouring our neighborhoods for two decades and change, and while some may think this is utterly frivolous, I will actually miss seeing those messages on the console every time I commit a change. They give the RepoMan personality and a soul, and thus, contribute to the uniqueness that is Gentoo.
--
Joshua Kinard
Gentoo/MIPS
kumba@gentoo.org
rsa6144/5C63F4E3F5C6C943 2015-04-27
177C 1972 1FB8 F254 BAD0 3E72 5C63 F4E3 F5C6 C943
"The past tempts us, the present confuses us, the future frightens us. And our lives slip away, moment by moment, lost in that vast, terrible in-between."
--Emperor Turhan, Centauri Republic
On 3/9/2022 16:47, Matt Turner wrote:
On Wed, Mar 9, 2022 at 1:37 PM Matthias Maier <tamiko@gentoo.org> wrote:
Just a quick though:
Looking at the man page of repoman it doesn't look to difficult to
replicate the behavior with pkgcheck. Meaning, we could think of
creating a drop-in replacement for "repoman [full]" (which would just
call pkgcheck) and "repoman commit" which actually does much more than
just prepending the git commit summary line: repoman commit does
- update the manifest
- bail out if files are not correctly "git add"ed
- add the output of [pkgcheck] as a comment to the git commit
description
I wouldn't block anyone from doing this, but it's not something I'm personally interested in pursuing. I see very little value here.
First, you're trying to justify replacing repoman on an entirely subjective opinion of "I think <foo> is superior", then when you get feedback on the idea, you dismiss that feedback with more opinion.
Why do you not see value here? The actions described are actually quite a few useful steps in the process of checking a change into the tree. If you expect developers to do those steps on their own, that increases, not decreases, the chances of making a mistake. Or are these steps already handled by one of the other scripts in the replacement packages you propose?
If so, please say so and identify which one(s).
My opinion is that any tool that replaces repoman should, at a minimum, replace like functionality with like functionality, plus benefits or enhancements. This looks more like a step backwards, not a step forwards.
--
Joshua Kinard
Gentoo/MIPS
kumba@gentoo.org
rsa6144/5C63F4E3F5C6C943 2015-04-27
177C 1972 1FB8 F254 BAD0 3E72 5C63 F4E3 F5C6 C943
"The past tempts us, the present confuses us, the future frightens us. And our lives slip away, moment by moment, lost in that vast, terrible in-between."
--Emperor Turhan, Centauri Republic
I wouldn't block anyone from doing this, but it's not something I'm personally interested in pursuing. I see very little value here.
First, you're trying to justify replacing repoman on an entirely subjective opinion of "I think <foo> is superior" ...
I wouldn't block anyone from doing this, but it's not something I'm
personally interested in pursuing. I see very little value here.
First, you're trying to justify replacing repoman on an entirely subjective >> opinion of "I think <foo> is superior" ...
Well, if you've ever tried it you'll notice that <foo> for <foo> != repoman actually finishes the checks within a finite amount of time. Kind of, the most blatant argument for ditching repoman, actually.
On Thu, Mar 10, 2022 at 10:28 AM Joshua Kinard <kumba@gentoo.org> wrote:
On 3/9/2022 16:47, Matt Turner wrote:
On Wed, Mar 9, 2022 at 1:37 PM Matthias Maier <tamiko@gentoo.org> wrote: >>>>
Just a quick though:
Looking at the man page of repoman it doesn't look to difficult to
replicate the behavior with pkgcheck. Meaning, we could think of
creating a drop-in replacement for "repoman [full]" (which would just
call pkgcheck) and "repoman commit" which actually does much more than >>>> just prepending the git commit summary line: repoman commit does
- update the manifest
- bail out if files are not correctly "git add"ed
- add the output of [pkgcheck] as a comment to the git commit
description
I wouldn't block anyone from doing this, but it's not something I'm
personally interested in pursuing. I see very little value here.
First, you're trying to justify replacing repoman on an entirely subjective >> opinion of "I think <foo> is superior", then when you get feedback on the
idea, you dismiss that feedback with more opinion.
Why do you not see value here? The actions described are actually quite a >> few useful steps in the process of checking a change into the tree. If you >> expect developers to do those steps on their own, that increases, not
decreases, the chances of making a mistake. Or are these steps already
handled by one of the other scripts in the replacement packages you propose? >> If so, please say so and identify which one(s).
My opinion is that any tool that replaces repoman should, at a minimum,
replace like functionality with like functionality, plus benefits or
enhancements. This looks more like a step backwards, not a step forwards.
I'd be interested in hearing your workflow, so we can capture it in
the table (mentioned earlier) so its clear how your existing workflow
will work with the new tools (or perhaps there is a gap, or we need to
craft / add additional tools?) I agree on the face it may not be
obvious what workflows look like.
On 10 Mar 2022, at 21:57, Joshua Kinard <kumba@gentoo.org> wrote:
I'd be interested in hearing your workflow, so we can capture it in
the table (mentioned earlier) so its clear how your existing workflow
will work with the new tools (or perhaps there is a gap, or we need to
craft / add additional tools?) I agree on the face it may not be
obvious what workflows look like.
My workflow is really rather standard when working in the tree itself. I work one package directory at a time, apply changes that I've tested outside of the tree in my local repo, eyeball everything a second time to make sure
I didn't miss something, regenerate the manifest, git add, run 'repoman full -d -x', fix any issues it finds (if any) and manifest/git add again, then 'repoman commit' and supply a commit message with sign-off. Lather, rinse, repeat for other packages.
On 3/10/2022 14:44, Andreas K. Huettel wrote:
I wouldn't block anyone from doing this, but it's not something I'm
personally interested in pursuing. I see very little value here.
First, you're trying to justify replacing repoman on an entirely subjective
opinion of "I think <foo> is superior" ...
Well, if you've ever tried it you'll notice that <foo> for <foo> != repoman actually finishes the checks within a finite amount of time. Kind of, the most blatant argument for ditching repoman, actually.
If this is a concern for some, has anyone looked into whether repoman can be fixed to be more efficient? If so, how was the determination made that it cannot be fixed and instead, needs to be replaced? It's been around for 20+ years. Surely someone has gotten annoyed enough to look at any issues it
has and attempt to fix them?
That said, I'm not terribly bothered by it. It is slow, don't get me wrong, but it's not slow enough that my workflow is significantly impacted. It catches most of the mistakes I've ever made before I make them so that I can fix them. For me, that's job well done.
--
Joshua Kinard
Gentoo/MIPS
kumba@gentoo.org
rsa6144/5C63F4E3F5C6C943 2015-04-27
177C 1972 1FB8 F254 BAD0 3E72 5C63 F4E3 F5C6 C943
"The past tempts us, the present confuses us, the future frightens us. And our lives slip away, moment by moment, lost in that vast, terrible in-between."
--Emperor Turhan, Centauri Republic
On Thu, Mar 10, 2022 at 10:27 AM Joshua Kinard <kumba@gentoo.org> wrote:
On 3/9/2022 16:00, Matt Turner wrote:
I'd like to deprecate and ultimately remove repoman. I believe that
dev-util/pkgcheck and pkgcommit (from app-portage/mgorny-dev-scripts)
are far superior replacements, and it makes sense to have people using
the same tool and seeing the same warnings as in the CI.
Are there any useful checks or behaviors of repoman that are missing
from pkgcheck and pkgcommit?
Thanks,
Matt
repoman has been //the// goto tool for checking in a change since before I >> was a developer in 2003. It does everything we need, in one simple tool.
Your proposal looks to replace repoman's functionality (and snark) with two >> or more packages, including tools from a package named after a fellow developer.
Additionally, "I believe that <foo> are far superior replacements" is an
entirely subjective opinion and, frankly, completely invalid as
justification to replace a tool that has worked fine for the last 20+ years. >> What is so fundamentally broken about repoman that cannot be fixed such
that it needs total replacement by multiple independent tools? Please
document. the pros and cons here so that we can all make an informed decision.
So here is the more basic argument, you can agree or disagree.
*The goal I want is for people to commit to the tree and not break things.*
If we could accomplish this with no tooling at all, that would be
great. Sadly humans are fallible and so we have tools to check their
work. Currently this tooling has two parts:
- pre-push tooling that you run prior to pushing.
- post-push CI tooling that informs you when you break the tree.
So holding to our goal of not breaking the tree, it's better for
developers to run the same QA tool in pre-push that CI is using,
because our metric for 'breaking the tree' is the CI tool and if the
CI tool passes locally, there is a strong likelihood it will not break
in CI either. Note this argument is generic, I'm not even saying which
tools are in use, or which tools are better, or anything like that.
Here we see Matt discussing a workflow he finds frustrating.
- A developer does a push.
- Their push breaks CI.
- He inquires about their workflow.
- He learns they did not run the CI tool in their pre-push workflow.
- He tells them they should run the CI tool in their pre-push workflow.
- This happens many times, causing this thread.
The point of the thread then is to convince people to run the CI tool
in pre-push, as a matter of policy, to reduce tree breakage and reduce
the occurrence of the above conversation.
So the generic argument above, we now get into the specifics.
I'm not opposed to making our tools better, but I think before anything can >> replace the RepoMan, several more boxes need to be ticked:
- functionality from multiple tools should be packaged into a single tool. >> * caveat: at least provide a wrapper that, using args, can invoke the
individual tools if needed.
I do not believe it's reasonable to provide a 'drop-in' backwards compatibility tool guarantee.
I believe we should provide a table of common repoman actions, with a
mapping to their replacements; so that users can understand how to do
similar actions.
- app-portage/mgorny-dev-scripts needs a new name. It's fine if it's
intended to only be a collection of useful scripts for individual developers >> on an as-needed basis, but if it's to be the Official Tool(TM), then it
needs to have a proper name. If not all of the scripts contained within it >> are applicable to the sole function of checking a change into the tree, then >> only the scripts that deal with change validation and committing should be >> broken out into a separate package, and ostensibly combined with any other >> tools/scripts into a single package, and preferably a single tool, to get
the job done.
I don't consider this a blocker, but I think it's mostly a discussion
between mattst88 and mgorny.
- all of our developer documentation needs to be updated to reference the
new tool and the new way of doing things, as well as a warning not to use
repoman any further after a set date. Additionally, a news item is probably >> advisable so that developers and proxy maintainers alike can get the message.
We should hold Matt accountable for updating any relevant development documentation, including the table I mentioned above.
I'm not sure a news item is strictly necessary, we might just p.mask
repoman with a link to the guide that Matt will need to write about
how repoman is being replaced.
- we need the snark. Gentoo is all about personality, and RepoMan has been >> scouring our neighborhoods for two decades and change, and while some may
think this is utterly frivolous, I will actually miss seeing those messages >> on the console every time I commit a change. They give the RepoMan
personality and a soul, and thus, contribute to the uniqueness that is Gentoo.
Cool.
I am going to nit-pick here, but pkgcheck pulls in pkgcore still. As far
asI know, pkgcore was meant to be a portage-like package manager, but it isn't maintained. So, can we break that dependency before we make
pkgcheck the official tool for qa checks? I would rather not have
pkgcore landing on everyone's systems unless it is usable. If I am wrong about pkgcore, please correct me and I'll be quiet, but if not let's
make pkgcheck independent from it before we deprecate repoman.
I stick to the officially-published method of checking and committing changes:
https://devmanual.gentoo.org/ebuild-maintenance/git/index.html
The two tools highlighted there for the bulk of the work is repoman
and pkgdev. repoman is cited twelve times, pkgdev is cited six times. pkgcheck is mentioned once. pkgcommit has no mentions.
From that, one should not be faulted for assuming that repoman is the
more important tool, if not preferred tool, with pkgdev coming in
second place.
pkgcheck comes across as entirely optional and even seems equivalent
to 'repoman full', and how would one know that pkgcommit even exists?
That all said, most of my checks are done pre-commit, not pre-push.Â
It's annoying trying to unwind a commit with a mistake in git, so I
aim to not have any in the commit itself. By the time I get to the
push stage, everything MUST be perfect before I hit 'git push'.Â
Cause once something is public, there's no going back. In a handful
of cases, I've gone as far as nuking my local copy of the tree, re-
cloning it, and re-doing everything again just to fix a mistake that
I should have caught pre-commit.
How about making an edit to repoman to have it throw a deprecation
warning for itself and to install whatever we all agree should
replace it?
On Thu, Mar 10, 2022 at 09:29:59AM -0800, Matt Turner wrote:
On Wed, Mar 9, 2022 at 11:09 PM Joonas Niilola <juippis@gentoo.org>
wrote:
On 9.3.2022 23.00, Matt Turner wrote:
I'd like to deprecate and ultimately remove repoman. I believe
that dev-util/pkgcheck and pkgcommit (from app-portage/mgorny-dev-scripts) are far superior replacements,
and it makes sense to have people using the same tool and
seeing the same warnings as in the CI.
Are there any useful checks or behaviors of repoman that are
missing from pkgcheck and pkgcommit?
Thanks,
Matt
I still fail to see the "why" in here. Repoman is better than
pure 'git commit' that I know some people still like to use, and
as long as it's kept maintained.
repoman is inferior to other tooling mentioned. The other tooling is actually run in CI. Developers should get the same warnings locally
as in CI. Restated another way: I'm tired of telling people to stop
using repoman or "pkgcheck would have caught that".
I am going to nit-pick here, but pkgcheck pulls in pkgcore still. As
far asI know, pkgcore was meant to be a portage-like package manager,
but it isn't maintained. So, can we break that dependency before we
make pkgcheck the official tool for qa checks? I would rather not have pkgcore landing on everyone's systems unless it is usable. If I am
wrong about pkgcore, please correct me and I'll be quiet, but if not
let's make pkgcheck independent from it before we deprecate repoman.
Thanks,
William
repoman is inferior to other tooling mentioned. The other tooling is
actually run in CI.
Developers should get the same warnings locally as in CI.
On Fri, Mar 11, 2022 at 9:14 AM Peter Stuge <peter@stuge.se> wrote:
Matt Turner wrote:
repoman is inferior to other tooling mentioned. The other tooling
is actually run in CI.
The problem seems to be that CI is running something other than
developers run, not the other way around.
Developers should get the same warnings locally as in CI.
I agree. And developers and their tools should not have to bend to
whatever CI does, I think the other way around makes more sense.
CI doesn't use repoman because of performance issues.
I think the problem is that making committers use the CI tool is
technically a fairly straightforward change; while everything you
discuss further in the thread is not; because it implies people will
work on improving tools that have shown little or no progress on
improving in the 15 years I've been in Gentoo.
What if pkgcore evolves to provide a portage-compatible API?
No API is planned and no one is working on it.
Then CI could use repoman without performance problems, and
developers would also enjoy improved performance, without spending
time on replacing tooling which already works for them.
The benefit of getting people to change their behavior is that the
benefits (less breakage, better tooling) are achieved now; as opposed
to in some hypothetical future where repoman is improved.
Note that we have been waiting for 'improved repoman' for about 8
years; so..I'm not willing to bet on it when we have better tooling
that works today and the primary complaint is that...what exactly?
The new workflow with pkgcheck was announced at the end of 2019: https://blogs.gentoo.org/mgorny/2019/12/12/a-better-ebuild-workflow-with-pure-git-and-pkgcheck
It's been 2 years, I think we can bring everyone into the fold here.
Looking into the future then maybe portage could even come to use
pkgcore for the low-level things that pkgcore does, then even users
could enjoy improved performance.
Many things could happen but again, if you talk to people who work on pkgcore; there is no concrete plan for this. So I don't see why we are betting on it happening.
If you read radhermit's blog: https://pkgcraft.github.io/posts/timesink-chronicles/ you can get one
take on the project and why it struggled.
-A
Right?
//Peter
Matt Turner wrote:
repoman is inferior to other tooling mentioned. The other tooling is actually run in CI.
The problem seems to be that CI is running something other than
developers run, not the other way around.
Developers should get the same warnings locally as in CI.
I agree. And developers and their tools should not have to bend to
whatever CI does, I think the other way around makes more sense.
CI doesn't use repoman because of performance issues.
What if pkgcore evolves to provide a portage-compatible API?
Then CI could use repoman without performance problems, and
developers would also enjoy improved performance, without spending
time on replacing tooling which already works for them.
Looking into the future then maybe portage could even come to use
pkgcore for the low-level things that pkgcore does, then even users
could enjoy improved performance.
Right?
//Peter
I stick to the officially-published method of checking and committing
changes:
https://devmanual.gentoo.org/ebuild-maintenance/git/index.html
The two tools highlighted there for the bulk of the work is repoman
and pkgdev. repoman is cited twelve times, pkgdev is cited six times.
pkgcheck is mentioned once. pkgcommit has no mentions.
From that, one should not be faulted for assuming that repoman is the
more important tool, if not preferred tool, with pkgdev coming in
second place. pkgcheck comes across as entirely optional and even
seems equivalent to 'repoman full', and how would one know that
pkgcommit even exists?
I believe the very purpose of this thread is to have a consensus/pre- announcement before actually editing the official documentation as part
of the process of deprecating repoman.
That all said, most of my checks are done pre-commit, not pre-push.
It's annoying trying to unwind a commit with a mistake in git, so I
aim to not have any in the commit itself. By the time I get to the
push stage, everything MUST be perfect before I hit 'git push'. Cause
once something is public, there's no going back. In a handful of
cases, I've gone as far as nuking my local copy of the tree, re-
cloning it, and re-doing everything again just to fix a mistake that I
should have caught pre-commit.
git commit --amend, interactive rebase, etc (all before push). I suggest
to learn them and embrace them. Re-cloning is not fun, just for
something that a quick interactive rebase or even a git reset -- hard destructive operation instead of re-clone would solve.
Also the benefit of using pkgcheck is to actually be able to make the
same checks that CI would do before you push, so you can amend your
commits to fix issues before they hit the server and CI and break the
tree. pkgcheck is so fast that it can do full tree checks in a
reasonable time (repoman would take days on a radiator mips when you go outside single package), and I believe has features to have it check all
your commits that haven't been pushed yet at once, checking only what it
can to not be too slow to not use (so you don't need to run the check
with each commit but for all of them once you commit - and if issues,
again, git interactive rebase).
On Thu, 10 Mar 2022, Alec Warner wrote:
I'm not sure a news item is strictly necessary, we might just p.mask
repoman with a link to the guide that Matt will need to write about
how repoman is being replaced.
The new workflow with pkgcheck was announced at the end of 2019: https://blogs.gentoo.org/mgorny/2019/12/12/a-better-ebuild-workflow-with-pure-git-and-pkgcheck
It's been 2 years, I think we can bring everyone into the fold here.
On 3/11/2022 13:25, Alec Warner wrote:
[snip]
The new workflow with pkgcheck was announced at the end of 2019:
https://blogs.gentoo.org/mgorny/2019/12/12/a-better-ebuild-workflow-with-pure-git-and-pkgcheck
It's been 2 years, I think we can bring everyone into the fold here.
I've searched my -dev archives for part of that URL, and the only hits I am getting is this e-mail thread. Was this URL previously shared on this mailing list or another? I do not track the Gentoo Blogs, so unless something is shared to the mailing lists, I will likely miss it.
That said, I will admit I am uncomfortable with post-commit, pre-push validation. I get that git is vastly different, and vastly superior, to CVS. Get it right the first time, and you don't have to worry about fixing it later -- CVS teaches you that very well, and it still works well for git workflows. Going back into git post-commit to fix things is still something I need to learn more about, as my git-fu is still pretty amateurish beyond the common basics. Especially when dealing with kernel patch maintenance
and maintaining lots of small, discrete changes that kernel upstream prefers.
I'd like to deprecate and ultimately remove repoman. I believe that dev-util/pkgcheck and pkgcommit (from app-portage/mgorny-dev-scripts)
Il giorno mer 9 mar 2022 alle ore 22:01 Matt Turner <mattst88@gentoo.org> ha scritto:
I'd like to deprecate and ultimately remove repoman. I believe that
dev-util/pkgcheck and pkgcommit (from app-portage/mgorny-dev-scripts)
Hi using `repoman manifest` in scripts here, what would be the correct replacement for that?
On 11 Mar 2022, at 19:39, Joshua Kinard <kumba@gentoo.org> wrote:
On 3/11/2022 03:54, Mart Raudsepp wrote:> Ühel kenal päeval, N, 10.03.2022 kell 18:18, kirjutas Joshua Kinard:
I stick to the officially-published method of checking and committing
changes:
https://devmanual.gentoo.org/ebuild-maintenance/git/index.html
The two tools highlighted there for the bulk of the work is repoman
and pkgdev. repoman is cited twelve times, pkgdev is cited six times.
pkgcheck is mentioned once. pkgcommit has no mentions.
From that, one should not be faulted for assuming that repoman is the
more important tool, if not preferred tool, with pkgdev coming in
second place. pkgcheck comes across as entirely optional and even
seems equivalent to 'repoman full', and how would one know that
pkgcommit even exists?
I believe the very purpose of this thread is to have a consensus/pre-
announcement before actually editing the official documentation as part
of the process of deprecating repoman.
I feel that the documentation should have had more mentions of these newer tools as their adoption by other developers accelerated. Documentation doesn't have to have a fixed point in time when it fully changes over. It can change organically, like almost everything else in the project.
[snip]
Also the benefit of using pkgcheck is to actually be able to make the
same checks that CI would do before you push, so you can amend your
commits to fix issues before they hit the server and CI and break the
tree. pkgcheck is so fast that it can do full tree checks in a
reasonable time (repoman would take days on a radiator mips when you go
outside single package), and I believe has features to have it check all
your commits that haven't been pushed yet at once, checking only what it
can to not be too slow to not use (so you don't need to run the check
with each commit but for all of them once you commit - and if issues,
again, git interactive rebase).
Speed is really not a big issue for me. I run repoman from my amd64 dev
box, and it's like, maybe 10-13 seconds at most during 'repoman full'? And my MIPS systems, while not the slowest of slow of that arch, they do teach you patience over the years.
The other bits you mention about pkgcheck do sound useful, though. But I am a stickler for official documentation, because my risk aversion level when committing to a public repo that can affect hundreds of thousands of users
is *extremely* high. When I first signed up as a dev and we had the
On 11 Mar 2022, at 19:51, Joshua Kinard <kumba@gentoo.org> wrote:
On 3/11/2022 13:25, Alec Warner wrote:
[snip]
The new workflow with pkgcheck was announced at the end of 2019:
https://blogs.gentoo.org/mgorny/2019/12/12/a-better-ebuild-workflow-with-pure-git-and-pkgcheck
It's been 2 years, I think we can bring everyone into the fold here.
I've searched my -dev archives for part of that URL, and the only hits I am getting is this e-mail thread. Was this URL previously shared on this mailing list or another? I do not track the Gentoo Blogs, so unless something is shared to the mailing lists, I will likely miss it.
That said, I will admit I am uncomfortable with post-commit, pre-push validation. I get that git is vastly different, and vastly superior, to CVS. Get it right the first time, and you don't have to worry about fixing it later -- CVS teaches you that very well, and it still works well for git workflows. Going back into git post-commit to fix things is still something I need to learn more about, as my git-fu is still pretty amateurish beyond the common basics. Especially when dealing with kernel patch maintenance
and maintaining lots of small, discrete changes that kernel upstream prefers.
On 10 Mar 2022, at 23:18, Joshua Kinard <kumba@gentoo.org> wrote:
On 3/10/2022 14:58, Alec Warner wrote:
On Thu, Mar 10, 2022 at 10:27 AM Joshua Kinard <kumba@gentoo.org> wrote: >>>>
On 3/9/2022 16:00, Matt Turner wrote:
I'd like to deprecate and ultimately remove repoman. I believe that
dev-util/pkgcheck and pkgcommit (from app-portage/mgorny-dev-scripts) >>>>> are far superior replacements, and it makes sense to have people using >>>>> the same tool and seeing the same warnings as in the CI.
Are there any useful checks or behaviors of repoman that are missing >>>>> from pkgcheck and pkgcommit?
Thanks,
Matt
repoman has been //the// goto tool for checking in a change since before I >>>> was a developer in 2003. It does everything we need, in one simple tool. >>>> Your proposal looks to replace repoman's functionality (and snark) with two
or more packages, including tools from a package named after a fellow developer.
Additionally, "I believe that <foo> are far superior replacements" is an >>>> entirely subjective opinion and, frankly, completely invalid as
justification to replace a tool that has worked fine for the last 20+ years.
What is so fundamentally broken about repoman that cannot be fixed such >>>> that it needs total replacement by multiple independent tools? Please >>>> document. the pros and cons here so that we can all make an informed decision.
Matt could've given more details about why pkgcheck is superior but I think this is actually showing/exposing the problem again: we've assumed that everybody should really know repoman lacks a significant number of the checks pkgcheck has, as well as being much slower.
Those are the two reasons why it's superior.
This goal I agree with, and I am rather pedantic about. If not mildly
So here is the more basic argument, you can agree or disagree.
*The goal I want is for people to commit to the tree and not break things.* >>
fearful of. We've all been there at least once in our dev-lives. It's
almost a rite of passage, if you will, to break the tree with a dumb commit >> at least once. Goal is to never repeat that mistake again.
Right. I spend a fair amount of time fixing issues with repoman doesn't
find but pkgcheck / CI does, or filing bugs for them.
If we could accomplish this with no tooling at all, that would be
great. Sadly humans are fallible and so we have tools to check their
work. Currently this tooling has two parts:
- pre-push tooling that you run prior to pushing.
- post-push CI tooling that informs you when you break the tree.
So holding to our goal of not breaking the tree, it's better for
developers to run the same QA tool in pre-push that CI is using,
because our metric for 'breaking the tree' is the CI tool and if the
CI tool passes locally, there is a strong likelihood it will not break
in CI either. Note this argument is generic, I'm not even saying which
tools are in use, or which tools are better, or anything like that.
Here we see Matt discussing a workflow he finds frustrating.
- A developer does a push.
- Their push breaks CI.
- He inquires about their workflow.
- He learns they did not run the CI tool in their pre-push workflow.
- He tells them they should run the CI tool in their pre-push workflow.
- This happens many times, causing this thread.
The point of the thread then is to convince people to run the CI tool
in pre-push, as a matter of policy, to reduce tree breakage and reduce
the occurrence of the above conversation.
I stick to the officially-published method of checking and committing changes:
https://devmanual.gentoo.org/ebuild-maintenance/git/index.html
The two tools highlighted there for the bulk of the work is repoman and
pkgdev. repoman is cited twelve times, pkgdev is cited six times. pkgcheck >> is mentioned once. pkgcommit has no mentions.
pkgcommit is just a wrapper around git and pkgcheck, so it is there.
It's not like you aren't allowed to make your own wrappers or aliases or scripts, right?
Back in mid-2002, it was exactly Gentoo's snarkness that tipped the scales >> for me. I'd just given up on FreeBSD-4.x on a sparc64 machine (old Sun
Blade 100) and LFS turned out to not be a lot of fun, but Gentoo worked fine >> on it. All of the colors on the terminal gave it zing and pop, and made it >> rather fun to work with. rpm and apt-get were dull; emerge was cheeky and >> playful! Still is to this day.
I have no objection to (and in fact would rather welcome) contributions
to make other tools more "Gentoo-like". Could you make a PR or provide
some patch to add some more personality to them?
On 11 Mar 2022, at 19:39, Joshua Kinard <kumba@gentoo.org> wrote:
On 3/11/2022 03:54, Mart Raudsepp wrote:> Ühel kenal päeval, N, 10.03.2022 >> kell 18:18, kirjutas Joshua Kinard:
I stick to the officially-published method of checking and committing
changes:
https://devmanual.gentoo.org/ebuild-maintenance/git/index.html
The two tools highlighted there for the bulk of the work is repoman
and pkgdev. repoman is cited twelve times, pkgdev is cited six times. >>>> pkgcheck is mentioned once. pkgcommit has no mentions.
From that, one should not be faulted for assuming that repoman is the
more important tool, if not preferred tool, with pkgdev coming in
second place. pkgcheck comes across as entirely optional and even
seems equivalent to 'repoman full', and how would one know that
pkgcommit even exists?
I believe the very purpose of this thread is to have a consensus/pre-
announcement before actually editing the official documentation as part
of the process of deprecating repoman.
I feel that the documentation should have had more mentions of these newer >> tools as their adoption by other developers accelerated. Documentation
doesn't have to have a fixed point in time when it fully changes over. It >> can change organically, like almost everything else in the project.
Well, I've done that. I've been adding pkgcheck and pkgdev to the devmanual over time, and to the wiki.
[snip]
Also the benefit of using pkgcheck is to actually be able to make the
same checks that CI would do before you push, so you can amend your
commits to fix issues before they hit the server and CI and break the
tree. pkgcheck is so fast that it can do full tree checks in a
reasonable time (repoman would take days on a radiator mips when you go
outside single package), and I believe has features to have it check all >>> your commits that haven't been pushed yet at once, checking only what it >>> can to not be too slow to not use (so you don't need to run the check
with each commit but for all of them once you commit - and if issues,
again, git interactive rebase).
Speed is really not a big issue for me. I run repoman from my amd64 dev
box, and it's like, maybe 10-13 seconds at most during 'repoman full'? And >> my MIPS systems, while not the slowest of slow of that arch, they do teach >> you patience over the years.
The other bits you mention about pkgcheck do sound useful, though. But I am >> a stickler for official documentation, because my risk aversion level when >> committing to a public repo that can affect hundreds of thousands of users >> is *extremely* high. When I first signed up as a dev and we had the
It is already mentioned in the devmanual, but we can add it in more places
if you specify which.
I've filed a PR against devmanual.git to remove references to repoman
and replace them with references to pkgdev where appropriate. Most
everywhere already had pkgdev/pkgcheck text in place so there wasn't
much to do. See: https://github.com/gentoo/devmanual/pull/274
the extra checks that pkgcheck runs haven't really applied to me.
On 11-03-2022 13:19:34 -0800, Matt Turner wrote:
I've filed a PR against devmanual.git to remove references to repoman
and replace them with references to pkgdev where appropriate.
Just wondering, is there a "migration guide" or something? I've never
used pkg* since joining in 2005. I can derive some things from the
first look at the below commit, but an "expert opinion" to just map the standard things from repoman to appropriate commands would be nice.
The devmanual has already been updated to contain information about pkgdev, in March 2021. See https://gitweb.gentoo.org/proj/devmanual.git/log/?qt=grep&q=pkgdev
I have opened a devmanual pull request to remove references to repoman that might suggest that it is still an appropriate tool to use. See https://github.com/gentoo/devmanual/pull/274
The next steps once the devmanual change is committed, I think, are
- Give the wiki similar treatment as the devmanual, replacing and removing references to repoman that would suggest it is an appropriate tool to use.
- Modify repoman to emit a warning informing users of its deprecation
- After some period of time, maybe 6 months, give last rites to repoman
- Delete repoman from portage.git
(and of course adding any features/behaviors we find lacking in pkgdev, et al)
I've filed a Council bug to approve the plan for deprecating repoman.
See https://bugs.gentoo.org/835013
pkgcheck is now considered the primary Gentoo tool for ebuild QA. repoman is no longer considered sufficient. Council condones the deprecation and removal of repoman by the portage team.
On Wed, Mar 09, 2022 at 01:00:37PM -0800, Matt Turner wrote:
I'd like to deprecate and ultimately remove repoman. I believe that dev-util/pkgcheck and pkgcommit (from app-portage/mgorny-dev-scripts)
There's also dev-util/pkgdev as an alternative to pkgcommit with more features for those that want them that will feel similar to repoman
- pkgdev commit -c/-b bugnum (repoman commit -c/-b bugnum)
- pkgdev manifest (repoman manifest)
- pkgdev push (runs pkgcheck scan --commits before pushing)
- updates copyright year / manifest on commit
- basic commit message templates, like auto "cat/pkg: add <version>" on
bumps without manual editing -- prepend cat/pkg: like other tools
if nothing specific
And other handy features like pkgdev mask --rites days
are far superior replacements, and it makes sense to have people using
the same tool and seeing the same warnings as in the CI.
Are there any useful checks or behaviors of repoman that are missing
from pkgcheck and pkgcommit?
Thanks,
Matt
--
ionen
On Sat, 19 Mar 2022, Zoltan Puskas wrote:
I've been using both repoman _and_ pkgcheck becasue I was not sure which
one covers all the checks I need to make. In fact [Pull Requests wiki](https://wiki.gentoo.org/wiki/Project:GitHub/Pull_requests) has a
big red warning box with the following message, and I quote:
"CI is not an excuse not to run repoman full, at least for the packages
being committed."
On Sat, 19 Mar 2022, Zoltan Puskas wrote:
[Please don't top-post.]
I've been using both repoman _and_ pkgcheck becasue I was not sure
which
one covers all the checks I need to make. In fact [Pull Requests wiki](https://wiki.gentoo.org/wiki/Project:GitHub/Pull_requests) has
a
big red warning box with the following message, and I quote:
"CI is not an excuse not to run repoman full, at least for the
packages
being committed."
That warning box also says: "Furthermore, pkgcheck is imperfect (yet
very fast) and may fail to notice some subtle breakages (especially if
USE flag masking/forcing) is involved."
Looks like the warning was added by mgorny in 2016 [1]. Have these
issues with pkgcheck been fixed in the meantime?
| Sysop: | Keyop |
|---|---|
| Location: | Huddersfield, West Yorkshire, UK |
| Users: | 475 |
| Nodes: | 16 (2 / 14) |
| Uptime: | 18:17:36 |
| Calls: | 9,487 |
| Calls today: | 6 |
| Files: | 13,617 |
| Messages: | 6,121,091 |