Hi All,

I was hoping for some advise regarding how I could improve the glusterfs package for users (and myself).  At least those using openrc, but I
suspect similar may be applicable to systemd, but I have no idea how
systemd handles network mounts so perhaps someone could chip in here on
that front too.

Specifically the mounting of glusterfs file systems currently has a few problems (glusterd if server=localhost, network, dns(?) and fuse availability).  For now the focus is on the fuse aspect since that's the biggest annoyance by far.

Mounting happens via the netmount service.

In order for glusterfs to mount successfully the fuse module needs to be available when mount.glusterfs is invoked.  This can be achieved in one
of two ways:

1.  Compile the module statically into the kernel.
2.  Arrange for fuse service to be started prior to netmount (using say /etc/conf.d/netmount rc_need="fuse")

I make note that there is specific code in /etc/init.d/netmount in
depend() to handle nfsclient (if there are any nfs and nfs4 mounts it automatically (unless if the filesystems are also labaled noauto) to
want nfsclient.

Two questions:

1.  Would a PR against https://github.com/openrc/openrc/ to add special
case code for glusterfs into netmount have a reasonable chance of being accepted?  I don't like this, it just pushes towards an ever-growing
list of special cases, but it's possibly preferable to having users to
figure out they need to edit /etc/conf.d/netmount to add rc_need="fuse" ?

2.  Would it not be an improvement to consider having a more generic
mechanism for other packages to add dependency requirements for
netmount, for example:

/etc/netmound.depend.d/glusterfs(.sh?) could contain a function called depend_glusterfs() { } which gets invoked if a glusterfs filesystem will
want to be mounted, which I would suggest would have something like:

depend_glusterfs()
{
    use glusterd
    need net fuse
}

Then if the answer to 2 is yes (which I feel it would be), then a few implementation details (Will attempt a PR):

What would be the best location for having these files dropped?
Generally I'd say let's stay out of /etc/ for this since these are system-controlled dependencies, however, some users may have things
rigged and may want to be able to edit or even outright ignore these dependencies ... ?

/lib/netmount.dependencies.d/ being over-shadowed by /etc/netmount.dependencies.d?

So any file for which an equivalent in /etc/netmount.dependencies.d/
exist is ignored in /lib/ (similar to udev/rules.d)?

Source all files in those locations, or try them based on filesystem
types to be mounted only (first /etc then /lib variants)?

Kind regards,
Jaco

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --------------zoUe3W50PMEhey0WPc9N0BRy
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

On 8.4.2024 12.51, Jaco Kroon wrote:

In order for glusterfs to mount successfully the fuse module needs to be available when mount.glusterfs is invoked.  This can be achieved in one
of two ways:

1.  Compile the module statically into the kernel.
2.  Arrange for fuse service to be started prior to netmount (using say /etc/conf.d/netmount rc_need="fuse")

3. Add "/sbin/modprobe -q fuse" to the init.d file's start_pre()
function, ExecStartPre with systemd, and make the ebuild warn about CONFIG_FUSE_FS with linux-info.eclass.

-- juippis


--------------zoUe3W50PMEhey0WPc9N0BRy--

-----BEGIN PGP SIGNATURE-----

iQGTBAEBCgB9FiEEltRJ9L6XRmDQCngHc4OUK43AaWIFAmYXcQ5fFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDk2 RDQ0OUY0QkU5NzQ2NjBEMDBBNzgwNzczODM5NDJCOERDMDY5NjIACgkQc4OUK43A aWLXzAf6AxioXurYugabr0s6Zz9XgfP0BWjnb2DZWJY/TMkBpEGtIQ++r2XrLOWl RNxuNZ2wyL5oaRxu8/28uudPvswrseakguK/QqrVBsUN0pjYTJI3FNDaStN+2qO3 h17DR6h55mRB1WagSCiQhzSl9Zs03RSzl8OFQPKy3nxhUx4HV0TUJFrYDvskdMYE EBy0WUj4HHvnSbD44CLQeuUbpQz5NyKK+Xs+SdTswoTWlXjHBDLxN/bxFc9iy+TU 0clD7SeXEPSm3fDB9JUY+e473ZsAH3vhrTpD+KCho60JsUO7usTxCh0yd+V8xmLm rRoJ2bXX6KrPgPbXHx4l/pTtvNnNTQ==
=5sOt
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi Joonas,

Thanks for the below.  Further comments there.

On 2024/04/11 07:11, Joonas Niilola wrote:

On 8.4.2024 12.51, Jaco Kroon wrote:

In order for glusterfs to mount successfully the fuse module needs to be
available when mount.glusterfs is invoked.  This can be achieved in one
of two ways:

1.  Compile the module statically into the kernel.
2.  Arrange for fuse service to be started prior to netmount (using say
/etc/conf.d/netmount rc_need="fuse")

3. Add "/sbin/modprobe -q fuse" to the init.d file's start_pre()
function, ExecStartPre with systemd, and make the ebuild warn about CONFIG_FUSE_FS with linux-info.eclass.

The latter can certainly be done and makes sense (only required if
you're using the fuse mount, so if USE=fuse at least).

The former doesn't make sense to do blindly in /etc/init.d/netmount
(which belongs to sys-apps/openrc, not glusterfs).

If you look at /etc/init.d/netmount it has some special logic in
depend() to want nfsclient if (and only if) there is at least one
filesystem with fs type nfs or nfs4.

The logic for depending on /etc/init.d/fuse should be similar, but I
don't think it makes sense to keep indefinitely expanding that depend()
for every possible future filesystem that may have some special need
like this.  So I think what we should rather do is find all fstab
entries with _netdev and !noauto's fstype, and iterate those and add the relevant want's from there (for openrc at least), in some mechanisms
where packages other than openrc can *supply* the relevant dependency
list (eg, glusterfs package would say that it want's net, dns and fuse).

Something similar for systemd would be great, but I'd have to study up
on systemd a bit before I can comment in greater detail.

At an absolute minimum I think we should amend netmount to add "use
fuse" such that if fuse is added to the relevant runlevels it will start
before netmount (and then I an arrange that  message be added to the
glusterfs ebuild that fuse should be added to the default (where
netmount is) runlevel).  As it is one can rc_need=fuse in /etc/conf.d/netmount, or rc_use=fuse and add fuse to default runlevel,
but I believe we can do better than either of these options.

Kind regards,
Jaco

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --------------iCp7L4td0S0Z0yPSgouUjlYG
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hey,

On 11.4.2024 9.14, Jaco Kroon wrote:

The latter can certainly be done and makes sense (only required if
you're using the fuse mount, so if USE=fuse at least).

The former doesn't make sense to do blindly in /etc/init.d/netmount
(which belongs to sys-apps/openrc, not glusterfs).

well I was thinking about putting that into glusterfs's init file.

-- juippis

--------------iCp7L4td0S0Z0yPSgouUjlYG--

-----BEGIN PGP SIGNATURE-----

iQGTBAEBCgB9FiEEltRJ9L6XRmDQCngHc4OUK43AaWIFAmYXtUNfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDk2 RDQ0OUY0QkU5NzQ2NjBEMDBBNzgwNzczODM5NDJCOERDMDY5NjIACgkQc4OUK43A aWIFRgf/UOisKYqT4cnMybJzFEYg+/OHZAicQBppwZDFJ9VhU3gJkMuwq95MLt7X dCjKr/UpapoMPSfpQaoPSQekoMTuWBnvnTgrxstqrKv1iVGzzngnSC9Cvy3wlvcw 46+/tHWVnO0fLuKxMIesH/sNdRg5yrrXTGNjKM3sxgRS1bd6Qg1OhqinULvVTdo1 +uj0wqZWms8lS8VIFGeLUsqyTbHjRqKIc+jzqNccSQi/SlsB5hkkKRUetfgOaSll Sf0Q3MMc/wtiVwsSvyKNed4GZbsnzo6wFYKhvUA5LeL3N0zqdFUKGU8mqbbmCi3Y KOyW3JG8Gl0lmO5L6PCezf/YcVkVNg==
=T+M0
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi Joonas,

On 2024/04/11 12:02, Joonas Niilola wrote:

Hey,

On 11.4.2024 9.14, Jaco Kroon wrote:

The latter can certainly be done and makes sense (only required if
you're using the fuse mount, so if USE=fuse at least).

The former doesn't make sense to do blindly in /etc/init.d/netmount
(which belongs to sys-apps/openrc, not glusterfs).

well I was thinking about putting that into glusterfs's init file.

Well, that's an obvious assumption now that I think about it, but
incorrect for what I'm looking at.

There's two init scripts for glusterfs, glusterd and glusterfsd.

glusterfsd init script is being dragged along for historic purposes and
comes from before I got involved, and I believe this was the way bricks
were brought up prior to glusterfs version 3.0, and it does look like
there is (u)mount stuff in there too.  This init script already has
stuff for "need fuse" if it's mounting a glusterfs filesystem.  It looks interesting even now for *mounting* file systems, but in my opinion not
for managing volumes.

IMHO the more modern/better way is to bring up glusterd on nodes that
*host* the volumes, ie, where bricks reside, and to then mount
filesystems from /etc/fstab using netmount as part of mounting all
network filesystem.  Otherwise you need to duplicate the init config for
every mountpoint and specify a large number of arguments in those ... in
my opinion it just gets messy quite quickly.

glusterd then manages starting/stopping of brick, shd and other
processes related to any volumes.

netmount handles mounting of network (including glusterfs) filesystems.

In many scenarios the storage nodes and those that consume them are independent.  In this scenario glusterd (along with bricks and shd's)
will run on the storage nodes but not on the "compute" nodes, and there
is no dependency between glusterd and netmount.

glusterd does need to start before netmount if (and only if) there are glusterfs mounts in /etc/fstab that depends on the local glusterd for
finding volume information.  This is hard(ish) to determine (reliably),
but given "fstabinfo" in openrc-run I could amend /etc/init.d/glusterd's
depend to do "before netmount" iff there are filesystems in /etc/fstab
that's relevant.  We don't (currently) have such deployments, and we
generally do have glusterfs mounts where we run bricks, even if only to
be able to inspect what's going on in the "constructed" filesystem, so
the explicit "before netmount" in glusterd doesn't bother me too much personally (even if it starts before netmount and it's not needed, who
cares?  So lots of effort for little gain).

glusterfs and fuse.glusterfs has already been added into /lib/rc/rc-functions.sh.  So technically I no longer need to flag my
mounts with _netdev.

Incidentally:  I *suspect* the noauto detection in netmount's depend
will only work if noauto is the *first* option for any given nfs mountpoint.

Anyway, I would thus like to suggest two "tweaks" to openrc here:

1.  net_fs_list needs to become more dynamic such that other packages
(such as glusterfs) can add to the list dynamically.
2.  packages should be allowed to hook into netmount's depend() phase.

If 2 isn't acceptable, I'll just send the desired changes for 2 directly
into openrc as a PR, and that kinda makes point 1 pointless as well.

Kind regards,
Jaco

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Mon, Apr 8, 2024 at 5:51 AM Jaco Kroon <jaco@uls.co.za> wrote:

Hi All,

I was hoping for some advise regarding how I could improve the glusterfs package for users (and myself). At least those using openrc, but I
suspect similar may be applicable to systemd, but I have no idea how
systemd handles network mounts so perhaps someone could chip in here on
that front too.

Specifically the mounting of glusterfs file systems currently has a few problems (glusterd if server=localhost, network, dns(?) and fuse availability). For now the focus is on the fuse aspect since that's the biggest annoyance by far.

Mounting happens via the netmount service.

In order for glusterfs to mount successfully the fuse module needs to be available when mount.glusterfs is invoked. This can be achieved in one
of two ways:

1. Compile the module statically into the kernel.
2. Arrange for fuse service to be started prior to netmount (using say /etc/conf.d/netmount rc_need="fuse")

This doesn't sound right. Why does the kernel module need to be loaded explicitly?
The kernel should auto-load the module when /dev/fuse is opened for
the first time.
/dev/fuse should get created via tmpfiles.d via kmod-static-nodes.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)