• Getting MIME headers right (was: I'm hacking on trn...want to join me?)

    From Richard@21:1/5 to All on Wed Apr 12 15:36:39 2023
    [Please do not mail me a copy of your followup]

    Michael =?ISO-8859-1?Q?B=E4uerle?= <michael.baeuerle@gmx.net> spake the secret code
    <AABkNrMFcqkAAAFo.A3.flnews@WStation5.stz-e.de> thusly:

    I think such mistakes would not occur if trn would automatically convert
    the content to quote into the encoding used by the editor.
    The source encoding is declared in the MIME header of the article.

    The source encoding *MAY* be declared in the MIME header of the article. There's no guarantee that it actually is declared as such. We've
    already stated that malformed messages can be injected into news due
    to user error (e.g. they didn't configure their editor to add
    appropriate MIME headers).

    The target encoding should be the one the editor is using (manually >configured, if trn cannot automatically detect it).

    How can trn know anything about your editor? It's entirely outside of
    trn's control.

    The conversion itself can be done with iconv.

    This is a unix-centric assumption. It *could* be done with iconv.

    If the user has configured e.g. US-ASCII to quote an article written in >Unicode, replacement characters should be inserted if the encoding
    conversion is not possible (instead of simply copying the bytes that are
    then interpreted wrong by the editor and the recipients).

    All of this seems outside the control of trn and mostly a consequence
    of how you edit the article. The existing NEWSPOSTER environment variable
    can be configured to a script that would do all that you propose.
    --
    "The Direct3D Graphics Pipeline" free book <http://tinyurl.com/d3d-pipeline>
    The Terminals Wiki <http://terminals-wiki.org>
    The Computer Graphics Museum <http://computergraphicsmuseum.org>
    Legalize Adulthood! (my blog) <http://legalizeadulthood.wordpress.com>

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Adam H. Kerman@21:1/5 to Richard on Wed Apr 12 15:48:54 2023
    legalize+jeeves@mail.xmission.com (Richard) wrote:
    Michael =?ISO-8859-1?Q?B=E4uerle?= <michael.baeuerle@gmx.net> spake:

    How did you get undecoded encoded word on the attribution line?

    . . .

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Richard@21:1/5 to All on Wed Apr 12 16:59:44 2023
    [Please do not mail me a copy of your followup]

    "Adam H. Kerman" <ahk@chinet.com> spake the secret code <u16jt6$32uql$3@dont-email.me> thusly:

    legalize+jeeves@mail.xmission.com (Richard) wrote:
    Michael =?ISO-8859-1?Q?B=E4uerle?= <michael.baeuerle@gmx.net> spake:

    How did you get undecoded encoded word on the attribution line?

    %[from] in my ATTRIBUTION setting.
    --
    "The Direct3D Graphics Pipeline" free book <http://tinyurl.com/d3d-pipeline>
    The Terminals Wiki <http://terminals-wiki.org>
    The Computer Graphics Museum <http://computergraphicsmuseum.org>
    Legalize Adulthood! (my blog) <http://legalizeadulthood.wordpress.com>

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Michael =?ISO-8859-1?Q?B=E4uerle?=@21:1/5 to Richard on Wed Apr 12 18:31:58 2023
    Richard wrote:
    Michael Bäuerle wrote:

    I think such mistakes would not occur if trn would automatically convert the content to quote into the encoding used by the editor.
    The source encoding is declared in the MIME header of the article.

    The source encoding *MAY* be declared in the MIME header of the article. There's no guarantee that it actually is declared as such. We've
    already stated that malformed messages can be injected into news due
    to user error (e.g. they didn't configure their editor to add
    appropriate MIME headers).

    Please forget incoming malformed messages completely. If something is
    already broken, it is completely acceptable that you need to repair it manually. My proposal is not about repairing malformed articles.

    What should be expected from a newsreader is correct behaviour with
    RFC 5536 conformant messages.
    Because RFC 5536 requires MIME, every article with 8-bit data in the
    body, but without a MIME declaration in the header, is malformed by
    definition.

    The target encoding should be the one the editor is using (manually configured, if trn cannot automatically detect it).

    How can trn know anything about your editor? It's entirely outside of
    trn's control.

    To be honest I don't know how trn works, I only see the results.

    If trn does not know what the editor expects, the user should be able
    to configure the encoding.

    The conversion itself can be done with iconv.

    This is a unix-centric assumption. It *could* be done with iconv.

    Nobody will complain if something else is used, as long as it works.

    If the user has configured e.g. US-ASCII to quote an article written in Unicode, replacement characters should be inserted if the encoding conversion is not possible (instead of simply copying the bytes that are then interpreted wrong by the editor and the recipients).

    All of this seems outside the control of trn and mostly a consequence
    of how you edit the article. The existing NEWSPOSTER environment variable can be configured to a script that would do all that you propose.

    Not every user is a programmer too.

    If this must be implemented with a script, it should be shipped with
    the newsreader and the documentation should explain how to produce
    RFC 5536 conformant articles with it.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Richard@21:1/5 to All on Wed Apr 12 17:21:03 2023
    [Please do not mail me a copy of your followup]

    Michael =?ISO-8859-1?Q?B=E4uerle?= <michael.baeuerle@gmx.net> spake the secret code
    <AABkNtz+CKEAAAdh.A3.flnews@WStation5.stz-e.de> thusly:

    Richard wrote:
    Michael Bäuerle wrote:
    The target encoding should be the one the editor is using (manually
    configured, if trn cannot automatically detect it).

    How can trn know anything about your editor? It's entirely outside of
    trn's control.

    To be honest I don't know how trn works, I only see the results.

    trn is a traditional command-line news reader and doesn't directly
    post anything at all. When you request to post a follow-up or new
    article, trn invokes your editor to prepare the message (including all
    headers appearing "naked" in the editor) and then confirms that you
    want to post the edited message. If confirmed, trn injects the
    article into the newsfeed with a program called inews that does the
    actual posting. The source for inews comes with trn and is
    distributed with trn but the act of injecting a news article is
    considered the responsibility of a program separate from trn and can
    be configured to be some other program besides inews.

    Similarly, composing an article -- including all it's headers -- is a responsibility of an editor program and not trn.

    If trn does not know what the editor expects, the user should be able
    to configure the encoding.

    They can, but often don't :).

    All of this seems outside the control of trn and mostly a consequence
    of how you edit the article. The existing NEWSPOSTER environment variable >> can be configured to a script that would do all that you propose.

    Not every user is a programmer too.

    True, but trn assumes a certain level of knowledge in order to use its
    more advanced features, such as configuring things through environment variables.

    Because trn doesn't own "the whole process" like a canned GUI-based
    news reader on a macOS or Windows system, it assumes the user knows
    how to deal with such things on their own.

    If this must be implemented with a script, it should be shipped with
    the newsreader and the documentation should explain how to produce
    RFC 5536 conformant articles with it.

    There really isn't any way to write such a script and make it useful
    to everyone. Trn can supply examples (and does) but invoking it unconditionally would be a mistake.
    --
    "The Direct3D Graphics Pipeline" free book <http://tinyurl.com/d3d-pipeline>
    The Terminals Wiki <http://terminals-wiki.org>
    The Computer Graphics Museum <http://computergraphicsmuseum.org>
    Legalize Adulthood! (my blog) <http://legalizeadulthood.wordpress.com>

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Adam H. Kerman@21:1/5 to Richard on Wed Apr 12 17:58:13 2023
    legalize+jeeves@mail.xmission.com (Richard) wrote:
    "Adam H. Kerman" <ahk@chinet.com> spake:
    legalize+jeeves@mail.xmission.com (Richard) wrote:
    Michael =?ISO-8859-1?Q?B=E4uerle?= <michael.baeuerle@gmx.net> spake:

    How did you get undecoded encoded word on the attribution line?

    %[from] in my ATTRIBUTION setting.

    Right, but you aren't decoding first before creating the attribution line.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Richard@21:1/5 to All on Wed Apr 12 18:52:08 2023
    [Please do not mail me a copy of your followup]

    "Adam H. Kerman" <ahk@chinet.com> spake the secret code <u16rfl$emd$1@dont-email.me> thusly:

    legalize+jeeves@mail.xmission.com (Richard) wrote:
    "Adam H. Kerman" <ahk@chinet.com> spake:
    legalize+jeeves@mail.xmission.com (Richard) wrote:
    Michael =?ISO-8859-1?Q?B=E4uerle?= <michael.baeuerle@gmx.net> spake:

    How did you get undecoded encoded word on the attribution line?

    %[from] in my ATTRIBUTION setting.

    Right, but you aren't decoding first before creating the attribution line.

    The From: header value appears in the attribution the way it appears in
    the message.
    --
    "The Direct3D Graphics Pipeline" free book <http://tinyurl.com/d3d-pipeline>
    The Terminals Wiki <http://terminals-wiki.org>
    The Computer Graphics Museum <http://computergraphicsmuseum.org>
    Legalize Adulthood! (my blog) <http://legalizeadulthood.wordpress.com>

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Adam H. Kerman@21:1/5 to Richard on Wed Apr 12 22:15:53 2023
    legalize+jeeves@mail.xmission.com (Richard) wrote:
    "Adam H. Kerman" <ahk@chinet.com> spake:
    legalize+jeeves@mail.xmission.com (Richard) wrote:
    "Adam H. Kerman" <ahk@chinet.com> spake: >>>>legalize+jeeves@mail.xmission.com (Richard) wrote:
    Michael =?ISO-8859-1?Q?B=E4uerle?= <michael.baeuerle@gmx.net> spake:

    How did you get undecoded encoded word on the attribution line?

    %[from] in my ATTRIBUTION setting.

    Right, but you aren't decoding first before creating the attribution line.

    The From: header value appears in the attribution the way it appears in
    the message.

    I checked my environment variables and the environment in trnrc. I just
    don't see what's sending the encoded word on From to be decoded.

    When I'm reading, if I press "v" I see all headers and the encoded text.
    If I then press "F", quoted text from the precursor article that was
    encoded has not been decoded but that doesn't apply to forming the
    attribution line.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Richard@21:1/5 to All on Wed Apr 12 23:43:07 2023
    [Please do not mail me a copy of your followup]

    "Adam H. Kerman" <ahk@chinet.com> spake the secret code <u17aip$2if3$2@dont-email.me> thusly:

    legalize+jeeves@mail.xmission.com (Richard) wrote:
    "Adam H. Kerman" <ahk@chinet.com> spake:
    legalize+jeeves@mail.xmission.com (Richard) wrote:
    "Adam H. Kerman" <ahk@chinet.com> spake: >>>>>legalize+jeeves@mail.xmission.com (Richard) wrote:
    Michael =?ISO-8859-1?Q?B=E4uerle?= <michael.baeuerle@gmx.net> spake:

    How did you get undecoded encoded word on the attribution line?

    %[from] in my ATTRIBUTION setting.

    Right, but you aren't decoding first before creating the attribution line.

    The From: header value appears in the attribution the way it appears in
    the message.

    I checked my environment variables and the environment in trnrc. I just
    don't see what's sending the encoded word on From to be decoded.

    No encoding or decoding is happening. It's just literally putting the
    value of the header into the body of the article.

    When I'm reading, if I press "v" I see all headers and the encoded text.
    If I then press "F", quoted text from the precursor article that was
    encoded has not been decoded but that doesn't apply to forming the >attribution line.

    I'm running:

    Trn version: 4.0-test77 (Sep 1, 2010).

    Perhaps you are running the UTF-8 hacked derivative from acli.
    --
    "The Direct3D Graphics Pipeline" free book <http://tinyurl.com/d3d-pipeline>
    The Terminals Wiki <http://terminals-wiki.org>
    The Computer Graphics Museum <http://computergraphicsmuseum.org>
    Legalize Adulthood! (my blog) <http://legalizeadulthood.wordpress.com>

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Adam H. Kerman@21:1/5 to Richard on Thu Apr 13 00:42:08 2023
    Richard <> wrote:
    [Please do not mail me a copy of your followup]

    "Adam H. Kerman" <ahk@chinet.com> spake the secret code ><u17aip$2if3$2@dont-email.me> thusly:

    legalize+jeeves@mail.xmission.com (Richard) wrote:
    "Adam H. Kerman" <ahk@chinet.com> spake: >>>>legalize+jeeves@mail.xmission.com (Richard) wrote:
    "Adam H. Kerman" <ahk@chinet.com> spake: >>>>>>legalize+jeeves@mail.xmission.com (Richard) wrote:
    Michael =?ISO-8859-1?Q?B=E4uerle?= <michael.baeuerle@gmx.net> spake:

    How did you get undecoded encoded word on the attribution line?

    %[from] in my ATTRIBUTION setting.

    Right, but you aren't decoding first before creating the attribution line. >>
    The From: header value appears in the attribution the way it appears in >>>the message.

    I checked my environment variables and the environment in trnrc. I just >>don't see what's sending the encoded word on From to be decoded.

    No encoding or decoding is happening. It's just literally putting the
    value of the header into the body of the article.

    When I'm reading, if I press "v" I see all headers and the encoded text.
    If I then press "F", quoted text from the precursor article that was >>encoded has not been decoded but that doesn't apply to forming the >>attribution line.

    I'm running:

    Trn version: 4.0-test77 (Sep 1, 2010).

    Perhaps you are running the UTF-8 hacked derivative from acli.

    No MIME headers in my articles unless I add them.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Frank Slootweg@21:1/5 to Adam H. Kerman on Thu Apr 13 16:44:42 2023
    Adam H. Kerman <ahk@chinet.com> wrote:
    legalize+jeeves@mail.xmission.com (Richard) wrote:
    "Adam H. Kerman" <ahk@chinet.com> spake:
    legalize+jeeves@mail.xmission.com (Richard) wrote:
    "Adam H. Kerman" <ahk@chinet.com> spake: >>>>legalize+jeeves@mail.xmission.com (Richard) wrote:
    Michael =?ISO-8859-1?Q?B=E4uerle?= <michael.baeuerle@gmx.net> spake:

    How did you get undecoded encoded word on the attribution line?

    %[from] in my ATTRIBUTION setting.

    Right, but you aren't decoding first before creating the attribution line.

    The From: header value appears in the attribution the way it appears in
    the message.

    I checked my environment variables and the environment in trnrc. I just
    don't see what's sending the encoded word on From to be decoded.

    When I'm reading, if I press "v" I see all headers and the encoded text.
    If I then press "F", quoted text from the precursor article that was
    encoded has not been decoded but that doesn't apply to forming the attribution line.

    FWIW, tin works similarly.

    I.e. it decodes the encoded From text, both to display the author of
    the currently viewed article and to form the attribution line when
    composing a followup.

    And - as happened here - if some user agent does not decode the
    encoded From text and uses that encoded From text in an attribution
    line, then that encoded text is left encoded, because it's now part of
    the body (not the headers) of the article and hence will not be decoded
    unless MIME headers say to do so (but Richard's followup to Michael did
    not contain any MIME headers).

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Richard@21:1/5 to All on Thu Apr 13 17:27:55 2023
    [Please do not mail me a copy of your followup]

    Frank Slootweg <this@ddress.is.invalid> spake the secret code <u19iic.m3o.1@ID-201911.user.individual.net> thusly:

    And - as happened here - if some user agent does not decode the
    encoded From text and uses that encoded From text in an attribution
    line, then that encoded text is left encoded, because it's now part of
    the body (not the headers) of the article and hence will not be decoded >unless MIME headers say to do so (but Richard's followup to Michael did
    not contain any MIME headers).

    Even if I had added MIME headers, it would still appear encoded in the
    body, because in trn %[header] gives the value of the named header field
    and doesn't do anything else to the value. The value was QP encoded,
    so it showed up in the body as QP encoded.
    --
    "The Direct3D Graphics Pipeline" free book <http://tinyurl.com/d3d-pipeline>
    The Terminals Wiki <http://terminals-wiki.org>
    The Computer Graphics Museum <http://computergraphicsmuseum.org>
    Legalize Adulthood! (my blog) <http://legalizeadulthood.wordpress.com>

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)