• Musings about redirection and (only) new information overwriting existi

    From Janis Papanagnou@21:1/5 to All on Sun Aug 14 13:40:29 2022
    Occasionally it happens that I want to redirect output to a file, but
    create that file or overwrite an existing file only if there's actually
    new data available. (So I don't want to and cannot use '>' or '>>'.)

    Since I haven't seen that feature in shell I usually use awk to achieve
    that function by something like

    ... | awk -v fn="some-file" '{ print > fn }'

    I used that pattern already a couple of times so that I now have it in a
    local bin-directory as executable file 'cf'

    # cf - conditionally create file
    awk -v fn="${1:?}" '{ print > fn }'

    using it in contexts like

    news-process | cf latest-news

    where the latest-news file gets not overwritten if there's no newer news.


    Redirecting with '>' to a file in shell will always overwrite the file,
    unless 'noclobber' shell option is set; in that case you need '>|' to
    overwrite an existing file.

    In ksh there's also the '>;' redirection available. It writes output to
    a temporary file and creates/overwrites the file only if no error occurs.

    The tee(1) command has also no option to create or overwrite files only conditionally if data is present, as far as I can see.

    Is there some tool or shell function that I missed that does conditional overwrites as described?


    To reproduce standard input to standard output as well I added option -t
    so that I can see what's getting written to that file (or to process any
    output further in a pipeline).

    #!/bin/ksh
    #
    # cf - conditionally create file if data is present

    t=0
    while getopts ":t" opt
    do
    case ${opt} in
    (t) t=1 ;;
    (\?) printf "Usage: cf [-t] filename\n" ; exit 1 ;;
    esac
    done
    shift OPTIND-1
    fn=${1:?}
    awk -v fn="${fn}" -v t="${t}" '{ print > fn } t'


    I can use that function in my environment but I think it would better
    fit in shell, maybe as a new shell redirection ('>@', like ksh's '>;'),
    or have it as feature of existing standard tools, maybe in tee(1) with
    a new option supporting this semantics, like the -a is used to support
    '.

    Janis

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From =?UTF-8?B?T8SfdXo=?=@21:1/5 to Janis Papanagnou on Tue Aug 16 08:52:41 2022
    On 8/14/22 2:40 PM, Janis Papanagnou wrote:
    Occasionally it happens that I want to redirect output to a file, but
    create that file or overwrite an existing file only if there's actually
    new data available. (So I don't want to and cannot use '>' or '>>'.)

    Since I haven't seen that feature in shell I usually use awk to achieve
    that function by something like

    ... | awk -v fn="some-file" '{ print > fn }'

    Does this have any advantage over using a temporary file? Like:

    ... > temp-file
    test -s temp-file && mv temp-file some-file

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Helmut Waitzmann@21:1/5 to All on Tue Aug 16 17:38:03 2022
    Oğuz <oguzismailuysal@gmail.com>:
    On 8/14/22 2:40 PM, Janis Papanagnou wrote:
    Occasionally it happens that I want to redirect output to a file,
    but create that file or overwrite an existing file only if
    there's actually new data available. (So I don't want to and
    cannot use '>' or '>>'.)

    Since I haven't seen that feature in shell I usually use awk to
    achieve that function by something like

    ... | awk -v fn="some-file" '{ print > fn }'

    Does this have any advantage over using a temporary file?


    It depends.  While


    ... > temp-file
    test -s temp-file && mv temp-file some-file


    copies the access permissions from “temp-file” to “some-file” and replaces “some-file” if it already exists,


    ... | awk -v fn="some-file" '{ print > fn }'


    as well as


    ... > temp-file &&
    if test -s temp-file
    then
    cat -- temp-file >| some-file &&
    rm -f -- temp-file
    fi


    will overwrite “some-file” rather than replace it.  Also, if “some-file” doesn't already exist, it will be just created rather
    than being created and having the access modes modified according to
    the access modes of “temp-file”.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Janis Papanagnou@21:1/5 to Helmut Waitzmann on Wed Aug 17 13:02:45 2022
    On 16.08.2022 17:38, Helmut Waitzmann wrote:
    Oğuz <oguzismailuysal@gmail.com>:
    On 8/14/22 2:40 PM, Janis Papanagnou wrote:
    Occasionally it happens that I want to redirect output to a file, but
    create that file or overwrite an existing file only if there's
    actually new data available. (So I don't want to and cannot use '>'
    or '>>'.)

    Since I haven't seen that feature in shell I usually use awk to
    achieve that function by something like

    ... | awk -v fn="some-file" '{ print > fn }'

    Does this have any advantage over using a temporary file?

    I haven't pondered about advantages or disadvantages of the one or
    the other option. Here are just a few obvious thoughts I have...

    It doesn't create a temporary file. - An advantage for itself; since
    it happens - as we see in your test/mv based code below demonstrated -
    that you forget to clean up that temporary file. (So it's getting yet
    more complex than the workaround already is if you want to maintain a
    tidy runtime environment.)

    It's also less complex; an additional test and a conditional command.
    The awk pattern doesn't create the file in the first place, if it's
    not needed.

    Also if you're at the system disk memory limit your code will create
    an additional file requiring (if only temporary) additional space
    that the system might not be able to allocate.

    If you dislike awk you can of course put your test/mv commands set
    in a file 'cf' (or whatever you call it) and it behaves (mostly) the
    same. The main point of my post was to have a compact pattern of a
    command I find useful, and with the tee-extension I also suggested,
    I can also use it in a pipe. The other point of my post was whether
    that semantics is already present in shell in some way I missed, or
    in some tools; I wouldn't want to use my own tool if there's already
    some standard tool supporting it.


    It depends. [...]
    [ considerations about access modes ]

    Janis

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Helmut Waitzmann@21:1/5 to All on Wed Aug 17 17:55:09 2022
    Janis Papanagnou <janis_papanagnou@hotmail.com>:

    I haven't pondered about advantages or disadvantages of the one or
    the other option. Here are just a few obvious thoughts I have...

    […]

    I totally agree with you (therefore I deleted your explanations).


    The other point of my post was whether that semantics is already
    present in shell in some way I missed, or in some tools;

    I don't know of any.  Of course one can do that using the shell and “cat”:


    (
    unset -v -- line &&
    lf="$( printf '%s\n' '' '.' )" && lf="${lf%.}" &&
    if
    IFS= read -r -- line && line="${line}${lf}"
    ${line:+:} false
    then
    exec > some-file &&
    printf '%s' "$line" &&
    cat
    fi
    )


    I wouldn't want to use my own tool if there's already some standard
    tool supporting it.

    I don't know of a better way to do that than yours.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)