1. Forum
  2. Usenet
  3. LINUX.DEBIAN.PROJECT

  • Re: Need to run script, but blocked out of packages.debian.org

    From =?utf-8?Q?=C3=89tienne?= Mollier@21:1/5 to All on Mon Dec 19 21:10:01 2022
    Hi Nicholas,

    Nicholas Hubbard, on 2022-12-19:
    Hey, I wrote a script to download the .debian.tar.xz file for every
    package on packages.debian.org.

    You sound like you would like to build a mirror (or sort of) of
    Debian source code. Please kindly lookup the documentation on
    setting up a mirror[1], you will find information as of whether
    this makes sense for your use case, and methods to get archives
    without hammering web front-ends.

    [1]: https://www.debian.org/mirror/ftpmirror

    I was running the script and all of a
    sudden it stopped working, and I am now unable to access
    packages.debian.org from my home IP. It seems that I got (rightfully)
    flagged as a bot. I was wondering if there was anyway somebody could
    make an exception for me so I can run my script without getting blocked
    out. I only need to run the script once. I would be happy to share the
    exact code I am running, and would be fine scheduling the script to run
    at a low-traffic time of day.

    There's always some place where the sun shines. But if your
    concerned about using low-traffic time of day, you can get the
    .debian.tar.xz files from your nearest public repository, which
    may have low-ish traffic early in your morning.

    Have a nice day, :)
    --
    Étienne Mollier <emollier@emlwks999.eu>
    Fingerprint: 8f91 b227 c7d6 f2b1 948c 8236 793c f67e 8f0d 11da
    Sent from /dev/pts/2, please excuse my verbosity.
    On air: Kaipa - Screwed-Upness

    -----BEGIN PGP SIGNATURE-----

    iQIzBAABCgAdFiEEj5GyJ8fW8rGUjII2eTz2fo8NEdoFAmOgw4QACgkQeTz2fo8N Edomnw/+NsjUJn43orj7zqQgwwyuR7k+VbGWXZiNvrolf5+emHWRiLk8+pMw+TKv zKqYrIS8kYvAejz00ezAuxSZuEwtCh5UIi/j96JK9SyDLtKqaUsEtBkYAk4AnKI0 WZJkOQcmWqjMu837IMOvY58vp0LjkYc50Gb0tdn6BFt9g0QoTGrv+wfWg3AAZTCs hizsMthhhK0OJRYzFJVs4EJWOO8mKAhnq/r4pkOkc3gi7Xex03yZiJcxrz4aftUC r89jDXv/e1ql6CJSAUYOz66XUG0M8prEFRk8jk1Uvi+IyV94052Yf1i+7hExYzrF S+M/Nf8zsEibH9ZO6Vd+u1ullt/q3ONGvv/JBmZ415RipFzNpwiBL/1GJEYT+EiI aNc8eowlxxTOoVQPr9T1gI+iL49UtL2zJv8ISl7LVxuPG0DaNr2VS5FJ2RGyP7um 8HdvovNSaLfF2EZHDY5TZGcQBJpLMDLMBLXold4C0PAKSeIIe3/XAChkBWg4lyW9 jTwmUhPuZVt6TjxmS2dX+CpIzh+NbVwZ3EXj026lLNft6QQpVsvaAH27TFUiQRyX 2xnwvGEbI/77aTcOSMV1WrM+OdEKV4JLVr7aP5G4LNUtz1C4ncey2FBWd+KuPiRV gDWbtOaoxGmwDtU35463dgte9zs5gKDMB04LXRoStMI0PHfff0o=
    =ZExy
    -----E
  • From Nicholas Hubbard@21:1/5 to All on Mon Dec 19 20:50:01 2022
    Hey, I wrote a script to download the .debian.tar.xz file for every
    package on packages.debian.org. I was running the script and all of a
    sudden it stopped working, and I am now unable to access
    packages.debian.org from my home IP. It seems that I got (rightfully)
    flagged as a bot. I was wondering if there was anyway somebody could
    make an exception for me so I can run my script without getting blocked
    out. I only need to run the script once. I would be happy to share the
    exact code I am running, and would be fine scheduling the script to run
    at a low-traffic time of day.

    Thanks,
    Nicholas

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Roberto A. Foglietta@21:1/5 to All on Mon Dec 19 21:20:01 2022
    Il Lun 19 Dic 2022, 20:48 Nicholas Hubbard <nicholashubbard@posteo.net> ha scritto:

    Hey, I wrote a script to download the .debian.tar.xz file for every
    package on packages.debian.org. I was running the script and all of a
    sudden it stopped working, and I am now unable to access
    packages.debian.org from my home IP. It seems that I got (rightfully)
    flagged as a bot. I was wondering if there was anyway somebody could
    make an exception for me so I can run my script without getting blocked
    out. I only need to run the script once. I would be happy to share the
    exact code I am running, and would be fine scheduling the script to run
    at a low-traffic time of day.


    Hi Nicholas,

    wait that the block ends and then limit the number of pararell connections this will not block you or at least the block will last less than one
    minute, supposing that is server is configured in a reasonable way.

    In this file, you will find an example of limitations on 8 downloading processes in parallel

    https://github.com/robang74/tinycore-editor/blob/main/tinycore/provides/tcmkdepsdb.sh

    Best regards, R-

    <div dir="auto"><div class="gmail_quote" dir="auto"><div dir="ltr" class="gmail_attr">Il Lun 19 Dic 2022, 20:48 Nicholas Hubbard &lt;<a href="mailto:nicholashubbard@posteo.net" target="_blank" rel="noreferrer">nicholashubbard@posteo.net</a>&gt; ha
    scritto:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hey, I wrote a script to download the .debian.tar.xz file for every<br>
    package on <a href="http://packages.debian.org" rel="noreferrer noreferrer noreferrer" target="_blank">packages.debian.org</a>. I was running the script and all of a<br>
    sudden it stopped working, and I am now unable to access<br>
    <a href="http://packages.debian.org" rel="noreferrer noreferrer noreferrer" target="_blank">packages.debian.org</a> from my home IP. It seems that I got (rightfully)<br>
    flagged as a bot. I was wondering if there was anyway somebody could<br>
    make an exception for me so I can run my script without getting blocked<br> out. I only need to run the script once. I would be happy to share the<br> exact code I am running, and would be fine scheduling the script to run<br>
    at a low-traffic time of day.<br></blockquote></div><div dir="auto"><br></div><div dir="auto">Hi Nicholas,</div><div dir="auto"><br></div><div dir="auto"> wait that the block ends and then limit the number of pararell connections this will not block you
    or at least the block will last less than one minute, supposing that is server is configured in a reasonable way.</div><div dir="auto"><br></div><div dir="auto"> In this file, you will find an example of limitations on 8 downloading processes in
    parallel </div><div dir="auto"><br></div><div dir="auto"><a href="https://github.com/robang74/tinycore-editor/blob/main/tinycore/provides/tcmkdepsdb.sh">https://github.com/robang74/tinycore-editor/blob/main/tinycore/provides/tcmkdepsdb.sh</a></div><div
    dir="auto"><br></div><div dir="auto">Best regards, R-</div></div>

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Nicholas Hubbard@21:1/5 to emollier@emlwks999.eu on Mon Dec 19 23:00:02 2022
    I see, a mirror should work fine for me! Thanks for the help!

    Nicholas
    Étienne Mollier <emollier@emlwks999.eu> writes:

    [[PGP Signed Part:Undecided]]
    Hi Nicholas,

    Nicholas Hubbard, on 2022-12-19:
    Hey, I wrote a script to download the .debian.tar.xz file for every
    package on packages.debian.org.

    You sound like you would like to build a mirror (or sort of) of
    Debian source code. Please kindly lookup the documentation on
    setting up a mirror[1], you will find information as of whether
    this makes sense for your use case, and methods to get archives
    without hammering web front-ends.

    [1]: https://www.debian.org/mirror/ftpmirror

    I was running the script and all of a
    sudden it stopped working, and I am now unable to access
    packages.debian.org from my home IP. It seems that I got (rightfully)
    flagged as a bot. I was wondering if there was anyway somebody could
    make an exception for me so I can run my script without getting blocked
    out. I only need to run the script once. I would be happy to share the
    exact code I am running, and would be fine scheduling the script to run
    at a low-traffic time of day.

    There's always some place where the sun shines. But if your
    concerned about using low-traffic time of day, you can get the
    .debian.tar.xz files from your nearest public repository, which
    may have low-ish traffic early in your morning.

    Have a nice day, :)

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Santiago Vila@21:1/5 to All on Wed Dec 21 15:40:01 2022
    El 19/12/22 a las 20:31, Nicholas Hubbard escribió:
    Hey, I wrote a script to download the .debian.tar.xz file for every
    package on packages.debian.org.

    You might want to download and parse this file:

    https://deb.debian.org/debian/dists/sid/main/source/Sources.xz

    and maybe you don't need to use packages.debian.org at all.

    Thanks.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)