Mirror an OBS deb repository locally

Story

I run an OBS repository for all my packages, and it is available at the main site: https://build.opensuse.org/project/show/home:bgstack15.

But I wanted to mirror this for myself, so I don’t have to configure all my systems to point outward to get updates. I already host a Devuan ceres mirror for myself, and so mirroring this Open Build System repository is the last step to be self-hosting entirely for all systems except the mirror server.

I first dabbled with debmirror, but it kept wanting to try rsync despite my best configuration, plus it really insists on using the dists/ directory which isn’t used in the OBS deb repo design. So, I researched scraping down a whole site, and I found httrack which exists to serve a local copy of an Internet site. Bingo!

After a few hours of work, here is my solution for mirroring an OBS deb repo locally.

Solution

Create a user who will own the files and execute the httrack command, because httrack didn’t want to be run as root. Also, this new user can’t munge other data.

useradd obsmirror

Configure a script (available at gitlab)

#!/bin/sh
# File: /etc/installed/obsmirror.sh
# License: CC-BY-SA 4.0
# Author: bgstack15
# Startdate: 2020-01-05 18:01
# Title: Script that scrapes down OBS site to serve a copy to intranet
# Purpose: save down my OBS site so I can serve it locally
# History:
# Usage:
#    in a cron job: /etc/cron.d/mirror.cron
#       50	12	*	*	*	root	/etc/installed/obsmirror.sh 1>/dev/null 2>&1
# Reference:
#    https://unix.stackexchange.com/questions/114044/how-to-make-wget-download-recursive-combining-accept-with-exclude-directorie?rq=1
#    man 1 httrack
#    https://software.opensuse.org//download.html?project=home%3Abgstack15&package=freefilesync
# Improve:
#    use some text file as a list of recently-synced URLs, and if today's URL matches a recent one, then run the httrack with the --update flag. Probably keep a running list forever.
# Documentation:
#    Download the release key and trust it.
#       curl -s http://repo.example.com/mirror/obs/Release.key | apt-key add -
#    Use a sources.list.d/ file with contents:
#       deb https://repo.example.com/mirror/obs/ /
# Dependencies:
#    binaries: curl httrack grep head tr sed awk chmod chown find rm ln
#    user: obsmirror

logfile="/var/log/obsmirror/obsmirror.$( date "+%FT%H%M%S" ).log"
{
   test "${DEBUG:-NONE}" = "FULL" && set -x
   inurl="http://download.opensuse.org/repositories/home:/bgstack15/Debian_Unstable"
   workdir=/tmp/obs-stage
   outdir=/var/www/mirror/obs
   thisuser=obsmirror
   echo "logfile=${logfile}"

   mkdir -p "${workdir}" ; chmod "0711" "${workdir}" ; chown "${thisuser}:$( id -Gn obsmirror )" "${workdir}" 
   cd "${workdir}"
   # get page contents
   step1="$( curl -s -L "${inurl}/all" )"
   # get first listed package
   step2="$( echo "${step1}" | grep --color=always -oE 'href="[a-zA-Z0-9_.+\-]+\.deb"' | head -n1 | grep -oE '".*"' | tr -d '"' )"
   # get full url to a package
   step3="$( curl -s -I "${inurl}/all/${step2}" | awk '/Location:/ {print $2}' )"
   # get directory of the mirror to save down
   step4="$( echo "${step3}" | sed -r -e "s/all\/${step2}//;" -e 's/\s*$//;' )"
   # get domain of full url
   domainname="$( echo "${step3}" | grep -oE '(ht|f)tps?:\/\/[^\/]+\/' | cut -d'/' -f3 )"
   echo "TARGET URL: ${step4}"
   test -z "${DRYRUN}" && {
      # clean workdir of specific domain name in use right now.
      echo su "${thisuser}" -c "rm -rf \"${workdir:-SOMETHING}/${domainname:-SOMETHING}\""
      su "${thisuser}" -c "rm -rf \"${workdir:-SOMETHING}/${domainname:-SOMETHING}\"*"
      # have to skip the orig.tar.gz files because they are large and slow down the sync process significantly.
      echo su "${thisuser}" -c "httrack \"${step4}\" -*.orig.t* -v --mirror --update -s0 -r3 -%e0 \"${workdir}\""
      time su "${thisuser}" -c "httrack ${step4} -*.orig.t* -v --mirror --update -s0 -r3 -%e0 ${workdir}"
   }
   # -s0 ignore robots.txt
   # -r3 only go down 3 links
   # -%e0 follow 0 links to external sites

   # find most recent directory of that level
   levelcount="$(( $( printf "%s" "${inurl}" | tr -dc '/' | wc -c ) - 1 ))"
   subdir="$( find "${workdir}" -mindepth "${levelcount}" -maxdepth "${levelcount}" -type d -name 'Debian_Unstable' -printf '%T@ %p\n' | sort -n -k1 | head -n1 | awk '{print $2}' )"

   # if the work directory actually synced
   if test -n "${subdir}" ;
   then

      printf "%s " "DIRECTORY SIZE:"
      du -sxBM "${subdir:-.}"
      mkdir -p "$( dirname "${outdir}" )"
      # get current target of symlink
      current_target="$( find "${outdir}" -maxdepth 0 -type l -printf '%l\n' )"

      # if the current link is pointing to a different directory than this subdir
      if test "${current_target}" != "${subdir}" ;
      then
         # then replace it with a link to this one
         test -L "${outdir}" && unlink "${outdir}"
         echo ln -sf "${subdir}" "${outdir}"
         ln -sf "${subdir}" "${outdir}"
      fi

   else
      echo "ERROR: No subdir found, so cannot update the symlink."
   fi

   # disable the index.html with all the httrack comments and original site links
   find "${workdir}" -iname '*index.html' -exec rm {} +
} 2>&1 | tee -a "${logfile}"

And place this in cron!

#       50	12	*	*	*	root	/etc/installed/obsmirror.sh 1>/dev/null 2>&1

Explanation of script

So the logic is a little convoluted, because the OBS front page actually redirects downloads to various mirrors where the files are kept. So I needed to learn what the actual site is, and then pull down that whole site.
I couldn’t just use httrack –getfiles because it makes just a flat directory, which breaks the Packages contents’ accuracy to the paths of the package files. But I didn’t want the whole complex directory structure, just the repository structure. So I make a symlink to it in my actual web contents location.

Depending on third-party packages or repos

A recent incident has caught my attention, where a Ubuntu PPA owner decided to restrict access to his PPAs after some bad feelings he got from an interaction with a stingy company.

As many in the reddit thread commented, it is unwise to pull in dependencies from third-parties. You cannot always expect them to remain available, or always trust them. In fact, you probably shouldn’t trust them. Obviously, there is a wide spectrum of opinions on the topic. This isn’t a moral issue, but it is an important possibly business-continuity one.

Disclosures for my public repositories

I package some programs myself, for Fedora/CentOS and Devuan ceres. Of course they have dependencies, and a few even have some third-party dependencies. For some of my projects, I have actually taken the effort to package up their dependencies as well so my repositories are sufficient (with main distro repos of course) for the actual package I care about.

I made the decision on CentOS 6, because I don’t actually have any extant systems myself, that I will rely on third-party repositories for some key dependencies. I am uncertain this information is publicly viewable on the copr, and I do not wish to hide it. The copr front page shows that the various repos are “[Modified]” but they do not provide links to what additional repositories they depend on. And the EPEL8 dependencies are because there were no official repos hosting certain packages yet, but that should rectified over time.

Holiday greetings

Also, happy holidays or Merry Christmas or whatever holiday greetings you want. This post was originally published on December 22, 2019 so it’s the last one before Christmas.

Thoughts about yum repo server and defined repositories

The internal server used as a yum repository runs EL6, and it servers EL6 and EL7 (read: CentOS) yum repos. An admin tried installing mkisofs, which wanted to come by default from the c7-base repository. It also wanted to upgrade bash and glibc. Well, installing a post-usrmerge bash (CentOS 7+) on CentOS6 caused all sorts of havoc. I had to load a rescue iso, boot, and copy /usr/bin/bash and /usr/bin/sh to the correct locations. Then my system would actually boot again.

I was getting an interesting error:

init: Failed to spawn rcS pre-start process: unable to execute: No such file or directory
init: Failed to spawn rcS post-stop process: unable to execute: No such file or directory

Also, kernel options rghb and quiet are really annoying and I always disable them.

So, the moral of the story is: always be very careful running yum on a yum repo server. Double-check what repos your package will pull in.

List yum repos that have packages installed and are still defined

yum repolist all $( yum list installed | awk '$NF ~ /@/ && $NF !~ /anaconda/ {split($NF,a,"@");print a[2];}' | sort | uniq ) | awk 'NR > 2 && $1 !~ /repolist/ {split($1,a,"/");print a[1]}'

The idea is to list each installed package which also lists the repo it was installed from, and then organize those repos. Then list them and show the ones that are still defined in yum’s repository files.

reposync fails: “No more mirrors to try.”

Symptom

The following message occurs, when running a yum reposync.

# time /usr/bin/reposync --source --repoid=EL7 -m --download-metadata --download_path="/var/www/html/yum/EL/EL7_Mirror"
Repository hosting is listed more than once in the configuration
firefox-60.6.0-3.0.1.el7_6.src FAILED
firefox-60.6.0-3.0.1.el7_6.src: [Errno 256] No more mirrors to try.                            ]  0.0 B/s |    0 B  --:--:-- ETA<[pre>

This could occur during an ansible play.

TASK [sync preprod EL7 to prod] ******************************************************************************************************************
fatal: [yumserver01]: FAILED! => {"changed": true, "cmd": "/usr/bin/reposync --source --repoid=EL7 -m --download-metadata --download_path=\"/var/www/html/yum/EL/EL7_Mirror\"", "delta": "0:04:35.805969", "end": "2019-04-17 08:27:22.119744", "msg": "non-zero return code", "rc": 1, "start": "2019-04-17 08:22:46.313775", "stderr": "", "stderr_lines": [], "stdout": "Repository hosting is listed more than once in the configuration\n\rfirefox-60.6.1-1.0.1.el7_6.src FAILED                                          \n\r(1/34): firefox-60.6.0-3.0 0% [                 ]  0.0 B/s |    0 B   --:-- ETA \r\rfirefox-60.6.0-3.0.1.el7_6.src FAILED                                          \n\r(1/34): firefox-60.6.0-3.0 0% [                 ]  0.0 B/s |    0 B   --:-- ETA \r\r(1/34): freerdp-1.0.2-15.e 0% [                 ]  0.0 B/s | 2.5 MB   --:-- ETA \r\rfreerdp-1.0.2-15.el7_6.1.src.r FAILED                                          \n\r(1/34): freerdp-1.0.2-15.e 0% [                 ]  0.0 B/s | 2.5 MB   --:-- ETA \r\rfreerdp-1.0.2-15.el7_6.1.src.r FAILED                                          \n\r(1/34): freerdp-1.0.2-15.e 0% [                 ]  0.0 B/s | 2.5 MB   --:-- ETA \r\rfreerdp-1.0.2-15.el7_6.1.src.r FAILED                                          \n\r(1/34): freerdp-1.0.2-15.e 0% [                 ]  0.0 B/s | 2.5 MB   --:-- ETA \r\rfreerdp-1
.............................. TRUNCATED ..............................
 0.0 B/s | 2.5 MB   --:-- ETA ", "", "tzdata-java-2019a-1.el7.noarch FAILED                                          ", "", "(1/34): tzdata-java-2019a- 0% [                 ]  0.0 B/s | 2.5 MB   --:-- ETA ", "libwsman1-2.6.3-6.git4391e5c.el7_6.i686: [Errno 256] No more mirrors to try.", "openwsman-client-2.6.3-6.git4391e5c.el7_6.i686: [Errno 256] No more mirrors to try.", "tuned-utils-2.10.0-6.0.1.el7_6.3.noarch: [Errno 256] No more mirrors to try.", "openwsman-server-2.6.3-6.git4391e5c.el7_6.x86_64: [Errno 256] No more mirrors to try.", "sbd-1.3.1-8.2.el7_6.1.x86_64: [Errno 256] No more mirrors to try.", "libwsman1-2.6.3-6.git4391e5c.el7_6.x86_64: [Errno 256] No more mirrors to try.", "sbd-1.3.1-8.2.el7_6.1.src: [Errno 256] No more mirrors to try.", "freerdp-1.0.2-15.el7_6.1.x86_64: [Errno 256] No more mirrors to try.", "freerdp-plugins-1.0.2-15.el7_6.1.x86_64: [Errno 256] No more mirrors to try.", "openwsman-client-2.6.3-6.git4391e5c.el7_6.x86_64: [Errno 256] No more mirrors to try.", "ocfs2-tools-1.8.6-11.el7.x86_64: [Errno 256] No more mirrors to try.", "python-2.7.5-77.0.1.el7_6.src: [Errno 256] No more mirrors to try.", "tuned-2.10.0-6.0.1.el7_6.3.noarch: [Errno 256] No more mirrors to try.", "firefox-60.6.0-3.0.1.el7_6.src: [Errno 256] No more mirrors to try.", "freerdp-libs-1.0.2-15.el7_6.1.i686: [Errno 256] No more mirrors to try.", "freerdp-1.0.2-15.el7_6.1.src: [Errno 256] No more mirrors to try.", "freerdp-libs-1.0.2-15.el7_6.1.x86_64: [Errno 256] No more mirrors to try.", "ghostscript-9.07-31.el7_6.10.i686: [Errno 256] No more mirrors to try.", "ocfs2-tools-1.8.6-11.el7.src: [Errno 256] No more mirrors to try.", "tzdata-java-2019a-1.el7.noarch: [Errno 256] No more mirrors to try.", "ghostscript-9.07-31.el7_6.10.x86_64: [Errno 256] No more mirrors to try.", "openwsman-2.6.3-6.git4391e5c.el7_6.src: [Errno 256] No more mirrors to try.", "tuned-profiles-cpu-partitioning-2.10.0-6.0.1.el7_6.3.noarch: [Errno 256] No more mirrors to try.", "firefox-60.6.1-1.0.1.el7_6.src: [Errno 256] No more mirrors to try.", "thunderbird-60.6.1-1.0.1.el7_6.src: [Errno 256] No more mirrors to try.", "openwsman-server-2.6.3-6.git4391e5c.el7_6.i686: [Errno 256] No more mirrors to try.", "libssh2-1.4.3-12.el7_6.2.src: [Errno 256] No more mirrors to try.", "pcs-0.9.165-6.0.3.el7_6.1.x86_64: [Errno 256] No more mirrors to try.", "openwsman-python-2.6.3-6.git4391e5c.el7_6.x86_64: [Errno 256] No more mirrors to try.", "pcs-0.9.165-6.0.3.el7_6.1.src: [Errno 256] No more mirrors to try.", "tuned-2.10.0-6.0.1.el7_6.3.src: [Errno 256] No more mirrors to try.", "tzdata-2019a-1.el7.src: [Errno 256] No more mirrors to try.", "ghostscript-9.07-31.el7_6.10.src: [Errno 256] No more mirrors to try."]}
        to retry, use: --limit @/etc/ansible/retries/reposync-prod.retry
PLAY RECAP **********************************************************************************************************************************************
yumserver01               : ok=1    changed=0    unreachable=0    failed=1

Possible causes

The original rpm files could be malformed, so even though the reposync client is successfully downloading whatever the server provided, the rpm does not match the checksum in the yum metadata files.

Fix #1

  1. Run the command manually to learn the whole list. Save it down to a file.
    # time /usr/bin/reposync --source --repoid=EL7 -m --download-metadata --download_path="/var/www/html/yum/EL/EL7_Mirror"
    Repository hosting is listed more than once in the configuration
    firefox-60.6.0-3.0.1.el7_6.src FAILED
    firefox-60.6.0-3.0.1.el7_6.src: [Errno 256] No more mirrors to try.                            ]  0.0 B/s |    0 B  --:--:-- ETA

    The file used in the documented example consisted of about 30 packages.
    file: bad.in

    firefox-60.6.0-3.0.1.el7_6.src
    firefox-60.6.1-1.0.1.el7_6.src
    freerdp-1.0.2-15.el7_6.1.src
    freerdp-1.0.2-15.el7_6.1.x86_64
    freerdp-libs-1.0.2-15.el7_6.1.i686
    freerdp-libs-1.0.2-15.el7_6.1.x86_64
    freerdp-plugins-1.0.2-15.el7_6.1.x86_64
    ghostscript-9.07-31.el7_6.10.i686
    ghostscript-9.07-31.el7_6.10.src
  2. On the reposync client delete or move to a temp directory the failed files. You could use a shell script similar to the following.
    #!/bin/sh
    INFILE=/root/bad.in
    INDIR=/var/www/html/yum/EL/EL7_Mirror/EL7
    OUTDIR=/root/orig/
    mkdir -p "${OUTDIR}"
    find $(
    for word in $( cat "${INFILE}" ) ;
    do
       find "${INDIR}" -name "${word}*rpm"
    done
    ) -print -exec mv {} "${OUTDIR}" \;
    
  3. On the repo server, manually download the files. You could modify the input file to have the upstream URLs.
    bad.in.url

    http://upstream.example.com/repo/EL/EL7/latest/x86_64/getPackage/tuned-utils-2.10.0-6.0.1.el7_6.3.noarch.rpm
    http://upstream.example.com/repo/EL/EL7/latest/x86_64/getPackage/tzdata-java-2019a-1.el7.noarch.rpm
    http://upstream.example.com/repo/EL/EL7/latest/x86_64/getPackageSource/firefox-60.6.0-3.0.1.el7_6.src.rpm
    http://upstream.example.com/repo/EL/EL7/latest/x86_64/getPackageSource/firefox-60.6.1-1.0.1.el7_6.src.rpm
    http://upstream.example.com/repo/EL/EL7/latest/x86_64/getPackageSource/freerdp-1.0.2-15.el7_6.1.src.rpm
    .... TRUNCATED .....                                                                                                                                                     
    

    You could use a shell script similar to the following.

    #!/bin/sh
    INFILE=/root/bad.in.url
    INDIR=/var/www/html/yum/EL/EL7_Mirror/EL7/getPackage/
    INDIRSRC=/var/www/html/yum/EL/EL7_Mirror/EL7/getPackageSource/
     
    for word in $( cat "${INFILE}" ) ;
    do
       if echo "${word}" | grep -q getPackageSource ;
       then
          cd "${INDIRSRC}"
       else
          cd "${INDIR}"
       fi
       #echo "$( pwd ) --> ${word}"
       wget "${word}"
    done
  4. Rerun the reposync manually and make sure it works.

dnf install build deps

If you want to build a package, but need all of its buildrequires packages, use this command:

sudo dnf builddep wxGTK3

I leave my source repositories off, so be sure to do any –enablerepo=fedora-source,updates-source as necessary.

For debian family

Try a cool tool named “mk-build-deps” as documented over at https://www.guyrutenberg.com/2017/09/23/use-mk-build-deps-instead-of-apt-get-build-dep/

References

Weblinks

Automatically install build dependencies prior to building an RPM package [stackoverflow.com]

Fixing problem Repository ceres InRelease changed its Label value from Master to Devuan

tl;dr

rm /var/lib/apt/lists/*

The fix

If you encounter an error that resembles the following, on Devuan GNU/Linux, there is a fix for it!

# sudo apt-get update
Reading package lists... Done
E: Repository 'http://packages.roundr.devuan.org/merged ceres InRelease' changed its 'Label' value from 'Master' to 'Devuan'
N: This must be accepted explicitly before updates for this repository can be applied. See apt-secure(8) manpage for details.

There’s a new label in use, it seems. Big deal, except for the fact you can’t really get around it. The apt-secure(8) page does not seem to provide any answers.

To view the current labels for the enabled repos:

# apt policy
Package files:
 100 /var/lib/dpkg/status
     release a=now
 500 http://packages.devuan.org/merged ceres/non-free i386 Packages
     release v=1.0.0,o=Devuan,a=unstable,n=ceres,l=Master,c=non-free,b=i386
     origin packages.devuan.org
 500 http://packages.devuan.org/merged ceres/contrib i386 Packages
     release v=1.0.0,o=Devuan,a=unstable,n=ceres,l=Master,c=contrib,b=i386
     origin packages.devuan.org
 500 http://packages.devuan.org/merged ceres/main i386 Packages
     release v=1.0.0,o=Devuan,a=unstable,n=ceres,l=Master,c=main,b=i386
     origin packages.devuan.org
Pinned packages:

The fix is to remove the cached lists for the repositories and fetch it all again.

rm /var/lib/apt/lists/*

That’s all there is to it! Then run apt-get update again, and you’re back on your way.

References

Weblinks

  1. man page apt_preferences(5)

Local resources

  1. bash autocomplete for apt and apt-get

Setup Yum Repository with Security Metadata

Define repository

Prepare the repo file on the server, so clients can download it.

cd /var/www/html/yum
cat <<'EOF' > hosting.repo
[hosting]
name=Hosting Delivery
baseurl=http://yum5.ipa.example.com/yum/hosting/
enabled=0
gpgcheck=0
EOF

Make or update repository

Use createrepo tool to make the repository. A wrapper script for creating or updating the existing repository is shown here.

tf=/usr/local/bin/updaterepo.sh
cat <<'EOF' > "${tf}"
#!/bin/sh
# reference:
#    https://gitlab.com/bgstack15/mirror/blob/master/usr/share/mirror/examples/rpm/update-smith122rpm.sh

# Prepare directory and files
test -z "${UR_REPODIR}" && UR_REPODIR=/var/www/html/yum/hosting
test -z "${UR_BASEURL}" && UR_BASEURL=http://yum5.ipa.example.com/yum/hosting
test -z "${UR_OWNERSHIP}" && UR_OWNERSHIP="root.root"
test -z "${UR_FILETYPES}" && UR_FILETYPES="rpm"

find "${UR_REPODIR}" -exec chown "${UR_OWNERSHIP}" {} + 1>/dev/null 2>&1
find "${UR_REPODIR}" -type f -exec chmod "0664" {} + 1>/dev/null 2>&1
find "${UR_REPODIR}" -type d -exec chmod "0775" {} + 1>/dev/null 2>&1
chmod 0754 "$0"
restorecon -RF "${UR_REPODIR}"

# Prepare repo for rpm
cd "${UR_REPODIR}"
createrepo -v -u "${UR_BASEURL}" --basedir "${UR_REPODIR}" --simple-md-filenames --no-database --update --pretty .
EOF

Run this script.

/usr/local/bin/updaterepo.sh

Manually make the security metadata

The security metadata that yum interprets is stored in updateinfo.xml.gz. To make this file and include it in repomd.xml, you need to prepare it and learn some information about it.

This is a trim example of updateinfo.xml. Please see the epel metadata for a full example. I do not have an automatic process for generating this file yet.

tf=updateinfo.xml
cat <<'EOF' > "${tf}"
<?xml version="1.0" encoding="UTF-8"?>
<updates>
  <update status="final" type="security" version="1" from="bgstack15@gmail.com">
    <id>HELP-210217</id>
    <title>bgscripts-core update</title>
    <release>Enterprise Linux 7</release>
    <issued date="2018-04-02"/>
    <rights>CC-BY-SA 4.0</rights>
    <description>bgscripts-core
[1.3-8]
- latest version from upstream
</description>
    <solution>This update is internal to the company.</solution>
    <references>
      <reference href="https://gitlab.com/bgstack15/bgscripts" type="self" title="bgscripts-core" />
    </references>
    <pkglist>
      <collection short="bgscripts">
        <name>bgscripts suite</name>
        <package name="bgscripts-core" version="1.3-8" release="" epoch="0" arch="noarch">
          <filename>bgscripts-core-1.3-8.noarch.rpm</filename>
          <sum type="md5">eaa20075720bf12d6e837a4f546241ab</sum>
        </package>
     </collection>
    </pkglist>
  </update>
</updates>
EOF

Update the repo metadata to include updateinfo.xml

A yum repository includes metadata of the package metadata, and stores this meta-metadata in repomd.xml. Insert the metadata for this new file, updateinfo.xml in the repomd file.
This script is an update version of updaterepo.sh, which was listed earlier in this document.

tf=/usr/local/bin/updaterepo.sh
        cat <<'EOF' > "${tf}"
#!/bin/sh
# reference:
#    https://gitlab.com/bgstack15/mirror/blob/master/usr/share/mirror/examples/rpm/update-smith122rpm.sh

# Prepare directory and files
test -z "${UR_REPODIR}" && UR_REPODIR=/var/www/html/yum/hosting
test -z "${UR_BASEURL}" && UR_BASEURL=http://yum5.ipa.example.com/yum/hosting
test -z "${UR_OWNERSHIP}" && UR_OWNERSHIP="root.root"
test -z "${UR_FILETYPES}" && UR_FILETYPES="rpm"
test -z "${UR_UPDATEINFO_INPUT}" && UR_UPDATEINFO_INPUT=/var/www/html/yum/build-hosting-repo/updateinfo.xml

find "${UR_REPODIR}" -exec chown "${UR_OWNERSHIP}" {} + 1>/dev/null 2>&1
find "${UR_REPODIR}" -type f -exec chmod "0664" {} + 1>/dev/null 2>&1
find "${UR_REPODIR}" -type d -exec chmod "0775" {} + 1>/dev/null 2>&1
chmod 0754 "$0"
restorecon -RF "${UR_REPODIR}"

# Prepare basic repo
cd "${UR_REPODIR}"
createrepo -v -u "${UR_BASEURL}" --basedir "${UR_REPODIR}" --simple-md-filenames --no-database --update --pretty .

# Inject custom updateinfo
# this task assumes the repomd file does not include node <data type="updateinfo"> yet.
UR_repomd="${UR_REPODIR}/repodata/repomd.xml"
UR_updateinfo_gz_short="repodata/updateinfo.xml.gz"
UR_updateinfo_gz="${UR_REPODIR}/${UR_updateinfo_gz_short}"

if ! test -e "${UR_UPDATEINFO_INPUT}" ;
then
   # file is absent, so decide how to fail.
   :
else
   # file exists, so continue with custom injection

   # learn open-size and open-checksum
   UR_updateinfo_opensize="$( /usr/bin/stat -c "%s" "${UR_UPDATEINFO_INPUT}" )"
   UR_updateinfo_openchecksum="$( /usr/bin/sha256sum "${UR_UPDATEINFO_INPUT}" | awk '{print $1}' )"

   # compress file and learn size and checksum
   /usr/bin/gzip < "${UR_UPDATEINFO_INPUT}" > "${UR_updateinfo_gz}"
   UR_updateinfo_size="$( /usr/bin/stat -c "%s" "${UR_updateinfo_gz}" )"
   UR_updateinfo_checksum="$( /usr/bin/sha256sum "${UR_updateinfo_gz}" | awk '{print $1}' )"
   UR_updateinfo_timestamp="$( /usr/bin/stat -c "%Y" "${UR_updateinfo_gz}" )"

   # insert information into repomd
   this_string="<data type=\"updateinfo\">
  <checksum type=\"sha256\">${UR_updateinfo_checksum}</checksum>
  <open-checksum type=\"sha256\">${UR_updateinfo_openchecksum}</open-checksum>
  <location xml:base=\"${UR_BASEURL}\" href=\"${UR_updateinfo_gz_short}\"/>
  <timestamp>${UR_updateinfo_timestamp}</timestamp>
  <size>${UR_updateinfo_size}</size>
  <open-size>${UR_updateinfo_opensize}</open-size>
</data>"

   {
      sed -r -e '/<\/repomd>/d' "${UR_repomd}"
      printf "%s\n%s\n" "${this_string}" "</repomd>"
   } > "${UR_repomd}.$$"
   /bin/touch --reference "${UR_repomd}" "${UR_repomd}.$$"
   /bin/mv -f "${UR_repomd}.$$" "${UR_repomd}"
fi
EOF

Summary

Using bash to modify xml files is obviously not ideal. However, this xml file is simple enough so this ugly mechanism suffices. For teams that know how to manage custom yum repositories and also want to just use yum update –security, this process should be a good basis or even complete solution!

Appendices

Appendix A: http proxy

If you use an http proxy for your yum traffic, the proxy might cache old versions of the metadata or package files. A quick and dirty way to clean up a squid proxy of the metadata file follows.

time squidclient -h localhost -r -p 3128 -m PURGE http://yum5.ipa.example.com/yum/hosting/repodata/updateinfo.xml.gz

Squid unfortunately does not allow recursive purging, so you will have to loop over all the metadata files and any package files you want to ensure get cleared.

References

Local file /var/cache/yum/x86_64/7Server/epel/69b82df00108c0ac8ac82fafbd0f3b89cc98d8dfe4fa350af7a23331a878fea2-updateinfo.xml.bz2

List available packages from one repository

last updated 2020-01-07

For dnf

dnf list available --disablerepo=* --enablerepo=reponame

For dpkg (low-level package manager for apt)

ff() { 
   for file in "/etc/apt/sources.list.d/${1}.list";
   do 
      for repo in $( awk '$1 ~ /^deb$/ {print $2}' "${file}" | sed -r -e 'sX\/X_Xg;' -e 's/\<https?.__//g;' ) ;
      do
         awk '$1 ~ /Package:/ {print $2}' /var/lib/apt/lists/${repo}Packages | sort | uniq
      done
   done;
}
ff reponame

The story

For some reason it is harder to manage packages with apt: This is a main reason I don’t like to use it. I had to go write this crazy one-liner function to accomplish the same task that dnf provides with just two flags.
Also, the apt command here shows all the packages from that repository, regardless of its installed state. The dnf command will show only the ones available that are not already installed.

Update yum repo with an easy script

Similar to how you can Build an apt repository on CentOS and update it with new packages with a simple script, you can do the same with a yum rpm repository.

cat <<'EOFUPDATE' > ./update-yumrepo.sh
#!/bin/sh

# working directory
repodir=/mnt/mirror/bgscripts/
cd ${repodir}
chmod 0644 *rpm 1>/dev/null 2>&1

# create the package index
createrepo .
EOFUPDATE
chmod u+x ./update-yumrepo.sh