Update osc password for Open Build Service

After the Open Build Service’s effort to migrate to the new authentication system, I went ahead and just rotated my password.

adrianSuSE wrote 2 days ago: OBS has switched to the new authentification system: https://idp-portal-info.suse.com

Of course my osc commands started failing on the command line.

The man page for osc is a little obtuse. You have to omit the word “section” when changing the password.

osc config https://api.opensuse.org --change-password

And then osc will be back on track!

Mirror an OBS repository locally — update 1

Previously, I wrote about how to mirror an Open Build Service apt repository. The original script used httrack and some crazy logic.

Due to the unsatisfactory performance of the first version, I rewrote the entire script, twice! I am much more satisfied with my v3 of the script, which is hosting in the same place as before.

So this rewrite is in some ways way simpler than it was before.

Now, the script has a dedicated function for downloading a file, and it only does so if there is no checksum passed, or if the checksum of the local file is not the one passed to the function. How an apt repo works is it has checksums of all of its files in various metadata files, which are the only explicitly named files we download first. We loop through the filenames in the Packages file, and pass those filenames and checksums to the getter function. If the local file has the same checksum, we skip that file.

A flag exists for choosing to download the dpkg sources as well.

And then, for some reason I cannot quite explain, a few files of mine never downloaded and matched the checksums in the Package file. The deb files install, so they’re valid, and I trust my own repo. So I decided to just re-sign all the apt repo files. That is, I rebuild the apt repo entirely and then sign it with my own gpg key for my internal network.

And I finally split out the config into a config file so I can provide an example, and not store my actual data on the Internet!

#!/bin/sh
# File: /etc/installed/obsmirror.sh
# Location: https://gitlab.com/bgstack15/former-gists/tree/master/obsmirror.sh
# Author: bgstack15
# Startdate: 2020-03-03 08:43
# SPDX-License-Identifier: CC-BY-SA-4.0
# Title: Script that scrapes down OBS site to serve a copy to intranet
# Purpose: save down my OBS site so I can serve it locally
# History:
#    2020-01-05 v1: begin which used httrack
#    2020-02-28 v2: complete rewrite to exclude httrack
#    2020-03-03 v3: complete rewrite to get explicit files and loop through their contents, and rebuild apt repo
# Usage:
#    in a cron job: /etc/cron.d/mirror.cron
#       50	12	*	*	*	root	OBSMIRROR_CONF=/etc/installed/obsmirror.conf /etc/installed/obsmirror.sh 1>/dev/null 2>&1
# Reference:
#    https://software.opensuse.org//download.html?project=home%3Abgstack15&package=freefilesync
#    /mnt/public/www/repo/devuan-deb/update-devuan-deb.sh
#    https://medium.com/sqooba/create-your-own-custom-and-authenticated-apt-repository-1e4a4cf0b864
# Improve:
# Documentation:
#    Download the release key and trust it.
#       curl -s http://repo.example.com/mirror/obs/Release.key | apt-key add -
#    Use a sources.list.d/ file with contents:
#       deb https://repo.example.com/mirror/obs/ /
# Dependencies:
#    binaries: wget sed awk
#    user: obsmirror
umask 0002

test -n "${OBSMIRROR_CONF}" && . "${OBSMIRROR_CONF}"
test -z "${logfile}" && logfile="/tmp/var/log/obsmirror/obsmirror.$( date "+%FT%H%M%S" ).log"
test -z "${inurl}" && inurl="http://download.opensuse.org/repositories/home:/bgstack15/Debian_Unstable"
test -z "${workdir}" && workdir=/tmp/obs
test -z "${thisuser}" && thisuser=obsmirror
# also use include_sources resign_repo gpg_passfile gpg_keyfile DEBUG

get_file() {
   # call: get_file "${tu}" "${md5sum}"
   ___tu="${1}"
   ___sum="${2}"
   tn="${___tu##${inurl}}"
   tf="${workdir}/${tn}" ; tf="$( readlink -m "${tf}" )"
   td="$( dirname "${tf}" )"
   test -d "${td}" || mkdir -p "${td}"
   gotten="skipped   "
   if test -z "${DRYRUN}" ;
   then
      if test -z "${___sum}" || test "$( md5sum "${tf}" 2>/dev/null | awk '{print $1}' )" != "${___sum}" ;
      then
         wget --content-disposition --no-verbose --quiet -O "${tf}" "${___tu}" && gotten=DOWNLOADED
      fi
   fi
   test -n "${VERBOSE}" && echo "${gotten} ${___tu} -> ${tf}"
}

wget_verbose=--quiet
test -n "${VERBOSE}" && unset wget_verbose
{
   test "${DEBUG:-NONE}" = "FULL" && set -x
   echo "logfile=${logfile}"

   # These files define an apt repo
   for word in InRelease Packages Packages.gz Release Release.gpg Release.key Sources Sources.gz ;
   do
      get_file "${inurl}/${word}"
   done

   # loop through named packages and download them
   #for word in $( awk '/Filename:/{print $2}' "${workdir}/Packages" ) ;
   awk '/Filename:|MD5/{print $2}' "${workdir}/Packages" | xargs -n2 | while read word sum
   do
      get_file "$( echo "${word}" | sed -r -e "s@^\.@${inurl}@;" )" "${sum}"
      #echo "a=${a}   b=${b}"
   done 

   # loop through dsc, orig.tar.gz, and debian.tar.xz files
   test -n "${include_sources}" && {
      for word in $( sed -n -r -e '/Files:/,/^\s*$/{/^ /p;}' ${workdir}/Sources | awk '{print $NF}' ) ;
      do
         get_file "${inurl}/${word}"
      done
   }

   test -n "${resign_repo}" && {
      # rebuild release files
      repodir="${workdir}"
      cd "${repodir}"
      dpkg-scanpackages -m . > Packages
      gzip -9c < Packages > Packages.gz
      # create the Release file
      PKGS="$(wc -c Packages)"
      PKGS_GZ="$(wc -c Packages.gz)"
      old_headers1="$( grep -E '^(Archive|Codename|Origin|Label|Architectures):' Release )"
      old_headers2="$( grep -E '^(Description):' Release )"
      cat < Release
${old_headers1}
Date: $(date -u '+%a, %d %b %Y %T %Z')
${old_headers2}
MD5Sum:
 $(md5sum Packages  | cut -d" " -f1) $PKGS
 $(md5sum Packages.gz  | cut -d" " -f1) $PKGS_GZ
SHA1:
 $(sha1sum Packages  | cut -d" " -f1) $PKGS
 $(sha1sum Packages.gz  | cut -d" " -f1) $PKGS_GZ
SHA256:
 $(sha256sum Packages | cut -d" " -f1) $PKGS
 $(sha256sum Packages.gz | cut -d" " -f1) $PKGS_GZ
EOF
      test -e "${gpg_passfile}" && gpg --batch --yes --passphrase-file "${gpg_passfile}" --pinentry-mode loopback -abs -o Release.gpg Release
      test -e "${gpg_passfile}" && gpg --batch --yes --passphrase-file "${gpg_passfile}" --pinentry-mode loopback --clearsign -o InRelease Release
      # and because we are resigning it, replace Release.key with the one we used
      test -e "${gpg_keyfile}" && cp -p "${gpg_keyfile}" Release.key
   }

   chown -R "${thisuser}:$( id -G "${thisuser}" | awk '{print $1}' )" "${workdir}"
} 2>&1 | tee -a "${logfile}"

Package for devuan: keyboard-leds-trayicons

I recently purchased a ThinkPad P50s and I love it! However, it doesn’t have indicators for capslock and numlock. So I wrote a tool to simulate them! Keyboard-leds-trayicons is my solution. You can go install it in Devuan Ceres from my OBS.

Reading the status of capslock and numlock is trivial on the command line:

xset q | awk 'function d2b(d,b) {while(d) {b=d%2b;d=int(d/2)}return(b)} /LED/{print d2b($NF)}'

The awk is there to convert the provided decimal value into a binary. But how do you poll this, and display it in your X11 session in an unobtrusive way? With a POSIX shell script, of course! But what about the actual icons? Use my fork of mktrayicon. My fork adds a few patches, and you can also get it at the OBS link above.

capslock and numlock indicators in system tray, both on

capslock and numlock indicators in system tray, capslock off and numlock on

Obviously most people have no need for this package. And by now anyone without the indicators probably doesn’t need them. This project was 99% for myself, but I like to share.

The default icons are a bit boring, which fits my style. But the tool uses the xdg icon spec, so you can add your own icons for capslock-on and similar. Patches are welcome to the project, so if you really think this package should include more icons, or use yours by default, please open a merge request!

Found: OBS service documentation

For some unknown reason, it is really hard to find good documentation on how to use the _service file for the Open Build Service.

The Open Build Service (OBS) is a project from the openSUSE team, and they run a public instance at https://build.opensuse.org/. This server application provides a build environment for many distributions’ package methods. I use it for my public Devuan packages (see my “OBS” link in the links section of this site, usually a sidebar on each page).

The _service file has some generic documentation in the main obs online man pages. I find this documentation extremely lacking in explaining how to use a service file fully. And I’ve learned by examples extant on the public OBS instance already.

So, just to be clear, a _service file is a file in an obs project, that defines a number of steps for the server to take. By default, each service runs for each commit, or whenever you select the “trigger services” button on the web page for a project.

I finally found a list of available service names to use! And the links take you to the projects on the Internet where they live and are fully documented. There are quite a few I plan on using in the future… like regex_replace.

Example _service file

With my service file set up to pull down the source debian/ directory as debian.tar.xz, and the entire source as a tar.gz, and extract out my .dsc file from that source, I can manage my OBS project way easier.

<services>
   <service name="tar_scm">
      <param name="scm">git</param>
      <param name="url">https://gitlab.com/bgstack15/fluxbox-themes-stackrpms.git</param>
      <param name="subdir">debian</param>
      <param name="filename">debian</param>
      <param name="revision">local-dsc-file</param>
      <param name="version">_none_</param>
   </service>
   <service name="recompress">
      <param name="file">*.tar</param>
      <param name="compression">xz</param>
   </service>
   <service name="tar_scm">
      <param name="scm">git</param>
      <param name="url">https://gitlab.com/bgstack15/fluxbox-themes-stackrpms.git</param>
      <param name="revision">local-dsc-file</param>
      <param name="version">_none_</param>
   </service>
   <service name="recompress">
      <param name="file">*.tar</param>
      <param name="compression">gz</param>
   </service>
   <service name="extract_file">
      <param name="archive">*.tar.gz</param>
      <param name="files">*/*.dsc</param>
   </service>
</services>

The .dsc file is modified from the output from a proper debuild -us -uc locally. I learned the trick of zeroing out the checksums from another OBS project, waterfox-classic-kpe.

Format: 3.0 (quilt)
Source: fluxbox-themes-stackrpms
Binary: fluxbox-themes-stackrpms
Architecture: all
Version: 0.0.1-1+devuan
Maintainer: Ben Stack 
Homepage: https://bgstack15.wordpress.com/
Standards-Version: 4.1.4
Build-Depends: debhelper (>= 12~)
Package-List:
 fluxbox-themes-stackrpms deb x11 optional arch=all
Files:
 00000000000000000000000000000000 1 fluxbox-themes-stackrpms_0.0.1.orig.tar.gz
 00000000000000000000000000000000 1 fluxbox-themes-stackrpms_0.0.1-1+devuan.debian.tar.xz

And since the filenames do not match, I think that section is not even necessary. Thankfully OBS figures out what tarballs to use where for the debuild. To quote my original distro (Korora): “Standing on the shoulders of giants.”

Conclusion

I can control my osc package with only one file in the obs source control: the _service file! This reduces the need to pass multiple assets, which can be large.

References

Internet searches

How I actually found the main weblink open build service services recompress

Weblinks

https://en.opensuse.org/openSUSE:Build_Service_Concept_SourceService#All_OBS_services_available

Mirror an OBS deb repository locally

Story

I run an OBS repository for all my packages, and it is available at the main site: https://build.opensuse.org/project/show/home:bgstack15.

But I wanted to mirror this for myself, so I don’t have to configure all my systems to point outward to get updates. I already host a Devuan ceres mirror for myself, and so mirroring this Open Build System repository is the last step to be self-hosting entirely for all systems except the mirror server.

I first dabbled with debmirror, but it kept wanting to try rsync despite my best configuration, plus it really insists on using the dists/ directory which isn’t used in the OBS deb repo design. So, I researched scraping down a whole site, and I found httrack which exists to serve a local copy of an Internet site. Bingo!

After a few hours of work, here is my solution for mirroring an OBS deb repo locally.

Solution

Create a user who will own the files and execute the httrack command, because httrack didn’t want to be run as root. Also, this new user can’t munge other data.

useradd obsmirror

Configure a script (available at gitlab)

#!/bin/sh
# File: /etc/installed/obsmirror.sh
# License: CC-BY-SA 4.0
# Author: bgstack15
# Startdate: 2020-01-05 18:01
# Title: Script that scrapes down OBS site to serve a copy to intranet
# Purpose: save down my OBS site so I can serve it locally
# History:
# Usage:
#    in a cron job: /etc/cron.d/mirror.cron
#       50	12	*	*	*	root	/etc/installed/obsmirror.sh 1>/dev/null 2>&1
# Reference:
#    https://unix.stackexchange.com/questions/114044/how-to-make-wget-download-recursive-combining-accept-with-exclude-directorie?rq=1
#    man 1 httrack
#    https://software.opensuse.org//download.html?project=home%3Abgstack15&package=freefilesync
# Improve:
#    use some text file as a list of recently-synced URLs, and if today's URL matches a recent one, then run the httrack with the --update flag. Probably keep a running list forever.
# Documentation:
#    Download the release key and trust it.
#       curl -s http://repo.example.com/mirror/obs/Release.key | apt-key add -
#    Use a sources.list.d/ file with contents:
#       deb https://repo.example.com/mirror/obs/ /
# Dependencies:
#    binaries: curl httrack grep head tr sed awk chmod chown find rm ln
#    user: obsmirror

logfile="/var/log/obsmirror/obsmirror.$( date "+%FT%H%M%S" ).log"
{
   test "${DEBUG:-NONE}" = "FULL" && set -x
   inurl="http://download.opensuse.org/repositories/home:/bgstack15/Debian_Unstable"
   workdir=/tmp/obs-stage
   outdir=/var/www/mirror/obs
   thisuser=obsmirror
   echo "logfile=${logfile}"

   mkdir -p "${workdir}" ; chmod "0711" "${workdir}" ; chown "${thisuser}:$( id -Gn obsmirror )" "${workdir}" 
   cd "${workdir}"
   # get page contents
   step1="$( curl -s -L "${inurl}/all" )"
   # get first listed package
   step2="$( echo "${step1}" | grep --color=always -oE 'href="[a-zA-Z0-9_.+\-]+\.deb"' | head -n1 | grep -oE '".*"' | tr -d '"' )"
   # get full url to a package
   step3="$( curl -s -I "${inurl}/all/${step2}" | awk '/Location:/ {print $2}' )"
   # get directory of the mirror to save down
   step4="$( echo "${step3}" | sed -r -e "s/all\/${step2}//;" -e 's/\s*$//;' )"
   # get domain of full url
   domainname="$( echo "${step3}" | grep -oE '(ht|f)tps?:\/\/[^\/]+\/' | cut -d'/' -f3 )"
   echo "TARGET URL: ${step4}"
   test -z "${DRYRUN}" && {
      # clean workdir of specific domain name in use right now.
      echo su "${thisuser}" -c "rm -rf \"${workdir:-SOMETHING}/${domainname:-SOMETHING}\""
      su "${thisuser}" -c "rm -rf \"${workdir:-SOMETHING}/${domainname:-SOMETHING}\"*"
      # have to skip the orig.tar.gz files because they are large and slow down the sync process significantly.
      echo su "${thisuser}" -c "httrack \"${step4}\" -*.orig.t* -v --mirror --update -s0 -r3 -%e0 \"${workdir}\""
      time su "${thisuser}" -c "httrack ${step4} -*.orig.t* -v --mirror --update -s0 -r3 -%e0 ${workdir}"
   }
   # -s0 ignore robots.txt
   # -r3 only go down 3 links
   # -%e0 follow 0 links to external sites

   # find most recent directory of that level
   levelcount="$(( $( printf "%s" "${inurl}" | tr -dc '/' | wc -c ) - 1 ))"
   subdir="$( find "${workdir}" -mindepth "${levelcount}" -maxdepth "${levelcount}" -type d -name 'Debian_Unstable' -printf '%T@ %p\n' | sort -n -k1 | head -n1 | awk '{print $2}' )"

   # if the work directory actually synced
   if test -n "${subdir}" ;
   then

      printf "%s " "DIRECTORY SIZE:"
      du -sxBM "${subdir:-.}"
      mkdir -p "$( dirname "${outdir}" )"
      # get current target of symlink
      current_target="$( find "${outdir}" -maxdepth 0 -type l -printf '%l\n' )"

      # if the current link is pointing to a different directory than this subdir
      if test "${current_target}" != "${subdir}" ;
      then
         # then replace it with a link to this one
         test -L "${outdir}" && unlink "${outdir}"
         echo ln -sf "${subdir}" "${outdir}"
         ln -sf "${subdir}" "${outdir}"
      fi

   else
      echo "ERROR: No subdir found, so cannot update the symlink."
   fi

   # disable the index.html with all the httrack comments and original site links
   find "${workdir}" -iname '*index.html' -exec rm {} +
} 2>&1 | tee -a "${logfile}"

And place this in cron!

#       50	12	*	*	*	root	/etc/installed/obsmirror.sh 1>/dev/null 2>&1

Explanation of script

So the logic is a little convoluted, because the OBS front page actually redirects downloads to various mirrors where the files are kept. So I needed to learn what the actual site is, and then pull down that whole site.
I couldn’t just use httrack –getfiles because it makes just a flat directory, which breaks the Packages contents’ accuracy to the paths of the package files. But I didn’t want the whole complex directory structure, just the repository structure. So I make a symlink to it in my actual web contents location.

How I use the OBS to build and host dpkgs for Devuan

Introduction

I have started using the public instance of the Open Build Service (OBS), aka openSUSE Build Service.
This post documents my process for taking a package upstream, my packaging recipe (to use the OBS parlance), and getting a hosted package. If you want to duplicate my efforts with your own packages, I hope this helps.

The process

Install osc

The openbuild service command line tool is available in the Devuan ceres repos already, as package name osc.

Select what upstream package to build

My example will use FreeFileSync, because I already bundle it for Devuan and it only takes a few minutes to compile.
Additionally, because the upstream provides only a zip file, I am using my collaborative Opensource Tracking repo for the tarball which dpkg seemed to require and I gave up investigating how to get it to use a zip file as a source.

Prepare to use ocs locally

Osc seems to operate pretty similar to version control, with commits and so on.
If necessary, initialize osc and checkout the project. On the openSUSE OBS instance, it’s probably the home project.

mkdir -p ~/osc ; cd ~/osc
osc checkout home:bgstack15

Build package with osc

Make a new package, either on cli or on the web interface.

osc mkpac freefilesync

Source: Reference 3
Retrieve the upstream source tarball, and prepare the debian.tar.xz file.
I store my dpkg intructions in the exploded directory form in git. So to assemble the debian.tar.xz, I have a few additional steps.
In another location, extract the source tarball, and copy in the debian/ directory. Outside the directory from the tarball, run dpkg-source.

cd ~/deb
tar -zxf freefilesync_10.13.orig.tar.gz
cp -pr ~/dev/stackrpms/freefilesync/debian ./FreeFileSync-10.13/
dpkg-source -b FreeFileSync-10.13

Now the assets required by OBS should exist. Copy in the .dsc and debian tarball to the osc project directory.

[bgstack15@myhost|/home/bgstack15/osc/home:bgstack15/freefilesync]$ ls -al
total 2116
-rw-r--r-- 1 bgstack15 bgstack15    9588 Jun 28 13:49 freefilesync_10.13-1+devuan.debian.tar.xz
-rw-r--r-- 1 bgstack15 bgstack15    1073 Jun 28 13:49 freefilesync_10.13-1+devuan.dsc
-rw-rw-r-- 1 bgstack15 bgstack15 2147432 Jun 28 13:14 freefilesync_10.13.orig.tar.gz

I can perform a local build to ensure it builds correctly.

osc build --local-package Debian_Testing x86_64

That will run for a while, and have to download all the build dependencies on the first run too.
If all that was successful, it’s time to add the assets and commit.

osc add *
osc commit

Build package on OBS

The assets are now the public OBS.
debian tarball, dsc, and upstream tarball
My builds triggered right away when I committed the changes. It took time for build workers to kick off and return the results, but my packages were published within a few hours!

If you want to tell the OBS to rebuild a package, select the status message of the Build Results section.

At the top of the log page, select the “Trigger Rebuild” button.

Or you could run osc rebuild command.

Using the repository

Of course the reason you want to use the OBS is to build packages to install them! A pretty front page is available for a project. Here’s my freefilesync one. It shows up as debian unstable, but it should work on devuan too.

Install the apt key

wget -nv https://download.opensuse.org/repositories/home:bgstack15/Debian_Unstable/Release.key -O Release.key
apt-key add - > Release.key
apt-get update

Install the packages

You can inspect and make sure the package is in your metadata and coming from the expected repo.

$ apt-cache policy freefilesync
freefilesync:
  Installed: (none)
  Candidate: 10.13-1+devuan
  Version table:
     10.13-1+devuan 500
        500 http://download.opensuse.org/repositories/home:/bgstack15/Debian_Unstable  Packages

Install the package!

apt-get install freefilesync

Final thoughts

I tried using a _service file (example) to automate the build tasks. It involves having the .dsc files available (such as in source control), which is generated from dpkg-source -b dirname-of-package/. If I have to do all that, and upload the dsc file, and then have the build nodes do all the same work, it’s not really worth it to me. Also, I never got it working because I’m not as smart as that guy in the example.

References

A random, fellow Devuan user thinks it’s OK to use the OBS debian repos for Devuan packages.
Steven Pusser’s Pale Moon project was a great example to me.
Beginnerʼs Guide | Open Build Service
My debuild instructions:

cl ; time debuild -us -uc 2>&1 | tee -a ~/log/debuild.$( basename "$( pwd )" ).$( date "+%F" ).log ; echo $? ; debuild -- clean 1>/dev/null 2>&1 ;