Deplist improved, with suggested and recommended entries

STUB
make sure to document how to use `make DEPLIST DEPTYPE=sug DISTRO=devuan` and default of `make DEPLIST DEPTYPE=dep DISTRO=devuan` and make DEPLIST DEPTYPE=rec DISTRO=devuan

Here is my improved method for using target deplist.

In the Makefile:

deplist:
   @# deplist 2020-04-18 input must be comma separated
   @# DEPTYPE( dep , rec , sug ) for depends, recommends, or suggests
   @if test -z "${DISTRO}" ; then ${echobin} "Please run \`make deplist\` with DISTRO= one of: `make deplist_opts 2>&1 1>/dev/null | ${xargsbin}`. Aborted." 1>&2 ; exit 1 ; fi
   @if ! ${echobin} "${DEPTYPE}" | grep -qE "^(dep|rec|sug)$$" ; then ${echobin} "Please run \`make deplist\` with DEPTYPE= one of: dep, rec, sug. Undefined will use \`dep\`. Aborted." 1>&2 ; exit 1; fi
   @${grepbin} -h --exclude-dir='doc' -riIE "\<${DEPTYPE}-" ${SRCDIR} | ${awkbin} -v "domain=${DISTRO}" -v "deptype=${DEPTYPE}" 'tolower($$2) ~ deptype"-"domain {$$1="";$$2="";print}' | tr ',' '\n' | ${sortbin} | ${uniqbin} | ${sedbin} -r -e 's/^\s*//' -e "s/\s*\$$/${SEPARATOR}/" | ${xargsbin}

deplist_opts:
   @# deplist_opts 2020-04-18 find all available dependency domains
   @${grepbin} -h -o -riIE '\<(dep|rec|sug)-[^\ :]+:' ${SRCDIR} | ${sedbin} -r -e 's/(dep|rec|sug)-//;' -e 's/:$$//;' | ${sortbin} | ${uniqbin} 1>&2

Observe how these all depend on variables being defined earlier:

awkbin     :=$(shell which awk)
chmodbin   :=$(shell which chmod)
cpbin      :=$(shell which cp)
echobin    :=$(shell which echo)
falsebin   :=$(shell which false)
findbin    :=$(shell which find)
grepbin    :=$(shell which grep)
gzipbin    :=$(shell which gzip)
installbin :=$(shell which install)
rmbin      :=$(shell which rm)
rmdirbin   :=$(shell which rmdir)
sedbin     :=$(shell which sed)
sortbin    :=$(shell which sort)
truebin    :=$(shell which true)
uniqbin    :=$(shell which uniq)
xargsbin   :=$(shell which xargs)

In the source files in the project, you can use entries like these:

# dep-devuan: mawk | gawk, lightdm, upower
# dep-raw: awk, grep, sed, lightdm
# rec-devuan: lightdm-gtk-greeter
# sug-devuan: logout-manager
# dep-centos: lightdm, upower

Obviously this is a poor substitute for the make_shlibs from debhelper (which probably just wraps around ldd). But for my shell-driven packages, it’s pretty nice if you add the dependencies to each individual file.

Makefile trick: deplist

I whipped together a snippet for a Makefile I plan on using more in the future.

This target, deplist, searches the entire source tree for my “# Dependencies:” tags, by distribution name, and lists them on standard out.

deplist:
	@if test -z "$(DISTRO)" ; then echo "Please run \`make deplist\` with DISTRO= one of: `make deplist_opts 2>&1 1>/dev/null | xargs`. Aborted." ; exit 1 ; fi
	@grep -h --exclude='Makefile' --exclude-dir='doc' -A5 -riIE dependencies $(SRCDIR) | \
	   awk -v 'distro=$(DISTRO)' 'tolower($$0) ~ distro {$$1="";$$2="";print}' | \
	   awk 'BEGIN{cmd="xargs -n1"} $$0 !~ /\(/{print $$0 | cmd ; close(cmd);} $$0 ~ /\(/{print;}' | \
	   sort | uniq | sed -r -e 's/$$/$(SEPARATOR)/' | xargs

deplist_opts:
	@echo "el7" 1>&2
	@echo "devuan" 1>&2

And, of course, probably place these in your .PHONY list because these are not real files to be built.

.PHONY: clean install uninstall list deplist deplist_opts

Every file in the project that has an external dependency should have some comments in this format:

# Dependencies:
#    devuan: python3-tk python3-pil
#    el7: python36 python36-pil

To use this target in a debuild recipe, so that it will dynamically build the dpkg dependency list, use:

override_dh_gencontrol:
	printf "misc:Depends=" > debian/${APPNAME}.substvars
	make -C src deplist DISTRO=devuan SEPARATOR=',' | grep -vE 'make\[[0-9]' >> debian/${APPNAME}.substvars
	dh_gencontrol

References

Inspiration from rpm’s Dynamic Build Dependency feature.

Original research.

Package for devuan: freefilesync

I already have a number of posts about FreeFileSync, an open-source software that provides a nice gui for managing file synchronization task and jobs.

While the project itself is open source, there is a non-free release available to monetary donors that includes a few additional features. The package for Devuan is the GPL release.

You can go check out the build artifacts for the dpkg on my OBS. To install the package, follow the instructions on the download page.

The FreeFileSync project does not publish a source tree. While the source is available in a standard tarball, no diffs are presented between versions. I participate in a group that does track the changes in code between FreeFileSync releases.

What’s really interesting to note is that the build available on the OBS uses gtk3, which has an instability in some of the icon functions. Unfortunately Debian, the upstream of Devuan GNU+Linux, has removed some of the more-stable gtk2 libs because gtk2 is really old.

If you are looking to compile FreeFileSync on Devuan with gtk2 bindings, you need these exact packages which are probably available at the Debian archive site.

  • libwxbase3.0-0v5_3.0.4+dfsg-14_amd64.deb
  • libwxbase3.0-0v5_3.0.4+dfsg-14_i386.deb
  • libwxbase3.0-dev_3.0.4+dfsg-14_amd64.deb
  • libwxbase3.0-dev_3.0.4+dfsg-14_i386.deb
  • libwxgtk3.0-0v5_3.0.4+dfsg-14_amd64.deb
  • libwxgtk3.0-0v5_3.0.4+dfsg-14_i386.deb
  • libwxgtk3.0-dev_3.0.4+dfsg-14_amd64.deb
  • libwxgtk3.0-dev_3.0.4+dfsg-14_i386.deb
  • wx3.0-headers_3.0.4+dfsg-14_all.deb

Package for devuan: xdgmenumaker

After a healthy discussion on the Devuan forum about how to get xdg-compliant menus for non-xdg-compliant window managers, I went and packaged up xdgmenumaker for Devuan!

You can go install this utility as an easy dpkg from my OBS, which includes a few of my own wrapper tools. One is a nice menu entry for rebuilding the fluxbox menus.

The nature of xdg menus is to use the system-wide /usr/share/applications directory and its contents, and also include your local ~/.local/share/applications (or other $XDG_DATA_DIRS/applications/ locations as defined), so using the system-wide xdg fluxbox menu would not be complete. Therefore, I highly recommend adding to your ~/.fluxbox/startup script:

/usr/bin/xdgmenumaker -f fluxbox -i > ~/.fluxbox/xdg-menu &

And adjust your ~/.fluxbox/menu file to include at least these entries:

[include] (.fluxbox/xdg-menu)
#[include] (/etc/xdgmenumaker/fluxbox)

I recommend the above commented line, because in case for some reason your per-user stuff is broken,
The dpkg I build provides a menu entry for reloading it, so you don’t have to go to the command line and run the above command.
screenshot showing "Update xdg menu (fluxbox)" menu entry

Install build dependencies from source files, dpkg and rpm

For a dpkg

cd $packagedir ;
mk-build-deps
sudo apt-get install ./${package}-build-deps*.deb

Source: ubuntu – Given a debian source package – How do I install the build-deps? – Server Fault

For an rpm

yum-builddep my-package.spec

Or

dnf builddep my-package.spec

fedora – Automatically install build dependencies prior to building an RPM package – Stack Overflow/13228992#13228992

Build dpkg with Jenkins project

Here is how I build dpkgs in Jenkins. I have not yet tried the jenkinsfile syntax, but I’m sure that’s a better way to do it.

Prerequisites

Install the Debian Package Builder plugin. It adds a nice option for a build step which simplifies the debuild process.
Now, I use a single repository to store my rpm specs and dpkg debian/ directories. It’s not ideal, I realize, but it was based on the architecture I understood at the time, as well as it was modeled after a few upstream places I rip off follow.
So I had to set up the Gitlab plugins:

My debuild workflow in Jenkins

Make a new item, of type Freestyle project.
screenshot of Jenkins wui where the user is about to make a new project named "veracrypt."
To load my repository with its specs and debian/ directories, I pointed to my public gitlab repo which is configured elsewhere in Jenkins.
Screenshot of jenkins project showing General settings, Gitlab connection
I am pulling down the updates branch, of my main git repo, and saving to a local subdirectory.
Screenshot of project configuration showing SCM, and additional behavior of "Check out to a sub-directory"
I’m behind a NAT, so I don’t expect to easily set up a webhook. But I really wish I would bother to get it hooked up so any repo changes would do this. I’ll have to check that out in the future.
For the build steps, I have some shell running before and after the main “Build debian package” section.
Screenshot of build steps from jenkins project, showing parts of the shell statments and Build debian package step.
I had to revamp the debian/watch file in my veracrypt sources, because I wasn’t smart enough to get it to reliably download from sourceforge (oh how the might have fallen). But uscan is a wonderful tool that downloads the source tarball for a package.
My first full shell step:

uscan -v -ddd --destdir ../../ --symlink work/veracrypt ; mkdir -p dpkg ; tar -zx -C dpkg --strip-components 1 -f $( find . -maxdepth 1 -iregex '.*\/veracrypt_[0-9]+.*orig.*tar.*z.?' | head -n1 ) ; cp -pr work/veracrypt/debian dpkg/

The syntax got a little weird when I was switching between the bzip2 and the gzip file. Also, I don’t always know the filename, ergo the find command.
And then, the Build debian package step. The source tarball was extracted to the dpkg/ location, and the debian dir copied there. I really appreciate the ability to specify where the debian/ directory is.
And the final shell step deploys the files to my nfs share for final processing by hand. I hand-curate what’s in my own repo. I don’t have a proper pool/ setup like the real Devuan repos, so I curate it myself, plus my volume is low enough I can do it all myself.

mkdir -p /mnt/public/Public/${JOB_NAME} ;
cp -p *.build *.buildinfo *.changes *.deb *.debian.tar* *.dsc /mnt/public/Public/${JOB_NAME}/

And that’s it! For now I will trigger these builds manually.

Customize binary package for Devuan

Devuan bans some packages so I need to host them myself, with the dependencies altered to work in Devuan. Specifically, I wrote this solution for python3-ipalib (python-ipalib originally, back in February 2019).

I wrote a process that helps Devuan join a freeipa domain. It involves manually manipulating the install dependencies for package python3-ipalib. There’s no need to rebuild the package; you can just alter the listed dependencies. This script turns the manual process into a single command.

#!/bin/sh
# File: /mnt/public/www/smith122/repo/devuan-deb/newest-package-for-devuan.sh
# License: CC-BY-SA 4.0
# Author: bgstack15
# Startdate: 2019-11-29 21:28
# Title: Script that Manipulates Dependencies for a Binary Dpkg
# Purpose: Download latest version of python3-ipalib and modify it for internal use
# History:
# Usage:
#    /mnt/public/www/smith122/repo/devuan-deb/
#    Next steps include running update-devuan-deb.sh to update the apt repo.
# Reference:
#    Original research
#    dpkg-architecture --list-known
# Improve:
# Dependencies:
#    curl, dpkg-deb, sed, grep

# FUNCTIONS
#customize_package "${PACKAGE}" "${DEPENDS_REMOVE}" "${DEPENDS_ADD}" "${SOURCE_DIR}" "${OUT_DIR}"
customize_package() {
   # weaknesses: this might struggle with adding or removing "foobar (>= 0.1.1~bz2)" or other complex version numbers.
   ___cp_package="${1}"
   ___cp_depends_remove="${2}"
   ___cp_depends_add="${3}"
   ___cp_source_dir="${4}"
   ___cp_out_dir="${5}"

   # Learn latest version to fetch
   ___cp_newfile="$( curl -L -s "${___cp_source_dir}" | grep -oE ">${___cp_package}.*<\/a" | sed -r -e 's/^>//;' -e 's/<\/a$//;' )" echo "Discovered filename ${___cp_newfile}" curl -L -s "${___cp_source_dir}/${___cp_newfile}" -o "${___cp_newfile}" 1>/dev/null 2>&1
   ___cp_tmpdir="temp_${$}_deb"
   mkdir -p "${___cp_tmpdir}"
   dpkg-deb -R "${___cp_newfile}" "${___cp_tmpdir}"

   # Remove the requested dependencies
   echo "${___cp_depends_remove}" | tr ',' '\n' | while read word ;
   do
      if test "${word}" != "" ;
      then
         echo "Removing dependency ${word}"
         sed -i -r -e "s/[:,]\s+${word}[^,]{0,13}(,)?/\1/;" "${___cp_tmpdir}/DEBIAN/control"
      fi
   done

   # Add the requested dependencies
   echo "${___cp_depends_add}" | tr ',' '\n' | while read word ;
   do
      if test "${word}" != "" ;
      then
         echo "Adding dependency ${word}"
         sed -i -r -e "/^Depends:/s/$/, ${word}/" "${___cp_tmpdir}/DEBIAN/control"
      fi
   done
   # Remove trailing comma, just in case
   sed -i -r -e '/^Depends:/s/,\s*$//;' "${___cp_tmpdir}/DEBIAN/control"

   # Calculate new file name
   ___cp_newfile2="$( echo "${___cp_newfile}" | sed -r -e 's/((_(amd64|i386|all))?\..{1,6})$/+stackrpms\1/;' )"
   dpkg-deb -b "${___cp_tmpdir}" "${___cp_newfile2}"

   # Move to outdir
   test -e "${___cp_newfile2}" && mv "${___cp_newfile2}" "${___cp_out_dir}/"

   # Clean up
   rm -rf "${___cp_tmpdir}" "${___cp_newfile}"
}

PACKAGE=python3-ipalib
DEPENDS_REMOVE="systemd"
DEPENDS_ADD=""
SOURCE_DIR="http://ftp.us.debian.org/debian/pool/main/f/freeipa"
OUT_DIR="/mnt/public/www/smith122/repo/devuan-deb/p"
customize_package "${PACKAGE}" "${DEPENDS_REMOVE}" "${DEPENDS_ADD}" "${SOURCE_DIR}" "${OUT_DIR}"

If for some reason new packages come up with different dependencies I need to alter, it will just be a one-liner at the bottom.

References

Weblinks

https://gitlab.com/bgstack15/former-gists/tree/master/newest-package-for-devuan

How I host a Devuan ceres mirror for myself

Building on what I learned a long time ago for building an apt repo on CentOS, and some guidance from the Devuan forum, I have now validated my local Devuan mirror.

I chose to use debmirror because it can be limited to just certain suites (that’s Debian-speak for “version”) and package repositories. I exclusively use Devuan ceres (testing) because I need the FreeIPA packages. And also CentOS had an rpm of debmirror.

I run my script on Sunday and weekdays.
Cron job:

0	9	*	*	0-5	root	/etc/installed/debmirror-devuan.sh 1>/dev/null 2>&1

And my script:

#!/usr/bin/env sh
# File: /etc/installed/debmirror-devuan.sh
# Startdate: 2019-10-02 21:08
# Usage: by cron
{ debmirror --keyring /var/storage1/shares/public/Support/Platforms/devuan/keyrings/devuan-archive-keyring.gpg --verbose --config-file=/etc/debmirror.conf.devuan-ceres ; } 2>&1 | tee -a /var/storage1/shares/public/Support/Systems/storage1/var/log/debmirror/debmirror.devuan.$( date "+%F" ).log

Pretty basic, right? I had to prep the keyrings directory with the contents of a devuan install’s /usr/share/keyrings, but calling the debmirror was pretty easy. I pipe it to some logging as well.

Of course the debmirror.conf for this is extremely relevant.

# File: debmirror.conf.devuan-ceres
# Location: storage1:/etc/
# Author: bgstack15
# Startdate: 2019-09-27 22:43
# Title: Debmirror config for Devuan Ceres i386 and amd64 only
# Purpose: To provide a local mirror of devuan unstable so my clients can traverse only the local network
# History:
# Usage:
#    { time sudo debmirror --keyring /var/storage1/shares/public/Support/Platforms/devuan/keyrings/devuan-archive-keyring.gpg --verbose --config-file=/etc/debmirror.conf.devuan-ceres ; } 2>&1 | tee -a /var/storage1/shares/public/Support/Systems/storage1/var/log/debmirror/debmirror.devuan.$( date "+%F" ).log
# Reference:
#    https://dev1galaxy.org/viewtopic.php?id=1152
#    https://dev1galaxy.org/viewtopic.php?id=1571
#    https://duckduckgo.com/?q=devuan+debmirror
#    http://pkgmaster.devuan.org/devuan_mirror_walkthrough.txt
#    http://ftp.debian.org/debian/dists/sid/main/
#    http://pkgmaster.devuan.org/devuan/dists/ceres/
#    http://pkgmaster.devuan.org/merged/dists/ceres/
#    how to get working keyring https://ilostmynotes.blogspot.com/2008/05/debmirror-of-ubuntu-archive-validating.html
# Improve:
# Documentation:
#    To get the gpg keyring, copy d2-02a:/usr/share/keyrings/ to /mnt/public/Support/Platforms/devuan/keyrings.
#    This setup will copy all the debian+devuan packages into one devuan/ directory. No amprolla or apache rewrites are necessary.
# Location of the local mirror (use with care)
$mirrordir="/mnt/narcissus/mirror/devuan";

# Output options
$verbose=0;
$progress=0;
$debug=0;

# Download options
$host="packages.devuan.org";
$user="anonymous";
$passwd="anonymous@";
$remoteroot="merged";
$download_method="http";
@dists="ceres";
@sections="main,main/debian-installer,contrib,non-free";
@arches="i386,amd64";
# @ignores="";
# @excludes="";
# @includes="";
# @excludes_deb_section="";
# @limit_priority="";
$omit_suite_symlinks=0;
$skippackages=0;
# @rsync_extra="doc,tools";
$i18n=0;
$getcontents=1;
$do_source=1;
$max_batch=0;
# @di_dists="dists";
# @di_archs="arches";

# Save mirror state between runs; value sets validity of cache in days
$state_cache_days=0;

# Security/Sanity options
$ignore_release_gpg=0;
$ignore_release=0;
$check_md5sums=0;
$ignore_small_errors=0;

# Cleanup
$cleanup=0;
$post_cleanup=1;

# Locking options
$timeout=300;

# Rsync options
$rsync_batch=200;
$rsync_options="-aIL --partial";

# FTP/HTTP options
$passive=0;
# $proxy="http://proxy:port/";

# Dry run
$dry_run=0;

# Don't keep diff files but use them
$diff_mode="use";

# The config file must return true or perl complains.
# Always copy this.
1;

References

Weblinks

  1. How to mirror ascii? / Other Issues / Dev1 Galaxy Forum
  2. Make a mirror of devuan package repository with a modified debmirror / Documentation / Dev1 Galaxy Forum
  3. http://pkgmaster.devuan.org/devuan_mirror_walkthrough.txt
  4. http://ftp.debian.org/debian/dists/sid/main/
  5. http://pkgmaster.devuan.org/devuan/dists/ceres/
  6. http://pkgmaster.devuan.org/merged/dists/ceres/
  7. how to get working keyring: Technical notes, my online memory: Debmirror of ubuntu archive, with valid gpg keys

Web searches

  1. devuan debmirror

How I use the OBS to build and host dpkgs for Devuan

Introduction

I have started using the public instance of the Open Build Service (OBS), aka openSUSE Build Service.
This post documents my process for taking a package upstream, my packaging recipe (to use the OBS parlance), and getting a hosted package. If you want to duplicate my efforts with your own packages, I hope this helps.

The process

Install osc

The openbuild service command line tool is available in the Devuan ceres repos already, as package name osc.

Select what upstream package to build

My example will use FreeFileSync, because I already bundle it for Devuan and it only takes a few minutes to compile.
Additionally, because the upstream provides only a zip file, I am using my collaborative Opensource Tracking repo for the tarball which dpkg seemed to require and I gave up investigating how to get it to use a zip file as a source.

Prepare to use ocs locally

Osc seems to operate pretty similar to version control, with commits and so on.
If necessary, initialize osc and checkout the project. On the openSUSE OBS instance, it’s probably the home project.

mkdir -p ~/osc ; cd ~/osc
osc checkout home:bgstack15

Build package with osc

Make a new package, either on cli or on the web interface.

osc mkpac freefilesync

Source: Reference 3
Retrieve the upstream source tarball, and prepare the debian.tar.xz file.
I store my dpkg intructions in the exploded directory form in git. So to assemble the debian.tar.xz, I have a few additional steps.
In another location, extract the source tarball, and copy in the debian/ directory. Outside the directory from the tarball, run dpkg-source.

cd ~/deb
tar -zxf freefilesync_10.13.orig.tar.gz
cp -pr ~/dev/stackrpms/freefilesync/debian ./FreeFileSync-10.13/
dpkg-source -b FreeFileSync-10.13

Now the assets required by OBS should exist. Copy in the .dsc and debian tarball to the osc project directory.

[bgstack15@myhost|/home/bgstack15/osc/home:bgstack15/freefilesync]$ ls -al
total 2116
-rw-r--r-- 1 bgstack15 bgstack15    9588 Jun 28 13:49 freefilesync_10.13-1+devuan.debian.tar.xz
-rw-r--r-- 1 bgstack15 bgstack15    1073 Jun 28 13:49 freefilesync_10.13-1+devuan.dsc
-rw-rw-r-- 1 bgstack15 bgstack15 2147432 Jun 28 13:14 freefilesync_10.13.orig.tar.gz

I can perform a local build to ensure it builds correctly.

osc build --local-package Debian_Testing x86_64

That will run for a while, and have to download all the build dependencies on the first run too.
If all that was successful, it’s time to add the assets and commit.

osc add *
osc commit

Build package on OBS

The assets are now the public OBS.
debian tarball, dsc, and upstream tarball
My builds triggered right away when I committed the changes. It took time for build workers to kick off and return the results, but my packages were published within a few hours!

If you want to tell the OBS to rebuild a package, select the status message of the Build Results section.

At the top of the log page, select the “Trigger Rebuild” button.

Or you could run osc rebuild command.

Using the repository

Of course the reason you want to use the OBS is to build packages to install them! A pretty front page is available for a project. Here’s my freefilesync one. It shows up as debian unstable, but it should work on devuan too.

Install the apt key

wget -nv https://download.opensuse.org/repositories/home:bgstack15/Debian_Unstable/Release.key -O Release.key
apt-key add - > Release.key
apt-get update

Install the packages

You can inspect and make sure the package is in your metadata and coming from the expected repo.

$ apt-cache policy freefilesync
freefilesync:
  Installed: (none)
  Candidate: 10.13-1+devuan
  Version table:
     10.13-1+devuan 500
        500 http://download.opensuse.org/repositories/home:/bgstack15/Debian_Unstable  Packages

Install the package!

apt-get install freefilesync

Final thoughts

I tried using a _service file (example) to automate the build tasks. It involves having the .dsc files available (such as in source control), which is generated from dpkg-source -b dirname-of-package/. If I have to do all that, and upload the dsc file, and then have the build nodes do all the same work, it’s not really worth it to me. Also, I never got it working because I’m not as smart as that guy in the example.

References

A random, fellow Devuan user thinks it’s OK to use the OBS debian repos for Devuan packages.
Steven Pusser’s Pale Moon project was a great example to me.
Beginnerʼs Guide | Open Build Service
My debuild instructions:

cl ; time debuild -us -uc 2>&1 | tee -a ~/log/debuild.$( basename "$( pwd )" ).$( date "+%F" ).log ; echo $? ; debuild -- clean 1>/dev/null 2>&1 ;