Building an entire Debian derivative from Git

Andrej Shadura

Collabora

25 July 2019

Presenter Notes

Apertis

Presenter Notes

Apertis

Presenter Notes

OBS: the bad

  • versioning internally resembles Git
  • CLI mimics Subversion (¡¿why?!)
  • branching is copying

Presenter Notes

OBS: the ugly

Presenter Notes

OBS: the ugly

Presenter Notes

OBS: the ugly

Presenter Notes

OBS: the ugly

Presenter Notes

Why not use Git?

Presenter Notes

Indeed, why not

  • We can store source packages in Git
  • OBS would just build packages

Presenter Notes

Debian → Apertis

We cannot take Debian Git trees:

  • different workflows
  • not always consistent
  • dgit (producing consistent Git trees) is used by a minority
  • some people may still use hg, bzr, svn, darcs and cvs or no VCS (they probably don’t, but they can)

Presenter Notes

Apertis already uses Git

  • mostly git-buildpackage/DEP-14 style
    • but with some differences
  • for select packages only:
    • packages where Apertis is the upstream
    • systemd, AppArmor, GLib…

Presenter Notes

DEP-14

  • https://dep.debian.net/deps/dep14/
  • Proposed by Raphaël Hertzog in 2014, revised in November 2016
  • Builds on an existing git-buildpackage workflow, refining it for multiple upstream releases, Debian releases and derivatives
  • Unfortunately, not used as much as it could be

Presenter Notes

DEP-14

Presenter Notes

DEP-14 in Apertis

Presenter Notes

What about upstream tarballs?

Presenter Notes

pristine-tar

  • ships a SuSE version of bzip2
  • ships bits of a different outdated bzip2
  • ships a zlib-based gzip that is heavily based on NetBSD's gzip

Presenter Notes

Frankenstein’s compressor

Presenter Notes

pristine-tar

  • is a wonderfully magical piece of art
  • which is best to stay away from
  • because it often breaks
  • because of how it is
  • and xz makes it only worse

Presenter Notes

Git LFS instead of pristine-tar

  • store binaries as is
  • no fighting with old or new compressors
  • supported by GitLab
  • protocol can be easily reimplemented if needed

Presenter Notes

Git LFS instead of pristine-tar

  • pristine-lfs was born
  • written in Python instead of Perl
  • uses the Go implementation of LFS underneath
  • almost option-compatible with pristine-tar
  • currently a working prototype

Presenter Notes

How pristine-lfs stores tarballs

 1 diff --git a/.gitattributes b/.gitattributes
 2 new file mode 100644
 3 index 0000000..d5fc5f8
 4 --- /dev/null
 5 +++ b/.gitattributes
 6 @@ -0,0 +1 @@
 7 +*.tar.* filter=lfs diff=lfs merge=lfs -text
 8 diff --git a/dash_0.5.10.2.orig.tar.gz b/dash_0.5.10.2.orig.tar.gz
 9 new file mode 100644
10 index 0000000..d3fda1e
11 --- /dev/null
12 +++ b/dash_0.5.10.2.orig.tar.gz
13 @@ -0,0 +1,3 @@
14 +version https://git-lfs.github.com/spec/v1
15 +oid sha256:3c663919dc5c66ec991da14c7cf7e0be8ad00f3db73986a987c118862b5f6071
16 +size 225196

Presenter Notes

dash

 1 $ git diff debian/0.5.10.2-4..apertis/0.5.10.2-4
 2 diff --git a/debian/apertis/gitlab-ci.yml b/debian/apertis/gitlab-ci.yml
 3 new file mode 100644
 4 index 0000000..272299c
 5 --- /dev/null
 6 +++ b/debian/apertis/gitlab-ci.yml
 7 @@ -0,0 +1,4 @@
 8 +include:
 9 +  - project: 'infrastructure/ci-package-builder'
10 +    file: '/ci-package-builder.yml'
11 +

Presenter Notes

base-files

Presenter Notes

tar-gplv2

Presenter Notes

GitLab CI pipeline

Install build-dependencies, check out tarballs from LFS,

build the source package, upload to :snapshots

Presenter Notes

New, better workflows

  • We can now use git merge to get updates from Debian!
  • CI automation attempts to merge debian/* to apertis/* and if it succeeds, submits a merge request
  • git blame now gives sensible output, allowing to trace where the changes come from and for what purpose

Presenter Notes

Still work in progress

  • Packages are being imported as we speak
  • There are still a few bugs in the tools
  • Some packages need build dependencies to build source packages, but installing them is expensive
  • GitLab needs a YAML file at a specific location in the repo
    • remote files can’t be used
    • Auto DevOps could be an alternative, but requires patching GitLab

Presenter Notes

How can Debian benefit

  • Move all Debian to Git? Not likely any time soon.
    • It’s actually now work in progress, thanks to Ian, Sean and others on the dgit team
  • LFS instead of pristine-tar?
    • git-buildpackage integration doesn’t yet exist
    • The are wrapper scripts which will be integrated into pristine-lfs:
    • import-dsc to import all tarballs from a said .dsc
    • export-tar to export tarball(s) corresponding to the currently checked out package version

Presenter Notes

pristine-lfs

Presenter Notes

Presenter Notes