Building an entire Debian derivative from Git

Andrej Shadura

Collabora

8 June 2019

Presenter Notes

Apertis

Presenter Notes

Apertis

Presenter Notes

OBS: the bad

  • versioning internally resembles Git
  • CLI mimics Subversion (¡¿why?!)
  • branching is copying

Presenter Notes

OBS: the ugly

Presenter Notes

OBS: the ugly

Presenter Notes

OBS: the ugly

Presenter Notes

OBS: the ugly

Presenter Notes

Why not use Git?

Presenter Notes

Indeed, why

Presenter Notes

Debian → Apertis

We cannot take Debian Git trees:

  • different workflows
  • not always consistent
  • dgit (producing consistent Git trees) is used by a minority
  • some people may still use hg, bzr, svn, darcs and cvs or no VCS (they probably don’t, but they can)

Presenter Notes

Apertis already uses Git

  • mostly git-buildpackage/DEP-14 style
    • but with some differences
  • for select packages only:
    • packages where Apertis is the upstream
    • systemd, AppArmor, GLib…

Presenter Notes

DEP-14

  • https://dep.debian.net/deps/dep14/
  • Proposed by Raphaël Hertzog in 2014, revised in November 2016
  • Builds on an existing git-buildpackage workflow, refining it for multiple upstream releases, Debian releases and derivatives
  • Unfortunately, not used as much as it could be

Presenter Notes

DEP-14

Presenter Notes

DEP-14 in Apertis

Presenter Notes

What about upstream tarballs?

Presenter Notes

pristine-tar

  • ships a SuSE version of bzip2
  • ships bits of a different outdated bzip2
  • ships a zlib-based gzip that is heavily based on NetBSD's gzip

Presenter Notes

Frankenstein’s compressor

Presenter Notes

pristine-tar

  • is a wonderfully magical piece of art
  • which is best to stay away from
  • because it often breaks
  • because of how it is
  • and xz makes it only worse

Presenter Notes

Git LFS instead of pristine-tar

  • store binaries as is
  • no fighting with old or new compressors
  • supported by GitLab
  • protocol can be easily reimplemented if needed

Presenter Notes

Git LFS instead of pristine-tar

  • pristine-lfs was born
  • written in Python instead of Perl
  • uses the Go implementation of LFS underneath
  • almost option-compatible with pristine-tar
  • currently a working prototype

Presenter Notes

How pristine-lfs stores tarballs

 1 diff --git a/.gitattributes b/.gitattributes
 2 new file mode 100644
 3 index 0000000..d5fc5f8
 4 --- /dev/null
 5 +++ b/.gitattributes
 6 @@ -0,0 +1 @@
 7 +*.tar.* filter=lfs diff=lfs merge=lfs -text
 8 diff --git a/dash_0.5.10.2.orig.tar.gz b/dash_0.5.10.2.orig.tar.gz
 9 new file mode 100644
10 index 0000000..d3fda1e
11 --- /dev/null
12 +++ b/dash_0.5.10.2.orig.tar.gz
13 @@ -0,0 +1,3 @@
14 +version https://git-lfs.github.com/spec/v1
15 +oid sha256:3c663919dc5c66ec991da14c7cf7e0be8ad00f3db73986a987c118862b5f6071
16 +size 225196

Presenter Notes

dash

 1 $ git diff debian/0.5.10.2-4..apertis/0.5.10.2-4
 2 diff --git a/debian/apertis/gitlab-ci.yml b/debian/apertis/gitlab-ci.yml
 3 new file mode 100644
 4 index 0000000..272299c
 5 --- /dev/null
 6 +++ b/debian/apertis/gitlab-ci.yml
 7 @@ -0,0 +1,4 @@
 8 +include:
 9 +  - project: 'infrastructure/ci-package-builder'
10 +    file: '/ci-package-builder.yml'
11 +

Presenter Notes

base-files

Presenter Notes

tar-gplv2

Presenter Notes

GitLab CI pipeline

Install build-dependencies, check out tarballs from LFS,

build the source package, upload to :snapshots

Presenter Notes

Still work in progress

  • Currently only a fraction of packages has been imported
  • Bugs in tools are still being fixed
  • Some packages need build dependencies to build source packages, but installing them is expensive
  • GitLab needs a YAML file at a specific location in the repo
    • remote files can’t be used
    • Auto DevOps could be an alternative, but requires patching GitLab

Presenter Notes

How can Debian benefit

  • Move all Debian to Git? Not likely any time soon.
  • LFS instead of pristine-tar? More likely.
    • git-buildpackage integration doesn’t yet exist

Presenter Notes

Presenter Notes