path: root/README

Kubuntu Automation, for obsoleting people [http://dilbert.com/strip/2015-12-09]

Index:
1 - Configuration
2 - Installation instructions for Kubuntu Automation
3 - Example to update KDE Frameworks packaging for a new release
4 - Example to prepare KDE Frameworks packaging for backports
5 - Example upload KDE Frameworks to the archive
6 - Merging from debian branches
7 - FAQ

+-----------------------------------------------------------------------------+
| 1 - Configuration                                                           |
+-----------------------------------------------------------------------------+

* 1.1 - Add following to your .ssh/config and make sure you have an ssh-agent running and the key unlocked
         Host racnoss.kde.org
             User ftpubuntu

* 1.2 - Make sure you have set this environment variables in your ~/.bashrc
         export DEBEMAIL="user@mailhost.com"
         export DEBFULLNAME="John Doe"

+-----------------------------------------------------------------------------+
| 2 - Installation instructions for Kubuntu Automation                        |
+-----------------------------------------------------------------------------+

 See the INSTALL file

+-----------------------------------------------------------------------------+
| 3 - Example to update KDE Frameworks packaging for a new release            |
+-----------------------------------------------------------------------------+

* 3.1 - Edit ~/ka-metadata/versions.json and set the proper value

* 3.2 - If you didn't already clone all the repositories with
         $ git-clone-all -r frameworks
        This will create by default a layout of directories like this:
         user@host:~/kubuntu-dev/frameworks$ tree -L 3
         .
         ├── attica
         │   └── git
         │       └── debian
         ├── baloo
         │   └── git
         │       └── debian
         ├── bluez-qt
         │   └── git
         │       └── debian
         ├── breeze-icons
         │   └── git
         │       └── debian
         ├── extra-cmake-modules
         │   └── git
         │       └── debian
         [...]

* 3.3 - Merge any branch you consider appropiate (origin/kubuntu_unstable, ...) [1]
        you can do this with
         user@host:~/kubuntu-dev/frameworks$ do-all git merge <branch>
        [1] if you are merging debian branches, please don't do it with "do-all git merge",
        see the section "Merging from debian branches" below

* 3.4 - Update the package metadata with
         user@host:~/kubuntu-dev/frameworks$ ka-update-metadata -d artful -r frameworks
        This metadata is going to be used later by other scripts such as the build pages 
        scripts, or gbp-newrelease, so it's very important to update it.
        Among the metadata there is a build dependencies map which is going to be used
        for the build depends version bumping which is going to be done automatically by
        gbp-newrelease.
        If the Frameworks release in question needs an specific version of Qt you can
        update the Qt map first and then merge it into the frameworks map with
         user@host:~/kubuntu-dev/frameworks$ ka-update-metadata -d zesty -r artful -m qt
        NOTE: Remember to commit and push the resulting files to the ka-metadata git
              repository.

* 3.5 - (OPTIONAL) If you want, before building the source packages, you can download all
        the orig tarballs in advance with:
         $ download-tarballs -r frameworks
        If you don't do this, gbp-* will download the tarball automatically if it wasn't
        already downloaded.

* 3.6 - Do the actual updating to the new upstream release
         user@host:~/kubuntu-dev/frameworks$ do-all gbp-newrelease
        This will make the appropiate changes and will build the initial version for the
        staging PPA.
        If something fails to source-build, it will report it at the end of the do-all
        output. So, let's say attica fails to build and we want to investigate the problem,
        we can do
         user@host:~/kubuntu-dev/frameworks/attica$ gbp-ppa
        to see again what failed, change things (such as updating patches) and build
        again and again with gbp-ppa until we fix the issue.
        Once you get everything building fine from source you will be able to upload the source
        packages you have created to the staging PPA's

* 3.7 - At this point some packages might have failed to source-build because a patch doesn't
        apply anymore, to fix that you might need to use quilt and unpack the upstream KDE
        source code, you can do this with:
         user@host:~/staging/frameworks/kcoreaddons/git$ unpack-source
        once you are done you can remove the upstream code with:
         user@host:~/staging/frameworks/kcoreaddons/git$ git clean -xdff
        NOTE: be careful when doing 'git clean -xdff' because if you have uncommited files
              under debian/* it will remove them.

* 3.8 - Pushing the changes to git: first of all pause the kci clicking build now at
         http://kci.pangea.pub/view/mgmt/job/mgmt_pause_integration/
        once its paused you can push everything to git with
         user@host:~/kubuntu-dev/frameworks$ git-push-all

* 3.9 - Upload the packages to the staging ppa with
         user@host:~/kubuntu-dev/frameworks$ do-all uploadsource

* 3.10 - Set up a cron job that generates the status page using kubuntu-ppa-build-status-cron:
      - There is a cronjob running on weegie that builds http://qa.kubuntu.co.uk/ppa-status/
        To update the job, edit the relevant config file in ~/ka-metadata/build-status-conf/
        and push your changes.
        The cronjob will pick up your changes for the next refresh run.

      - Alternatively, set it up yourself:
        copy kubuntu-ppa-build-status-cron.conf.tmpl to kubuntu-ppa-build-status-cron.conf
        and fill the necessary variables.
        CREDENTIALS is the user:pw part from the https apt ppa url if a private ppa is used

* 3.11 - If you need to retry builds you can do it with
          $ kubuntu-retry-builds -d artful -r frameworks
         Note that Launchpad sometimes marks incorrectly as 'Failed to build'
         packages which should be actually in 'Dependency wait', if that's the
         case you can pass '--force' to 'kubuntu-retry-builds' so it will also
         retry the packages which failed to build and not just the ones in the
         'Dependency wait' state:
          $ kubuntu-retry-builds --force -d artful -r frameworks
         Please don't abuse the '--force' option and use it only when it's actually needed.

+-----------------------------------------------------------------------------+
| 4 - Example to prepare KDE Frameworks packaging for backports               |
+-----------------------------------------------------------------------------+

Let's see how to backport the packages we prepared in the previous section.
This is an example for xenial.

* 4.1 - In a set of frameworks clones created with 'git-clone-all' (see section 3.2)
        checkout the corresponding _backports branch.
        a) If the branches doesn't exist in your local clones but they already exist in the server
           you can create and checkout them with
            $ do-all git checkout -b kubuntu_xenial_backports origin/kubuntu_xenial_backports
        b) If the branches doesn't exist neither on your local clones nor in the server you can
           create and checkout them with
            $ do-all git checkout -b kubuntu_xenial_backports kubuntu_<name_of_devel_release>_archive
           (But make sure the kubuntu_<name_of_devel_release>_archive branches are updated first)

* 4.2 - Merge the branches from the current development release into kubuntu_xenial_backports
        with
         $ do-all git checkout kubuntu_xenial_backports
         $ do-all git merge --no-edit origin/kubuntu_<name_of_devel_release>_archive
        NOTE: don't miss the '--no-edit' option for 'git merge', otherwise the editor may pop up
              for any and all of the dozens of packages you are working on

* 4.3 - Update the package metadata with
         user@host:~/kubuntu-dev/frameworks$ ka-update-metadata -d xenial -r frameworks
        This is needed for 'kubuntu-retry-builds'

* 4.4 - Build the source packages from that branches with
         $ do-all gbp-ppa -d xenial

* 4.5 - Pushing the changes to git: first of all pause the kci clicking build now at
         http://kci.pangea.pub/view/mgmt/job/mgmt_pause_integration/
        once its paused you can push everything to git with
         user@host:~/kubuntu-dev/frameworks$ git-push-all

* 4.6 - Upload the source packages to the staging PPA with
         $ do-all uploadsource

* 4.7 - Set up the status page editing 'build-status-conf/frameworks-bp.conf'

* 4.8 - If you need to retry builds you can do it with
         $ kubuntu-retry-builds -d xenial -r frameworks
        Note that Launchpad sometimes marks incorrectly as 'Failed to build'
        packages which should be actually in 'Dependency wait', if that's the
        case you can pass '--force' to 'kubuntu-retry-builds' so it will also
        retry the packages which failed to build and not just the ones in the
        'Dependency wait' state:
         $ kubuntu-retry-builds --force -d xenial -r frameworks

+-----------------------------------------------------------------------------+
| 5 - Example upload KDE Frameworks to the archive                            |
+-----------------------------------------------------------------------------+

* 5.1 - First of all, build the source packages to be uploaded with
         user@host:~/kubuntu-dev/frameworks$ do-all gbp-archive
        Then do the upload with
         user@host:~/kubuntu-dev/frameworks$ do-all uploadsource -t ubuntu
        And finally push the git changes (no need to pause CI this time)
         user@host:~/kubuntu-dev/frameworks$ git-push-all --force-paused

* 5.2 - If you need to retry builds in the main archive you can do it with
         $ kubuntu-retry-builds --archive -d zesty -r frameworks
        Note that Launchpad sometimes marks incorrectly as 'Failed to build'
        packages which should be actually in 'Dependency wait', if that's the
        case you can pass '--force' to 'kubuntu-retry-builds' so it will also
        retry the packages which failed to build and not just the ones in the
        'Dependency wait' state:
         $ kubuntu-retry-builds --force --archive -d zesty -r frameworks

+-----------------------------------------------------------------------------+
| 6 - Merging from debian branches                                            |
+-----------------------------------------------------------------------------+

You shouldn't merge debian branches doing 'git merge' directly. Why not?
 1. Our Vcs-* fields are different than the ones from debian packaging.
 2. Our versioned build depends might be different compared with debian.
 3. Our Standards-Version fields might be different compared with debian.
 4. Maybe some things from debian should be kept out of our packaging. For
    instance,  right now debian is adding 'Breaks' against all reverse
    dependencies for Frameworks and Plasma packages, probably to avoid partial
    migrations by Britney to debian testing. This is very inconvenient for us
    because:
    I)  it pollutes the apt-get debug output when you do
        $ apt-get -o debug::pkgproblemresolver=1 dist-upgrade
    II) it makes harder to package managers to find a solution for dist-upgrades
        increasing the risk of getting 'apt-get dist-upgrade' removing things it
        shouldn't remove

So you could end up having unneccesary conflicts due to 1. 2. and 3. which you
would have to resolve. You also might merge some undesired things like that one
pointed out in 4.

To perform debian merges we have some scripts to automate some tasks to do a
proper merge, fixing some issues in advance.

How to do the merges with the KA tooling? we will show an example below for one 
particular package (all the commands are suposed to be executed in the git
repository of the package in question) but we could automate it for a whole package
set (Frameworks/Plasma/Applications):

* 6.1 - Check the Standards-Version from debian:
         $ git fetch debian
         $ git checkout debian/master
        Inspect debian/control and take note of the value, then
         $ git checkout kubuntu_zesty_archive
        Inspect debian/control and if the value is lower, do:
         $ update-sv-field <new_value>
        This will do a commit in the kubuntu_zesty_archive branch, example:
         $ git show
         commit b6dd4d3f394f2a2734642f31c33746b458a02d96
         Author: José Manuel Santamaría Lema <panfaust@gmail.com>
         Date:   Thu Oct 27 20:20:24 2016 +0200
         
             NOCI
             Update Standards-Version to 3.9.8
         
         diff --git a/debian/control b/debian/control
         index 6962e6f..7553cfb 100644
         --- a/debian/control
         +++ b/debian/control
         @@ -15,7 +15,7 @@ Build-Depends: cmake (>= 2.8.12~),
                         qtchooser,
                         qttools5-dev (>= 5.6.1~),
                         qttools5-dev-tools (>= 5.6.1~)
         -Standards-Version: 3.9.6
         +Standards-Version: 3.9.8
          Homepage: https://projects.kde.org/projects/kdesupport/extra-cmake-modules
          Vcs-Browser: https://code.launchpad.net/~kubuntu-packagers/kubuntu-packaging/+git/extra-cmake-modules
          Vcs-Git: https://git.launchpad.net/~kubuntu-packagers/kubuntu-packaging/+git/extra-cmake-modules


* 6.2 - Check the debian revision: in case debian has a higher version than our packaging
        the changelog merge won't work properly, for instance if they had on top of their changelog 
        the version 5.26.0-1 and we have the 5.26.0-0ubuntu1 their changelog entry would end up
        on top of the resulting changelog after the merge, so to avoid this you can bump the version
        of our top changelog block with:
         $ bump-ubuntu-revision
        This will create a commit on kubuntu_zesty_archive like this:
         $ git show
         commit 8acef268219982c0b37ece31b51b4020b2ad6552
         Author: José Manuel Santamaría Lema <panfaust@gmail.com>
         Date:   Thu Oct 27 21:28:45 2016 +0200
         
             Bump ubuntu revision to -1ubuntu1 to make the merges from debian easier.
         
         diff --git a/debian/changelog b/debian/changelog
         index c4743f7..bb85aee 100644
         --- a/debian/changelog
         +++ b/debian/changelog
         @@ -1,4 +1,4 @@
         -extra-cmake-modules (5.27.0-0ubuntu1) UNRELEASED; urgency=low
         +extra-cmake-modules (5.27.0-1ubuntu1) UNRELEASED; urgency=low
          
            [ Rik Mills ]
            * New upstream release (5.27.0)


* 6.3 - Check the wrap-and-sort status: in order to reduce the number of conflicts
        you should make sure that the current kubuntu branch is ka-wrap-and-sort'ed
        before merging, you can do this with:
         $ ka-wrap-and-sort
        check the output of "git diff" and if it reports anything commit the results
        to the current kubuntu branch

* 6.4 - Merge from the debian/master branch with:
         $ ka-debian2kubuntu-merge

        This will create a temporary preprocessed branch with a commit on top of it, e.g.:
         $ git show debian-master-tmp_2016-10-27_193120
         commit 60da136cfee161f94ca284501c1d3b95f2b90970
         Author: José Manuel Santamaría Lema <panfaust@gmail.com>
         Date:   Thu Oct 27 21:31:20 2016 +0200
         
             Prepare for merging into Kubuntu branch
         
         diff --git a/debian/control b/debian/control
         index a78b0d7..3763a2e 100644
         --- a/debian/control
         +++ b/debian/control
         @@ -6,12 +6,12 @@ Uploaders: Maximiliano Curia <maxy@debian.org>
          Build-Depends: cmake (>= 2.8.12~),
                         debhelper (>= 9),
                         pkg-kde-tools (>= 0.15.16~),
         -               qttools5-dev (>= 5.4),
         -               qttools5-dev-tools (>= 5.4)
         +               qttools5-dev (>= 5.6.1~),
         +               qttools5-dev-tools (>= 5.6.1~)
          Standards-Version: 3.9.8
          Homepage: https://projects.kde.org/projects/kdesupport/extra-cmake-modules
         -Vcs-Browser: https://anonscm.debian.org/git/pkg-kde/frameworks/extra-cmake-modules.git
         -Vcs-Git: https://anonscm.debian.org/git/pkg-kde/frameworks/extra-cmake-modules.git
         +Vcs-Browser: https://code.launchpad.net/~kubuntu-packagers/kubuntu-packaging/+git/extra-cmake-modules
         +Vcs-Git: https://git.launchpad.net/~kubuntu-packagers/kubuntu-packaging/+git/extra-cmake-modules
          
          Package: extra-cmake-modules
          Architecture: any
        
         At this point this temporary branch was tried to merge into kubuntu_zesty_archive,
         so you can see the conflicts with:
          $ git diff

         So if you have conflicts at this point you have two options:
          a) Drop the merge to do it other day
          b) Fix the conflicts and go ahead with the merge
         You can do a) this way:
          $ git merge --abort
          $ ka-debian2kubuntu-clean # This will remove any temporary branch
         You can do b) this way (once you fixed the conflicts):
          $ git add <file1 with conflicts>
          $ git add <file2 with conflicts>
          $ git commit
          $ dch "Merge with debian/master."
          $ git commit -am "Update changelog."
          $ ka-debian2kubuntu-clean # This will remove any temporary branch
         And don't forget to push and upload the changed package to the staging PPA
          $ git push origin kubuntu_zesty_archive
          $ gbp-ppa
          $ uploadsource

+-----------------------------------------------------------------------------+
| 7 - FAQ                                                                     |
+-----------------------------------------------------------------------------+

* I don't have my PGP key in the host where I'm building my packages. How can I
  make gbp-* not sign the packages after building the source?
  a) You can use "gbp-whatever -- -us -uc", the options after '--' are passed to
     'gbp buildpackage'
  b) You can configure gbp-* to not sign the packages having this in your
     ~/.kubuntu-automation.conf:
      [gbp-* signing]
      _default = off

* How can I have a single 'upload' directory instead of having one per package?
  - Make sure you have this in your ~/.kubuntu-automation.conf:
    [areas]
    upload-area = ../../upload

* How can I execute 2 commands with 'do-all'? 'do-all command1 && command2'
  doesn't do what I expect...
  - You can try this:
    $ do-all bash -c "command1 && command2"