EnvironmentFiles used by systemd

In a previous posting, Environment variables set by systemd, variables were directly set within the systemd unit file. This is fine for a small amount of modifications, but in some case these environment variables are provided by another package on the system or need to be the same for multiple services.

We have modified our Python application to print all environment variables that are set to make this example easier.

#!/usr/bin/env python3

import os

for param in os.environ.keys():
    print("%20s %s" % (param, os.environ[param]))

We create the first environment file /usr/local/env/file1 with the content as below to assign string values to variables. Just as in the systemd unit file no string interpolation is being done, only lines with an equal sign are processed and everything after a hash sign is ignored.

FVAR1="test1"
FVAR2="test2"

We also create a second environment file /usr/local/env/file2 with the content below. Directly we see that variable FVAR1 is also be declared in this environment file.

FVAR1="TEST1"
FVAR3="Test3"

To use the environment files we need to declare them in the systemd unit file below. The line for file1 shows that we require the file to be present otherwise the service will fail, but for file2 the filename has been preceded by a hyphen to indicate to systemd that the file is optional and no error will be generated if it is absent.

[Unit]
Description=App Service

[Service]
Type=simple
EnvironmentFile=/usr/local/env/file1
EnvironmentFile=-/usr/local/env/file2
Environment="VAR0=hello world"
ExecStart=/usr/local/bin/app

After restarting the application with systemd all the environment variables that were set are shown in the system journal. The most interesting variable shown is FVAR1 as we declared it in two files earlier and we see that the value set in file1 was replaced by the value set in file2 that was processed later.

$ sudo systemctl daemon-reload
$ sudo systemctl restart app.service
$ sudo journalctl -u app
Aug 12 09:43:35 server.example.org systemd[1]: Started App Service.
Aug 12 09:43:35 server.example.org app[4483]: LANG en_US.UTF-8
Aug 12 09:43:35 server.example.org app[4483]: VAR0 hello world
Aug 12 09:43:35 server.example.org app[4483]: FVAR2 test2
Aug 12 09:43:35 server.example.org app[4483]: FVAR3 Test3
Aug 12 09:43:35 server.example.org app[4483]: FVAR1 TEST1
Aug 12 09:43:35 server.example.org app[4483]: PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin

While the purpose isn’t directly clear for everyone in the beginning, the most common use-case for enterprises is to manage environment variables that need to be set for Oracle for example. As lot of developers and engineers struggle why ORACLE_HOME isn’t being set while /etc/profile.d/oraenv.sh is present and works fine on other platforms like IBM AIX.

Environment variables set by systemd

Applications sometimes need environment variables to be set for triggering certain behavior like giving debug output or to route traffic via a HTTP-proxy for example. A common way is to modify the start-stop script, but with systemd on most Linux systems, like Debian and Red Hat based distributions, this can also directly set within the unit file and you don’t have to export the variables anymore.

The Python script below that we run via systemd checks if environment variable VAR1 has been set and will generate a different output based on that.

#!/usr/bin/env python3

import os

if os.environ.get('VAR1'):
    a = os.environ['VAR1']
else:
    a = 'default'

print(a);

Running the Python script also shows the output difference as the second command doesn’t print the string “default” anymore to the terminal, but the text “test” that we set via the environment variable.

$ ./main.py 
default
$ VAR1=test ./main.py 
test

Setting the environment variables via systemd is done by adding the attribute Environment to the Service section of the unit file for the service. After a systemctl daemon-load the environment variable will be set when you start or restart the service.

...
[Service]
Environment="VAR1=hello"
...

If more variables need to be set, then more Environment attributes can be added to the Service section.

...
[Service]
Environment="VAR1=hello"
Environment="VAR2=world"
...

While it may break some human workflows in the beginning, but in long term it is a good step for following the infrastructure as code path as Ansible could be used for managing these variables. Also storing these kind of variables in the same way makes both troubleshooting and collecting settings for an audit easier.

Connecting to legacy servers with OpenSSH

Phasing out legacy cryptographic algorithms can always be an interesting endeavor as terminating to early breaks stuff and to late it can lead to a compromise. OpenSSH disabled DSA with version 7.0 in March 2015 as 5 years earlier it was discovered that DSA was compromised and labelled as insecure. Normally this shouldn’t be a problem with a normal software life cycle, but sometimes you will encounter a legacy box that will not be upgraded as it will break things. Now it will stop new connections being setup from upgraded to machines as with SSH.

$ ssh user@server.example.org Unable to negotiate with server.example.org port 22: no matching host key type found. Their offer: ssh-dss

For an incidental connection from the command line the algorithm can be enabled again to connect with a legacy machine.

$ ssh -o HostKeyAlgorithms=+ssh-dss user@server.example.org

For automated processed or when scripts can’t be modified a setting for OpenSSH can also be set in $HOME/.ssh/config for the account depending on this option to be set.

Host server.example.org
  HostKeyAlgorithms=+ssh-dss

Re-enabling broken algorithms like DSA should only be done for a limited time and scope. In a lot of commercial environments these algorithms aren’t allowed to be enabled again. Also in most cases the code to run these obsolete algorithms can be removed in a later version as already is the case with SSL 3.0 and earlier for example.

Setting a different libvirt uri for Vagrant

HashiCorp Vagrant normally selects the right hypervisor, but the version shipped with Fedora 30 prefers to run within the QEMU user session of the hypervisor. A Vagrantfile it would match the default behavior which doesn’t require any system privileges is shown below.

# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|
  config.vm.synced_folder ".", "/vagrant", disabled: true
  config.vm.define "test01" do |test01|
    config.vm.box = "centos/7"
    config.vm.box_version = "1902.01"
    config.vm.hostname = "test01.localdomain"
    config.vm.provider :libvirt do |domain|
      domain.uri = 'qemu:///session'
    end
  end
end

In some cases a virtual machine needs to run on QEMU system level and that can be done by changing the domain.uri from “qemu:///session” to “qemu:///system”. Vagrant now creates the virtual machine at the system level of the hypervisor and isn’t depending on any user environment to run.

# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|
  config.vm.synced_folder ".", "/vagrant", disabled: true
  config.vm.define "test01" do |test01|
    config.vm.box = "centos/7"
    config.vm.box_version = "1902.01"
    config.vm.hostname = "test01.localdomain"
    config.vm.provider :libvirt do |domain|
      domain.uri = 'qemu:///system'
    end
  end
end

Membership of group libvirt maybe required or the right permissions with sudo for example. The first isn’t really advised except on local development system and then even sudo is still advised to reduce any accidental errors.

Using bare variables in Ansible 2.8

Ansible 2.8 was released in May 2019 and later in May came to Fedora 30 in package form. So the first tests could be done to see what needed to be done to switch from 2.7 to 2.8 and don’t generate a lot of stopped GitLab CI-jobs due to new warnings and errors. So let start with one warning that needs to be resolved before the 2.12 release and also is given on many third-party roles.

- name: Enable EPEL repository
  package:
    name: epel-release
    state: present
  when: platform_repo_epel

The example code above is simple enough to get the warning about CONDITIONAL_BARE_VARS. We could opt for disabling the warning in ansible.cfg and move forward, but as this is technical debt we don’t want to get more and resolve the current debt as quickly as possible.

TASK [role.platform : Enable EPEL repository] *******************************
[DEPRECATION WARNING]: evaluating platform_repo_epel as a bare variable, this 
behaviour will go away and you might need to add |bool to the expression in the
 future. Also see CONDITIONAL_BARE_VARS configuration toggle.. This feature 
will be removed in version 2.12. Deprecation warnings can be disabled by 
setting deprecation_warnings=False in ansible.cfg.

First we try to resolve this technical debt in the traditional way and making it a Boolean comparison and this stops Ansible from complaining as it is not a bare variable anymore.

- name: Enable EPEL repository
  package:
    name: epel-release
    state: present
  when: platform_repo_epel == True

Now Ansible lint starts to give a notification, added in version 4.0.0, as you shouldn’t do a Boolean comparison this was. And while it is technical correct, we also want this linting notification gone to pass the CI-pipeline.

[601] Don't compare to literal True/False
/path/to/ansible/project/roles/platform/tasks/main:6
  when: platform_repo_epel == True

In original message from Ansible there was already a hit on how to resolve this and by adding a Boolean filter both Ansible keeps on running correctly and Ansible lint is also happy.

- name: Enable EPEL repository
  package:
    name: epel-release
    state: present
  when: platform_repo_epel|bool

While these kind of modifications seem non-trivial and a test in your CI-pipeline could easily be set to “allow_failure=true”, but it makes code more readable for yourself and others.