Tag: docker

Watch out for Apparmor!

I’ve been hit by Apparmor a couple of times now. First with Samba, then with Openldap. AppArmor is a mandatory access control (MAC) system that restricts the capabilities of applications on a Linux system. While it can enhance the security of a Linux system, it can also cause issues with certain applications. Here are some apps that AppArmor can break and workarounds for each.

  1. Docker

Docker is a popular containerization technology that allows users to package and run applications in isolated environments. AppArmor can cause issues with Docker by blocking access to certain system resources required by Docker containers. To work around this issue, you can create a custom AppArmor profile for Docker that allows it to access the necessary resources.

To create a custom AppArmor profile for Docker, you can create a new profile file in the /etc/apparmor.d/ directory with the following contents:

# Profile for Docker
profile docker-container {
  # Allow access to necessary system resources
  /var/lib/docker/** rw,
  /var/run/docker.sock rw,
  /sys/fs/cgroup/** rw,
  /proc/sys/** rw,
  /etc/hostname r,
  /etc/hosts r,
  /etc/resolv.conf r,
  /etc/passwd r,
  /etc/group r,
  /etc/shadow r,
  /etc/gshadow r,
}

After creating the profile file, you can load it into the AppArmor kernel by running the following command:

sudo apparmor_parser -r /etc/apparmor.d/docker-container
  1. Apache

Apache is a widely used web server that can also be affected by AppArmor. If Apache is running in a restricted environment, it may not be able to access certain files or directories. To resolve this issue, you can modify the AppArmor profile for Apache to allow access to the necessary resources.

To modify the AppArmor profile for Apache, you can edit the existing profile file located in /etc/apparmor.d/usr.sbin.apache2 and add the necessary permissions. For example, to allow Apache to access the /var/www/html/ directory, you can add the following line to the profile:

/var/www/html/** r,

After making the necessary changes, you can reload the AppArmor profile by running the following command:

sudo service apparmor reload
  1. MySQL

MySQL is a popular open-source relational database management system that can be affected by AppArmor. If AppArmor is blocking access to MySQL, you may experience issues with database connectivity. To work around this issue, you can modify the AppArmor profile for MySQL to allow access to the necessary resources.

To modify the AppArmor profile for MySQL, you can edit the existing profile file located in /etc/apparmor.d/usr.sbin.mysqld and add the necessary permissions. For example, to allow MySQL to access the /var/lib/mysql/ directory, you can add the following line to the profile:

/var/lib/mysql/** rwk,

After making the necessary changes, you can reload the AppArmor profile by running the following command:

sudo service apparmor reload
  1. Nginx

Nginx is a high-performance web server that can also be affected by AppArmor. If Nginx is running in a restricted environment, it may not be able to access certain files or directories required for its operation. To resolve this issue, you can modify the AppArmor profile for Nginx to allow access to the necessary resources.

To modify the AppArmor profile for Nginx, you can edit the existing profile file located in /etc/apparmor.d/usr.sbin.nginx and add the necessary permissions. For example, to allow Nginx to access the /var/www/html/ directory, you can add the following line to the profile:

/var/www/html/** r,

After making the necessary changes, you can reload the AppArmor profile by running the following command:

sudo service apparmor reload
  1. OpenSSH

OpenSSH is a widely used remote access tool that can also be affected by AppArmor. If AppArmor is blocking access to OpenSSH, you may not be able to establish a remote connection to your Linux system. To work around this issue, you can modify the AppArmor profile for OpenSSH to allow access to the necessary resources.

To modify the AppArmor profile for OpenSSH, you can edit the existing profile file located in /etc/apparmor.d/usr.sbin.sshd and add the necessary permissions. For example, to allow OpenSSH to access the /var/log/auth.log file, you can add the following line to the profile:

/var/log/auth.log rw,

After making the necessary changes, you can reload the AppArmor profile by running the following command:

sudo service apparmor reload
  1. Samba

To modify the AppArmor profile for Samba, you can edit the existing profile file located in /etc/apparmor.d/usr.sbin.smbd and add the necessary permissions. For example, to allow Samba to access the /mnt/share/ directory, you can add the following line to the profile:

/mnt/share/** rw,

After making the necessary changes, you can reload the AppArmor profile by running the following command:

sudo service apparmor reload
  1. OpenLDAP

To modify the AppArmor profile for OpenLDAP, you can create a new profile file in the /etc/apparmor.d/ directory with the following contents:

# Profile for OpenLDAP
profile slapd {
  # Allow access to necessary system resources
  /var/lib/ldap/ r,
  /var/lib/ldap/** rw,
  /var/run/slapd/** rw,
  /etc/ldap/slapd.conf r,
  /etc/ldap/slapd.d/ r,
  /etc/ldap/slapd.d/** r,
  /usr/sbin/slapd mr,
  /usr/sbin/slapd.debug mr,
  /usr/sbin/slapd-{slave,monitor} ix,
  /usr/sbin/slapd.dbg mr,
  /usr/sbin/slapd-sock rw,
  /usr/sbin/slapd-sock-debug rw,
  /usr/sbin/slaptest mr,
}

After creating the profile file, you can load it into the AppArmor kernel by running the following command:

sudo apparmor_parser -r /etc/apparmor.d/slapd

By modifying AppArmor profiles for specific applications in this way, you can ensure that your applications have the necessary permissions to function correctly while still maintaining the security benefits of AppArmor.

AppArmor can cause issues with various applications on a Linux system, but these issues can usually be resolved by modifying the AppArmor profile for the affected application. By following the steps outlined above, you can ensure that your applications are functioning correctly while still maintaining the security benefits of AppArmor.

Clean up your old Kubernetes persistent data!

If you have ever removed a node from a Kubernetes cluster and then added it back, you may have encountered some issues with persistent data. Persistent data is any data that survives beyond the lifecycle of a pod, such as databases, logs, or configuration files. Kubernetes uses persistent volumes (PVs) and persistent volume claims (PVCs) to manage persistent data across the cluster.

However, sometimes these resources may not be cleaned up properly when a node is deleted or drained. This can cause problems when you try to reuse the node for another cluster or add it back to the same cluster. For example, you may see errors like:

  • Failed to attach volume \”pvc-1234\” on node \”node1\”: volume is already attached to node \”node2\”
  • Failed to mount volume \”pvc-5678\” on pod \”pod1\”: mount failed: exit status 32
  • Failed to create subPath directory for volumeMount \”data\” of container \”db\”: mkdir /var/lib/kubelet/pods/abcd-efgh/volumes/kubernetes.io~nfs/data: file exists

To avoid these issues, you need to clean up your old Kubernetes persistent data before adding a node back to a cluster. Here are some steps you can follow:

Step 1: Delete or unbind any PVCs associated with the node

The first step is to delete or unbind any PVCs that are associated with the node you want to remove. A PVC is a request for storage by a user or a pod. It binds to a PV that provides the actual storage backend. When you delete a PVC, it also releases the PV that it was bound to, unless the PV has a reclaim policy of Retain.

To list all the PVCs in your cluster, you can use the command:

kubectl get pvc --all-namespaces

To delete a PVC, you can use the command:

kubectl delete pvc <pvc-name> -n <namespace>

Alternatively, you can unbind a PVC from a PV without deleting it by editing the PVC spec and removing the volumeName field. This will make the PVC available for binding to another PV.

To edit a PVC, you can use the command:

kubectl edit pvc <pvc-name> -n <namespace>

Step 2: Delete any PVs that are not bound to any PVCs

The next step is to delete any PVs that are not bound to any PVCs. A PV is a piece of storage in the cluster that has been provisioned by an administrator or dynamically provisioned using storage classes. It is a resource in the cluster just like a node. PVs have a lifecycle independent of any pod that uses them.

To list all the PVs in your cluster, you can use the command:

kubectl get pv

To delete a PV, you can use the command:

kubectl delete pv <pv-name>

Note that deleting a PV does not necessarily delete the underlying storage device or volume. Depending on the type of storage and the reclaim policy of the PV, you may need to manually delete the storage device or volume from your cloud provider or storage server.

Step 3: Delete any leftover data on the node

The final step is to delete any leftover data on the node that you want to remove. This may include directories or files that were created by Kubernetes or by your applications. For example, you may need to delete:

  • The /etc/cni/net.d directory that contains CNI (Container Network Interface) configuration files
  • The /var/lib/kubelet directory that contains kubelet data such as pods, volumes, plugins, etc.
  • The /var/lib/etcd directory that contains etcd data if the node was running an etcd member
  • The /var/lib/docker directory that contains docker data such as images, containers, volumes, etc.
  • Any other application-specific data directories or files that were mounted or created on the node

To delete these directories or files, you can use commands like:

sudo rm -rf /etc/cni/net.d
sudo rm -rf /var/lib/kubelet
sudo rm -rf /var/lib/etcd
sudo rm -rf /var/lib/docker
sudo rm -rf /path/to/your/application/data

Be careful when using these commands and make sure you are deleting only what you intend

Source: Conversation with Bing, 3/27/2023(1) A complete storage guide for your Kubernetes storage problems. https://bing.com/search?q=Kubernetes+persistent+data Accessed 3/27/2023.
(2) Persistent Volumes | Kubernetes. https://kubernetes.io/docs/concepts/storage/persistent-volumes/?ref=hack-slash Accessed 3/27/2023.
(3) Kubernetes Persistent Volumes – Ultimate Guide – Knowledge Base by …. https://phoenixnap.com/kb/kubernetes-persistent-volumes Accessed 3/27/2023.
(4) Data persistence on Kubernetes – SQL Server Big Data Clusters. https://learn.microsoft.com/en-us/sql/big-data-cluster/concept-data-persistence?view=sql-server-ver15 Accessed 3/27/2023.
(5) A complete storage guide for your Kubernetes storage problems. https://www.cncf.io/blog/2020/04/28/a-complete-storage-guide-for-your-kubernetes-storage-problems/ Accessed 3/27/2023.
(6) Data Persistence in Kubernetes | Kubernetes Volumes simply explained …. https://dev.to/techworld_with_nana/data-persistence-in-kubernetes-kubernetes-volumes-simply-explained-852 Accessed 3/27/2023.