Lesson 6: Defending container Infrastructure

Introduction

Hello everyone, in the past few weeks, we have gone through the container reconnaissance, container attack surface and how to exploit various scenarios of docker misconfiguration.

This week, we will see how to secure/defend containers from the attacks.

Let’s follow a similar lesson plan we had in the attack modules.

1. Image-based defenses
2. Daemon-based defenses
3. OS and kernel defenses
4. Network-based defenses
5. Application-based defenses

We will cover image based defenses and some daemon based defenses today. OS, Network, and Application based defenses will be covered in the next few lessons.

Image-based defenses

We have already seen the risks of vulnerable components, malware and sensitive files in docker images. Let’s ensure container images are following best practices and are free from security vulnerabilities, malicious software.

Free images (Dockerfile available)

We recommend organizations embed these checks as part of their DevOps processes so we can find and fix these issues early on.

Considering we have access to Dockerfile, we can divide the tools into two broad categories.

1. Linting (SAST) of Dockerfile for best practices.
2. Component analysis (SCA) of docker images using Clair (CVE scanning).

 We can run these tools on a Dockerfile and a Docker image. Let’s dive deeper into these two tools.

Hadolint

The Hadolint authors describe it as “A smarter Dockerfile linter that helps you build best practice Docker images. The linter is parsing the Dockerfile into an Abstract Syntax Tree (AST) and performs rules on top of the AST”.

It’s a static analysis technique to ensure DevOps follows best practices. Let’s try to run this linter on the following Dockerfile (obviously save it in the course VM before running the next command).

# FROM python base image
 FROM python:2-alpine
 # COPY startup script
 RUN wget https://github.com/secfigo/django.nv/archive/1.1.zip /app
 WORKDIR /app
 RUN apk add --no-cache gawk sed bash grep bc coreutils
 RUN pip install -r requirements.txt
 RUN chmod +x reset_db.sh && bash reset_db.sh
 # EXPOSE port 8000 for communication to/from server
 EXPOSE 8000
 # CMD specifcies the command to execute container starts running.
 CMD ["/app/run_app_docker.sh"]

Now run the Hadolint linter on a Dockerfile.

$ docker run --rm -i hadolint/hadolint < Dockerfile 

But why is it an issue? apart from the risk of breaking the build and adding uncertainty in the build, it’s difficult to find out the exact version of the package used (by just reading Dockerfile) as it’s not mentioned in the Dockerfile.

The above issue can be solved by scanning the docker images where the exact version of the package will be installed on an image layer.

Clair can help you in solving this problem. Clair achieves this by scanning the exact version of a component to figure out vulnerable components in a docker image.

Clair Scanner, static analysis of docker images

Clair is an open-source project that helps in finding vulnerable components in a docker image.

Running the Clair scanner basically involves the three broad steps:
1. Clair downloads vulnerability metadata from a configured set of sources (NVD, CVE, etc,) and stores this data in a database.
2. The Clair Clients (CLI, wrappers) can use the Clair API to index the container images.
3. Clients can then use the Clair API to query the database for vulnerabilities of a particular image.

As shown in the figure above, Clair downloads the NVD database then starts the API server which will be, then used by Clair CLI to scan a docker image.

Let’s convert this process into a command and run the Clair scanner now.
1. Clair downloads vulnerability metadata from a configured set of sources (NVD, CVE, etc,)

$ docker run -d --name db arminc/clair-db:latest

Store the vulnerability data in a database.

$ docker run -p 6060:6060 --link db:postgres -d --name clair \
--restart on-failure arminc/clair-local-scan:v2.0.1

2. Download the Clair CLI scanner

$ wget https://github.com/arminc/clair-scanner/releases/download/v8/clair-scanner_linux_amd64
 $ chmod +x clair-scanner_linux_amd64

3. Use the Clair Scanner to scan a particular image. We are going to use ubuntu:latest image in this exercise.

$ docker pull ubuntu:latest

Push ubuntu image to docker registry as Clair needs it to be in the registry.

$ docker tag ubuntu:latest localhost:5000/myubuntu:latest && \ 
 docker push localhost:5000/myubuntu:latest

4. Wait for 10-15 minutes for the database to download the latest vulnerability information. If you have slow internet speeds then you might have to wait up to an hour as well.

$ HOST_IP=$(ip -4 addr show docker0 | grep -Po 'inet \K[\d.]+')
$./clair-scanner_linux_amd64 -c http://localhost:6060 --ip ${HOST_IP} \
 localhost:5000/myubuntu:latest

Clair output

We can see a typical Clair scan output on a ubuntu docker image below. You can clearly see, Clair has found multiple vulnerable packages in the latest ubuntu image.

Are they all exploitable vulnerable? maybe, try figuring it out 🙂

Task 1: Static Analysis of Nginx Docker image.
Please scan for security issues in an Nginx image using Clair-scanner.

Free images (Dockerfile not available)

If you do not have a Dockerfile for an image, you can use Dive and Whaler tools. We have already covered these two tools in our previous lessons.

You can also explore another tool called Dagda which is great for CVE scanning, antivirus scanning and much more. It’s beyond the scope of this course but if you are interested, you can explore our expert course where we cover this topic extensively.

Malicious images

As discussed in the previous lesson, we can use ClamAV Anti-virus (clamscan) software to scan for malware.

Secrets stored inside the images.

Let’s analyze hysnsec/secrets docker image for secrets using Inspect tool and Trufflehog tool.

Find secrets in the code

Let’s download and run hysnsec/secrets container for which we don’t have Dockerfile.

$ docker pull hysnsec/secrets
$ docker run -p 8000:8000 -d --name secrets hysnsec/secrets

Now install Trufflehog and run it on the present working directory ( . <– dot)

$ docker exec -it secrets /bin/sh -c "pip install trufflehog && trufflehog ." 

As we can see, there are encrypted passwords stored in the image filesystem. We can then go ahead, scan all images regularly and fail the developer builds if we find secrets.

Find secrets in environment variables

You can also use the docker inspect command to check out secrets in the environment variables.

$ docker inspect secrets -f "{{json .Config.Env}}"

Daemon-based defenses

We have to also ensure the Docker engine is configured properly with the right set of permissions and settings. Special care has to be taken to configure host kernel parameters like Namespaces, Cgroups, capabilities.

NCC Group has created a nice diagram of different container technologies and their security mechanisms.

As you can see, Docker has strong defaults but there’s still scope for improvements.

Docker bench security, dynamic analysis tool

Docker released a script to check if docker daemon is set up properly, it’s called docker bench security, Docker describes it as “A script that checks for dozens of common best-practices around deploying Docker containers in production. The tests are all automated, and are inspired by the CIS Docker Community Edition Benchmark v1.1.0.”

You can run it via the following command

$ docker run -it --net host --pid host --userns host --cap-add audit_control \
 -e DOCKER_CONTENT_TRUST=$DOCKER_CONTENT_TRUST \
 -v /var/lib:/var/lib \
 -v /var/run/docker.sock:/var/run/docker.sock \
 -v /usr/lib/systemd:/usr/lib/systemd \
 -v /etc:/etc --label docker_bench_security \
 docker/docker-bench-security

Note: Take a look at the permissions we have given to the docker bench security container, it needs root access to the tool for accessing, configurations and other settings.

A typical output from the docker bench for security is shown below.

Task 2: Docker bench security
Please use the docker-bench-security script to scan a docker host for security issues and try to fix at least a few security findings.

We as security engineers should utilize these tools extensively to find security issues before hackers do.

Reference and Further Reading

Refer to lesson 4 and lesson 5 of Practical DevSecOps’s Container security course.

Conclusion

This week, we have learned more about how to defend docker infrastructure using static analysis techniques like linting, component analysis, finding secrets and docker benchmark scanning. Unfortunately, the security folks give vague and general defense techniques, we hope this lesson addresses specific recommendations you can provide to your clients and in-house teams.

Next week, we will be discussing advanced defenses like namespaces, capabilities, SecComp and AppArmor profiles.