Creating Splunk universal forwarder using Alpine base image - splunk

I am trying to create a Splunk universal forwarder image using the alpine:3.8 base image.
FROM alpine:3.8
ENV SPLUNK_PRODUCT universalforwarder
ENV SPLUNK_VERSION 6.3.1
ENV SPLUNK_BUILD f3e41e4b37b2
ENV SPLUNK_FILENAME splunkforwarder-${SPLUNK_VERSION}-${SPLUNK_BUILD}-Linux-x86_64.tgz
ENV SPLUNK_SERVER_HOST testapp:9997
ENV SPLUNK_HOME /opt/splunk
ENV SPLUNK_GROUP splunk
ENV SPLUNK_USER splunk
ENV SPLUNK_BACKUP_DEFAULT_ETC /var/opt/splunk
ENV SPLUNK_INDEX test
ENV FORWARD_HOSTNAME InstanceId
# Here we install GNU libc (aka glibc) and set C.UTF-8 locale as default.
RUN apk --no-cache add ca-certificates wget \
&& wget -q -O /etc/apk/keys/sgerrand.rsa.pub https://alpine-pkgs.sgerrand.com/sgerrand.rsa.pub \
&& wget https://github.com/sgerrand/alpine-pkg-glibc/releases/download/2.28-r0/glibc-2.28-r0.apk \
&& apk add glibc-2.28-r0.apk \
&& rm -rf /var/lib/apt/lists/*
# add splunk:splunk user
RUN addgroup --system ${SPLUNK_GROUP} \
&& adduser --system --ingroup ${SPLUNK_GROUP} ${SPLUNK_USER}
# Download official Splunk release, verify checksum and unzip in /opt/splunk
# Also backup etc folder, so it will be later copied to the linked volume
RUN apk add sudo curl\
&& mkdir -p ${SPLUNK_HOME} \
&& curl -o /tmp/${SPLUNK_FILENAME} https://download.splunk.com/products/${SPLUNK_PRODUCT}/releases/${SPLUNK_VERSION}/linux/${SPLUNK_FILENAME} \
&& curl -o /tmp/${SPLUNK_FILENAME}.md5 https://download.splunk.com/products/${SPLUNK_PRODUCT}/releases/${SPLUNK_VERSION}/linux/${SPLUNK_FILENAME}.md5 \
&& tar xzf /tmp/${SPLUNK_FILENAME} --strip 1 -C ${SPLUNK_HOME} \
&& rm /tmp/${SPLUNK_FILENAME} \
&& rm /tmp/${SPLUNK_FILENAME}.md5 \
&& mkdir -p /var/opt/splunk \
&& cp -R ${SPLUNK_HOME}/etc ${SPLUNK_BACKUP_DEFAULT_ETC} \
&& rm -fR ${SPLUNK_HOME}/etc \
&& chown -R ${SPLUNK_USER}:${SPLUNK_GROUP} ${SPLUNK_HOME} \
&& chown -R ${SPLUNK_USER}:${SPLUNK_GROUP} ${SPLUNK_BACKUP_DEFAULT_ETC} \
&& rm -rf /var/lib/apt/lists/*
COPY ./config /tmp/splunk
COPY patch-entrypoint.sh /sbin/entrypoint.sh
RUN chmod +x /sbin/entrypoint.sh
# Ports Splunk Daemon, Network Input, HTTP Event Collector
EXPOSE 8089/tcp 1514 8088/tcp
WORKDIR /opt/splunk
# Configurations folder, var folder for everyting (indexes, logs, kvstore)
VOLUME [ "/opt/splunk/etc", "/opt/splunk/var" ]
ENTRYPOINT ["/sbin/entrypoint.sh"]
CMD ["start-service"]
Now I am facing a couple of issues here:
When I am running /opt/splunkforwarder/bin/splunk start --accept-license I am getting /opt/splunkforwarder/bin/splunk: not found.
I am using custom output.conf file. It's in config folder.
[tcpout]
defaultGroup = abc
disabled = false
[tcpout:abc]
server = _OUTPUT_SERVERS_
autoLB = true
compressed = false
useACK = true
sendCookedData = true
entrypoint.sh is the script which I am using to replace the environment variable from output.config and restart the Splunk but again restart is not working. How can I fix this?

AFAIK, alpine:3.8 doesn't ship with glibc, which Splunk requires. Is is possible that this is causing issues? Have you tried with a different base image?

Related

Running Bazel in offline mode ? (bazel 5.1.1, Tensor 2.9.1, CentOS)

I am trying to install Tensor 2.9.1 using bazel 5.1.1 on Centos via Jenkins so here what I am doing (simplified the code below for easier reading)
ENV BAZELRC ENV BAZELRC "/serving/.bazelrc"
WORKDIR /
RUN mkdir /bazel && cd /bazel \
&& curl -fSsL -O https://github.com/bazelbuild/bazel/releases/download/0.5.2/bazel-0.5.2-installer-linux-x86_64.sh \
&& curl -fSsL -o /bazel/LICENSE.txt https://raw.githubusercontent.com/bazelbuild/bazel/master/LICENSE \
&& chmod +x bazel-*.sh && ./bazel-0.5.2-installer-linux-x86_64.sh \
&& cd / && git clone --recurse-submodules https://github.com/tensorflow/serving \
&& cd serving && git checkout tags/2.9.1 \
&& git submodule init && git submodule update --recursive \
&& cd /serving && bazel build -c opt tensorflow_serving/...
The last command fails trying to connect to an external mirror (we are not allowed to connect externally due to security reasons )
Wondering if is possible to run Bazel in an offline mode ? Example pre-download all the required tar.gz from https://mirror.bazel.build/github.com/tensorflow/tensorflow/archive/ somewhere ? And where
Error
[2022-07-21T10:46:34.873Z] [0m[91mStarting local Bazel server and connecting to it...
[2022-07-21T10:46:36.273Z] [0m[91mLoading: 0 packages loaded
[2022-07-21T10:46:41.062Z] [0m[91mWARNING: Download from https://mirror.bazel.build/github.com/tensorflow/tensorflow/archive/d8ce9f9c301d021a69953134185ab728c1c248d3.tar.gz failed: class java.io.FileNotFoundException GET returned 404 Not Found
[2022-07-21T10:46:42.460Z] [0m[91mLoading: 0 packages loaded

How to run Apache as non-root user?

I'm building an image from the following Dockerfile and following command docker build --rm -f "Dockerfile" -t non_root_image_plz_work .:
DockerFile
FROM node:14.7.0-buster-slim AS apache_for_selenium
# Create non-root group and user
RUN addgroup --system shared-folder \
&& adduser --system --home /var/cache/shared-folder --group shared-folder --uid 1001
# Make Port accessable
EXPOSE 80/tcp
# Set Node env.Name
ENV NODE_ENV=dev
RUN apt-get -qq update && apt-get -qq install -y --no-install-recommends nano git openssl bash musl curl apache2 apache2-utils systemd && \
systemctl enable apache2 && npm config set registry http://localhost:5000/repository/repo && \
npm i -g pm2 serve && mkdir /usr/share/shared-folder
RUN ln -sf /dev/stdout /var/log/apache2/access.log && \
ln -sf /dev/stderr /var/log/apache2/error.log
WORKDIR /usr/share/shared-folder
COPY . /usr/share/shared-folder/
RUN npm install && npm cache clean --force && npm cache verify && \
rm /var/www/html/index.html && \
ln -s /usr/share/shared-folder/mochawesome-report /var/www/html/mochawesome-report && \
chown www-data -R /var/www/html/mochawesome-report && chgrp www-data -R /var/www/html/mochawesome-report
VOLUME /usr/share/shared-folder/mochawesome-report
USER 1001
CMD [ "sh", "-c", "service apache2 start ; pm2-runtime process.yml --no-daemon" ]
When I try to run the image using docker run non_root_image_plz_work, I get the following error:
Error after running docker run command:
Starting Apache httpd web server: apache2 failed!
The apache2 configtest failed. ... (warning).
Output of config test was:
mkdir: cannot create directory '/var/run/apache2': Permission denied
chown: changing ownership of '/var/lock/apache2.3FGoa8Y71E': Operation not permitted
It seems to be a permissions issue, as if I'm not properly running the container as a non-root user. Any suggestions on how I can get the container to run properly as a non-root user?
Note: I used a dummy registry in the Dockerfile for I don't want to show the actual registry, thanks
In Docker, all folders are owned by root. Without knowing your directory structure, I guess your problem is, that your user 1001 (or the setup programm which is run with 1001's permission) tries to access directories that (probably) are owned by root.
Either you can try:
Change your permissions of the folders.
This can be used of you know which folders are accessed and want to prevent further permission issues.
chmod -R 777 /path/to/folder
Give your user proper permissions.
Here is a very quick walkthrough. Please comment if it didn't slove your problem and I'll try to update this for a more specific answer.
A small example (taken from here).
You can setup your non-root-user foo with passwordless access:
RUN \
groupadd -g 1001 foo && useradd -u 1001 -g foo -G sudo -m -s /bin/bash 1001
&& \
sed -i /etc/sudoers -re 's/^%sudo.*/%sudo ALL=(ALL:ALL) NOPASSWD: ALL/g' && \
sed -i /etc/sudoers -re 's/^root.*/root ALL=(ALL:ALL) NOPASSWD: ALL/g' && \
sed -i /etc/sudoers -re 's/^#includedir.*/## **Removed the include directive** ##"/g' && \
echo "foo ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers; su - foo -c id
Hint: You will probably need to install sudo
apt-get install sudo
Now, try running the entrypoint (or your commad) with sudo.
EDIT:
I've updated the answer to match your Docker-File. Have a look at it. The user nonroot is assigned uuid 1001 and added to /etc/sudoers. Also your command is now run with sudo which should prevent the permission issues.
FROM node:14.7.0-buster-slim AS apache_for_selenium
# Create non-root group and user
RUN addgroup --system shared-folder \
&& adduser --system --home /var/cache/shared-folder --ingroup shared-folder --uid 1001 nonroot
# Make Port accessable
EXPOSE 80/tcp
# Set Node env.Name
ENV NODE_ENV=dev
RUN apt-get -qq update && apt-get -qq install -y --no-install-recommends \
sudo nano git openssl bash musl curl apache2 apache2-utils systemd \
&& systemctl enable apache2
#\
# && #npm config set registry http://localhost:5000/repository/repo && \
#npm i -g pm2 serve && mkdir /usr/share/shared-folder
RUN ln -sf /dev/stdout /var/log/apache2/access.log && \
ln -sf /dev/stderr /var/log/apache2/error.log
WORKDIR /usr/share/shared-folder
COPY . /usr/share/shared-folder/
RUN npm install && npm cache clean --force && npm cache verify && \
rm /var/www/html/index.html && \
ln -s /usr/share/shared-folder/mochawesome-report /var/www/html/mochawesome-report && \
chown www-data -R /var/www/html/mochawesome-report && chgrp www-data -R /var/www/html/mochawesome-report
VOLUME /usr/share/shared-folder/mochawesome-report
RUN \
sed -i /etc/sudoers -re 's/^%sudo.*/%sudo ALL=(ALL:ALL) NOPASSWD: ALL/g' && \
sed -i /etc/sudoers -re 's/^root.*/root ALL=(ALL:ALL) NOPASSWD: ALL/g' && \
sed -i /etc/sudoers -re 's/^#includedir.*/## **Removed the include directive** ##"/g' && \
echo "nonroot ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers
USER nonroot
CMD [ "sudo sh", "-c", "service apache2 start ; pm2-runtime process.yml --no-daemon" ]
The problem here: apache is special user. it only can start by root user.
you can not start apache by another user. That why you got permission deny.
Seen i saw your dockerfile. your user created is normal user.
Try make script like below with name apache-start :
#!/bin/sh
set -e
# Apache gets grumpy about PID files pre-existing
rm -f /usr/local/apache2/logs/httpd.pid
exec httpd -DFOREGROUND "$#"
and your docker file should be like
FROM node:14.7.0-buster-slim AS apache_for_selenium
# Create non-root group and user
RUN addgroup --system shared-folder \
&& adduser --system --home /var/cache/shared-folder --group shared-folder --uid 1001
# Make Port accessable
EXPOSE 80/tcp
# Set Node env.Name
ENV NODE_ENV=dev
RUN apt-get -qq update && apt-get -qq install -y --no-install-recommends nano git openssl bash musl curl apache2 apache2-utils systemd && \
systemctl enable apache2 && npm config set registry http://localhost:5000/repository/repo && \
npm i -g pm2 serve && mkdir /usr/share/shared-folder
RUN ln -sf /dev/stdout /var/log/apache2/access.log && \
ln -sf /dev/stderr /var/log/apache2/error.log
WORKDIR /usr/share/shared-folder
COPY . /usr/share/shared-folder/
RUN npm install && npm cache clean --force && npm cache verify && \
rm /var/www/html/index.html && \
ln -s /usr/share/shared-folder/mochawesome-report /var/www/html/mochawesome-report && \
chown www-data -R /var/www/html/mochawesome-report && chgrp www-data -R /var/www/html/mochawesome-report
VOLUME /usr/share/shared-folder/mochawesome-report
COPY apache-start /usr/local/bin/
CMD ["apache-start"]
USER 1001
Another option is to switch to podman tool which is an alternative to Docker. With podman you can run containers (the same images you use in Docker) but with normal users. That has a lot of benefits specially from security point of view.

Docker Container from php:5.6-apache as root

This would be related to Docker php:5.6-Apache Development Environment missing permissions on volume mount
I have tried pretty much everything to make the mounted volume be readable by www-data, my current solution is trying to move by scripts the folders needed by the application to /var and giving the proper permissions to be writable by www-data but that is becoming hard to maintain.
Giving the fact that it's a development environment I don't mind being a security hole so I would like to run apache as root and I get
Error: Apache has not been designed to serve pages while running as
root. There are known race conditions that will allow any local user
to read any file on the system. If you still desire to serve pages as
root then add -DBIG_SECURITY_HOLE to the CFLAGS line in your
src/Configuration file and rebuild the server. It is strongly
suggested that you instead modify the User directive in your
httpd.conf file to list a non-root user.
Is there any easy way I can accomplish this using the docker image php:5.6-apache?
This is my docker-compose.yml
version: '2'
services:
api:
container_name: api
privileged: true
build:
context: .
dockerfile: apigility/Dockerfile
ports:
- "2020:80"
volumes:
- /ft/code/api:/var/www:rw
And this is my Dockerfile:
FROM php:5.6-apache
USER root
RUN apt-get update \
&& apt-get install -y sudo openjdk-7-jdk \
&& echo "www-data ALL=NOPASSWD: ALL" >> /etc/sudoers
RUN apt-get install -y git zlib1g-dev libmcrypt-dev nano vim --no-install-recommends \
&& apt-get clean \
&& rm -r /var/lib/apt/lists/* \
&& docker-php-ext-install mcrypt zip \
&& curl -sS https://getcomposer.org/installer \
| php -- --install-dir=/usr/local/bin --filename=composer \
&& a2enmod rewrite \
&& sed -i 's!/var/www/html!/var/www/public!g' /etc/apache2/apache2.conf \
&& echo "AllowEncodedSlashes On" >> /etc/apache2/apache2.conf \
&& cp /usr/src/php/php.ini-production /usr/local/etc/php/php.ini \
&& printf '[Date]\ndate.timezone=UTC' > /usr/local/etc/php/conf.d/timezone.ini
WORKDIR /var/www
Why not to do exactly what it says in the question you referred to?
RUN usermod -u 1000 www-data
RUN groupmod -g 1000 www-data
This is not a hack. It's a proper solution to the problem you have in the development environment.
So, I managed to make the mounted data available for www-data by using the part of the answer in the related post but another step is required for it to work.
After you run docker-machine start default you need to ssh into it and run the following:
sudo mkdir --parents /code [where /code is the shared folder in virtualbox]
sudo mount -t vboxsf -o uid=999,gid=999 code /code [this is to make sure the uid and gid is 999 for the next part to work]
Then in your Dockerfile add
RUN usermod -u 999 www-data \
&& groupmod -g 999 www-data
After it's mounted, /code will have the owner www-data, and problem solved!
Another and better solution.
Add this in your dockerfile
RUN cd ~ \
&& apt-get -y install dpkg-dev debhelper libaprutil1-dev libapr1-dev libpcre3-dev liblua5.1-0-dev autotools-dev \
&& apt-get source apache2.2-common \
&& cd apache2-2.4.10 \
&& export DEB_CFLAGS_SET="-g -O2 -fstack-protector-strong -Wformat -Werror=format-security -DBIG_SECURITY_HOLE" \
&& dpkg-buildpackage -b \
&& cd .. \
&& dpkg -i apache2-bin_2.4.10-10+deb8u7_amd64.deb \
&& dpkg -i apache2.2-common_2.4.10-10+deb8u7_amd64.deb
After that, you could be able to run apache as root.
PS : apache2-2.4.10, apache2-bin_2.4.10-10+deb8u7_amd64.deb and apache2.2-common_2.4.10-10+deb8u7_amd64.deb could change according to your source

php app files permission after docker build

I'm trying to build docker image with SuiteCRM using this Dockerfile:
FROM php:5.4-apache
RUN a2enmod rewrite
RUN apt-get update \
&& apt-get install -y apt-utils \
&& apt-get install -y libpng12-dev libjpeg-dev mercurial zip nano \
&& docker-php-ext-configure gd --with-jpeg-dir=/usr/lib \
&& docker-php-ext-install gd \
&& docker-php-ext-install mysqli \
&& docker-php-ext-install pdo_mysql \
&& apt-get -y install re2c libmcrypt-dev \
&& docker-php-ext-install mcrypt \
&& apt-get -y install zlib1g-dev \
&& docker-php-ext-install zip \
&& apt-get purge --auto-remove -y zlib1g-dev \
&& apt-get -y install libssl-dev libc-client2007e-dev libkrb5-dev \
&& docker-php-ext-configure imap --with-imap-ssl --with-kerberos \
&& docker-php-ext-install imap mbstring json \
&& rm -rf /var/lib/apt/lists/*
RUN curl -k -L -o suitecrm.zip "https://suitecrm.com/component/dropfiles/?task=frontfile.download&id=35"
RUN unzip -q suitecrm.zip -d /var/www/
RUN rm suitecrm.zip
RUN rm -rf /var/www/html && mv /var/www/suitecrm-7.2.2-max /var/www/html
RUN rm -rf /var/www/suitecrm-7.2.2-max
RUN chown -R www-data:www-data /var/www/html
RUN chmod -R 755 /var/www/html
RUN chmod -R 775 cache custom modules themes data upload config_override.php
EXPOSE 80
CMD ["apache2-foreground"]
Then image built and ran I've got an errors:
Warning: include(include/MVC/preDispatch.php): failed to open stream: Permission denied in /var/www/html/index.php on line 42
...
File owner is www-data:www-data
$ docker exec -t suite_web_dev ls -la index.php
-rwxr-xr-x 1 www-data www-data 2525 Mar 2 18:04 index.php
$ docker exec -t suite_web_dev ls -la include/MVC/preDispatch.php
-rwxr-xr-x 1 www-data www-data 2766 Mar 2 18:04 include/MVC/preDispatch.php
If I exclude form Dockerfile lines where it downloading App and changing files permissions, and call these commands after image start, everything works fine.
RUN curl -k -L -o ...
...
RUN chmod -R 775 cache custom modules themes data upload config_override.php
What differens between changing permission at build and after run? What I need to do for build image with App code?
Upd:
docker runs under boot2docker:
Boot2Docker-cli version: v1.7.1
Docker version 1.7.1
The permission does not seem to be on a file, but on 'open stream' operation.
This could be one of the causes:
When you install your application in the docker file, the hostname of the final container will no longer be the same as the hostname of the temporary container used while building the image. The app installer might fetch the hostname and store it in some configuration file.
If that is the case, then, when you run the container, you should execute a script which replaces the hostname in the config.

Docker - Cannot start Redis Service

I'm installation Redis, setting up init.d, placed the redis.conf beside init.d.
Then using CMD service init.d start to start Redis.
However, Redis-Server does not start, and there are no indiciation in the log file that the service failed to start.
Installing Redis and Placing redis.conf to the etc/init.d folder
Commands:
# add our user and group first to make sure their IDs get assigned consistently, regardless of whatever dependencies get added
RUN groupadd -r redis && useradd -r -g redis redis
RUN apt-get update > /dev/null \
&& apt-get install -y curl > /dev/null 2>&1 \
&& rm -rf /var/lib/apt/lists/* > /dev/null 2>&1
# grab gosu for easy step-down from root
RUN gpg --keyserver pool.sks-keyservers.net --recv-keys B42F6819007F00F88E364FD4036A9C25BF357DD4
RUN curl -o /usr/local/bin/gosu -SL "https://github.com/tianon/gosu/releases/download/1.2/gosu-$(dpkg --print-architecture)" > /dev/null 2>&1 \
&& curl -o /usr/local/bin/gosu.asc -SL "https://github.com/tianon/gosu/releases/download/1.2/gosu-$(dpkg --print-architecture).asc" > /dev/null 2>&1 \
&& gpg --verify /usr/local/bin/gosu.asc > /dev/null 2>&1 \
&& rm /usr/local/bin/gosu.asc > /dev/null 2>&1 \
&& chmod +x /usr/local/bin/gosu > /dev/null 2>&1
ENV REDIS_VERSION 3.0.1
ENV REDIS_DOWNLOAD_URL http://download.redis.io/releases/redis-3.0.1.tar.gz
ENV REDIS_DOWNLOAD_SHA1 fe1d06599042bfe6a0e738542f302ce9533dde88
# for redis-sentinel see: http://redis.io/topics/sentinel
RUN buildDeps='gcc libc6-dev make'; \
set -x \
&& apt-get update > /dev/null && apt-get install -y $buildDeps --no-install-recommends > /dev/null 2>&1 \
&& rm -rf /var/lib/apt/lists/* > /dev/null 2>&1 \
&& mkdir -p /usr/src/redis > /dev/null 2>&1 \
&& curl -sSL "$REDIS_DOWNLOAD_URL" -o redis.tar.gz > /dev/null 2>&1 \
&& echo "$REDIS_DOWNLOAD_SHA1 *redis.tar.gz" | sha1sum -c - > /dev/null 2>&1 \
&& tar -xzf redis.tar.gz -C /usr/src/redis --strip-components=1 > /dev/null 2>&1 \
&& rm redis.tar.gz > /dev/null 2>&1 \
&& make -C /usr/src/redis > /dev/null 2>&1 \
&& make -C /usr/src/redis install > /dev/null 2>&1 \
&& cp /usr/src/redis/utils/redis_init_script /etc/init.d/redis_6379
&& rm -r /usr/src/redis > /dev/null 2>&1 \
&& apt-get purge -y --auto-remove $buildDeps > /dev/null 2>&1
RUN mkdir /data && chown redis:redis /data
VOLUME [/data]
WORKDIR /data
CMD Service init.d start
Command:
RUN touch /var/redis/6379/redis-6379-log.txt
RUN chmod 777 /var/redis/6379/redis-6379-log.txt
ENV REDISPORT 6379
ADD $app$/redis-config.txt /etc/redis/$REDISPORT.conf
CMD service /etc/init.d/redis_6379 start
If I use shellinabox to access the container, and if I type in
/etc/init.d/redis_6379 start
Redis server will start, but it won't start in the dockerfile. Why is this?
It seems that you cannot use background processes, but instead you need something called supervisord.
To Install:
RUN apt-get install -y supervisor
RUN mkdir -p /var/log/supervisor
ADD $app$/supervisord.conf /etc/supervisor/conf.d/supervisord.conf
CMD /usr/bin/supervisord
Configuration File:
[supervisord]
nodaemon=true
[program:shellinabox]
command=/bin/bash -c "cd /tmp && exec /opt/shellinabox/shellinaboxd --no-beep --service ${service}"
[program:redis-server]
command=/bin/bash -c "redis-server /etc/redis/${REDISPORT}.conf"
What happens is that after the command is executed, it will start both programs shelllinabox and redis-server.
Thanks everyone for the help!
In general, you can't use an init script inside a Docker container. These scripts are typically designed to start a service "in the background", which means that even if the service starts, the script ultimately exits.
If this is the first process in your Docker container, Docker will see it exit, which will cause it to clean up the container. You will need to arrange for redis to run in the foreground in your container, or you will need to arrange to run some sort of process supervisor in your container.
Consider looking at the official resource container to see one way of setting things up. You can see the Dockerfiles in the github repository.