Can't start Kibana (ELK stack) with Docker Compose

  • I have two OMV4 machines, let's call them OLD and NEW.
    They both:
    - are running the OMV 4.1.31-1 version
    - have Docker base path in "Settings" tab set to a shared folder named "Docker" with identical privileges
    - have Docker Compose v1.25.0 installed


    When I try to run this YAML file with Docker Compose:

    ... everything runs fine on the OLD machine.


    However, when using the identical YAML file on the NEW machine elasticsearch starts as it should and is available from the host, but it can't start Kibana.
    http://NEW:5601/ returns "Kibana server is not ready yet" no matter how long I wait.


    Docker log for the Kibana container keeps repeating these two messages in endless loop:
    "Unable to revive connection: http://elasticsearch:9200/" and "No living connections"
    On both machines the network elk_default is successfully created and DNS names are assigned to the containers listed in the YAML file.


    Command docker network inspect elk_default returns this on the NEW machine (the non-working one):


    For comparison, this is the output from the OLD machine (where everything works fine) for the same command:


    This might be connected with another issue described here I'm experiencing on the NEW machine.


    Any ideas?

  • Whats the output of
    docker network ls command.?
    and also can we maybe get docker logs elasticsearch , atleast for a few lines?
    You could also docker -ti on kibana docker and see if you can ping elasticsearch.
    your docker compose is pretty much straightforward ELK stack

  • Whats the output of docker network ls command.?
    and also can we maybe get docker logs elasticsearch , atleast for a few lines?
    You could also docker -ti on kibana docker and see if you can ping elasticsearch.
    your docker compose is pretty much straightforward ELK stack

    Docker network list (identical on both machines, except for the NETWORK ID column):


    NETWORK ID NAME DRIVER SCOPE
    1cef399269c7 bridge bridge local
    fa38b50d55f3 elk_default bridge local
    eeec400025a8 host host local
    7e8abdb4779e none null local


    As I stated in the original post, elasticsearch itself is running properly. I can access and use it via the host mapped port, no problem there.
    The docker-compose.yml file is valid, because it works properly on another machine.


    The elasticsearch log is attached.
    I can normally ping the elasticsearch and logstash containers from the kibana container by their IP addresses, but not by their DNS names.
    However, I can ping both the kibana and logstash containers from the elasticsearch container by their DNS names.


    So, the elasticsearch container can use DNS names of other two containers (kibana and logstash) to access them, but the other two can't.

  • Due to the recent global situation, I've had enough time last weekend to install OMV over 20 times on three different machines and (finally) figure out the cause of the problem.

    I've used a shared folder for the Docker base path. The original problem occured when I've set "read only" or "no access" for "others" on that shared folder ("administrator" and "users" had full access). Giving r/w permissions to the "docker" group while "others" were set to "no access" didin't help.


    No error message pointed to that problem, so it would be nice if the text "The location of the Docker base path (this setting is optional and defaults to /var/lib/docker if unset). The plugin must be enabled for a change to be committed" could be expanded to include a note about the necessary permissions.

  • Well, you do not say what version of OMV you are running. But for the more recent builds of OMV 5, using a /sharedfolder path for the Docker base path is not allowed and the software will not let you do that in the WebGUI.


    It may still be possible to make that mistake anyway if /etc/docker/daemon.json is manually hand edited. I do not know and I am not going to try it to find out.

    --
    Google is your friend and Bob's your uncle!


    OMV AMD64 7.x on headless Chenbro NR12000 1U 1x 8m Quad Core E3-1220 3.1GHz 32GB ECC RAM.

  • Well, you do not say what version of OMV you are running. But for the more recent builds of OMV 5, using a /sharedfolder path for the Docker base path is not allowed and the software will not let you do that in the WebGUI.


    It may still be possible to make that mistake anyway if /etc/docker/daemon.json is manually hand edited. I do not know and I am not going to try it to find out.

    As the thread tag at the top of the page says, it's OMV 4.

    The problem is not tied to the /sharedfolders path, but the permissions for "others" on the shared folder. It would also occur if I used /srv/dev-disk-by-label... scheme.


    Docker images can get quite large. Forcing their placement on the system drive can be impractical or even impossible if you use, say, a USB drive.

    • Offizieller Beitrag

    For someone who doesn't know what this is the output from docker network inspect elk_default on each machine is different, the working one loads;

    elasticsearch

    kibana

    logstash


    the none working one loads

    elasticsearch

    logstash

    kibana


    would that make a difference? but it's just an observation from the output.

    Raid is not a backup! Would you go skydiving without a parachute?


    OMV 6x amd64 running on an HP N54L Microserver

  • geaves, this shouldn't be a problem. The order of items is irelevant since Kibana container repeatedly retries to connect to the Elasticsearch container on each failure (it's not limited to some time window).

    In both listings all the containers were (reportedly) assigned unique IPv4 addresses and expected DNS names. They are visible to each other via IP addresses, but it gets complicated with DNS name visibility (see original post). What makes it peculiar is that the problem goes away when you add rwx permissions for the Docker host folder to "others" while there's nothing in the logs and inspect outputs that would suggest that.

Jetzt mitmachen!

Sie haben noch kein Benutzerkonto auf unserer Seite? Registrieren Sie sich kostenlos und nehmen Sie an unserer Community teil!