Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[#84] Fix pinpoint-hbase connection configurations #85

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

[#84] Fix pinpoint-hbase connection configurations #85

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions pinpoint-hbase/Dockerfile_standalone
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
FROM java:8-jdk

LABEL maintainer="Roy Kim <[email protected]>"

ARG PINPOINT_VERSION=${PINPOINT_VERSION:-2.1.0}

ENV HBASE_REPOSITORY=http://apache.mirrors.pair.com/hbase
ENV HBASE_SUB_REPOSITORY=http://archive.apache.org/dist/hbase

ENV HBASE_VERSION=1.2.6
ENV BASE_DIR=/opt/hbase
ENV HBASE_HOME=${BASE_DIR}/hbase-${HBASE_VERSION}


COPY hbase-site-standalone.xml hbase-site.xml
COPY hbase-env.sh hbase-env.sh
COPY /build/scripts/initialize-hbase.sh /usr/local/bin/
COPY /build/scripts/check-table.sh /usr/local/bin/

RUN chmod a+x /usr/local/bin/initialize-hbase.sh \
&& chmod a+x /usr/local/bin/check-table.sh \
&& mkdir -p ${BASE_DIR} \
&& cd ${BASE_DIR} \
&& curl -fSL "${HBASE_REPOSITORY}/${HBASE_VERSION}/hbase-${HBASE_VERSION}-bin.tar.gz" -o hbase.tar.gz || curl -fSL "${HBASE_SUB_REPOSITORY}/${HBASE_VERSION}/hbase-${HBASE_VERSION}-bin.tar.gz" -o hbase.tar.gz \
&& tar xfvz hbase.tar.gz \
&& mv ../../hbase-site.xml ../../${HBASE_HOME}/conf/hbase-site.xml \
&& mv ../../hbase-env.sh ../../${HBASE_HOME}/conf/hbase-env.sh \
&& curl -SL "https://raw.githubusercontent.com/naver/pinpoint/v${PINPOINT_VERSION}/hbase/scripts/hbase-create.hbase" -o ${BASE_DIR}/hbase-create.hbase

VOLUME ["/home/pinpoint/hbase", "/home/pinpoint/zookeeper"]

CMD /usr/local/bin/initialize-hbase.sh && tail -f $HBASE_HOME/logs/*
18 changes: 13 additions & 5 deletions pinpoint-hbase/docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ services:
pinpoint-hbase:
build:
context: .
dockerfile: Dockerfile
dockerfile: Dockerfile_standalone
args:
- PINPOINT_VERSION=${PINPOINT_VERSION}

Expand All @@ -15,20 +15,28 @@ services:
- /home/pinpoint/hbase
- /home/pinpoint/zookeeper
expose:
# # zookeeper
# - "2181"
# HBase Master API port
- "60000"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how about also changing 60000 to 16000 and all the code below that is related (https://hbase.apache.org/book.html#hbase_default_configurations)

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, I think that is more related to the others.

# HBase Master Web UI
- "16010"
# Regionserver API port
- "60020"
- "16201"
# HBase Regionserver web UI
- "16030"
# Regionserver web UI port
- "16301"
ports:
- "${EXTERNAL_HBASE_PORT:-2181}:2181"
- "60000:60000"
- "16010:16010"
- "60020:60020"
- "16030:16030"
- "16301:16301"
- "16201:16201"
restart: always
networks:
- pinpoint

networks:
pinpoint:
external:
name: pinpoint-zookeeper_pinpoint
42 changes: 42 additions & 0 deletions pinpoint-hbase/hbase-site-standalone.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
<configuration>
<property>
<name>hbase.rootdir</name>
<value>file:///home/pinpoint/hbase</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
Comment on lines +6 to +9
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if it's stand-alone mode. this should be removed

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is necessary, otherwise region server ipc port will be randomized whenever the hbase container runs.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why does region server ipc port not be randomized?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The pinpoint collector should connect to hbase region server ipc.
I found a similar problem on stackoverflow.

https://stackoverflow.com/questions/43524733/how-to-specify-rpc-server-port-for-hbase

<property>
<name>zookeeper.znode.parent</name>
<value>/hbase</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>zoo1,zoo2,zoo3</value>
</property>
Comment on lines +14 to +17
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if it's stand-alone mode. Is this even necessary? At least, zoo1 is not the zookeeper hbase needs to connect. It should be the internal one.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be obsolete, I just want to use the same yaml file in pinpoint-zookeeper directory.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you are misunderstading the architecture of stand-alone hbase.

<property>
<name>hbase.zookeeper.property.clientPort</name>
<value>2181</value>
</property>
<property>
<name>hbase.master.port</name>
<value>60000</value>
</property>
<property>
<name>hbase.regionserver.port</name>
<value>16201</value>
</property>
<property>
<name>hbase.regionserver.hostname</name>
<value>localhost</value>
</property>
<property>
<name>hbase.regionserver.ipc.address</name>
<value>0.0.0.0</value>
</property>
<property>
<name>hbase.master.ipc.address</name>
<value>0.0.0.0</value>
</property>
Comment on lines +30 to +41
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this necessary when the hbase is in stand-alone mode?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is.
In order to pinpoint collector and web module that run in host need to know ipc address to connect hbase in docker container.

I had a wrong choice on the word 'stand-alone'.
It is more related to docker container not the hbase itself.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you are using not referring to stand-alone hbase. What is this docker file for?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is for the one runs pinpoint collector and web module in IDE or host environment, and want to run hbase as container. because running with hbase binary is too troublesome.

Copy link
Contributor

@RoySRose RoySRose Sep 18, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cwJohnPark

I've understood your intentions. But with the below configuration, this will ONLY work IF HBase is on localhost.

<property>
        <name>hbase.regionserver.hostname</name>
        <value>localhost</value>
</property>

Actually there is a solution for below

This is for the one runs pinpoint collector and web module in IDE or host environment
should be taken care of by adding the hbase containers IP address in the host file.

I know adding a line in the host file may be annoying. But since the reason for this is due to the networking architecture of docker containers. I couldn't find any appropriate solution for this yet.
Please let me know what you think about this, and welcome if there is any better solution.

</configuration>