V0.6.0 #165

oguzhanunlu · 2018-05-30T15:07:56Z

release process

blog post (under review)
discourse post
rc & qa (note to self: never trust in local dev env!)
doc update (prep a branch to be merged)
Prepared for release

oguzhanunlu · 2018-05-30T15:30:55Z

Hey @BenFradet , just added as reviewer, I'll ping you once I'm about to finish implementation phase.

oguzhanunlu · 2018-06-28T01:48:43Z

Hey @BenFradet , I'm done with the implementation phase and would like you to review my code changes.

I merged 4 tickets into #23 , see #23's page for included issues.

Also, only blocker for QA is that there is an issue with Caddy configuration (every endpoint except /kibana works as expected, probably due to kibana version bump, looking into that). I hope this is not a blocker for code review.

In the meantime, I'll solve the issue with /kibana endpoint, write a QA journal, prepare blog post first draft and a separate wiki branch to be merged.

BenFradet

did a first pass 👍

BenFradet · 2018-06-28T09:43:47Z

provisioning/resources/configs/Caddyfile

@@ -5,7 +5,6 @@
    /kibana
    /elasticsearch
    /control-plane
-    /_plugin


did the es plugin path change?

Elasticsearch has removed site plugins as of 5.x, but head plugin can be used as a browser extension which works without a problem.

BenFradet · 2018-06-28T09:58:55Z

provisioning/resources/control-plane/main.go

+			return
+		}
+
+		file, handler, err := req.FormFile("igluserverjson")


why this filename, isn't it supposed to be hocon?

Right, I'll correct.

BenFradet · 2018-06-28T09:59:33Z

provisioning/resources/control-plane/main.go

+		fileContentBytes, err := ioutil.ReadAll(file)
+		fileContent := string(fileContentBytes)
+
+		f, err := os.OpenFile(config.Dirs.Config+"/"+handler.Filename, os.O_WRONLY|os.O_CREATE, 0666)


what if there is a / at the end of config.Dirs.Config?

No issues, I tested and works without a problem.

BenFradet · 2018-06-28T10:01:10Z

provisioning/resources/control-plane/main.go

+		defer f.Close()
+
+		f.Truncate(0)
+		f.Seek(0, 0)


can we add a comment here on what this is doing

BenFradet · 2018-06-28T10:03:55Z

provisioning/resources/elasticsearch/config/log4j2.properties

+appender.rolling.strategy.action.type = Delete
+appender.rolling.strategy.action.basepath = ${sys:es.logs.base_path}
+appender.rolling.strategy.action.condition.type = IfLastModified
+appender.rolling.strategy.action.condition.age = 7D


isn't that too long?

this wasn't acted on

How many days of log should be kept @BenFradet ? Is a week unnecessarily long here?

3 days seems long enough to me, what do you reckon @jbeemster ?

BenFradet · 2018-06-28T10:09:57Z

provisioning/roles/docker/files/docker-compose.yml

+    container_name: elasticsearch
+    restart: always
+    environment:
+      - "bootstrap.memory_lock=true"


what does this do? can we add a comment

It is explained here. I'll add a short comment.

BenFradet · 2018-06-28T10:10:48Z

provisioning/roles/docker/files/docker-compose.yml

+    restart: always
+    environment:
+      - "bootstrap.memory_lock=true"
+      - "ES_JAVA_OPTS=-Xms4g -Xmx4g"


should we be starting with 4g? cc @jbeemster

what's the reasoning behind those settings?

we need to take the size of the machine into account now c.f. #173, Xms and Xmx should indeed be equal, c.f. https://www.elastic.co/guide/en/elasticsearch/reference/6.2/heap-size.html

Let's decide and I'll act accordingly regarding heap size. From the page @BenFradet referenced, it is indicated as following;

Set Xmx to no more than 50% of your physical RAM, to ensure that there is enough physical RAM left for kernel file system caches

My only reasoning is to make ES able to stay up under high load, as much as possible. I remember us talking about publishing to t2.large as of this release and it has 8GB ram, hence 4GB allocation to ES.

Should it be less or is it fine, what do you think? @BenFradet @jbeemster

I think 50% is good so that would be:

type ES loader enrich collector

t2.large 4G 0.5G 0.5G 0.5G

t2.xlarge 8G 1.5G 1.5G 1.5G

t2.2xlarge 16G 3G 3G 3G

BenFradet · 2018-06-28T10:12:42Z

provisioning/roles/docker/files/docker-compose.yml

+        max-size: "1M"
+        max-file: "10"
+    environment:
+      - "SP_JAVA_OPTS=-Xms512m -Xmx512m"


I wouldn't specify Xms for any of them

why was this removed altogether, I would still specify Xmx

Specified only Xmx now.

BenFradet · 2018-06-28T10:14:43Z

provisioning/roles/docker/tasks/main.yml

+
+- name: Ensure file permissions
+  become: yes
+  shell: chown ubuntu:ubuntu -R /home/ubuntu/snowplow && chmod 755 -R /home/ubuntu/snowplow


why is this necessary, can we add a comment?

I forgot to delete it, I used that for test purposes. 4th role already does that.

BenFradet · 2018-06-28T10:16:12Z

provisioning/roles/sp_mini_7_setup_init/tasks/main.yml

+
+- name: Increase mmap count to recommended 262144 for Elasticsearch
+  become: yes
+  shell: echo "vm.max_map_count=262144" >> /etc/sysctl.conf && service procps start


this isn't in the container why would it affect es?

I think it is because containers share the same kernel as host OS.

BenFradet · 2018-07-03T14:41:32Z

provisioning/roles/docker/tasks/main.yml

+# rebooting kibana and retrying works out
+- name: Likely to fail attempt to create ES indexes & Kibana index patterns & NSQ topics
+  become: yes
+  shell: sh {{init_dir}}/create.sh


what's the reasoning for this

Kibana 5.x had problems with creating index patterns (they still don't have a stable API documentation, unfortunately) and I guaranteed index pattern creation using a dirty way. Anyway, 6.x doesn't have aforementioned issue and I removed this part.

BenFradet · 2018-07-03T14:53:20Z

provisioning/roles/sp_mini_7_setup_init/tasks/main.yml

+    special_time: weekly
+    job: /usr/bin/curl -s -X POST http://localhost:9200/good/_delete_by_query -d '{ "query" :{ "range" :{ "collector_tstamp" :{ "lt" :"now-1w/d" } } } }' > /dev/null 2>&1
+
+- cron:


can we have something more specific as far as names are concerned instead of 3 times cron?

chuwy · 2018-07-04T09:33:52Z

Am I right that we did not bump Iglu Server to 0.3.0 in this (or previous) release? If we didn't - why? It looked like a quite serious upgrade. If we did - it probably does not work as expected due snowplow/iglu#378

Edit: seems we did. This must be QA'ed with public schemas.

oguzhanunlu · 2018-08-06T16:51:14Z

Hey @chuwy , @BenFradet has been through his QA and he thinks it is good to go. Will you also QA or should we release tomorrow?

chuwy · 2018-08-07T05:09:41Z

Good to go from me.

oguzhanunlu added this to the Version 0.6.0 milestone May 30, 2018

oguzhanunlu self-assigned this May 30, 2018

oguzhanunlu requested a review from BenFradet May 30, 2018 15:23

oguzhanunlu force-pushed the v0.6.0 branch 2 times, most recently from 9bd8639 to 405eb43 Compare June 18, 2018 23:54

oguzhanunlu force-pushed the v0.6.0 branch 12 times, most recently from 5d43590 to 7d75caf Compare June 28, 2018 01:30

BenFradet reviewed Jun 28, 2018

View reviewed changes

oguzhanunlu force-pushed the v0.6.0 branch 5 times, most recently from 8a9f14d to 7d6fba6 Compare June 29, 2018 19:25

BenFradet reviewed Jul 3, 2018

View reviewed changes

oguzhanunlu force-pushed the v0.6.0 branch from bb47350 to 22b46c5 Compare July 11, 2018 12:22

oguzhanunlu force-pushed the v0.6.0 branch 9 times, most recently from 6d47589 to 759799e Compare July 20, 2018 09:32

oguzhanunlu force-pushed the v0.6.0 branch 7 times, most recently from 5931a8c to 0f2f3a5 Compare July 30, 2018 17:51

oguzhanunlu force-pushed the v0.6.0 branch 4 times, most recently from 352183f to 247992e Compare August 6, 2018 16:21

oguzhanunlu closed this Aug 7, 2018

oguzhanunlu reopened this Aug 7, 2018

oguzhanunlu closed this Aug 7, 2018

oguzhanunlu force-pushed the v0.6.0 branch from 247992e to 33a95c9 Compare August 7, 2018 08:56

oguzhanunlu deleted the v0.6.0 branch August 7, 2018 08:56

oguzhanunlu restored the v0.6.0 branch June 14, 2019 16:07

oguzhanunlu deleted the v0.6.0 branch June 14, 2019 16:12

type	ES	loader	enrich	collector
t2.large	4G	0.5G	0.5G	0.5G
t2.xlarge	8G	1.5G	1.5G	1.5G
t2.2xlarge	16G	3G	3G	3G

V0.6.0 #165

V0.6.0 #165

Conversation

oguzhanunlu commented May 30, 2018 • edited Loading

release process

oguzhanunlu commented May 30, 2018

oguzhanunlu commented Jun 28, 2018 • edited Loading

BenFradet left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chuwy commented Jul 4, 2018 • edited Loading

oguzhanunlu commented Aug 6, 2018

chuwy commented Aug 7, 2018

oguzhanunlu commented May 30, 2018 •

edited

Loading

oguzhanunlu commented Jun 28, 2018 •

edited

Loading

chuwy commented Jul 4, 2018 •

edited

Loading