Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dataflow runner options: disk type & streaming engine #906

Merged
merged 2 commits into from
Aug 1, 2020

Conversation

pyalex
Copy link
Collaborator

@pyalex pyalex commented Jul 29, 2020

What this PR does / why we need it:

This PR extends execution parameteres for Dataflow Job according to official documentation.

Next parameter were added:

  • workerDiskType
  • enableStreamingEngine

Next parameter were updated:

  • zone -> workerZone (due to deprecation)

Which issue(s) this PR fixes:

Fixes #

Does this PR introduce a user-facing change?:


new options introduced in application.yaml

feast:
  jobs:
    runners:
    - name: dataflow
      options:
        enableStreamingEngine: false
        workerDiskType: compute.googleapis.com/projects/asia-east1-a/diskTypes/pd-ssd
        workerZone: asia-east1-a  # renamed "zone" due to deprecation

@pyalex pyalex added the kind/feature New feature or request label Aug 1, 2020
@pyalex
Copy link
Collaborator Author

pyalex commented Aug 1, 2020

/retest

@pyalex
Copy link
Collaborator Author

pyalex commented Aug 1, 2020

/test test-end-to-end-batch-dataflow

@pyalex
Copy link
Collaborator Author

pyalex commented Aug 1, 2020

/retest

@woop
Copy link
Member

woop commented Aug 1, 2020

/lgtm

@feast-ci-bot
Copy link
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: pyalex, woop

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@pyalex
Copy link
Collaborator Author

pyalex commented Aug 1, 2020

/test test-end-to-end-redis-cluster

@feast-ci-bot feast-ci-bot merged commit 294faa5 into feast-dev:master Aug 1, 2020
@mrzzy mrzzy mentioned this pull request Aug 2, 2020
pyalex pushed a commit that referenced this pull request Aug 2, 2020
* diskType & streamingEnginer

* edit infra docs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants