Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature: full backups #2710

Merged
merged 30 commits into from
Nov 8, 2018
Merged

Feature: full backups #2710

merged 30 commits into from
Nov 8, 2018

Commits on Oct 15, 2018

  1. saving state

    srfrog committed Oct 15, 2018
    Configuration menu
    Copy the full SHA
    4c52c27 View commit details
    Browse the repository at this point in the history

Commits on Oct 16, 2018

  1. Merge branch 'master' of github.com:dgraph-io/dgraph into feature/roa… 

    …dmap-backups
    srfrog committed Oct 16, 2018
    Configuration menu
    Copy the full SHA
    cd769f5 View commit details
    Browse the repository at this point in the history

  2. saving changes

    srfrog committed Oct 16, 2018
    Configuration menu
    Copy the full SHA
    0de29ea View commit details
    Browse the repository at this point in the history

Commits on Oct 17, 2018

  1. saving state

    srfrog committed Oct 17, 2018
    Configuration menu
    Copy the full SHA
    396c38a View commit details
    Browse the repository at this point in the history

  2. added backup dir

    srfrog committed Oct 17, 2018
    Configuration menu
    Copy the full SHA
    7eb2172 View commit details
    Browse the repository at this point in the history

Commits on Oct 24, 2018

  1. Merge branch 'master' of github.com:/dgraph-io/dgraph into feature/ro… 

    …admap-backups
    srfrog committed Oct 24, 2018
    Configuration menu
    Copy the full SHA
    f6b8b10 View commit details
    Browse the repository at this point in the history

Commits on Oct 25, 2018

  1. renamed dgraphee tree to ee. 

    moved orchestrate to its own pkg within worker.
adding backup to worker and alpha.
    srfrog committed Oct 25, 2018
    Configuration menu
    Copy the full SHA
    cd1b88d View commit details
    Browse the repository at this point in the history

Commits on Oct 30, 2018

  1. trying to get the handler and file writers working.

    srfrog committed Oct 30, 2018
    Configuration menu
    Copy the full SHA
    c4701a4 View commit details
    Browse the repository at this point in the history

  2. added destination parameter. handler support to destination URI schem… 

    …e. additional logging.
    srfrog committed Oct 30, 2018
    Configuration menu
    Copy the full SHA
    b5cfb77 View commit details
    Browse the repository at this point in the history

  3. file handler rename on same volume. added more comments and logging.

    srfrog committed Oct 30, 2018
    Configuration menu
    Copy the full SHA
    905f1b4 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'master' of github.com:/dgraph-io/dgraph into feature/ro… 

    …admap-backups
    srfrog committed Oct 30, 2018
    Configuration menu
    Copy the full SHA
    d62e439 View commit details
    Browse the repository at this point in the history

Commits on Oct 31, 2018

  1. changed worker to use stream pkg. updated protos for backup. fixed mi… 

    …slabelled request field.
    srfrog committed Oct 31, 2018
    Configuration menu
    Copy the full SHA
    8ac55fb View commit details
    Browse the repository at this point in the history

  2. logging changes for debugging

    srfrog committed Oct 31, 2018
    Configuration menu
    Copy the full SHA
    c8d9054 View commit details
    Browse the repository at this point in the history

  3. added some error checks, tweaked comments.

    srfrog committed Oct 31, 2018
    Configuration menu
    Copy the full SHA
    6ea4688 View commit details
    Browse the repository at this point in the history

Commits on Nov 1, 2018

  1. moved stream pkg out of worker. 

    removed binary encoding package, using encode/binary with size delimiter.
fixed race and added posting list values to backup.
fixed issue with file handler that was breaking badger.
refactored backup process to be simpler.
added generic Status proto for any service response.
added minio and used it for S3 backup uploads.
    srfrog committed Nov 1, 2018
    Configuration menu
    Copy the full SHA
    00a1cd0 View commit details
    Browse the repository at this point in the history

  2. removed unused const. format fixes.

    srfrog committed Nov 1, 2018
    Configuration menu
    Copy the full SHA
    2a317fb View commit details
    Browse the repository at this point in the history

Commits on Nov 3, 2018

  1. saving state

    srfrog committed Nov 3, 2018
    Configuration menu
    Copy the full SHA
    2101284 View commit details
    Browse the repository at this point in the history

  2. Initial pass at simplifying things.

    @manishrjain
    manishrjain committed Nov 3, 2018
    Configuration menu
    Copy the full SHA
    255f8f4 View commit details
    Browse the repository at this point in the history

Commits on Nov 6, 2018

  1. cleaned up redundant code. 

    renamed s3handler.send to s3handler.upload and removed all buffering.
s3handler tests that bucket exists before working, we cant assume a region.
s3handler.Close blocks until the upload is complete.
    srfrog committed Nov 6, 2018
    Configuration menu
    Copy the full SHA
    90a394a View commit details
    Browse the repository at this point in the history

  2. unused const

    srfrog committed Nov 6, 2018
    Configuration menu
    Copy the full SHA
    1963b7c View commit details
    Browse the repository at this point in the history

  3. missing space

    srfrog committed Nov 6, 2018
    Configuration menu
    Copy the full SHA
    b38ea24 View commit details
    Browse the repository at this point in the history

  4. added progress monitoring. fixed issues found by CI

    srfrog committed Nov 6, 2018
    Configuration menu
    Copy the full SHA
    03ae950 View commit details
    Browse the repository at this point in the history

Commits on Nov 7, 2018

  1. Small fixes here and there.

    @manishrjain
    manishrjain committed Nov 7, 2018
    Configuration menu
    Copy the full SHA
    3606eaa View commit details
    Browse the repository at this point in the history

  2. Rename handler files.

    @manishrjain
    manishrjain committed Nov 7, 2018
    Configuration menu
    Copy the full SHA
    7688e76 View commit details
    Browse the repository at this point in the history

Commits on Nov 8, 2018

  1. Both S3 uploads and file writes are tested to work.

    @manishrjain
    manishrjain committed Nov 8, 2018
    Configuration menu
    Copy the full SHA
    a2af931 View commit details
    Browse the repository at this point in the history

  2. renamed writer.cleapup to writer.close

    srfrog committed Nov 8, 2018
    Configuration menu
    Copy the full SHA
    f8dc111 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'master' of github.com:/dgraph-io/dgraph into feature/ro… 

    …admap-backups
    srfrog committed Nov 8, 2018
    Configuration menu
    Copy the full SHA
    24c9258 View commit details
    Browse the repository at this point in the history

  4. regenerated protos

    srfrog committed Nov 8, 2018
    Configuration menu
    Copy the full SHA
    4116b46 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'feature/roadmap-backups' of github.com:/dgraph-io/dgrap… 

    …h into feature/roadmap-backups
    srfrog committed Nov 8, 2018
    Configuration menu
    Copy the full SHA
    1beefbf View commit details
    Browse the repository at this point in the history

  6. removed unneeded fallthrough

    srfrog committed Nov 8, 2018
    Configuration menu
    Copy the full SHA
    7da0cae View commit details
    Browse the repository at this point in the history