Skyhook cephmods #72

KDahlgren · 2019-09-21T02:43:04Z

Edited copy from code to call transform function on src and append the results to the target object. Transform is currently hard-coded for fbx to arrow.
Src schema is currently hard-coded to lineitem.
Project-cols is currently hard-coded to orderkey.
Getting rid of the hard code will require parameterizing do_copy_get in PrimaryLogPG.cc.

… from do_copy_get which takes the src bl and generates an outbl which is appended to the target bl.

KDahlgren · 2019-09-21T02:45:03Z

Tested build against FBX, ARROW, FBU_Rows, and FBU_Cols quereies in vcluster. Everything works fine.

Here're the tests I used:

 #PREP FOR SKYHOOK-CEPHMODS BRANCH PR

# --------------------------------------------------------------------------- #
# COPY FROM TRANSFORM APPEND TESTS
../src/stop.sh; make -j12 vstart; ../src/stop.sh; ../src/vstart.sh -d -n -x; bin/rados mkpool tpchflatbuf ; bin/ceph osd pool set tpchflatbuf size 1 ; bin/rados mkpool tpchdata ; bin/ceph osd pool set tpchdata size 1 ; bin/rados mkpool paper_exps ; bin/ceph osd pool set paper_exps size 1 ;

bin/transform-copyfrom-merge --pool paper_exps ;

# --------------------------------------------------------------------------- #
# FBX TESTS
../src/stop.sh; make -j12 vstart; ../src/stop.sh; ../src/vstart.sh -d -n -x; bin/rados mkpool tpchflatbuf ; bin/ceph osd pool set tpchflatbuf size 1 ; bin/rados mkpool tpchdata ; bin/ceph osd pool set tpchdata size 1 ; bin/rados mkpool paper_exps ; bin/ceph osd pool set paper_exps size 1 ;

OBJ_TYPE=SFT_FLATBUF_FLEX_ROW;  # choose one of {SFT_FLATBUF_FLEX_ROW, SFT_ARROW, SFT_JSON}
OBJ_BASE_NAME=skyhook.${OBJ_TYPE}.lineitem;
for i in {0..1}; do 
    rm -rf ${OBJ_BASE_NAME}.$i;   # remove the old test objects
    wget https://users.soe.ucsc.edu/~jlefevre/skyhookdb/testdata/${OBJ_BASE_NAME}.$i;
done;
yes | PATH=$PATH:bin ../src/progly/rados-store-glob.sh tpchdata ${OBJ_BASE_NAME}.*;

bin/run-query --num-objs 2 --pool tpchdata --wthreads 1 --qdepth 10 --query flatbuf --table-name "lineitem" --quiet;
bin/run-query --num-objs 2 --pool tpchdata --wthreads 1 --qdepth 10 --query flatbuf --table-name "lineitem"  --select-preds "orderkey,lt,3"  --project-cols "orderkey,tax,comment,linenumber,returnflag" --quiet;
bin/run-query --num-objs 2 --pool tpchdata --wthreads 1 --qdepth 10 --query flatbuf --table-name "lineitem"  --select-preds "orderkey,geq,3"  --project-cols "orderkey,tax,comment,linenumber,returnflag" --quiet ;

bin/run-query --num-objs 2 --pool tpchdata --wthreads 1 --qdepth 10 --query flatbuf --table-name "lineitem" --quiet --use-cls;
bin/run-query --num-objs 2 --pool tpchdata --wthreads 1 --qdepth 10 --query flatbuf --table-name "lineitem"  --select-preds "orderkey,lt,3"  --project-cols "orderkey,tax,comment,linenumber,returnflag" --quiet --use-cls;
bin/run-query --num-objs 2 --pool tpchdata --wthreads 1 --qdepth 10 --query flatbuf --table-name "lineitem"  --select-preds "orderkey,geq,3"  --project-cols "orderkey,tax,comment,linenumber,returnflag" --quiet --use-cls;
 
# --------------------------------------------------------------------------- #
# ARROW TESTS ../src/stop.sh; make -j12 vstart; ../src/stop.sh; ../src/vstart.sh -d -n -x; bin/rados mkpool tpchflatbuf ; bin/ceph osd pool set tpchflatbuf size 1 ; bin/rados mkpool tpchdata ; bin/ceph osd pool set tpchdata size 1 ; bin/rados mkpool paper_exps ; bin/ceph osd pool set paper_exps size 1 ;

OBJ_TYPE=SFT_ARROW
OBJ_BASE_NAME=skyhook.${OBJ_TYPE}.lineitem;
for i in {0..1}; do 
    rm -rf ${OBJ_BASE_NAME}.$i;   # remove the old test objects
    wget https://users.soe.ucsc.edu/~jlefevre/skyhookdb/testdata/${OBJ_BASE_NAME}.$i;
done;
yes | PATH=$PATH:bin ../src/progly/rados-store-glob.sh tpchdata ${OBJ_BASE_NAME}.*;

bin/run-query --num-objs 2 --pool tpchdata --wthreads 1 --qdepth 10 --query flatbuf --table-name "lineitem" --quiet;
bin/run-query --num-objs 2 --pool tpchdata --wthreads 1 --qdepth 10 --query flatbuf --table-name "lineitem"  --select-preds "orderkey,lt,3"  --project-cols "orderkey,tax,comment,linenumber,returnflag" --quiet;
bin/run-query --num-objs 2 --pool tpchdata --wthreads 1 --qdepth 10 --query flatbuf --table-name "lineitem"  --select-preds "orderkey,geq,3"  --project-cols "orderkey,tax,comment,linenumber,returnflag" --quiet ;

bin/run-query --num-objs 2 --pool tpchdata --wthreads 1 --qdepth 10 --query flatbuf --table-name "lineitem" --quiet --use-cls;
bin/run-query --num-objs 2 --pool tpchdata --wthreads 1 --qdepth 10 --query flatbuf --table-name "lineitem"  --select-preds "orderkey,lt,3"  --project-cols "orderkey,tax,comment,linenumber,returnflag" --quiet --use-cls;
bin/run-query --num-objs 2 --pool tpchdata --wthreads 1 --qdepth 10 --query flatbuf --table-name "lineitem"  --select-preds "orderkey,geq,3"  --project-cols "orderkey,tax,comment,linenumber,returnflag" --quiet --use-cls;

# --------------------------------------------------------------------------- #
# FBU_ROWS TESTS
../src/stop.sh; make -j12 vstart; ../src/stop.sh; ../src/vstart.sh -d -n -x; bin/rados mkpool tpchflatbuf ; bin/ceph osd pool set tpchflatbuf size 1 ;

bin/fbwriter_fbu --filename testdata.txt --write_type rows --debug yes --schema_datatypes int,float,string,int --schema_attnames att0,att1,att2,att3 --table_name atable --nrows 6 --ncols 4 --targetoid obj --targetpool tpchflatbuf --targetformat SFT_FLATBUF_UNION_ROW --writeto ceph --schema_iskey 0,0,0,0 --schema_isnullable 0,0,0,0 --savedir . --rid_start_value 2 --rid_end_value 4 --obj_counter 0 ;

bin/run-query --num-objs 1 --pool tpchflatbuf --wthreads 1 --qdepth 10 --query flatbuf --select "*" ;

bin/run-query --num-objs 1 --pool tpchflatbuf --wthreads 1 --qdepth 10 --query flatbuf --select "att0,lt,25;att1,lt,25.0;"  --project-cols att0,att1,att2,att3 --data-schema "0 7 0 0 ATT0 ; 1 12 0 0 ATT1 ; 2 15 0 0 ATT2 ; 3 7 0 0 ATT3" ;

bin/run-query --num-objs 1 --pool tpchflatbuf --wthreads 1 --qdepth 10 --query flatbuf --select "att0,lt,25;att1,lt,25.0;"  --project-cols att0,att2 --data-schema "0 7 0 0 ATT0 ; 1 12 0 0 ATT1 ; 2 15 0 0 ATT2 ;3 7 0 0 ATT3" ;

bin/run-query --num-objs 1 --pool tpchflatbuf --wthreads 1 --qdepth 10 --query flatbuf --select "att0,lt,40;att3,lt,215;"  --project-cols att0,att1,att2,att3 --data-schema "0 7 0 0 ATT0 ; 1 12 0 0 ATT1 ; 2 15 0 0 ATT2 ; 3 7 0 0 ATT3" ;

bin/run-query --num-objs 1 --pool tpchflatbuf --wthreads 1 --qdepth 10 --query flatbuf --select-preds ";att0,sum,0;" --table-name "atable" --data-schema "0 7 0 0 ATT0 ; 1 12 0 0 ATT1 ; 2 15 0 0 ATT2 ;3 7 0 0 ATT3" ;

# ++++ use-cls ++++ 
bin/run-query --num-objs 1 --pool tpchflatbuf --wthreads 1 --qdepth 10 --query flatbuf --select "*" ;

bin/run-query --num-objs 1 --pool tpchflatbuf --wthreads 1 --qdepth 10 --query flatbuf --select "att0,lt,25;att1,lt,25.0;"  --project-cols att0,att1,att2,att3 --data-schema "0 7 0 0 ATT0 ; 1 12 0 0 ATT1 ; 2 15 0 0 ATT2 ; 3 7 0 0 ATT3" --use-cls ;

bin/run-query --num-objs 1 --pool tpchflatbuf --wthreads 1 --qdepth 10 --query flatbuf --select "att0,lt,25;att1,lt,25.0;"  --project-cols att0,att2 --data-schema "0 7 0 0 ATT0 ; 1 12 0 0 ATT1 ; 2 15 0 0 ATT2 ;3 7 0 0 ATT3" --use-cls ;

bin/run-query --num-objs 1 --pool tpchflatbuf --wthreads 1 --qdepth 10 --query flatbuf --select "att0,lt,40;att3,lt,215;"  --project-cols att0,att1,att2,att3 --data-schema "0 7 0 0 ATT0 ; 1 12 0 0 ATT1 ; 2 15 0 0 ATT2 ; 3 7 0 0 ATT3" --use-cls ;

bin/run-query --num-objs 1 --pool tpchflatbuf --wthreads 1 --qdepth 10 --query flatbuf --select-preds ";att0,sum,0;" --table-name "atable" --data-schema "0 7 0 0 ATT0 ; 1 12 0 0 ATT1 ; 2 15 0 0 ATT2 ;3 7 0 0 ATT3" --use-cls ;

# --------------------------------------------------------------------------- #
# FBU_COLS 1 TESTS
../src/stop.sh; make -j12 vstart; ../src/stop.sh; ../src/vstart.sh -d -n -x; bin/rados mkpool tpchflatbuf ; bin/ceph osd pool set tpchflatbuf size 1 ;

bin/fbwriter_fbu --filename testdata.txt --write_type cols --debug yes --schema_datatypes int,float,string,int --schema_attnames att0,att1,att2,att3 --table_name atable --nrows 6 --ncols 4 --targetoid obj --targetpool tpchflatbuf --targetformat SFT_FLATBUF_UNION_COL --writeto ceph --cols_per_fb 1 --schema_iskey 0,0,0,0 --schema_isnullable 0,0,0,0 --savedir . --rid_start_value 2 --rid_end_value 4 --obj_counter 0 ;

bin/run-query --num-objs 1 --pool tpchflatbuf --wthreads 1 --qdepth 10 --query flatbuf --select "*" ;

bin/run-query --num-objs 1 --pool tpchflatbuf --wthreads 1 --qdepth 10 --query flatbuf --select "att0,lt,25;att1,lt,25.0;"  --project-cols att0,att1,att2,att3 --data-schema "0 7 0 0 ATT0 ; 1 12 0 0 ATT1 ; 2 15 0 0 ATT2 ; 3 7 0 0 ATT3" ;

bin/run-query --num-objs 1 --pool tpchflatbuf --wthreads 1 --qdepth 10 --query flatbuf --select "att0,lt,25;att1,lt,25.0;"  --project-cols att0,att2 --data-schema "0 7 0 0 ATT0 ; 1 12 0 0 ATT1 ; 2 15 0 0 ATT2 ;3 7 0 0 ATT3" ;

bin/run-query --num-objs 1 --pool tpchflatbuf --wthreads 1 --qdepth 10 --query flatbuf --select "att0,lt,40;att3,lt,215;"  --project-cols att0,att1,att2,att3 --data-schema "0 7 0 0 ATT0 ; 1 12 0 0 ATT1 ; 2 15 0 0 ATT2 ; 3 7 0 0 ATT3" ;

bin/run-query --num-objs 1 --pool tpchflatbuf --wthreads 1 --qdepth 10 --query flatbuf --select-preds ";att0,sum,0;" --table-name "atable" --data-schema "0 7 0 0 ATT0 ; 1 12 0 0 ATT1 ; 2 15 0 0 ATT2 ;3 7 0 0 ATT3" ;

# ++++ use-cls ++++ 
bin/run-query --num-objs 1 --pool tpchflatbuf --wthreads 1 --qdepth 10 --query flatbuf --select "*" ;

bin/run-query --num-objs 1 --pool tpchflatbuf --wthreads 1 --qdepth 10 --query flatbuf --select "att0,lt,25;att1,lt,25.0;"  --project-cols att0,att1,att2,att3 --data-schema "0 7 0 0 ATT0 ; 1 12 0 0 ATT1 ; 2 15 0 0 ATT2 ; 3 7 0 0 ATT3" --use-cls ;

bin/run-query --num-objs 1 --pool tpchflatbuf --wthreads 1 --qdepth 10 --query flatbuf --select "att0,lt,25;att1,lt,25.0;"  --project-cols att0,att2 --data-schema "0 7 0 0 ATT0 ; 1 12 0 0 ATT1 ; 2 15 0 0 ATT2 ;3 7 0 0 ATT3" --use-cls ;

bin/run-query --num-objs 1 --pool tpchflatbuf --wthreads 1 --qdepth 10 --query flatbuf --select "att0,lt,40;att3,lt,215;"  --project-cols att0,att1,att2,att3 --data-schema "0 7 0 0 ATT0 ; 1 12 0 0 ATT1 ; 2 15 0 0 ATT2 ; 3 7 0 0 ATT3" --use-cls ;

bin/run-query --num-objs 1 --pool tpchflatbuf --wthreads 1 --qdepth 10 --query flatbuf --select-preds ";att0,sum,0;" --table-name "atable" --data-schema "0 7 0 0 ATT0 ; 1 12 0 0 ATT1 ; 2 15 0 0 ATT2 ;3 7 0 0 ATT3" --use-cls ;

# --------------------------------------------------------------------------- #
# FBU_COLS 2 TESTS
../src/stop.sh; make -j12 vstart; ../src/stop.sh; ../src/vstart.sh -d -n -x; bin/rados mkpool tpchflatbuf ; bin/ceph osd pool set tpchflatbuf size 1 ;

bin/fbwriter_fbu --filename testdata.txt --write_type cols --debug yes --schema_datatypes int,float,string,int --schema_attnames att0,att1,att2,att3 --table_name atable --nrows 6 --ncols 4 --targetoid obj --targetpool tpchflatbuf --targetformat SFT_FLATBUF_UNION_COL --writeto ceph --cols_per_fb 2 --schema_iskey 0,0,0,0 --schema_isnullable 0,0,0,0 --savedir . --rid_start_value 2 --rid_end_value 4 --obj_counter 0 ;

bin/run-query --num-objs 1 --pool tpchflatbuf --wthreads 1 --qdepth 10 --query flatbuf --select "*" ;

bin/run-query --num-objs 1 --pool tpchflatbuf --wthreads 1 --qdepth 10 --query flatbuf --select "att0,lt,25;att1,lt,25.0;"  --project-cols att0,att1,att2,att3 --data-schema "0 7 0 0 ATT0 ; 1 12 0 0 ATT1 ; 2 15 0 0 ATT2 ; 3 7 0 0 ATT3" ;

bin/run-query --num-objs 1 --pool tpchflatbuf --wthreads 1 --qdepth 10 --query flatbuf --select "att0,lt,25;att1,lt,25.0;"  --project-cols att0,att2 --data-schema "0 7 0 0 ATT0 ; 1 12 0 0 ATT1 ; 2 15 0 0 ATT2 ;3 7 0 0 ATT3" ;

bin/run-query --num-objs 1 --pool tpchflatbuf --wthreads 1 --qdepth 10 --query flatbuf --select "att0,lt,40;att3,lt,215;"  --project-cols att0,att1,att2,att3 --data-schema "0 7 0 0 ATT0 ; 1 12 0 0 ATT1 ; 2 15 0 0 ATT2 ; 3 7 0 0 ATT3" ;

bin/run-query --num-objs 1 --pool tpchflatbuf --wthreads 1 --qdepth 10 --query flatbuf --select-preds ";att0,sum,0;" --table-name "atable" --data-schema "0 7 0 0 ATT0 ; 1 12 0 0 ATT1 ; 2 15 0 0 ATT2 ;3 7 0 0 ATT3" ;

# ++++ use-cls ++++ 
bin/run-query --num-objs 1 --pool tpchflatbuf --wthreads 1 --qdepth 10 --query flatbuf --select "*" ;

bin/run-query --num-objs 1 --pool tpchflatbuf --wthreads 1 --qdepth 10 --query flatbuf --select "att0,lt,25;att1,lt,25.0;"  --project-cols att0,att1,att2,att3 --data-schema "0 7 0 0 ATT0 ; 1 12 0 0 ATT1 ; 2 15 0 0 ATT2 ; 3 7 0 0 ATT3" --use-cls ;

bin/run-query --num-objs 1 --pool tpchflatbuf --wthreads 1 --qdepth 10 --query flatbuf --select "att0,lt,25;att1,lt,25.0;"  --project-cols att0,att2 --data-schema "0 7 0 0 ATT0 ; 1 12 0 0 ATT1 ; 2 15 0 0 ATT2 ;3 7 0 0 ATT3" --use-cls ;

bin/run-query --num-objs 1 --pool tpchflatbuf --wthreads 1 --qdepth 10 --query flatbuf --select "att0,lt,40;att3,lt,215;"  --project-cols att0,att1,att2,att3 --data-schema "0 7 0 0 ATT0 ; 1 12 0 0 ATT1 ; 2 15 0 0 ATT2 ; 3 7 0 0 ATT3" --use-cls ;

bin/run-query --num-objs 1 --pool tpchflatbuf --wthreads 1 --qdepth 10 --query flatbuf --select-preds ";att0,sum,0;" --table-name "atable" --data-schema "0 7 0 0 ATT0 ; 1 12 0 0 ATT1 ; 2 15 0 0 ATT2 ;3 7 0 0 ATT3" --use-cls ;

# --------------------------------------------------------------------------- #
# FBU_COLS 4 TESTS
../src/stop.sh; make -j12 vstart; ../src/stop.sh; ../src/vstart.sh -d -n -x; bin/rados mkpool tpchflatbuf ; bin/ceph osd pool set tpchflatbuf size 1 ;

bin/fbwriter_fbu --filename testdata.txt --write_type cols --debug yes --schema_datatypes int,float,string,int --schema_attnames att0,att1,att2,att3 --table_name atable --nrows 6 --ncols 4 --targetoid obj --targetpool tpchflatbuf --targetformat SFT_FLATBUF_UNION_COL --writeto ceph --cols_per_fb 4 --schema_iskey 0,0,0,0 --schema_isnullable 0,0,0,0 --savedir . --rid_start_value 1 --rid_end_value 6 --obj_counter 0 ;

bin/run-query --num-objs 1 --pool tpchflatbuf --wthreads 1 --qdepth 10 --query flatbuf --select "*" ;

bin/run-query --num-objs 1 --pool tpchflatbuf --wthreads 1 --qdepth 10 --query flatbuf --select "att0,lt,25;att1,lt,25.0;"  --project-cols att0,att1,att2,att3 --data-schema "0 7 0 0 ATT0 ; 1 12 0 0 ATT1 ; 2 15 0 0 ATT2 ; 3 7 0 0 ATT3" ;

bin/run-query --num-objs 1 --pool tpchflatbuf --wthreads 1 --qdepth 10 --query flatbuf --select "att0,lt,25;att1,lt,25.0;"  --project-cols att0,att2 --data-schema "0 7 0 0 ATT0 ; 1 12 0 0 ATT1 ; 2 15 0 0 ATT2 ;3 7 0 0 ATT3" ;

bin/run-query --num-objs 1 --pool tpchflatbuf --wthreads 1 --qdepth 10 --query flatbuf --select "att0,lt,40;att3,lt,215;"  --project-cols att0,att1,att2,att3 --data-schema "0 7 0 0 ATT0 ; 1 12 0 0 ATT1 ; 2 15 0 0 ATT2 ; 3 7 0 0 ATT3" ;

bin/run-query --num-objs 1 --pool tpchflatbuf --wthreads 1 --qdepth 10 --query flatbuf --select-preds ";att0,sum,0;" --table-name "atable" --data-schema "0 7 0 0 ATT0 ; 1 12 0 0 ATT1 ; 2 15 0 0 ATT2 ;3 7 0 0 ATT3" ;

# ++++ use-cls ++++ 
bin/run-query --num-objs 1 --pool tpchflatbuf --wthreads 1 --qdepth 10 --query flatbuf --select "*" ;

bin/run-query --num-objs 1 --pool tpchflatbuf --wthreads 1 --qdepth 10 --query flatbuf --select "att0,lt,25;att1,lt,25.0;"  --project-cols att0,att1,att2,att3 --data-schema "0 7 0 0 ATT0 ; 1 12 0 0 ATT1 ; 2 15 0 0 ATT2 ; 3 7 0 0 ATT3" --use-cls ;

bin/run-query --num-objs 1 --pool tpchflatbuf --wthreads 1 --qdepth 10 --query flatbuf --select "att0,lt,25;att1,lt,25.0;"  --project-cols att0,att2 --data-schema "0 7 0 0 ATT0 ; 1 12 0 0 ATT1 ; 2 15 0 0 ATT2 ;3 7 0 0 ATT3" --use-cls ;

bin/run-query --num-objs 1 --pool tpchflatbuf --wthreads 1 --qdepth 10 --query flatbuf --select "att0,lt,40;att3,lt,215;"  --project-cols att0,att1,att2,att3 --data-schema "0 7 0 0 ATT0 ; 1 12 0 0 ATT1 ; 2 15 0 0 ATT2 ; 3 7 0 0 ATT3" --use-cls ;

bin/run-query --num-objs 1 --pool tpchflatbuf --wthreads 1 --qdepth 10 --query flatbuf --select-preds ";att0,sum,0;" --table-name "atable" --data-schema "0 7 0 0 ATT0 ; 1 12 0 0 ATT1 ; 2 15 0 0 ATT2 ;3 7 0 0 ATT3" --use-cls ;

KDahlgren and others added 10 commits September 17, 2019 15:06

Create run-copyfrom-merge.cc

16b49a0

Update CMakeLists.txt

7fe2f44

Create run-copyfrom-merge.cc_test

1e5341e

Update PrimaryLogPG.cc

a03bfb8

Update librados.h

9bbceeb

Update librados.hpp

6b22217

added basic copyfrom merge as baseline reference.

4b2b6ab

checkpoint save. builds and runs as expected. calling stub cls method…

fb0b4ad

… from do_copy_get which takes the src bl and generates an outbl which is appended to the target bl.

builds and runs as expected. hard coded tabular call in copy from.

403c576

minor changes in stub.

a4514b5

KDahlgren requested a review from jlefevre September 21, 2019 02:43

minor edit.

50e37ae

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Skyhook cephmods #72

Skyhook cephmods #72

KDahlgren commented Sep 21, 2019

KDahlgren commented Sep 21, 2019 •

edited

Loading

Skyhook cephmods #72

Are you sure you want to change the base?

Skyhook cephmods #72

Conversation

KDahlgren commented Sep 21, 2019

KDahlgren commented Sep 21, 2019 • edited Loading

KDahlgren commented Sep 21, 2019 •

edited

Loading