-
Notifications
You must be signed in to change notification settings - Fork 100
2.4 ChRIS FS DS DS plugin worflow: simulate a FreeSurfer MPC heatmap workflow
This page provides instructions for directly interacting with the CUBE backend to create the following Feed workflow:
⬤:N N -- Indicates the instance id of the plugin
F|DS(K) K -- Indicates plugin id
⬤:1 FS(12): pl-pacscopy
| -- OR --
| FS(16): pl-mri10yr06mo01da_normal
↓
⬤:2 DS(14): pl-freesurfer_pp
╱│╲
╱ │ ╲
╱ │ ╲
╱ │ ╲
╱ │ ╲
↓ ↓ ↓
⬤:3 ⬤:4 ⬤:5 DS(13): pl-mpcs
│ │ │
↓ ↓ ↓
⬤:6 ⬤:7 ⬤:8 DS(15): pl-z2labelmap
The set of operations are:
- instantiate CUBE;
- if using
pl-mri10yr06mo01da_normalthen simply run the plugin; - if using
pl-pacscopythen after instantiation:- clear the internal
pmanjob-IDjiddatabase; - pull a sample dataset from github;
- push into openstorage
- the the
pl-pacscopyplugin
- clear the internal
- if using
- run the FS data through
pl-freesurfer_ppto create a FreeSurfer output (pre calculated on the FS output) - run the FreeSurfer output through three parallel
pl-mpcsplugins, each generating a different output - run each of the 3 outputs through a
pl-z2labelmapto create one of three sets of heat maps.
Make sure you are in the base directory of the ChRIS_ultron_backEnd (CUBE) repo:
git clone https://github.com/FNNDSC/ChRIS_ultron_backEnd
cd ChRIS_ultron_backEndNow run either the full CUBE instantiation with tests:
*unmake* ; sudo rm -fr CHRIS_REMOTE_FS; rm -fr CHRIS_REMOTE_FS ; *make*or skip the unit and integration tests and the intro:
*destroy* ; sudo rm -fr CHRIS_REMOTE_FS; rm -fr CHRIS_REMOTE_FS ; *make* -U -I -sexport HOST_IP=$(ip route | grep -v docker | awk '{if(NF==11) print $9}' | head -n 1)
export HOST_PORT=8000This step is no longer necessary since the CUBE instantiation scripts also clear the pman data base. It might however be useful to refer to here, and also if a "manual" clearing of the DB is ever necessary.
pfurl --verb POST --raw --http ${HOST_IP}:5010/api/v1/cmd \
--jsonwrapper 'payload' --msg \
'{ "action": "DBctl",
"meta": {
"do": "clear"
}
}' --quiet --jsonpprintindent 4This option entails running the pl-mri10yr06mo-1da_normal to create a Feed.
First, we need to determine the ID of the plugin. Log into the CUBE API by executing a GET on the following URL:
http://${HOST_IP}:${HOST_PORT}/api/v1/plugins/either by using a web-client or a CLI call such as
pfurl --auth chris:chris1234 --verb GET \
--http ${HOST_IP}:${HOST_PORT}/api/v1/plugins/?limit=50 \
--content-type application/vnd.collection+json \
--quiet --jsonpprintindent 4and search for the id of the pl-mri10yr06mo01da_normal. For this example, let's assume this to be 12. Now, execute the plugin with
pfurl --auth chris:chris1234 --verb POST \
--http ${HOST_IP}:${HOST_PORT}/api/v1/plugins/12/instances/ \
--content-type application/vnd.collection+json \
--jsonwrapper 'template' --msg '
{"data":
[{"name":"dir",
"value":"/usr/src/data"}
]
}' \
--quiet --jsonpprintindent 4Note, that the instance ID of this run is 1.
This option entails pulling data from the FNNDSC repo, pushing into CUBE, and then running the pl-pacscopy plugin on this data to create a Feed. It is a more involved but arguably more complete example exercise.
First, pull the sample data set that will be used in this example:
git clone https://github.com/FNNDSC/anon-10-06-01.gitSet a convenience variable:
export DICOMDIR=$(pwd)/anon-10-06-01/dataOnce CUBE is up and in interactive mode, open a new terminal and cd to the CUBE repo dir (you might need to reset the convenience variables):
cd ChRIS_ultron_backEnd
export HOST_IP=$(ip route | grep -v docker | awk '{if(NF==11) print $9}')
export HOST_PORT=8000
export DICOMDIR=$(pwd)/anon-10-06-01/dataand push the data into CUBE openstorage. The PUSH operation relies on the command line apps:
-
httpto PUSH via the CUBE API; -
swiftto PUSH to a swift storage container directly;
Both apps are just a pip install away
pip install httpie
pip install swiftNow, register the DICOM files in swift storage.
cd utils/scripts
./swiftCtl.sh -A push -E dcm -D $DICOMDIR -P chris/uploads/DICOM/dataset1
./swiftCtl.sh -A push -E jpg -D $DICOMDIR -P chris/uploads/DICOM/dataset1
./swiftCtl.sh -A push -E html -D $DICOMDIR -P chris/uploads/DICOM/dataset1
./swiftCtl.sh -A push -E txt -D $DICOMDIR -P chris/uploads/DICOM/dataset1
./swiftCtl.sh -A push -E json -D $DICOMDIR -P chris/uploads/DICOM/dataset1verify the data is in fact in openstorage
./swiftCtl.shNow, execute the plugin with
pfurl --auth chris:chris1234 --verb POST \
--http ${HOST_IP}:${HOST_PORT}/api/v1/plugins/12/instances/ \
--content-type application/vnd.collection+json \
--jsonwrapper 'template' --msg '
{"data":
[{"name":"dir",
"value":"DICOM/dataset1"}
]
}' \
--quiet --jsonpprintindent 4Again, the choice of pl-pacscopy or pl-mri10yr06mo01da_normal should have been an either/or. In such a case, the relevant FS instance in the system has ID 1.
Query the status of the plugin job:
pfurl --auth chris:chris1234 \
--verb GET \
--http ${HOST_IP}:${HOST_PORT}/api/v1/plugins/instances/1/ \
--content-type application/vnd.collection+json \
--quiet --jsonpprintindent 4pfurl --auth chris:chris1234 \
--verb GET \
--http ${HOST_IP}:${HOST_PORT}/api/v1/1/files/ \
--content-type application/vnd.collection+json \
--quiet --jsonpprintindent 4Next in the chain is the dummy FreeSurfer plugin, pl-freesurfer_pp that copies to its output directory the results of a pre-calculated run. In this case, the pre-populated run corresponds to the DICOM data pulled by the FS plugin.
In the CLI case, we would run the pl-freesurfer_pp plugin with
mkdir in out && chmod 777 out
docker run --rm -v $(pwd)/in:/incoming -v $(pwd)/out:/outgoing \
fnndsc/pl-freesurfer_pp freesurfer_pp.py \
-a 10-06-01 \
-c stats,sag,cor,tra,3D \
/incoming /outgoing
In the context of CUBE, we need to determine the instance ID of the pl-freesurfer_pp, which can be found by a call to
pfurl --auth chris:chris1234 --verb GET \
--http ${HOST_IP}:${HOST_PORT}/api/v1/plugins/ \
--content-type application/vnd.collection+json \
--quiet --jsonpprintindent 4Let's assume this is plugin id 10. The equivalent call of the above CLI snippet in CUBE, using the previous id of 1 for incoming, becomes:
pfurl --auth chris:chris1234 --verb POST \
--http ${HOST_IP}:${HOST_PORT}/api/v1/plugins/10/instances/ \
--content-type application/vnd.collection+json \
--jsonwrapper 'template' --msg '
{"data":
[
{"name":"ageSpec",
"value":"10-06-01"},
{"name":"copySpec",
"value":"stats,sag,cor,tra,3D"},
{"name":"previous_id",
"value":"1"}
]
}' \
--quiet --jsonpprintindent 4
NOTE: Please be patient with this plugin. Due to the high volume of data, it can take several minutes to complete.
Query the results of the plugin call:
pfurl --auth chris:chris1234 \
--verb GET \
--http ${HOST_IP}:${HOST_PORT}/api/v1/plugins/instances/2/ \
--content-type application/vnd.collection+json \
--quiet --jsonpprintindent 4Since the pl-freesurfer_pp generated a lot of files, this pl-mpcs plugin executes slowly since all the upstream volume of data needs to be transferred to the pl-mpcs. Ironically, the plugin uses none of the inputs to generate its output.
At this point, we will simulate the running of three MPC plugins that use the output generated by the dummy FreeSurfer plugin. In the figure above, we note that the plugin ID of the pl-mpcs is 11, and the instance ID of the pl-freesurfer_pp run is 2. We will now simply call the pl-mpcs plugin three times, each time with the instance id of the FreeSurfer run as previous_id:
The equivalent CLI call for the plugin would be
mkdir in out && chmod 777 out
docker run --rm -v $(pwd)/in:/incoming -v $(pwd)/out:/outgoing \
fnndsc/pl-mpcs mpcs.py \
--random --seed 1 \
--posRange 3.0 --negRange -3.0 \
in outwhich in the context of CUBE becomes
pfurl --auth chris:chris1234 --verb POST \
--http ${HOST_IP}:${HOST_PORT}/api/v1/plugins/13/instances/ \
--content-type application/vnd.collection+json \
--jsonwrapper 'template' --msg '
{"data":
[
{"name":"random",
"value":true},
{"name":"seed",
"value":"1"},
{"name":"posRange",
"value":"3.0"},
{"name":"negRange",
"value":"-3.0"},
{"name":"previous_id",
"value":"2"}
]
}' \
--quiet --jsonpprintindent 4and query the status of the call with
pfurl --auth chris:chris1234 \
--verb GET \
--http ${HOST_IP}:${HOST_PORT}/api/v1/plugins/instances/3/ \
--content-type application/vnd.collection+json \
--quiet --jsonpprintindent 4This call can take some time to complete, mostly due to data transfers.
Repeat the plugin call two more times. Test the status of each with
pfurl --auth chris:chris1234 --verb POST \
--http ${HOST_IP}:${HOST_PORT}/api/v1/plugins/13/instances/ \
--content-type application/vnd.collection+json \
--jsonwrapper 'template' --msg '
{"data":
[
{"name":"random",
"value":true},
{"name":"seed",
"value":"1"},
{"name":"posRange",
"value":"3.0"},
{"name":"negRange",
"value":"-3.0"},
{"name":"previous_id",
"value":"2"}
]
}' \
--quiet --jsonpprintindent 4pfurl --auth chris:chris1234 \
--verb GET \
--http ${HOST_IP}:${HOST_PORT}/api/v1/plugins/instances/4/ \
--content-type application/vnd.collection+json \
--quiet --jsonpprintindent 4pfurl --auth chris:chris1234 --verb POST \
--http ${HOST_IP}:${HOST_PORT}/api/v1/plugins/13/instances/ \
--content-type application/vnd.collection+json \
--jsonwrapper 'template' --msg '
{"data":
[
{"name":"random",
"value":true},
{"name":"seed",
"value":"1"},
{"name":"posRange",
"value":"3.0"},
{"name":"negRange",
"value":"-3.0"},
{"name":"previous_id",
"value":"2"}
]
}' \
--quiet --jsonpprintindent 4pfurl --auth chris:chris1234 \
--verb GET \
--http ${HOST_IP}:${HOST_PORT}/api/v1/plugins/instances/5/ \
--content-type application/vnd.collection+json \
--quiet --jsonpprintindent 4
Now, finally, we will run the pl-z2labelmap on each branch of the pl-mpc output. The equivalent CLI call is
docker run --rm -v $(pwd)/in:/incoming -v $(pwd)/out:/outgoing \
fnndsc/pl-z2labelmap z2labelmap.py \
--scaleRange 2.0 --lowerFilter 0.8 \
--negColor B --posColor R \
/incoming /outgoingWhich, in each branch becomes:
pfurl --auth chris:chris1234 --verb POST \
--http ${HOST_IP}:${HOST_PORT}/api/v1/plugins/15/instances/ \
--content-type application/vnd.collection+json \
--jsonwrapper 'template' --msg '
{"data":
[
{"name":"imageSet",
"value":"../data/set1"},
{"name":"negColor",
"value":"B"},
{"name":"posColor",
"value":"R"},
{"name":"previous_id",
"value":"3"}
]
}' \
--quiet --jsonpprintindent 4pfurl --auth chris:chris1234 \
--verb GET \
--http ${HOST_IP}:${HOST_PORT}/api/v1/plugins/instances/6/ \
--content-type application/vnd.collection+json \
--quiet --jsonpprintindent 4pfurl --auth chris:chris1234 --verb POST \
--http ${HOST_IP}:${HOST_PORT}/api/v1/plugins/15/instances/ \
--content-type application/vnd.collection+json \
--jsonwrapper 'template' --msg '
{"data":
[
{"name":"imageSet",
"value":"../data/set1"},
{"name":"negColor",
"value":"B"},
{"name":"posColor",
"value":"R"},
{"name":"previous_id",
"value":"4"}
]
}' \
--quiet --jsonpprintindent 4pfurl --auth chris:chris1234 \
--verb GET \
--http ${HOST_IP}:${HOST_PORT}/api/v1/plugins/instances/7/ \
--content-type application/vnd.collection+json \
--quiet --jsonpprintindent 4pfurl --auth chris:chris1234 --verb POST \
--http ${HOST_IP}:${HOST_PORT}/api/v1/plugins/15/instances/ \
--content-type application/vnd.collection+json \
--jsonwrapper 'template' --msg '
{"data":
[
{"name":"imageSet",
"value":"../data/set1"},
{"name":"negColor",
"value":"B"},
{"name":"posColor",
"value":"R"},
{"name":"previous_id",
"value":"5"}
]
}' \
--quiet --jsonpprintindent 4pfurl --auth chris:chris1234 \
--verb GET \
--http ${HOST_IP}:${HOST_PORT}/api/v1/plugins/instances/8/ \
--content-type application/vnd.collection+json \
--quiet --jsonpprintindent 4-30-