add recordio doc #248
Merged
add recordio doc #248
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
gongweibao
reviewed
Jul 26, 2017
| ```python | ||
| import paddle.v2.dataset as dataset | ||
| dataset.convert(output_path = "./dataset", | ||
| reader = dataset.uci_housing.train(), |
There was a problem hiding this comment.
用户可能不理解reader的意思，或者需要再去看dataset.uci_housing.train(),是否直接把例子贴到这里？
There was a problem hiding this comment.
Done with a link to reader.
typhoonzero
reviewed
Jul 26, 2017
doc/usage_cn.md
Outdated
| @@ -68,6 +68,47 @@ scp -r my_training_data_dir/ user@tunnel-server:/mnt/hdfs_mulan/idl/idl-dl/mydir | |||
|
|
|||
| 在训练任务提交后，每个训练节点会把HDFS挂载在`/pfs/[datacenter_name]/home/[username]/`目录下这样训练程序即可使用这个路径读取训练数据并开始训练。 | |||
|
|
|||
| ### 使用[RecordIO](https://github.com/PaddlePaddle/recordio)对训练数据进行预处理 | |||
| 用户可以在本地将数据预先处理为RecordIO的格式，再上传至集群进行训练。 | |||
doc/usage_cn.md
Outdated
| def cluster_creator(filename): | ||
| import recordio | ||
| def reader(): | ||
| r = recordio.reader("./dataset/uci_housing_train*") |
There was a problem hiding this comment.
没有master的reader是需要在reader端dispatch文件的。这个例子还是按照dispatch的方法吧，master功能稳定之后再更新。
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixed #247