[MINOR][DOCS] JSON APIs related documentation fixes #17602
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
HyukjinKwon
commented
Apr 11, 2017
| `JSON Lines <http://jsonlines.org/>`_(newline-delimited JSON) is supported by default. | ||
| For JSON (one record per file), set the `wholeFile` parameter to ``true``. | ||
| `JSON Lines <http://jsonlines.org/>`_ (newline-delimited JSON) is supported by default. | ||
| For JSON (one record per file), set the ``wholeFile`` parameter to ``true``. |
HyukjinKwon
commented
Apr 11, 2017
| `JSON Lines <http://jsonlines.org/>`_(newline-delimited JSON) is supported by default. | ||
| For JSON (one record per file), set the `wholeFile` parameter to ``true``. | ||
| `JSON Lines <http://jsonlines.org/>`_ (newline-delimited JSON) is supported by default. | ||
| For JSON (one record per file), set the ``wholeFile`` parameter to ``true``. |
HyukjinKwon
commented
Apr 11, 2017
| * Loads a JSON file (<a href="http://jsonlines.org/">JSON Lines text format or | ||
| * newline-delimited JSON</a>) and returns the result as a `DataFrame`. | ||
| * Loads a JSON file and returns the results as a `DataFrame`. | ||
| * |
There was a problem hiding this comment.
This de-duplicates the documentation as it points the overloaded json() out below.
Before
After
HyukjinKwon
force-pushed
the
minor-json-documentation
branch
from
82aadaa
to
9043f01
Compare
HyukjinKwon
commented
Apr 11, 2017
| @@ -634,7 +634,9 @@ def saveAsTable(self, name, format=None, mode=None, partitionBy=None, **options) | |||
|
|
|||
| @since(1.4) | |||
| def json(self, path, mode=None, compression=None, dateFormat=None, timestampFormat=None): | |||
| """Saves the content of the :class:`DataFrame` in JSON format at the specified path. | |||
| """Saves the content of the :class:`DataFrame` in JSON format | |||
| (`JSON Lines text format or newline-delimited JSON <http://jsonlines.org/>`_) at the | |||
There was a problem hiding this comment.
Before
After
Note that this is not consistent with Scala/Java ones:
HyukjinKwon
commented
Apr 11, 2017
| @@ -883,7 +883,7 @@ Configuration of Parquet can be done using the `setConf` method on `SparkSession | |||
|
|
|||
| <div data-lang="scala" markdown="1"> | |||
| Spark SQL can automatically infer the schema of a JSON dataset and load it as a `Dataset[Row]`. | |||
| This conversion can be done using `SparkSession.read.json()` on either an RDD of String, | |||
| This conversion can be done using `SparkSession.read.json()` on either a `Dataset[String]`, | |||
HyukjinKwon
commented
Apr 11, 2017
| @@ -897,7 +897,7 @@ For a regular multi-line JSON file, set the `wholeFile` option to `true`. | |||
|
|
|||
| <div data-lang="java" markdown="1"> | |||
| Spark SQL can automatically infer the schema of a JSON dataset and load it as a `Dataset<Row>`. | |||
| This conversion can be done using `SparkSession.read().json()` on either an RDD of String, | |||
| This conversion can be done using `SparkSession.read().json()` on either a `Dataset<String>`, | |||
|
Test build #75689 has finished for PR 17602 at commit
|
|
Test build #75691 has finished for PR 17602 at commit
|
|
Test build #75690 has finished for PR 17602 at commit
|
|
Test build #75692 has finished for PR 17602 at commit
|
srowen
approved these changes
Apr 11, 2017
|
Merged to master |
|
Thank you @srowen. |
peter-toth
pushed a commit
to peter-toth/spark
that referenced
this pull request
Oct 6, 2018
## What changes were proposed in this pull request? This PR proposes corrections related to JSON APIs as below: - Rendering links in Python documentation - Replacing `RDD` to `Dataset` in programing guide - Adding missing description about JSON Lines consistently in `DataFrameReader.json` in Python API - De-duplicating little bit of `DataFrameReader.json` in Scala/Java API ## How was this patch tested? Manually build the documentation via `jekyll build`. Corresponding snapstops will be left on the codes. Note that currently there are Javadoc8 breaks in several places. These are proposed to be handled in apache#17477. So, this PR does not fix those. Author: hyukjinkwon <[email protected]> Closes apache#17602 from HyukjinKwon/minor-json-documentation.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR proposes corrections related to JSON APIs as below:
RDDtoDatasetin programing guideDataFrameReader.jsonin Python APIDataFrameReader.jsonin Scala/Java APIHow was this patch tested?
Manually build the documentation via
jekyll build. Corresponding snapstops will be left on the codes.Note that currently there are Javadoc8 breaks in several places. These are proposed to be handled in #17477. So, this PR does not fix those.