-
Notifications
You must be signed in to change notification settings - Fork 28.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[MINOR][DOCS] JSON APIs related documentation fixes #17602
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -883,7 +883,7 @@ Configuration of Parquet can be done using the `setConf` method on `SparkSession | |
|
||
<div data-lang="scala" markdown="1"> | ||
Spark SQL can automatically infer the schema of a JSON dataset and load it as a `Dataset[Row]`. | ||
This conversion can be done using `SparkSession.read.json()` on either an RDD of String, | ||
This conversion can be done using `SparkSession.read.json()` on either a `Dataset[String]`, | ||
or a JSON file. | ||
|
||
Note that the file that is offered as _a json file_ is not a typical JSON file. Each | ||
|
@@ -897,7 +897,7 @@ For a regular multi-line JSON file, set the `wholeFile` option to `true`. | |
|
||
<div data-lang="java" markdown="1"> | ||
Spark SQL can automatically infer the schema of a JSON dataset and load it as a `Dataset<Row>`. | ||
This conversion can be done using `SparkSession.read().json()` on either an RDD of String, | ||
This conversion can be done using `SparkSession.read().json()` on either a `Dataset<String>`, | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. |
||
or a JSON file. | ||
|
||
Note that the file that is offered as _a json file_ is not a typical JSON file. Each | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -173,8 +173,8 @@ def json(self, path, schema=None, primitivesAsString=None, prefersDecimal=None, | |
""" | ||
Loads JSON files and returns the results as a :class:`DataFrame`. | ||
`JSON Lines <http://jsonlines.org/>`_(newline-delimited JSON) is supported by default. | ||
For JSON (one record per file), set the `wholeFile` parameter to ``true``. | ||
`JSON Lines <http://jsonlines.org/>`_ (newline-delimited JSON) is supported by default. | ||
For JSON (one record per file), set the ``wholeFile`` parameter to ``true``. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. |
||
If the ``schema`` parameter is not specified, this function goes | ||
through the input once to determine the input schema. | ||
|
@@ -634,7 +634,9 @@ def saveAsTable(self, name, format=None, mode=None, partitionBy=None, **options) | |
|
||
@since(1.4) | ||
def json(self, path, mode=None, compression=None, dateFormat=None, timestampFormat=None): | ||
"""Saves the content of the :class:`DataFrame` in JSON format at the specified path. | ||
"""Saves the content of the :class:`DataFrame` in JSON format | ||
(`JSON Lines text format or newline-delimited JSON <http://jsonlines.org/>`_) at the | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. |
||
specified path. | ||
:param path: the path in any Hadoop supported file system | ||
:param mode: specifies the behavior of the save operation when data already exists. | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -405,8 +405,8 @@ def json(self, path, schema=None, primitivesAsString=None, prefersDecimal=None, | |
""" | ||
Loads a JSON file stream and returns the results as a :class:`DataFrame`. | ||
|
||
`JSON Lines <http://jsonlines.org/>`_(newline-delimited JSON) is supported by default. | ||
For JSON (one record per file), set the `wholeFile` parameter to ``true``. | ||
`JSON Lines <http://jsonlines.org/>`_ (newline-delimited JSON) is supported by default. | ||
For JSON (one record per file), set the ``wholeFile`` parameter to ``true``. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. |
||
|
||
If the ``schema`` parameter is not specified, this function goes | ||
through the input once to determine the input schema. | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -268,8 +268,8 @@ class DataFrameReader private[sql](sparkSession: SparkSession) extends Logging { | |
} | ||
|
||
/** | ||
* Loads a JSON file (<a href="http://jsonlines.org/">JSON Lines text format or | ||
* newline-delimited JSON</a>) and returns the result as a `DataFrame`. | ||
* Loads a JSON file and returns the results as a `DataFrame`. | ||
* | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. |
||
* See the documentation on the overloaded `json()` method with varargs for more details. | ||
* | ||
* @since 1.4.0 | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Output:
Example: