Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SQL] SPARK-1333 First draft of java API #248

Closed
wants to merge 6 commits into from
Closed

[SQL] SPARK-1333 First draft of java API #248

wants to merge 6 commits into from

Conversation

marmbrus
Copy link
Contributor

WIP: Some work remains...

  • Hive support
  • Tests
  • Update docs

Feedback welcome!

@AmplabJenkins
Copy link

Merged build triggered.

@AmplabJenkins
Copy link

Merged build started.

@AmplabJenkins
Copy link

Merged build finished.

@AmplabJenkins
Copy link

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13495/

mateiz
mateiz reviewed Mar 27, 2014
View reviewed changes
sql/core/src/main/scala/org/apache/spark/sql/api/java/Row.scala

def length: Int = row.length

def get(i: Int): Any =
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These guys should all have Scaladocs

@AmplabJenkins
Copy link

Merged build triggered. Build is starting -or- tests failed to complete.

@AmplabJenkins
Copy link

Merged build started. Build is starting -or- tests failed to complete.

@AmplabJenkins
Copy link

Merged build finished. Build is starting -or- tests failed to complete.

@AmplabJenkins
Copy link

Build is starting -or- tests failed to complete.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13617/

@AmplabJenkins
Copy link

Merged build triggered. Build is starting -or- tests failed to complete.

@AmplabJenkins
Copy link

Merged build started. Build is starting -or- tests failed to complete.

@marmbrus
Copy link
Contributor Author

marmbrus commented Apr 1, 2014

@mateiz Here's a more complete version. Note that this includes the sql/hql distinction we discussed, but only for Java. I'll do the scala one in a separate PR.

The docs here are also updated: http://people.apache.org/~pwendell/catalyst-docs/sql-programming-guide.html

@AmplabJenkins
Copy link

Merged build triggered. Build is starting -or- tests failed to complete.

@AmplabJenkins
Copy link

Merged build started. Build is starting -or- tests failed to complete.

@AmplabJenkins
Copy link

Merged build finished.

@AmplabJenkins
Copy link

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13618/

@AmplabJenkins
Copy link

Merged build finished.

@AmplabJenkins
Copy link

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13621/

@marmbrus
First draft of java API.
0b859c8
@marmbrus
Improve Java SQL API:
ab91750 
* Change JavaRow => Row
* Add support for querying RDDs of JavaBeans
* Docs
* Tests
* Hive support
@marmbrus
improve docs.
822f626
@AmplabJenkins
Copy link

Merged build triggered.

@AmplabJenkins
Copy link

Merged build started.

@AmplabJenkins
Copy link

Merged build finished.

@AmplabJenkins
Copy link

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13635/

@AmplabJenkins
Copy link

Merged build triggered.

@AmplabJenkins
Copy link

Merged build started.

@AmplabJenkins
Copy link

Merged build triggered.

@AmplabJenkins
Copy link

Merged build started.

@AmplabJenkins
Copy link

Merged build finished. All automated tests passed.

@AmplabJenkins
Copy link

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13704/

@AmplabJenkins
Copy link

Merged build finished.

@AmplabJenkins
Copy link

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13705/

@marmbrus
Copy link
Contributor Author

marmbrus commented Apr 3, 2014

Jenkins, test this please.

@AmplabJenkins
Copy link

Merged build triggered.

@AmplabJenkins
Copy link

Merged build started.

@AmplabJenkins
Copy link

Merged build finished. All automated tests passed.

@AmplabJenkins
Copy link

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13727/

@mateiz
Copy link
Contributor

mateiz commented Apr 3, 2014

Thanks Michael, I've merged this in.

@asfgit asfgit closed this in b8f5341 Apr 3, 2014
@marmbrus marmbrus deleted the javaSchemaRDD branch April 7, 2014 23:16
jhartlaub referenced this pull request in jhartlaub/spark May 27, 2014
@pwendell
Merge pull request alteryx#248 from colorant/branch-0.8
d03589d 
Fix POM file for mvn assembly on hadoop 2.2 Yarn

This is the fix for maven YARN build on hadoop 2.2
pdeyhim pushed a commit to pdeyhim/spark-1 that referenced this pull request Jun 25, 2014
@marmbrus @pdeyhim
[SQL] SPARK-1333 First draft of java API
ff77cfd 
WIP: Some work remains...
 * [x] Hive support
 * [x] Tests
 * [x] Update docs

Feedback welcome!

Author: Michael Armbrust <[email protected]>

Closes apache#248 from marmbrus/javaSchemaRDD and squashes the following commits:

b393913 [Michael Armbrust] @srowen 's java style suggestions.
f531eb1 [Michael Armbrust] Address matei's comments.
33a1b1a [Michael Armbrust] Ignore JavaHiveSuite.
822f626 [Michael Armbrust] improve docs.
ab91750 [Michael Armbrust] Improve Java SQL API: * Change JavaRow => Row * Add support for querying RDDs of JavaBeans * Docs * Tests * Hive support
0b859c8 [Michael Armbrust] First draft of java API.
liancheng pushed a commit to liancheng/spark that referenced this pull request Mar 17, 2017
@adrian-ionescu
[SC-6025][DIRECTORYCOMMIT] Separate Vacuum from Acls
96022f2 
The problem with PR apache#242 was that it renamed, but didn't completely decouple `DatabricksSqlParser` from Acls. As such, the Vacuum command was only recognized if Acl support was enabled (via `spark.session.extensions = AclExtensions` and `spark.databricks.acl.enabled = true`)

## What changes were proposed in this pull request?

- extract out all Acl-related commands from `DatabricksSqlCommandBuilder` into `AclCommandBuilder`
  - separate related test suites accordingly
- make Acl client optional for `DatabricksSqlParser`
- create new `DatabricksExtensions` class that injects `DatabricksSqlParser` without Acl support
- apply `DatabricksExtensions` by default

## How was this patch tested?

Ran all tests in `spark-sql`
Manually tested `Vacuum` from `sparkShell` and `sparkShellAcl`

Author: Adrian Ionescu <[email protected]>

Closes apache#248 from adrian-ionescu/db-parser.
foxish added a commit to lins05/spark that referenced this pull request May 3, 2017
@foxish @mccheah
Enable testing against GCE clusters (apache#243)
6b489c2 
* Part 1: making test code cluster-agnostic

* Final checked

* Move all test code into KubernetesTestComponents

* Addressed comments

* Fixed doc

* Restructure the test backends (apache#248)

* Restructured the test backends

* Address comments

* var -> val

* Comments

* removed deadcode
ash211 referenced this pull request in palantir/spark Jun 14, 2017
@foxish @ash211
Enable testing against GCE clusters
ed30529 
* Part 1: making test code cluster-agnostic

* Final checked

* Move all test code into KubernetesTestComponents

* Addressed comments

* Fixed doc

* Restructure the test backends (#248)

* Restructured the test backends

* Address comments

* var -> val

* Comments

* removed deadcode

(cherry picked from commit 6b489c2)
erikerlandson pushed a commit to erikerlandson/spark that referenced this pull request Jul 28, 2017
@foxish
Enable testing against GCE clusters (apache#243)
2c753de 
* Part 1: making test code cluster-agnostic

* Final checked

* Move all test code into KubernetesTestComponents

* Addressed comments

* Fixed doc

* Restructure the test backends (apache#248)

* Restructured the test backends

* Address comments

* var -> val

* Comments

* removed deadcode
mccheah pushed a commit to mccheah/spark that referenced this pull request Oct 12, 2017
@ash211
Merge pull request apache#248 from palantir/resync-kube
65f1048 
Resync with apache-spark-on-k8s upstream
Igosuki pushed a commit to Adikteev/spark that referenced this pull request Jul 31, 2018
Ss/link fixes (apache#248)
ce442c2 
* new directory name for services

* reformat metadata for new site
@sixPulseExcalibur sixPulseExcalibur mentioned this pull request Jul 3, 2019
Closed
bzhaoopenstack pushed a commit to bzhaoopenstack/spark that referenced this pull request Sep 11, 2019
@liusheng
Merge pull request apache#248 from theopenlab/q-for-fwaas
5c9631b 
Use legacy q-* to enable neutron fwaas
arjunshroff pushed a commit to arjunshroff/spark that referenced this pull request Nov 24, 2020
@rsotn-mapr @ekrivokonmapr
MapR [SPARK-174] Spark Core unit tests fixed (apache#248)
a2ac2a2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@marmbrus @AmplabJenkins @mateiz @srowen