Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-3007][SQL] Adds dynamic partitioning support #2616

Closed
wants to merge 28 commits into from
Closed

[SPARK-3007][SQL] Adds dynamic partitioning support #2616

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
6bb5880
Update HiveQl.scala
baishuo Aug 12, 2014
1867e23
Update SparkHadoopWriter.scala
baishuo Aug 12, 2014
adf02f1
Update InsertIntoHiveTable.scala
baishuo Aug 12, 2014
6af73f4
Update InsertIntoHiveTable.scala
baishuo Aug 12, 2014
98cfb1f
Update HiveCompatibilitySuite.scala
baishuo Aug 18, 2014
37c603b
Update InsertIntoHiveTable.scala
baishuo Aug 18, 2014
d452eb3
Update HiveQuerySuite.scala
baishuo Aug 18, 2014
051ba91
Update Cast.scala
baishuo Aug 18, 2014
8ad173c
Update InsertIntoHiveTable.scala
baishuo Aug 18, 2014
3f91665
Update Cast.scala
baishuo Aug 18, 2014
8e7268c
update file after test
baishuo Aug 19, 2014
cd822f0
do a little modify
baishuo Aug 19, 2014
b660e74
delete a empty else branch
baishuo Sep 3, 2014
caea6fb
modify code to pass scala style checks
baishuo Sep 3, 2014
207c6ac
modify for some bad indentation
baishuo Sep 3, 2014
761ecf2
modify according micheal's advice
baishuo Sep 9, 2014
997c990
use HiveConf.DEFAULTPARTITIONNAME to replace hive.exec.default.partit…
baishuo Sep 10, 2014
b821611
pass check style
baishuo Sep 10, 2014
d53daa5
Refactors dynamic partitioning support
liancheng Sep 16, 2014
6fb16d7
Fixes typo in test name, regenerated golden answer files
liancheng Sep 17, 2014
c47470e
Refactors InsertIntoHiveTable to a Command
liancheng Sep 17, 2014
9227181
Minor refactoring
liancheng Sep 17, 2014
26632c3
Adds more tests
liancheng Sep 17, 2014
0eed349
Addresses @yhuai's comments
liancheng Sep 18, 2014
9c6eb2d
Adds tests to verify dynamic partitioning folder layout
liancheng Sep 22, 2014
a132c80
Fixes output compression
liancheng Oct 1, 2014
f471c4b
PreInsertionCasts should take table partitions into account
liancheng Oct 1, 2014
21935b6
Adds back deleted trailing space
liancheng Oct 1, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions ...atibility/src/test/scala/org/apache/spark/sql/hive/execution/HiveCompatibilitySuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -220,6 +220,23 @@ class HiveCompatibilitySuite extends HiveQueryFileTest with BeforeAndAfter {
*/
override def whiteList = Seq(
"add_part_exist",
"dynamic_partition_skip_default",
"infer_bucket_sort_dyn_part",
"load_dyn_part1",
"load_dyn_part2",
"load_dyn_part3",
"load_dyn_part4",
"load_dyn_part5",
"load_dyn_part6",
"load_dyn_part7",
"load_dyn_part8",
"load_dyn_part9",
"load_dyn_part10",
"load_dyn_part11",
"load_dyn_part12",
"load_dyn_part13",
"load_dyn_part14",
"load_dyn_part14_win",
"add_part_multiple",
"add_partition_no_whitelist",
"add_partition_with_whitelist",
Expand Down
195 changes: 0 additions & 195 deletions sql/hive/src/main/scala/org/apache/spark/SparkHadoopWriter.scala
View file
Open in desktop

This file was deleted.

3 changes: 2 additions & 1 deletion sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -142,7 +142,8 @@ private[hive] class HiveMetastoreCatalog(hive: HiveContext) extends Catalog with
val childOutputDataTypes = child.output.map(_.dataType)
// Only check attributes, not partitionKeys since they are always strings.
// TODO: Fully support inserting into partitioned tables.
val tableOutputDataTypes = table.attributes.map(_.dataType)
val tableOutputDataTypes =
table.attributes.map(_.dataType) ++ table.partitionKeys.map(_.dataType)

if (childOutputDataTypes == tableOutputDataTypes) {
p
Expand Down
5 changes: 0 additions & 5 deletions sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveQl.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -837,11 +837,6 @@ private[hive] object HiveQl {
cleanIdentifier(key.toLowerCase) -> None
}.toMap).getOrElse(Map.empty)

if (partitionKeys.values.exists(p => p.isEmpty)) {
throw new NotImplementedError(s"Do not support INSERT INTO/OVERWRITE with" +
s"dynamic partitioning.")
}

InsertIntoTable(UnresolvedRelation(db, tableName, None), partitionKeys, query, overwrite)

case a: ASTNode =>
Expand Down
Loading