diff --git a/docs/reference/sql/language/syntax/describe-table.asciidoc b/docs/reference/sql/language/syntax/describe-table.asciidoc index dd2d27a5781d2..396be25bb5170 100644 --- a/docs/reference/sql/language/syntax/describe-table.asciidoc +++ b/docs/reference/sql/language/syntax/describe-table.asciidoc @@ -20,3 +20,8 @@ DESC table .Description `DESC` and `DESCRIBE` are aliases to <>. + +["source","sql",subs="attributes,callouts,macros"] +---- +include-tagged::{sql-specs}/docs.csv-spec[describeTable] +---- diff --git a/docs/reference/sql/language/syntax/select.asciidoc b/docs/reference/sql/language/syntax/select.asciidoc index 4a7c0534b68a3..b58173097b0ae 100644 --- a/docs/reference/sql/language/syntax/select.asciidoc +++ b/docs/reference/sql/language/syntax/select.asciidoc @@ -36,23 +36,26 @@ The general execution of `SELECT` is as follows: As with a table, every output column of a `SELECT` has a name which can be either specified per column through the `AS` keyword : -[source,sql] +["source","sql",subs="attributes,callouts,macros"] ---- -SELECT column AS c +include-tagged::{sql-specs}/docs.csv-spec[selectColumnAlias] ---- +Note: `AS` is an optional keyword however it helps with the readability and in some case ambiguity of the query +which is why it is recommended to specify it. + assigned by {es-sql} if no name is given: -[source,sql] +["source","sql",subs="attributes,callouts,macros"] ---- -SELECT 1 + 1 +include-tagged::{sql-specs}/docs.csv-spec[selectInline] ---- or if it's a simple column reference, use its name as the column name: -[source,sql] +["source","sql",subs="attributes,callouts,macros"] ---- -SELECT col FROM table +include-tagged::{sql-specs}/docs.csv-spec[selectColumn] ---- [[sql-syntax-select-wildcard]] @@ -61,11 +64,11 @@ SELECT col FROM table To select all the columns in the source, one can use `*`: ["source","sql",subs="attributes,callouts,macros"] --------------------------------------------------- -include-tagged::{sql-specs}/select.sql-spec[wildcardWithOrder] --------------------------------------------------- +---- +include-tagged::{sql-specs}/docs.csv-spec[wildcardWithOrder] +---- -which essentially returns all columsn found. +which essentially returns all(top-level fields, sub-fields, such as multi-fields are ignored] columns found. [[sql-syntax-from]] [float] @@ -83,17 +86,30 @@ where: `table_name`:: Represents the name (optionally qualified) of an existing table, either a concrete or base one (actual index) or alias. + + If the table name contains special SQL characters (such as `.`,`-`,etc...) use double quotes to escape them: -[source, sql] + +["source","sql",subs="attributes,callouts,macros"] ---- -SELECT ... FROM "some-table" +include-tagged::{sql-specs}/docs.csv-spec[fromTableQuoted] ---- The name can be a <> pointing to multiple indices (likely requiring quoting as mentioned above) with the restriction that *all* resolved concrete tables have **exact mapping**. +["source","sql",subs="attributes,callouts,macros"] +---- +include-tagged::{sql-specs}/docs.csv-spec[fromTablePatternQuoted] +---- + `alias`:: A substitute name for the `FROM` item containing the alias. An alias is used for brevity or to eliminate ambiguity. When an alias is provided, it completely hides the actual name of the table and must be used in its place. +["source","sql",subs="attributes,callouts,macros"] +---- +include-tagged::{sql-specs}/docs.csv-spec[fromTableAlias] +---- + [[sql-syntax-where]] [float] ==== WHERE Clause @@ -111,6 +127,11 @@ where: Represents an expression that evaluates to a `boolean`. Only the rows that match the condition (to `true`) are returned. +["source","sql",subs="attributes,callouts,macros"] +---- +include-tagged::{sql-specs}/docs.csv-spec[basicWhere] +---- + [[sql-syntax-group-by]] [float] ==== GROUP BY @@ -126,9 +147,79 @@ where: `grouping_element`:: -Represents an expression on which rows are being grouped _on_. It can be a column name, name or ordinal number of a column or an arbitrary expression of column values. +Represents an expression on which rows are being grouped _on_. It can be a column name, alias or ordinal number of a column or an arbitrary expression of column values. + +A common, group by column name: + +["source","sql",subs="attributes,callouts,macros"] +---- +include-tagged::{sql-specs}/docs.csv-spec[groupByColumn] +---- + +Grouping by output ordinal: + +["source","sql",subs="attributes,callouts,macros"] +---- +include-tagged::{sql-specs}/docs.csv-spec[groupByOrdinal] +---- + +Grouping by alias: + +["source","sql",subs="attributes,callouts,macros"] +---- +include-tagged::{sql-specs}/docs.csv-spec[groupByAlias] +---- + +And grouping by column expression (typically used along-side an alias): + +["source","sql",subs="attributes,callouts,macros"] +---- +include-tagged::{sql-specs}/docs.csv-spec[groupByExpression] +---- + +When a `GROUP BY` clause is used in a `SELECT`, _all_ output expressions must be either aggregate functions or expressions used for grouping or derivatives of (otherwise there would be more than one possible value to return for each ungrouped column). + +To wit: + +["source","sql",subs="attributes,callouts,macros"] +---- +include-tagged::{sql-specs}/docs.csv-spec[groupByAndAgg] +---- + +Expressions over aggregates used in output: + +["source","sql",subs="attributes,callouts,macros"] +---- +include-tagged::{sql-specs}/docs.csv-spec[groupByAndAggExpression] +---- + +Multiple aggregates used: + +["source","sql",subs="attributes,callouts,macros"] +---- +include-tagged::{sql-specs}/docs.csv-spec[groupByAndMultipleAggs] +---- + +[[sql-syntax-group-by-implicit]] +[float] +===== Implicit Grouping + +When an aggregation is used without an associated `GROUP BY`, an __implicit grouping__ is applied, meaning all selected rows are considered to form a single default, or implicit group. +As such, the query emits only a single row (as there is only a single group). + +A common example is counting the number of records: + +["source","sql",subs="attributes,callouts,macros"] +---- +include-tagged::{sql-specs}/docs.csv-spec[groupByImplicitCount] +---- + +Of course, multiple aggregations can be applied: -When a `GROUP BY` clause is used in a `SELECT`, _all_ output expressions must be either aggregate functions or expressions used for grouping or derivates of (otherwise there would be more than one possible value to return for each ungrouped column). +["source","sql",subs="attributes,callouts,macros"] +---- +include-tagged::{sql-specs}/docs.csv-spec[groupByImplicitMultipleAggs] +---- [[sql-syntax-having]] [float] @@ -147,13 +238,44 @@ where: Represents an expression that evaluates to a `boolean`. Only groups that match the condition (to `true`) are returned. -Both `WHERE` and `HAVING` are used for filtering however there are several differences between them: +Both `WHERE` and `HAVING` are used for filtering however there are several significant differences between them: . `WHERE` works on individual *rows*, `HAVING` works on the *groups* created by ``GROUP BY`` . `WHERE` is evaluated *before* grouping, `HAVING` is evaluated *after* grouping -Note that it is possible to have a `HAVING` clause without a ``GROUP BY``. In this case, an __implicit grouping__ is applied, meaning all selected rows are considered to form a single group and `HAVING` can be applied on any of the aggregate functions specified on this group. ` -As such a query emits only a single row (as there is only a single group), `HAVING` condition returns either one row (the group) or zero if the condition fails. +["source","sql",subs="attributes,callouts,macros"] +---- +include-tagged::{sql-specs}/docs.csv-spec[groupByHaving] +---- + +Further more, one can use multiple aggregate expressions inside `HAVING` even ones that are not used in the output (`SELECT`): + +["source","sql",subs="attributes,callouts,macros"] +---- +include-tagged::{sql-specs}/docs.csv-spec[groupByHavingMultiple] +---- + +[[sql-syntax-having-group-by-implicit]] +[float] +===== Implicit Grouping + +As indicated above, it is possible to have a `HAVING` clause without a ``GROUP BY``. In this case, the so-called <> is applied, meaning all selected rows are considered to form a single group and `HAVING` can be applied on any of the aggregate functions specified on this group. ` +As such, the query emits only a single row (as there is only a single group) and `HAVING` condition returns either one row (the group) or zero if the condition fails. + +In this example, `HAVING` matches: + +["source","sql",subs="attributes,callouts,macros"] +---- +include-tagged::{sql-specs}/docs.csv-spec[groupByHavingImplicitMatch] +---- + +//However `HAVING` can also not match, in which case an empty result is returned: +// +//["source","sql",subs="attributes,callouts,macros"] +//---- +//include-tagged::{sql-specs}/docs.csv-spec[groupByHavingImplicitNoMatch] +//---- + [[sql-syntax-order-by]] [float] @@ -178,30 +300,10 @@ IMPORTANT: When used along-side, `GROUP BY` expression can point _only_ to the c For example, the following query sorts by an arbitrary input field (`page_count`): -[source,js] --------------------------------------------------- -POST /_xpack/sql?format=txt -{ - "query": "SELECT * FROM library ORDER BY page_count DESC LIMIT 5" -} --------------------------------------------------- -// CONSOLE -// TEST[setup:library] - -which results in something like: - -[source,text] --------------------------------------------------- - author | name | page_count | release_date ------------------+--------------------+---------------+------------------------ -Peter F. Hamilton|Pandora's Star |768 |2004-03-02T00:00:00.000Z -Vernor Vinge |A Fire Upon the Deep|613 |1992-06-01T00:00:00.000Z -Frank Herbert |Dune |604 |1965-06-01T00:00:00.000Z -Alastair Reynolds|Revelation Space |585 |2000-03-15T00:00:00.000Z -James S.A. Corey |Leviathan Wakes |561 |2011-06-02T00:00:00.000Z --------------------------------------------------- -// TESTRESPONSE[s/\|/\\|/ s/\+/\\+/] -// TESTRESPONSE[_cat] +["source","sql",subs="attributes,callouts,macros"] +---- +include-tagged::{sql-specs}/docs.csv-spec[orderByBasic] +---- [[sql-syntax-order-by-score]] ==== Order By Score @@ -215,58 +317,22 @@ combined using the same rules as {es}'s To sort based on the `score`, use the special function `SCORE()`: -[source,js] --------------------------------------------------- -POST /_xpack/sql?format=txt -{ - "query": "SELECT SCORE(), * FROM library WHERE match(name, 'dune') ORDER BY SCORE() DESC" -} --------------------------------------------------- -// CONSOLE -// TEST[setup:library] - -Which results in something like: - -[source,text] --------------------------------------------------- - SCORE() | author | name | page_count | release_date ----------------+---------------+-------------------+---------------+------------------------ -2.288635 |Frank Herbert |Dune |604 |1965-06-01T00:00:00.000Z -1.8893257 |Frank Herbert |Dune Messiah |331 |1969-10-15T00:00:00.000Z -1.6086555 |Frank Herbert |Children of Dune |408 |1976-04-21T00:00:00.000Z -1.4005898 |Frank Herbert |God Emperor of Dune|454 |1981-05-28T00:00:00.000Z --------------------------------------------------- -// TESTRESPONSE[s/\|/\\|/ s/\+/\\+/ s/\(/\\\(/ s/\)/\\\)/] -// TESTRESPONSE[_cat] - -Note that you can return `SCORE()` by adding it to the where clause. This -is possible even if you are not sorting by `SCORE()`: - -[source,js] --------------------------------------------------- -POST /_xpack/sql?format=txt -{ - "query": "SELECT SCORE(), * FROM library WHERE match(name, 'dune') ORDER BY page_count DESC" -} --------------------------------------------------- -// CONSOLE -// TEST[setup:library] - -[source,text] --------------------------------------------------- - SCORE() | author | name | page_count | release_date ----------------+---------------+-------------------+---------------+------------------------ -2.288635 |Frank Herbert |Dune |604 |1965-06-01T00:00:00.000Z -1.4005898 |Frank Herbert |God Emperor of Dune|454 |1981-05-28T00:00:00.000Z -1.6086555 |Frank Herbert |Children of Dune |408 |1976-04-21T00:00:00.000Z -1.8893257 |Frank Herbert |Dune Messiah |331 |1969-10-15T00:00:00.000Z --------------------------------------------------- -// TESTRESPONSE[s/\|/\\|/ s/\+/\\+/ s/\(/\\\(/ s/\)/\\\)/] -// TESTRESPONSE[_cat] +["source","sql",subs="attributes,callouts,macros"] +---- +include-tagged::{sql-specs}/docs.csv-spec[orderByScore] +---- + +Note that you can return `SCORE()` by using a full-text search predicate in the `WHERE` clause. +This is possible even if `SCORE()` is not used for sorting: + +["source","sql",subs="attributes,callouts,macros"] +---- +include-tagged::{sql-specs}/docs.csv-spec[orderByScoreWithMatch] +---- NOTE: -Trying to return `score` from a non full-text queries will return the same value for all results, as -all are equilley relevant. +Trying to return `score` from a non full-text query will return the same value for all results, as +all are equally relevant. [[sql-syntax-limit]] [float] @@ -284,3 +350,10 @@ where count:: is a positive integer or zero indicating the maximum *possible* number of results being returned (as there might be less matches than the limit). If `0` is specified, no results are returned. ALL:: indicates there is no limit and thus all results are being returned. + +To return + +["source","sql",subs="attributes,callouts,macros"] +---- +include-tagged::{sql-specs}/docs.csv-spec[limitBasic] +---- \ No newline at end of file diff --git a/docs/reference/sql/language/syntax/show-columns.asciidoc b/docs/reference/sql/language/syntax/show-columns.asciidoc index a52c744f17a97..539c35c57952a 100644 --- a/docs/reference/sql/language/syntax/show-columns.asciidoc +++ b/docs/reference/sql/language/syntax/show-columns.asciidoc @@ -12,3 +12,8 @@ SHOW COLUMNS [ FROM | IN ] ? table .Description List the columns in table and their data type (and other attributes). + +["source","sql",subs="attributes,callouts,macros"] +---- +include-tagged::{sql-specs}/docs.csv-spec[showColumns] +---- diff --git a/docs/reference/sql/language/syntax/show-functions.asciidoc b/docs/reference/sql/language/syntax/show-functions.asciidoc index 964cdf39081c6..1e4220ef5295c 100644 --- a/docs/reference/sql/language/syntax/show-functions.asciidoc +++ b/docs/reference/sql/language/syntax/show-functions.asciidoc @@ -14,3 +14,34 @@ SHOW FUNCTIONS [ LIKE? pattern<1>? ]? .Description List all the SQL functions and their type. The `LIKE` clause can be used to restrict the list of names to the given pattern. + +["source","sql",subs="attributes,callouts,macros"] +---- +include-tagged::{sql-specs}/docs.csv-spec[showFunctions] +---- + +The list of functions returned can be customized based on the pattern. + +It can be an exact match: +["source","sql",subs="attributes,callouts,macros"] +---- +include-tagged::{sql-specs}/docs.csv-spec[showFunctionsLikeExact] +---- + +A wildcard for exactly one character: +["source","sql",subs="attributes,callouts,macros"] +---- +include-tagged::{sql-specs}/docs.csv-spec[showFunctionsLikeChar] +---- + +A wildcard matching zero or more characters: +["source","sql",subs="attributes,callouts,macros"] +---- +include-tagged::{sql-specs}/docs.csv-spec[showFunctionsLikeWildcard] +---- + +Or of course, a variation of the above: +["source","sql",subs="attributes,callouts,macros"] +---- +include-tagged::{sql-specs}/docs.csv-spec[showFunctionsWithPattern] +---- diff --git a/docs/reference/sql/language/syntax/show-tables.asciidoc b/docs/reference/sql/language/syntax/show-tables.asciidoc index 7772c39c6fc21..b401e9f7d900a 100644 --- a/docs/reference/sql/language/syntax/show-tables.asciidoc +++ b/docs/reference/sql/language/syntax/show-tables.asciidoc @@ -13,4 +13,36 @@ SHOW TABLES [ LIKE? pattern<1>? ]? .Description -List the tables available to the current user and their type. The `LIKE` clause can be used to restrict the list of names to the given pattern. +List the tables available to the current user and their type. + +["source","sql",subs="attributes,callouts,macros"] +---- +include-tagged::{sql-specs}/docs.csv-spec[showTables] +---- + +The `LIKE` clause can be used to restrict the list of names to the given pattern. + +The pattern can be an exact match: +["source","sql",subs="attributes,callouts,macros"] +---- +include-tagged::{sql-specs}/docs.csv-spec[showTablesLikeExact] +---- + +Multiple chars: +["source","sql",subs="attributes,callouts,macros"] +---- +include-tagged::{sql-specs}/docs.csv-spec[showTablesLikeWildcard] +---- + +A single char: +["source","sql",subs="attributes,callouts,macros"] +---- +include-tagged::{sql-specs}/docs.csv-spec[showTablesLikeOneChar] +---- + + +Or a mixture of single and multiple chars: +["source","sql",subs="attributes,callouts,macros"] +---- +include-tagged::{sql-specs}/docs.csv-spec[showTablesLikeMixed] +---- diff --git a/x-pack/qa/sql/no-security/src/test/java/org/elasticsearch/xpack/qa/sql/nosecurity/JdbcDocCsvSpectIT.java b/x-pack/qa/sql/no-security/src/test/java/org/elasticsearch/xpack/qa/sql/nosecurity/JdbcDocCsvSpectIT.java new file mode 100644 index 0000000000000..24e8c170cc39d --- /dev/null +++ b/x-pack/qa/sql/no-security/src/test/java/org/elasticsearch/xpack/qa/sql/nosecurity/JdbcDocCsvSpectIT.java @@ -0,0 +1,90 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +package org.elasticsearch.xpack.qa.sql.nosecurity; + +import com.carrotsearch.randomizedtesting.annotations.ParametersFactory; + +import org.apache.logging.log4j.Logger; +import org.elasticsearch.client.RestClient; +import org.elasticsearch.xpack.qa.sql.jdbc.CsvTestUtils.CsvTestCase; +import org.elasticsearch.xpack.qa.sql.jdbc.DataLoader; +import org.elasticsearch.xpack.qa.sql.jdbc.JdbcAssert; +import org.elasticsearch.xpack.qa.sql.jdbc.SpecBaseIntegrationTestCase; +import org.elasticsearch.xpack.qa.sql.jdbc.SqlSpecTestCase; + +import java.sql.Connection; +import java.sql.ResultSet; +import java.sql.SQLException; +import java.util.List; + +import static org.elasticsearch.xpack.qa.sql.jdbc.CsvTestUtils.csvConnection; +import static org.elasticsearch.xpack.qa.sql.jdbc.CsvTestUtils.executeCsvQuery; +import static org.elasticsearch.xpack.qa.sql.jdbc.CsvTestUtils.specParser; + +/** + * CSV test specification for DOC examples. + * While we could use the existing tests, their purpose is to test corner-cases which + * gets reflected in the dataset structure. + * The doc tests while redundant, try to be expressive first and foremost and sometimes + * the dataset isn't exactly convenient. + * + * Also looking around for the tests across the test files isn't trivial. + * + * That's not to say the two cannot be merged however that felt like too much of an effort + * at this stage and, to not keep things stalling, started with this approach. + */ +public class JdbcDocCsvSpectIT extends SpecBaseIntegrationTestCase { + + private final CsvTestCase testCase; + + @Override + protected String indexName() { + return "library"; + } + + @Override + protected void loadDataset(RestClient client) throws Exception { + DataLoader.loadDocsDatasetIntoEs(client); + } + + @ParametersFactory(shuffle = false, argumentFormatting = SqlSpecTestCase.PARAM_FORMATTING) + public static List readScriptSpec() throws Exception { + Parser parser = specParser(); + return readScriptSpec("/docs.csv-spec", parser); + } + + public JdbcDocCsvSpectIT(String fileName, String groupName, String testName, Integer lineNumber, CsvTestCase testCase) { + super(fileName, groupName, testName, lineNumber); + this.testCase = testCase; + } + + @Override + protected void assertResults(ResultSet expected, ResultSet elastic) throws SQLException { + Logger log = logEsResultSet() ? logger : null; + + // + // uncomment this to printout the result set and create new CSV tests + // + //JdbcTestUtils.logLikeCLI(elastic, log); + JdbcAssert.assertResultSets(expected, elastic, log, true); + } + + @Override + protected boolean logEsResultSet() { + return true; + } + + @Override + protected final void doTest() throws Throwable { + try (Connection csv = csvConnection(testCase.expectedResults); Connection es = esJdbc()) { + + // pass the testName as table for debugging purposes (in case the underlying reader is missing) + ResultSet expected = executeCsvQuery(csv, testName); + ResultSet elasticResults = executeJdbcQuery(es, testCase.query); + assertResults(expected, elasticResults); + } + } +} \ No newline at end of file diff --git a/x-pack/qa/sql/src/main/java/org/elasticsearch/xpack/qa/sql/jdbc/CsvSpecTestCase.java b/x-pack/qa/sql/src/main/java/org/elasticsearch/xpack/qa/sql/jdbc/CsvSpecTestCase.java index e37688eb90465..99e8432370471 100644 --- a/x-pack/qa/sql/src/main/java/org/elasticsearch/xpack/qa/sql/jdbc/CsvSpecTestCase.java +++ b/x-pack/qa/sql/src/main/java/org/elasticsearch/xpack/qa/sql/jdbc/CsvSpecTestCase.java @@ -6,14 +6,13 @@ package org.elasticsearch.xpack.qa.sql.jdbc; import com.carrotsearch.randomizedtesting.annotations.ParametersFactory; + import org.elasticsearch.xpack.qa.sql.jdbc.CsvTestUtils.CsvTestCase; -import org.elasticsearch.xpack.sql.jdbc.jdbc.JdbcConfiguration; import java.sql.Connection; import java.sql.ResultSet; import java.util.ArrayList; import java.util.List; -import java.util.Properties; import static org.elasticsearch.xpack.qa.sql.jdbc.CsvTestUtils.csvConnection; import static org.elasticsearch.xpack.qa.sql.jdbc.CsvTestUtils.executeCsvQuery; @@ -57,13 +56,4 @@ protected final void doTest() throws Throwable { assertResults(expected, elasticResults); } } - - // make sure ES uses UTC (otherwise JDBC driver picks up the JVM timezone per spec/convention) - @Override - protected Properties connectionProperties() { - Properties connectionProperties = new Properties(); - connectionProperties.setProperty(JdbcConfiguration.TIME_ZONE, "UTC"); - return connectionProperties; - } - } diff --git a/x-pack/qa/sql/src/main/java/org/elasticsearch/xpack/qa/sql/jdbc/CsvTestUtils.java b/x-pack/qa/sql/src/main/java/org/elasticsearch/xpack/qa/sql/jdbc/CsvTestUtils.java index fbbc2285ed123..ad26db3104758 100644 --- a/x-pack/qa/sql/src/main/java/org/elasticsearch/xpack/qa/sql/jdbc/CsvTestUtils.java +++ b/x-pack/qa/sql/src/main/java/org/elasticsearch/xpack/qa/sql/jdbc/CsvTestUtils.java @@ -190,7 +190,7 @@ public Object parse(String line) { } public static class CsvTestCase { - String query; - String expectedResults; + public String query; + public String expectedResults; } -} +} \ No newline at end of file diff --git a/x-pack/qa/sql/src/main/java/org/elasticsearch/xpack/qa/sql/jdbc/DataLoader.java b/x-pack/qa/sql/src/main/java/org/elasticsearch/xpack/qa/sql/jdbc/DataLoader.java index 9137e2028aa50..3a697e86ba287 100644 --- a/x-pack/qa/sql/src/main/java/org/elasticsearch/xpack/qa/sql/jdbc/DataLoader.java +++ b/x-pack/qa/sql/src/main/java/org/elasticsearch/xpack/qa/sql/jdbc/DataLoader.java @@ -35,18 +35,28 @@ public class DataLoader { public static void main(String[] args) throws Exception { try (RestClient client = RestClient.builder(new HttpHost("localhost", 9200)).build()) { - loadDatasetIntoEs(client); + loadEmpDatasetIntoEs(client); Loggers.getLogger(DataLoader.class).info("Data loaded"); } } protected static void loadDatasetIntoEs(RestClient client) throws Exception { - loadDatasetIntoEs(client, "test_emp"); - loadDatasetIntoEs(client, "test_emp_copy"); + loadEmpDatasetIntoEs(client); + } + + protected static void loadEmpDatasetIntoEs(RestClient client) throws Exception { + loadEmpDatasetIntoEs(client, "test_emp"); + loadEmpDatasetIntoEs(client, "test_emp_copy"); makeAlias(client, "test_alias", "test_emp", "test_emp_copy"); makeAlias(client, "test_alias_emp", "test_emp", "test_emp_copy"); } + public static void loadDocsDatasetIntoEs(RestClient client) throws Exception { + loadEmpDatasetIntoEs(client, "emp"); + loadLibDatasetIntoEs(client, "library"); + makeAlias(client, "employees", "emp"); + } + private static void createString(String name, XContentBuilder builder) throws Exception { builder.startObject(name).field("type", "text") .startObject("fields") @@ -54,7 +64,8 @@ private static void createString(String name, XContentBuilder builder) throws Ex .endObject() .endObject(); } - protected static void loadDatasetIntoEs(RestClient client, String index) throws Exception { + + protected static void loadEmpDatasetIntoEs(RestClient client, String index) throws Exception { XContentBuilder createIndex = JsonXContent.contentBuilder().startObject(); createIndex.startObject("settings"); { @@ -154,6 +165,50 @@ protected static void loadDatasetIntoEs(RestClient client, String index) throws new StringEntity(bulk.toString(), ContentType.APPLICATION_JSON)); } + protected static void loadLibDatasetIntoEs(RestClient client, String index) throws Exception { + XContentBuilder createIndex = JsonXContent.contentBuilder().startObject(); + createIndex.startObject("settings"); + { + createIndex.field("number_of_shards", 1); + createIndex.field("number_of_replicas", 1); + } + createIndex.endObject(); + createIndex.startObject("mappings"); + { + createIndex.startObject("book"); + { + createIndex.startObject("properties"); + { + createString("name", createIndex); + createString("author", createIndex); + createIndex.startObject("release_date").field("type", "date").endObject(); + createIndex.startObject("page_count").field("type", "short").endObject(); + } + createIndex.endObject(); + } + createIndex.endObject(); + } + createIndex.endObject().endObject(); + client.performRequest("PUT", "/" + index, emptyMap(), + new StringEntity(Strings.toString(createIndex), ContentType.APPLICATION_JSON)); + + StringBuilder bulk = new StringBuilder(); + csvToLines("library", (titles, fields) -> { + bulk.append("{\"index\":{\"_id\":\"" + fields.get(0) + "\"}}\n"); + bulk.append("{"); + for (int f = 0; f < titles.size(); f++) { + if (f > 0) { + bulk.append(","); + } + bulk.append('"').append(titles.get(f)).append("\":\"").append(fields.get(f)).append('"'); + } + bulk.append("}\n"); + }); + client.performRequest("POST", "/" + index + "/book/_bulk", singletonMap("refresh", "true"), + new StringEntity(bulk.toString(), ContentType.APPLICATION_JSON)); + + } + protected static void makeAlias(RestClient client, String aliasName, String... indices) throws Exception { for (String index : indices) { client.performRequest("POST", "/" + index + "/_alias/" + aliasName); diff --git a/x-pack/qa/sql/src/main/java/org/elasticsearch/xpack/qa/sql/jdbc/DebugCsvSpec.java b/x-pack/qa/sql/src/main/java/org/elasticsearch/xpack/qa/sql/jdbc/DebugCsvSpec.java index c0d3db026d8bd..c4d25f4311327 100644 --- a/x-pack/qa/sql/src/main/java/org/elasticsearch/xpack/qa/sql/jdbc/DebugCsvSpec.java +++ b/x-pack/qa/sql/src/main/java/org/elasticsearch/xpack/qa/sql/jdbc/DebugCsvSpec.java @@ -10,13 +10,11 @@ import org.apache.logging.log4j.Logger; import org.elasticsearch.test.junit.annotations.TestLogging; import org.elasticsearch.xpack.qa.sql.jdbc.CsvTestUtils.CsvTestCase; -import org.elasticsearch.xpack.sql.jdbc.jdbc.JdbcConfiguration; import java.sql.Connection; import java.sql.ResultSet; import java.sql.SQLException; import java.util.List; -import java.util.Properties; import static org.elasticsearch.xpack.qa.sql.jdbc.CsvTestUtils.csvConnection; import static org.elasticsearch.xpack.qa.sql.jdbc.CsvTestUtils.executeCsvQuery; @@ -65,12 +63,4 @@ protected final void doTest() throws Throwable { assertResults(expected, elasticResults); } } - - // make sure ES uses UTC (otherwise JDBC driver picks up the JVM timezone per spec/convention) - @Override - protected Properties connectionProperties() { - Properties connectionProperties = new Properties(); - connectionProperties.setProperty(JdbcConfiguration.TIME_ZONE, "UTC"); - return connectionProperties; - } } \ No newline at end of file diff --git a/x-pack/qa/sql/src/main/java/org/elasticsearch/xpack/qa/sql/jdbc/JdbcAssert.java b/x-pack/qa/sql/src/main/java/org/elasticsearch/xpack/qa/sql/jdbc/JdbcAssert.java index 801f40639fad1..47f531ebd1f9b 100644 --- a/x-pack/qa/sql/src/main/java/org/elasticsearch/xpack/qa/sql/jdbc/JdbcAssert.java +++ b/x-pack/qa/sql/src/main/java/org/elasticsearch/xpack/qa/sql/jdbc/JdbcAssert.java @@ -20,10 +20,20 @@ import java.util.TimeZone; import static java.lang.String.format; +import static java.sql.Types.BIGINT; +import static java.sql.Types.DOUBLE; +import static java.sql.Types.FLOAT; +import static java.sql.Types.INTEGER; +import static java.sql.Types.REAL; +import static java.sql.Types.SMALLINT; +import static java.sql.Types.TINYINT; import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertTrue; import static org.junit.Assert.fail; +/** + * Utility class for doing JUnit-style asserts over JDBC. + */ public class JdbcAssert { private static final Calendar UTC_CALENDAR = Calendar.getInstance(TimeZone.getTimeZone("UTC"), Locale.ROOT); @@ -32,14 +42,29 @@ public static void assertResultSets(ResultSet expected, ResultSet actual) throws } public static void assertResultSets(ResultSet expected, ResultSet actual, Logger logger) throws SQLException { + assertResultSets(expected, actual, logger, false); + } + + /** + * Assert the given result sets, potentially in a lenient way. + * When lenient is specified, the type comparison of a column is widden to reach a common, compatible ground. + * This means promoting integer types to long and floating types to double and comparing their values. + * For example in a non-lenient, strict case a comparison between an int and a tinyint would fail, with lenient it will succeed as + * long as the actual value is the same. + */ + public static void assertResultSets(ResultSet expected, ResultSet actual, Logger logger, boolean lenient) throws SQLException { try (ResultSet ex = expected; ResultSet ac = actual) { - assertResultSetMetadata(ex, ac, logger); - assertResultSetData(ex, ac, logger); + assertResultSetMetadata(ex, ac, logger, lenient); + assertResultSetData(ex, ac, logger, lenient); } } - // metadata doesn't consume a ResultSet thus it shouldn't close it public static void assertResultSetMetadata(ResultSet expected, ResultSet actual, Logger logger) throws SQLException { + assertResultSetMetadata(expected, actual, logger, false); + } + + // metadata doesn't consume a ResultSet thus it shouldn't close it + public static void assertResultSetMetadata(ResultSet expected, ResultSet actual, Logger logger, boolean lenient) throws SQLException { ResultSetMetaData expectedMeta = expected.getMetaData(); ResultSetMetaData actualMeta = actual.getMetaData(); @@ -81,8 +106,8 @@ public static void assertResultSetMetadata(ResultSet expected, ResultSet actual, } // use the type not the name (timestamp with timezone returns spaces for example) - int expectedType = expectedMeta.getColumnType(column); - int actualType = actualMeta.getColumnType(column); + int expectedType = typeOf(expectedMeta.getColumnType(column), lenient); + int actualType = typeOf(actualMeta.getColumnType(column), lenient); // since H2 cannot use a fixed timezone, the data is stored in UTC (and thus with timezone) if (expectedType == Types.TIMESTAMP_WITH_TIMEZONE) { @@ -92,6 +117,7 @@ public static void assertResultSetMetadata(ResultSet expected, ResultSet actual, if (expectedType == Types.FLOAT && expected instanceof CsvResultSet) { expectedType = Types.REAL; } + // when lenient is used, an int is equivalent to a short, etc... assertEquals("Different column type for column [" + expectedName + "] (" + JDBCType.valueOf(expectedType) + " != " + JDBCType.valueOf(actualType) + ")", expectedType, actualType); } @@ -99,12 +125,16 @@ public static void assertResultSetMetadata(ResultSet expected, ResultSet actual, // The ResultSet is consumed and thus it should be closed public static void assertResultSetData(ResultSet expected, ResultSet actual, Logger logger) throws SQLException { + assertResultSetData(expected, actual, logger, false); + } + + public static void assertResultSetData(ResultSet expected, ResultSet actual, Logger logger, boolean lenient) throws SQLException { try (ResultSet ex = expected; ResultSet ac = actual) { - doAssertResultSetData(ex, ac, logger); + doAssertResultSetData(ex, ac, logger, lenient); } } - private static void doAssertResultSetData(ResultSet expected, ResultSet actual, Logger logger) throws SQLException { + private static void doAssertResultSetData(ResultSet expected, ResultSet actual, Logger logger, boolean lenient) throws SQLException { ResultSetMetaData metaData = expected.getMetaData(); int columns = metaData.getColumnCount(); @@ -118,10 +148,33 @@ private static void doAssertResultSetData(ResultSet expected, ResultSet actual, } for (int column = 1; column <= columns; column++) { - Object expectedObject = expected.getObject(column); - Object actualObject = actual.getObject(column); - int type = metaData.getColumnType(column); + Class expectedColumnClass = null; + try { + String columnClassName = metaData.getColumnClassName(column); + + // fix for CSV which returns the shortName not fully-qualified name + if (!columnClassName.contains(".")) { + switch (columnClassName) { + case "Timestamp": + columnClassName = "java.sql.Timestamp"; + break; + case "Int": + columnClassName = "java.lang.Integer"; + break; + default: + columnClassName = "java.lang." + columnClassName; + break; + } + } + + expectedColumnClass = Class.forName(columnClassName); + } catch (ClassNotFoundException cnfe) { + throw new SQLException(cnfe); + } + + Object expectedObject = expected.getObject(column); + Object actualObject = lenient ? actual.getObject(column, expectedColumnClass) : actual.getObject(column); String msg = format(Locale.ROOT, "Different result for column [" + metaData.getColumnName(column) + "], " + "entry [" + (count + 1) + "]"); @@ -161,4 +214,20 @@ else if (type == Types.DOUBLE) { } } + /** + * Returns the value of the given type either in a lenient fashion (widened) or strict. + */ + private static int typeOf(int columnType, boolean lenient) { + if (lenient) { + // integer upcast to long + if (columnType == TINYINT || columnType == SMALLINT || columnType == INTEGER || columnType == BIGINT) { + return BIGINT; + } + if (columnType == FLOAT || columnType == REAL || columnType == DOUBLE) { + return REAL; + } + } + + return columnType; + } } \ No newline at end of file diff --git a/x-pack/qa/sql/src/main/java/org/elasticsearch/xpack/qa/sql/jdbc/JdbcTestUtils.java b/x-pack/qa/sql/src/main/java/org/elasticsearch/xpack/qa/sql/jdbc/JdbcTestUtils.java index 5062525f2b31e..1fa4e3fb9e062 100644 --- a/x-pack/qa/sql/src/main/java/org/elasticsearch/xpack/qa/sql/jdbc/JdbcTestUtils.java +++ b/x-pack/qa/sql/src/main/java/org/elasticsearch/xpack/qa/sql/jdbc/JdbcTestUtils.java @@ -6,10 +6,17 @@ package org.elasticsearch.xpack.qa.sql.jdbc; import org.apache.logging.log4j.Logger; +import org.elasticsearch.xpack.sql.plugin.CliFormatter; +import org.elasticsearch.xpack.sql.plugin.ColumnInfo; +import org.elasticsearch.xpack.sql.plugin.SqlQueryResponse; +import java.sql.JDBCType; import java.sql.ResultSet; import java.sql.ResultSetMetaData; import java.sql.SQLException; +import java.sql.Timestamp; +import java.util.ArrayList; +import java.util.List; public abstract class JdbcTestUtils { @@ -96,4 +103,37 @@ private static StringBuilder trimOrPad(StringBuilder buffer) { } return buffer; } + + public static void logLikeCLI(ResultSet rs, Logger logger) throws SQLException { + ResultSetMetaData metaData = rs.getMetaData(); + int columns = metaData.getColumnCount(); + + List cols = new ArrayList<>(columns); + + for (int i = 1; i <= columns; i++) { + cols.add(new ColumnInfo(metaData.getTableName(i), metaData.getColumnName(i), metaData.getColumnTypeName(i), + JDBCType.valueOf(metaData.getColumnType(i)), metaData.getColumnDisplaySize(i))); + } + + + List> data = new ArrayList<>(); + + while (rs.next()) { + List entry = new ArrayList<>(columns); + for (int i = 1; i <= columns; i++) { + Object value = rs.getObject(i); + // timestamp to string is similar but not ISO8601 - fix it + if (value instanceof Timestamp) { + Timestamp ts = (Timestamp) value; + value = ts.toInstant().toString(); + } + entry.add(value); + } + data.add(entry); + } + + SqlQueryResponse response = new SqlQueryResponse(null, cols, data); + CliFormatter formatter = new CliFormatter(response); + logger.info("\n" + formatter.formatWithHeader(response)); + } } \ No newline at end of file diff --git a/x-pack/qa/sql/src/main/java/org/elasticsearch/xpack/qa/sql/jdbc/SpecBaseIntegrationTestCase.java b/x-pack/qa/sql/src/main/java/org/elasticsearch/xpack/qa/sql/jdbc/SpecBaseIntegrationTestCase.java index 5a589f94d28d4..968fdb7ea458d 100644 --- a/x-pack/qa/sql/src/main/java/org/elasticsearch/xpack/qa/sql/jdbc/SpecBaseIntegrationTestCase.java +++ b/x-pack/qa/sql/src/main/java/org/elasticsearch/xpack/qa/sql/jdbc/SpecBaseIntegrationTestCase.java @@ -7,8 +7,10 @@ import org.apache.logging.log4j.Logger; import org.elasticsearch.client.ResponseException; +import org.elasticsearch.client.RestClient; import org.elasticsearch.common.Strings; import org.elasticsearch.common.SuppressForbidden; +import org.elasticsearch.xpack.sql.jdbc.jdbc.JdbcConfiguration; import org.junit.AfterClass; import org.junit.Before; @@ -27,6 +29,7 @@ import java.util.List; import java.util.Map; import java.util.Objects; +import java.util.Properties; /** * Tests that compare the Elasticsearch JDBC client to some other JDBC client @@ -49,11 +52,19 @@ public SpecBaseIntegrationTestCase(String fileName, String groupName, String tes @Before public void setupTestDataIfNeeded() throws Exception { - if (client().performRequest("HEAD", "/test_emp").getStatusLine().getStatusCode() == 404) { - DataLoader.loadDatasetIntoEs(client()); + if (client().performRequest("HEAD", "/" + indexName()).getStatusLine().getStatusCode() == 404) { + loadDataset(client()); } } + protected String indexName() { + return "test_emp"; + } + + protected void loadDataset(RestClient client) throws Exception { + DataLoader.loadEmpDatasetIntoEs(client); + } + @Override protected boolean preserveIndicesUponCompletion() { return true; @@ -94,6 +105,14 @@ protected ResultSet executeJdbcQuery(Connection con, String query) throws SQLExc return statement.executeQuery(query); } + // TODO: use UTC for now until deciding on a strategy for handling date extraction + @Override + protected Properties connectionProperties() { + Properties connectionProperties = new Properties(); + connectionProperties.setProperty(JdbcConfiguration.TIME_ZONE, "UTC"); + return connectionProperties; + } + protected boolean logEsResultSet() { return false; } diff --git a/x-pack/qa/sql/src/main/java/org/elasticsearch/xpack/qa/sql/jdbc/SqlSpecTestCase.java b/x-pack/qa/sql/src/main/java/org/elasticsearch/xpack/qa/sql/jdbc/SqlSpecTestCase.java index f1bcef6f750fc..3b5cae742d34b 100644 --- a/x-pack/qa/sql/src/main/java/org/elasticsearch/xpack/qa/sql/jdbc/SqlSpecTestCase.java +++ b/x-pack/qa/sql/src/main/java/org/elasticsearch/xpack/qa/sql/jdbc/SqlSpecTestCase.java @@ -7,14 +7,12 @@ import com.carrotsearch.randomizedtesting.annotations.ParametersFactory; -import org.elasticsearch.xpack.sql.jdbc.jdbc.JdbcConfiguration; import org.junit.ClassRule; import java.sql.Connection; import java.sql.ResultSet; import java.util.ArrayList; import java.util.List; -import java.util.Properties; /** * Tests comparing sql queries executed against our jdbc client @@ -67,12 +65,4 @@ protected final void doTest() throws Throwable { assertResults(expected, elasticResults); } } - - // TODO: use UTC for now until deciding on a strategy for handling date extraction - @Override - protected Properties connectionProperties() { - Properties connectionProperties = new Properties(); - connectionProperties.setProperty(JdbcConfiguration.TIME_ZONE, "UTC"); - return connectionProperties; - } } diff --git a/x-pack/qa/sql/src/main/resources/docs.csv-spec b/x-pack/qa/sql/src/main/resources/docs.csv-spec new file mode 100644 index 0000000000000..8bf74bc4a2f89 --- /dev/null +++ b/x-pack/qa/sql/src/main/resources/docs.csv-spec @@ -0,0 +1,639 @@ +// +// CSV spec used by the docs +// + +/////////////////////////////// +// +// Describe table +// +/////////////////////////////// + +describeTable +// tag::describeTable +DESCRIBE emp; + + column | type +--------------------+--------------- +birth_date |TIMESTAMP +dep |STRUCT +dep.dep_id |VARCHAR +dep.dep_name |VARCHAR +dep.dep_name.keyword|VARCHAR +dep.from_date |TIMESTAMP +dep.to_date |TIMESTAMP +emp_no |INTEGER +first_name |VARCHAR +first_name.keyword |VARCHAR +gender |VARCHAR +hire_date |TIMESTAMP +languages |TINYINT +last_name |VARCHAR +last_name.keyword |VARCHAR +salary |INTEGER + +// end::describeTable +; + +//describeTableAlias +// tag::describeTableAlias +//DESCRIBE employee; + +// column | type +//---------------+--------------- + +// end::describeTableAlias +//; + +// +// Show columns +// +showColumns +// tag::showColumns +SHOW COLUMNS IN emp; + + column | type +--------------------+--------------- +birth_date |TIMESTAMP +dep |STRUCT +dep.dep_id |VARCHAR +dep.dep_name |VARCHAR +dep.dep_name.keyword|VARCHAR +dep.from_date |TIMESTAMP +dep.to_date |TIMESTAMP +emp_no |INTEGER +first_name |VARCHAR +first_name.keyword |VARCHAR +gender |VARCHAR +hire_date |TIMESTAMP +languages |TINYINT +last_name |VARCHAR +last_name.keyword |VARCHAR +salary |INTEGER + +// end::showColumns +; + +//showColumnsInAlias +// tag::showColumnsInAlias +//SHOW COLUMNS FROM employee; + +// column | type +//---------------+--------------- + +// end::showColumnsInAlias +//; + +/////////////////////////////// +// +// Show Tables +// +/////////////////////////////// + +showTables +// tag::showTables +SHOW TABLES; + + name | type +---------------+--------------- +emp |BASE TABLE +employees |ALIAS +library |BASE TABLE + +// end::showTables +; + +showTablesLikeExact +// tag::showTablesLikeExact +SHOW TABLES LIKE 'emp'; + + name | type +---------------+--------------- +emp |BASE TABLE + +// end::showTablesLikeExact +; + +showTablesLikeWildcard +// tag::showTablesLikeWildcard +SHOW TABLES LIKE 'emp%'; + + name | type +---------------+--------------- +emp |BASE TABLE +employees |ALIAS + +// end::showTablesLikeWildcard +; + + +showTablesLikeOneChar +// tag::showTablesLikeOneChar +SHOW TABLES LIKE 'em_'; + + name | type +---------------+--------------- +emp |BASE TABLE + +// end::showTablesLikeOneChar +; + +showTablesLikeMixed +// tag::showTablesLikeMixed +SHOW TABLES LIKE '%em_'; + + name | type +---------------+--------------- +emp |BASE TABLE + +// end::showTablesLikeMixed +; + +/////////////////////////////// +// +// Show Functions +// +/////////////////////////////// + +showFunctions +// tag::showFunctions +SHOW FUNCTIONS; + + name | type +----------------+--------------- +AVG |AGGREGATE +COUNT |AGGREGATE +MAX |AGGREGATE +MIN |AGGREGATE +SUM |AGGREGATE +STDDEV_POP |AGGREGATE +VAR_POP |AGGREGATE +PERCENTILE |AGGREGATE +PERCENTILE_RANK |AGGREGATE +SUM_OF_SQUARES |AGGREGATE +SKEWNESS |AGGREGATE +KURTOSIS |AGGREGATE +DAY_OF_MONTH |SCALAR +DAY |SCALAR +DOM |SCALAR +DAY_OF_WEEK |SCALAR +DOW |SCALAR +DAY_OF_YEAR |SCALAR +DOY |SCALAR +HOUR_OF_DAY |SCALAR +HOUR |SCALAR +MINUTE_OF_DAY |SCALAR +MINUTE_OF_HOUR |SCALAR +MINUTE |SCALAR +SECOND_OF_MINUTE|SCALAR +SECOND |SCALAR +MONTH_OF_YEAR |SCALAR +MONTH |SCALAR +YEAR |SCALAR +WEEK_OF_YEAR |SCALAR +WEEK |SCALAR +ABS |SCALAR +ACOS |SCALAR +ASIN |SCALAR +ATAN |SCALAR +ATAN2 |SCALAR +CBRT |SCALAR +CEIL |SCALAR +CEILING |SCALAR +COS |SCALAR +COSH |SCALAR +COT |SCALAR +DEGREES |SCALAR +E |SCALAR +EXP |SCALAR +EXPM1 |SCALAR +FLOOR |SCALAR +LOG |SCALAR +LOG10 |SCALAR +MOD |SCALAR +PI |SCALAR +POWER |SCALAR +RADIANS |SCALAR +RANDOM |SCALAR +RAND |SCALAR +ROUND |SCALAR +SIGN |SCALAR +SIGNUM |SCALAR +SIN |SCALAR +SINH |SCALAR +SQRT |SCALAR +TAN |SCALAR +SCORE |SCORE + +// end::showFunctions +; + +showFunctionsLikeExact +// tag::showFunctionsLikeExact +SHOW FUNCTIONS LIKE 'ABS'; + + name | type +---------------+--------------- +ABS |SCALAR + +// end::showFunctionsLikeExact +; + +showFunctionsLikeWildcard +// tag::showFunctionsLikeWildcard +SHOW FUNCTIONS LIKE 'A%'; + + name | type +---------------+--------------- +AVG |AGGREGATE +ABS |SCALAR +ACOS |SCALAR +ASIN |SCALAR +ATAN |SCALAR +ATAN2 |SCALAR +// end::showFunctionsLikeWildcard +; + +showFunctionsLikeChar +// tag::showFunctionsLikeChar +SHOW FUNCTIONS LIKE 'A__'; + + name | type +---------------+--------------- +AVG |AGGREGATE +ABS |SCALAR +// end::showFunctionsLikeChar +; + +showFunctionsWithPattern +// tag::showFunctionsWithPattern +SHOW FUNCTIONS '%DAY%'; + + name | type +---------------+--------------- +DAY_OF_MONTH |SCALAR +DAY |SCALAR +DAY_OF_WEEK |SCALAR +DAY_OF_YEAR |SCALAR +HOUR_OF_DAY |SCALAR +MINUTE_OF_DAY |SCALAR + +// end::showFunctionsWithPattern +; + +/////////////////////////////// +// +// Select +// +/////////////////////////////// + +selectColumnAlias +// tag::selectColumnAlias +SELECT 1 + 1 AS result + + result +--------------- +2 + +// end::selectColumnAlias +; + +selectInline +// tag::selectInline +SELECT 1 + 1; + + (1 + 1) +--------------- +2 + +// end::selectInline +; + +selectColumn +// tag::selectColumn +SELECT emp_no FROM emp LIMIT 1; + + emp_no +--------------- +10001 + +// end::selectColumn +; + +selectQualifiedColumn +// tag::selectQualifiedColumn +SELECT emp.emp_no FROM emp LIMIT 1; + + emp_no +--------------- +10001 + +// end::selectQualifiedColumn +; + + +wildcardWithOrder +// tag::wildcardWithOrder +SELECT * FROM emp LIMIT 1; + + birth_date | emp_no | first_name | gender | hire_date | languages | last_name | salary +--------------------+---------------+---------------+---------------+--------------------+---------------+---------------+--------------- +1953-09-02T00:00:00Z|10001 |Georgi |M |1986-06-26T00:00:00Z|2 |Facello |57305 + +// end::wildcardWithOrder +; + +fromTable +// tag::fromTable +SELECT * FROM emp LIMIT 1; + + birth_date | emp_no | first_name | gender | hire_date | languages | last_name | salary +--------------------+---------------+---------------+---------------+--------------------+---------------+---------------+--------------- +1953-09-02T00:00:00Z|10001 |Georgi |M |1986-06-26T00:00:00Z|2 |Facello |57305 + + +// end::fromTable +; + +fromTableQuoted +// tag::fromTableQuoted +SELECT * FROM "emp" LIMIT 1; + + birth_date | emp_no | first_name | gender | hire_date | languages | last_name | salary +--------------------+---------------+---------------+---------------+--------------------+---------------+---------------+--------------- +1953-09-02T00:00:00Z|10001 |Georgi |M |1986-06-26T00:00:00Z|2 |Facello |57305 + +// end::fromTableQuoted +; + +fromTableQuoted +// tag::fromTablePatternQuoted +SELECT emp_no FROM "e*p" LIMIT 1; + + emp_no +--------------- +10001 + +// end::fromTablePatternQuoted +; + +fromTableAlias +// tag::fromTableAlias +SELECT e.emp_no FROM emp AS e LIMIT 1; + + emp_no +------------- +10001 + +// end::fromTableAlias +; + +basicWhere +// tag::basicWhere +SELECT last_name FROM emp WHERE emp_no = 10001; + + last_name +--------------- +Facello + +// end::basicWhere +; + +/////////////////////////////// +// +// Group By +// +/////////////////////////////// + +groupByColumn +// tag::groupByColumn +SELECT gender AS g FROM emp GROUP BY gender; + + g +--------------- +F +M + +// end::groupByColumn +; + +groupByOrdinal +// tag::groupByOrdinal +SELECT gender FROM emp GROUP BY 1; + + gender +--------------- +F +M + +// end::groupByOrdinal +; + +groupByAlias +// tag::groupByAlias +SELECT gender AS g FROM emp GROUP BY g; + + g +--------------- +F +M + +// end::groupByAlias +; + +groupByExpression +// tag::groupByExpression +SELECT languages + 1 AS l FROM emp GROUP BY l; + + l +--------------- +2 +3 +4 +5 +6 + + +// end::groupByExpression +; + +groupByAndAgg +// tag::groupByAndAgg +SELECT gender AS g, COUNT(*) AS c FROM emp GROUP BY gender; + + g | c +---------------+--------------- +F |37 +M |63 + +// end::groupByAndAgg +; + +groupByAndAggExpression +// tag::groupByAndAggExpression +SELECT gender AS g, ROUND(MIN(salary) / 100) AS salary FROM emp GROUP BY gender; + + g | salary +---------------+--------------- +F |260 +M |253 + +// end::groupByAndAggExpression +; + +groupByAndMultipleAggs +// tag::groupByAndMultipleAggs +SELECT gender AS g, KURTOSIS(salary) AS k, SKEWNESS(salary) AS s FROM emp GROUP BY gender; + + g | k | s +---------------+------------------+------------------- +F |1.8427808415250482|0.04517149340491813 +M |2.259327644285826 |0.40268950715550333 + +// end::groupByAndMultipleAggs +; + +groupByImplicitCount +// tag::groupByImplicitCount +SELECT COUNT(*) AS count FROM emp; + + count +--------------- +100 + +// end::groupByImplicitCount +; + +/////////////////////////////// +// +// Having +// +/////////////////////////////// + +groupByHaving +// tag::groupByHaving +SELECT languages AS l, COUNT(*) AS c FROM emp GROUP BY l HAVING c BETWEEN 15 AND 20; + + l | c +---------------+--------------- +1 |16 +2 |20 +4 |18 + +// end::groupByHaving +; + +groupByHavingMultiple +// tag::groupByHavingMultiple +SELECT MIN(salary) AS min, MAX(salary) AS max, MAX(salary) - MIN(salary) AS diff FROM emp GROUP BY languages HAVING diff - max % min > 0 AND AVG(salary) > 30000; + + min | max | diff +---------------+---------------+--------------- +25976 |73717 |47741 +29175 |73578 |44403 +26436 |74999 |48563 +27215 |74572 |47357 +25324 |73851 |48527 + +// end::groupByHavingMultiple +; + +groupByImplicitMultipleAggs +// tag::groupByImplicitMultipleAggs +SELECT MIN(salary) AS min, MAX(salary) AS max, AVG(salary) AS avg, COUNT(*) AS count FROM emp; + + min | max | avg | count +---------------+---------------+---------------+--------------- +25324 |74999 |48248 |100 + +// end::groupByImplicitMultipleAggs +; + +groupByHavingImplicitMatch +// tag::groupByHavingImplicitMatch +SELECT MIN(salary) AS min, MAX(salary) AS max FROM emp HAVING min > 25000; + + min | max +---------------+--------------- +25324 |74999 + +// end::groupByHavingImplicitMatch +; + +//groupByHavingImplicitNoMatch +// tag::groupByHavingImplicitNoMatch +//SELECT MIN(salary) AS min, MAX(salary) AS max FROM emp HAVING max > 75000; + +// min | max +//---------------+--------------- + +// end::groupByHavingImplicitNoMatch +//; + +/////////////////////////////// +// +// Order by +// +/////////////////////////////// + +orderByBasic +// tag::orderByBasic +SELECT * FROM library ORDER BY page_count DESC LIMIT 5; + + author | name | page_count | release_date +-----------------+--------------------+---------------+-------------------- +Peter F. Hamilton|Pandora's Star |768 |2004-03-02T00:00:00Z +Vernor Vinge |A Fire Upon the Deep|613 |1992-06-01T00:00:00Z +Frank Herbert |Dune |604 |1965-06-01T00:00:00Z +Alastair Reynolds|Revelation Space |585 |2000-03-15T00:00:00Z +James S.A. Corey |Leviathan Wakes |561 |2011-06-02T00:00:00Z + + + +// end::orderByBasic +; + +orderByScore +// tag::orderByScore +SELECT SCORE(), * FROM library WHERE match(name, 'dune') ORDER BY SCORE() DESC; + + SCORE() | author | name | page_count | release_date +---------------+---------------+-------------------+---------------+-------------------- +2.288635 |Frank Herbert |Dune |604 |1965-06-01T00:00:00Z +1.8893257 |Frank Herbert |Dune Messiah |331 |1969-10-15T00:00:00Z +1.6086555 |Frank Herbert |Children of Dune |408 |1976-04-21T00:00:00Z +1.4005898 |Frank Herbert |God Emperor of Dune|454 |1981-05-28T00:00:00Z + +// end::orderByScore +; + +orderByScoreWithMatch +// tag::orderByScoreWithMatch +SELECT SCORE(), * FROM library WHERE match(name, 'dune') ORDER BY page_count DESC; + + SCORE() | author | name | page_count | release_date +---------------+---------------+-------------------+---------------+-------------------- +2.288635 |Frank Herbert |Dune |604 |1965-06-01T00:00:00Z +1.4005898 |Frank Herbert |God Emperor of Dune|454 |1981-05-28T00:00:00Z +1.6086555 |Frank Herbert |Children of Dune |408 |1976-04-21T00:00:00Z +1.8893257 |Frank Herbert |Dune Messiah |331 |1969-10-15T00:00:00Z + +// end::orderByScoreWithMatch +; + + +/////////////////////////////// +// +// Limit +// +/////////////////////////////// + +limitBasic +// tag::limitBasic +SELECT first_name, last_name, emp_no FROM emp LIMIT 1; + + first_name | last_name | emp_no +---------------+---------------+--------------- +Georgi |Facello |10001 + +// end::limitBasic +; diff --git a/x-pack/qa/sql/src/main/resources/library.csv b/x-pack/qa/sql/src/main/resources/library.csv new file mode 100644 index 0000000000000..a93be21abe63e --- /dev/null +++ b/x-pack/qa/sql/src/main/resources/library.csv @@ -0,0 +1,25 @@ +name,author,release_date,page_count +Leviathan Wakes,James S.A. Corey,2011-06-02T00:00:00Z,561 +Hyperion,Dan Simmons,1989-05-26T00:00:00Z,482 +Dune,Frank Herbert,1965-06-01T00:00:00Z,604 +Dune Messiah,Frank Herbert,1969-10-15T00:00:00Z,331 +Children of Dune,Frank Herbert,1976-04-21T00:00:00Z,408 +God Emperor of Dune,Frank Herbert,1981-05-28T00:00:00Z,454 +Consider Phlebas,Iain M. Banks,1987-04-23T00:00:00Z,471 +Pandora's Star,Peter F. Hamilton,2004-03-02T00:00:00Z,768 +Revelation Space,Alastair Reynolds,2000-03-15T00:00:00Z,585 +A Fire Upon the Deep,Vernor Vinge,1992-06-01T00:00:00Z,613 +Ender's Game,Orson Scott Card,1985-06-01T00:00:00Z,324 +1984,George Orwell,1985-06-01T00:00:00Z,328 +Fahrenheit 451,Ray Bradbury,1953-10-15T00:00:00Z,227 +Brave New World,Aldous Huxley,1932-06-01T00:00:00Z,268 +Foundation,Isaac Asimov,1951-06-01T00:00:00Z,224 +The Giver,Lois Lowry,1993-04-26T00:00:00Z,208 +Slaughterhouse-Five,Kurt Vonnegut,1969-06-01T00:00:00Z,275 +The Hitchhiker's Guide to the Galaxy,Douglas Adams,1979-10-12T00:00:00Z,180 +Snow Crash,Neal Stephenson,1992-06-01T00:00:00Z,470 +Neuromancer,William Gibson,1984-07-01T00:00:00Z,271 +The Handmaid's Tale,Margaret Atwood,1985-06-01T00:00:00Z,311 +Starship Troopers,Robert A. Heinlein,1959-12-01T00:00:00Z,335 +The Left Hand of Darkness,Ursula K. Le Guin,1969-06-01T00:00:00Z,304 +The Moon is a Harsh Mistress,Robert A. Heinlein,1966-04-01T00:00:00Z,288 diff --git a/x-pack/qa/sql/src/main/resources/select.sql-spec b/x-pack/qa/sql/src/main/resources/select.sql-spec index 76562a07c86f7..ce57606e35b0c 100644 --- a/x-pack/qa/sql/src/main/resources/select.sql-spec +++ b/x-pack/qa/sql/src/main/resources/select.sql-spec @@ -3,9 +3,7 @@ // wildcardWithOrder -// tag::wildcardWithOrder SELECT * FROM test_emp ORDER BY emp_no; -// end::wildcardWithOrder column SELECT last_name FROM "test_emp" ORDER BY emp_no; columnWithAlias