Add support for UTF-8 feature extension. #722

ulvii · 2018-06-13T21:20:09Z

Support for UTF-8 feature extension that will be introduced in newer versions of SQL Server.

rene-ye · 2018-06-13T21:30:27Z

src/main/java/com/microsoft/sqlserver/jdbc/SQLServerConnection.java

@@ -4419,6 +4441,8 @@ else if (serverMajorVersion >= 9) // Yukon (9.0) --> TDS 7.2 // Prelogin disconn

        len2 = len2 + 1; // add 1 to length becaue of FeatureEx terminator

+        len2 = len2 + writeUTF8SupportFeatureRequest(false, tdsWriter);


This is a little convoluted, the function basically always returns "5". It also has a Boolean to turn on/off the actual writing. Shouldn't we remove the length calculation from this function and just have a static variable which equals 5? Can also remove the Boolean parameter.

This is done to keep the same structure for all feature extension requests, in case we calculate the length differently in the future. Take a look at writeFedAuthFeatureRequest, writeAEFeatureRequest, writeDataClassificationFeatureRequest, The methods are implemented the same way in .NET driver too.

It makes sense with fedAuth because the length is actually variable. For the others, it seems there's a set value(6 for AE, 5 for UTF8, 6 for data classification). I guess it's fine if we're consistent, but I'm just saying that other than the fedAuth function, the others don't really make a lot of sense.

rene-ye · 2018-06-13T21:31:09Z

src/main/java/com/microsoft/sqlserver/jdbc/SQLServerConnection.java

@@ -4598,6 +4622,8 @@ else if (serverMajorVersion >= 9) // Yukon (9.0) --> TDS 7.2 // Prelogin disconn
            writeFedAuthFeatureRequest(true, tdsWriter, fedAuthFeatureExtensionData);
        }

+        writeUTF8SupportFeatureRequest(true, tdsWriter);


Length is being "calculated" and returned, but we don't use the value. Seems like we're always using half of the functionalities.

yes I've never liked this. It's not really "calculated" at all the whole thing is just hardcoded

cheenamalhotra · 2018-06-13T21:34:02Z

src/test/java/com/microsoft/sqlserver/jdbc/unit/UTF8SupportTest.java

+    public static void setUp() throws ClassNotFoundException, SQLException {
+        databaseName = RandomUtil.getIdentifier("UTF8Database");
+        tableName = AbstractSQLGenerator.escapeIdentifier(RandomUtil.getIdentifier("RequestBoundaryTable"));
+        connection = PrepUtil.getConnection(getConfiguredProperty("mssql_jdbc_test_connection_properties"));


who closes this connection?

…a database

codecov-io · 2018-06-13T22:12:22Z

Codecov Report

Merging #722 into dev will decrease coverage by 0.08%.
The diff coverage is 50%.

@@             Coverage Diff              @@
##                dev     #722      +/-   ##
============================================
- Coverage     48.12%   48.03%   -0.09%     
+ Complexity     2629     2626       -3     
============================================
  Files           113      113              
  Lines         26631    26648      +17     
  Branches       4472     4477       +5     
============================================
- Hits          12816    12801      -15     
- Misses        11695    11721      +26     
- Partials       2120     2126       +6

Flag	Coverage Δ	Complexity Δ
#JDBC42	`47.52% <50%> (-0.11%)`	`2576 <2> (-4)`
#JDBC43	`47.98% <50%> (-0.01%)`	`2624 <2> (+6)`

Impacted Files	Coverage Δ	Complexity Δ
...om/microsoft/sqlserver/jdbc/SQLServerResource.java	`100% <ø> (ø)`	`4 <0> (ø)`	⬇️
...in/java/com/microsoft/sqlserver/jdbc/IOBuffer.java	`52.68% <0%> (-0.06%)`	`0 <0> (ø)`
...ava/com/microsoft/sqlserver/jdbc/SQLCollation.java	`92.1% <50%> (-0.43%)`	`14 <0> (ø)`
.../microsoft/sqlserver/jdbc/SQLServerConnection.java	`48.01% <53.84%> (+0.02%)`	`323 <2> (+2)`	⬆️
...rc/main/java/com/microsoft/sqlserver/jdbc/DDC.java	`43.84% <0%> (-1.57%)`	`107% <0%> (-1%)`
...c/main/java/com/microsoft/sqlserver/jdbc/Util.java	`60.91% <0%> (-0.44%)`	`88% <0%> (-1%)`
...m/microsoft/sqlserver/jdbc/SQLServerResultSet.java	`33.32% <0%> (-0.4%)`	`247% <0%> (-4%)`
...n/java/com/microsoft/sqlserver/jdbc/Parameter.java	`62.47% <0%> (-0.21%)`	`63% <0%> (ø)`
...om/microsoft/sqlserver/jdbc/SQLServerBulkCopy.java	`52.61% <0%> (-0.13%)`	`241% <0%> (ø)`
...rc/main/java/com/microsoft/sqlserver/jdbc/dtv.java	`63.24% <0%> (-0.12%)`	`0% <0%> (ø)`
... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c34e1a9...72d3b9b. Read the comment docs.

lilgreenbird · 2018-06-15T00:31:54Z

src/main/java/com/microsoft/sqlserver/jdbc/SQLServerConnection.java


+                if (1 > data.length) {
+                    if (connectionlogger.isLoggable(Level.SEVERE)) {
+                        connectionlogger.severe(toString() + " Unknown value for UTF8 support.");


use R_unknownUTF8SupportValue here too?

no, we don't localize the logger messages.

* 623 fix * 623 change stash * Prepared Statement Caching fix for 'handle not found' errors * Fix for PS Caching issue - Calling reset instead of on type def changes * Updated comparison * Change back assert check. * Adding call to removeReference back + Fix for Batch processes intermittent failures. * Removed DBName and made changes to resetPrepStmtHandle method * Check for null handle before proceed * Adding Old Constrcutor back to AKV Implementation * Making baseURL final * Remove unnecessary code. * Use Bulk Copy API for batch insert operation * Parse bug fixing and test added * bug fix + additional tests * change reflection for testing * more test changes * Add parsing logic for -- comment * refactoring * Update snapshot * Bug fix / testing change * Reflect comment change * Feature | AKV Old Constructor changes - Reformatted code + Deprecated old Constructor and added a new constructor with 1 param * Mark computed columns as IS_GENERATEDCOLUMN in the result set returned by getColumns() (#695) * Fix | getColumns() API, changed column name from SS_IS_COMPUTED to IS_AUTOINCREMENT per JDBC specs | issue #600 * Fix | getColumns() API, changed column name from SS_IS_COMPUTED to IS_GENERATEDCOLUMN per JDBC specs | issue #600 * fix issue with redirection * Fix | PS Caching - Remove commented lines * Trigger Appveyor test * Fix | AKV Old Constructor - Calling the other constructor instead. * Fix | Reversed null checks * Resolving alignment problems and comments * Refactor two Bulk files into a common parent * javadoc changes * Applied formatter * fix problem with precision / scale * Fix | Fix some of the Javadoc warnings (#702) * fix issue with setting all to true * Resolved maven build warnings and java warnings regarding deprecated API (#701) * Resolving maven warnings * Removing jreVersion property Does not make sense now that we use final name in the build itself. Only used in 1 place, hard-coding java version for different builds as that's what it represents anyways. * java warnings * make bamoo fixes * resource bundle for junit test error strings (#698) resource bundle for error message strings in junit tests * undo some changes made to SQLServerConnection * apply resource bundling changes * Add support for JDK 10 in both Maven and Gradle (#691) * Feature | Added support for JDK 10 in both Maven and Gradle - builds jre10 jars for the driver, replacing jre9 * JDK 10 | Merge 42 classes to base classes to reduce class redundancy. * JDK 10 | Attempt to run JDK 10 with Appveyor * Remove unwanted space * Updating Travis script to use JDK 10 * Testing without addons * Update script for Jacoco report to build 43 profile * Revert driver changes for 42 compliance - to be added in a separate PR * Revert Test class changes for 42 compliance - to be done in a separate PR * Reformatted code * Add ID to jacoco plugin execution task * Kerberos Constrained Delegation Impersonated Credential Expiry fix (#636) fix for automatic credential discarding * update felix to 3.5.0 * Revised implementation Decided to not dispose user created credentials at all. * Updated flag set location * changes for 6.5.3 preview release * Revert "changes for 6.5.3 preview release" This reverts commit 5c6ccd3. * Changes in preparation for 6.5.3 preview release (#710) * changes for preview release * requested changes * jre version update changes * snapshot updates post release * remove on_dw, and remove redundant fmtonly * formatting * fix for getSchema when using "-" in name * Reformatting + adding more tests * inherit the connection property in statement + fix issue with null / empty string being passed in as values * Request Boundary methods - beginRequest()/endRequest() implementation (#708) * Add | Request Boundary Methods - beginRequest()/endRequest() implementation * Fix | Remove unused import from AbstractTest * Fix | Applying review comments * Fix | Moving RequestBoundaryMethodsTest.java to connection package * added error message in resource file and changed files accordingly * comment revisions * use TestResource * test changes removed finals removed database creation tracking * drop database before creating * replaced dropDBIfExists with Utils function * added try-with-resources nest avoid manually closing statements, and safetly handles resources. * Fixing logic / adding more tests * dont use test database in tests * Change exception handling as per JDBC specs * Add | Add missing license headers (#725) * remove some comments * Enable verify data (#724) Fix to enable data verification in Junit tests. Also addresses intermittent failures with Time/Timestamp where the precision was being inaccurately judged. * Fix | Refactored socket creation to simplify handling of socket creation Refactors socket creation in SocketFinder.findSocket(...) to simplify handling of socket creation. When the host resolves to a single address the driver now defers to getConnectedSocket(...) to create the socket without spawning any threads. This happens regardless of whether we're running on an IBM JDK. Previously the single address case would still use NIO on an IBM JDK. On non-IBM JDKs the driver now handles both IPv4 and IPv6 addresses concurrently with a single shared timeout. Previously hosts that resolved to both types of addresses were allowed half the timeout for socket creation per address type with the resolution performed sequentially. * reflect comments * Add support for UTF-8 feature extension. (#722) * Add | Support for UTF8 changes * changed how logger works, refactored code in SQLServerBulkCommon due to that, changed exception being thrown to BatchUpdateException, added same logic for parsing in executeLargeBatch, and added tests accordingly. * add more tests, make the prepared statement property go away * Feature | Introduce support for "Data Classification Specifications" on fetched resultsets (#709) * Feature | Data Classification Project | Phase 1 (contains temporary skipping 2 bytes) * Feature | Data Classification - Removing extra bytes added before * Feature | Data Classification - Added new test class for testing Data Classification support in the driver * Remove one println * Feature | Repackaged newly added files for Data Classification + improvements in source code * Feature | Changing tokens to bytes instead of int * Feature | Making variables private * Formatted code + dropTable method called from Utils * Feature | Data Classification - Changes as per review comments * Fix | Review comment changes * Change exception codes to follow series * Fix Conflict issue * Added missing Javadocs and headers for all new files * Added getter/setter public for the useBulkCopyForBatchInsert connection property. * Change implementation of child classes a bit * Remove dependencies from tests that are from outside required libraries * also remove hex from DBTable * Fix bamboo problem + refactor test code * Replace all connection and statements with try blocks * change spacing * refactor code * refactoring * Fix | Making driver default compliant to JDBC 4.2 Specs and update ADAL4J dependency to 1.6.0 (#711) * Feature | Added support for JDK 10 in both Maven and Gradle - builds jre10 jars for the driver, replacing jre9 * JDK 10 | Merge 42 classes to base classes to reduce class redundancy. * JDK 10 | Attempt to run JDK 10 with Appveyor * Remove unwanted space * Updating Travis script to use JDK 10 * Testing without addons * Update script for Jacoco report to build 43 profile * Minor fix in formatting to avoid conflicts * moving driver specific functions for SQLServerPreparedStatement * Remove unwanted code + Update Adal4J library dependency * changes for CallableStatement repeptitive delcarations * Remove an extra bracket due to conflict * changes for ISQLServerConnection there are problems with moving all Driver sepcific public methods. SQLServerConnectionPoolProxy also implements this interface and there are many public APIs (such as preparedstmt cacheing stuff) which it doesn't implement, and cannot be moved into the interface at this time. * lambda touch-up should generally stick to 1 line if possible. * changes for ISQLServerDataSource * updates for ISQLServerResultSet * Improvements | Missing interface APIs added + Code improvements * More changes for Interface missing methods * Implemented missing methods in SQLServerConnectionPoolProxy * Removed ISQLServerConnection43 for duplicated method definitions * Added APIs in interface for SQLServerResultSet * More cleanup done * Fix minor issues * Fix test failures and implement Serialization for HashKey * Fix JavaDoc errors and warnigs * More changes for CallableStatement APIs * More changes for Statement and Prepared Statament public APIs * Javadoc fix * More changes for SQL Server Bulk Record interface * Callable Statement missing APIs for Interface * Add missing desciptions * Reverting pom.xml change for this PR * Attempt to resolve conflicts * Remove Interface as not needed. * Added missing docs * Changes for Clob/Blob classes for compliance * Update ADAL4J with latest version * Changes for Data Source classes * Minor fixes to the new changes * Fix for failing tests * More changes for compliance * Add Javadocs and class headers * Fixed Malformed HTML Error in Javadocs * javadoc changes * more javadoc changes to make the abbreviations more clear * fix unchecked warning issue * Change HashKey in the driver to 256 Hash * Add Interface back to SQLServerConnection43 class * Revert "Change HashKey in the driver to 256 Hash" This reverts commit e6bef4e. * Changes for exceptions to throw SQLServerException type * 6.5.4 preview release changelog (#731) Release | Changelog for 6.5.4 preview release (#731) * Fix Conflict issues with master branch

ulvii added 7 commits June 6, 2018 17:15

Add | Support for UTF8 changes - initial implementation

df9ef73

Fix | Fix line endings

fccdcbb

Fix | More line endings fix

b21fe49

Fix | Add newline to IOBuffer.java

26f914c

Merge remote-tracking branch 'upstream/dev' into utf8Support

a5ca684

Add | Adding a unit test for UTF8 support changes

4c5acb4

Fix | Add newline to the end of UTF8SupportTest.java

e95e34c

rene-ye reviewed Jun 13, 2018

View reviewed changes

cheenamalhotra reviewed Jun 13, 2018

View reviewed changes

ulvii added 3 commits June 13, 2018 14:36

Fix | Check if server supports UTF8 feature extension before creting …

5bcd514

…a database

Fix | Close connection at the end of UTF8SupportTest

6c1cad3

Fix | Add license header to UTF8SupportTest.java

8e67788

rene-ye previously approved these changes Jun 13, 2018

View reviewed changes

Fix | Add byte comparison for PreparedStatement fetch in UTF8SupportTest

d3de10c

ulvii dismissed rene-ye’s stale review via d3de10c June 14, 2018 21:12

lilgreenbird reviewed Jun 15, 2018

View reviewed changes

ulvii added this to the 6.5.4 milestone Jun 15, 2018

rene-ye previously approved these changes Jun 15, 2018

View reviewed changes

lilgreenbird previously approved these changes Jun 15, 2018

View reviewed changes

Merge branch 'dev' into utf8Support

d089d8f

cheenamalhotra previously approved these changes Jun 21, 2018

View reviewed changes

Merge branch 'dev' into utf8Support

0dac284

peterbae previously approved these changes Jun 21, 2018

View reviewed changes

Merge branch 'dev' into utf8Support

72d3b9b

cheenamalhotra dismissed stale reviews from peterbae, lilgreenbird, rene-ye, and themself via 72d3b9b June 21, 2018 19:45

cheenamalhotra approved these changes Jun 21, 2018

View reviewed changes

peterbae approved these changes Jun 21, 2018

View reviewed changes

rene-ye approved these changes Jun 22, 2018

View reviewed changes

lilgreenbird approved these changes Jun 22, 2018

View reviewed changes

ulvii merged commit abc9851 into microsoft:dev Jun 22, 2018

ulvii deleted the utf8Support branch June 22, 2018 22:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for UTF-8 feature extension. #722

Add support for UTF-8 feature extension. #722

ulvii commented Jun 13, 2018

rene-ye Jun 13, 2018

ulvii Jun 13, 2018

rene-ye Jun 13, 2018

rene-ye Jun 13, 2018

lilgreenbird Jun 15, 2018

cheenamalhotra Jun 13, 2018

codecov-io commented Jun 13, 2018 •

edited

Loading

lilgreenbird Jun 15, 2018

ulvii Jun 15, 2018

		@@ -4419,6 +4441,8 @@ else if (serverMajorVersion >= 9) // Yukon (9.0) --> TDS 7.2 // Prelogin disconn

		len2 = len2 + 1; // add 1 to length becaue of FeatureEx terminator

		len2 = len2 + writeUTF8SupportFeatureRequest(false, tdsWriter);

Add support for UTF-8 feature extension. #722

Add support for UTF-8 feature extension. #722

Conversation

ulvii commented Jun 13, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-io commented Jun 13, 2018 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-io commented Jun 13, 2018 •

edited

Loading