-
Notifications
You must be signed in to change notification settings - Fork 851
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix: Make warnings available as soon as they are received. #857
Changes from 3 commits
7762346
4fbdf29
e93883c
2b412e4
54b68ec
60aae91
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,35 @@ | ||
/* | ||
* Copyright (c) 2017, PostgreSQL Global Development Group | ||
* See the LICENSE file in the project root for more information. | ||
*/ | ||
|
||
package org.postgresql.util; | ||
|
||
import java.sql.SQLWarning; | ||
|
||
/** | ||
* Wrapper class for SQLWarnings that provides an optimisation to add | ||
* new warnings to the tail of the SQLWarning singly linked list, avoiding Θ(n) insertion time | ||
* of calling #setNextWarning on the head. By encapsulating this into a single object it allows | ||
* users(ie PgStatement) to atomically set and clear the warning chain. | ||
*/ | ||
public class PGSQLWarningWrapper { | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Should this be PSQLWarningWrapper? We already have PSQLWarning, PSQLState, PSQLSavepoint, ... On the other hand, I'm sure this ( There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. How about moving it to There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If package-private is possible, please do that. |
||
|
||
private final SQLWarning firstWarning; | ||
private SQLWarning lastWarning; | ||
|
||
public PGSQLWarningWrapper(SQLWarning warning) { | ||
firstWarning = warning; | ||
lastWarning = warning; | ||
} | ||
|
||
public void addWarning(SQLWarning sqlWarning) { | ||
lastWarning.setNextWarning(sqlWarning); | ||
lastWarning = sqlWarning; | ||
} | ||
|
||
public SQLWarning getFirstWarning() { | ||
return firstWarning; | ||
} | ||
|
||
} |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -24,8 +24,18 @@ | |
import java.sql.PreparedStatement; | ||
import java.sql.ResultSet; | ||
import java.sql.SQLException; | ||
import java.sql.SQLWarning; | ||
import java.sql.Statement; | ||
|
||
import java.util.concurrent.Callable; | ||
import java.util.concurrent.ExecutionException; | ||
import java.util.concurrent.ExecutorService; | ||
import java.util.concurrent.Executors; | ||
import java.util.concurrent.Future; | ||
import java.util.concurrent.ScheduledExecutorService; | ||
import java.util.concurrent.ScheduledFuture; | ||
import java.util.concurrent.TimeUnit; | ||
import java.util.concurrent.TimeoutException; | ||
import java.util.concurrent.atomic.AtomicInteger; | ||
import java.util.concurrent.atomic.AtomicReference; | ||
|
||
|
@@ -411,6 +421,104 @@ public void testWarningsAreCleared() throws SQLException { | |
stmt.close(); | ||
} | ||
|
||
@Test | ||
public void testWarningsAreAvailableAsap() | ||
throws SQLException, InterruptedException, ExecutionException { | ||
con.createStatement() | ||
.execute("CREATE OR REPLACE FUNCTION notify_then_sleep() RETURNS VOID AS " | ||
+ "$BODY$ " | ||
+ "BEGIN " | ||
+ "RAISE NOTICE 'Test 1'; " | ||
+ "RAISE NOTICE 'Test 2'; " | ||
+ "EXECUTE pg_sleep(2); " | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Alternative approach would be to use two connections, and make sure the outer transaction holds a row lock preventing notify_then_sleep from making progress. What do you think? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Great idea, I have now implemented this, it should speed up the test a bit and make it a little more robust. |
||
+ "END " | ||
+ "$BODY$ " | ||
+ "LANGUAGE plpgsql;"); | ||
con.createStatement().execute("SET SESSION client_min_messages = 'NOTICE'"); | ||
final PreparedStatement preparedStatement = con.prepareStatement("SELECT notify_then_sleep()"); | ||
ScheduledExecutorService executorService = Executors.newScheduledThreadPool(1); | ||
ScheduledFuture<SQLWarning> future = executorService.schedule(new Callable<SQLWarning>() { | ||
@Override | ||
public SQLWarning call() throws SQLException { | ||
return preparedStatement.getWarnings(); | ||
} | ||
}, 1000, TimeUnit.MILLISECONDS); | ||
preparedStatement.execute(); | ||
|
||
SQLWarning warning = future.get(); | ||
executorService.shutdown(); | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. can we have There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I didn't want to introduce a new field that is only used in a couple of tests. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I have added a try-finally condition to ensure the executor is shut down. |
||
|
||
assertNotNull(warning); | ||
assertEquals("First warning received not first notice raised", | ||
warning.getMessage(), "Test 1"); | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Please use the proper expected, actual argument order: http://junit.sourceforge.net/javadoc/org/junit/Assert.html#assertEquals(java.lang.String, java.lang.Object, java.lang.Object) |
||
assertEquals("Second warning received not second notice raised", | ||
warning.getNextWarning().getMessage(), "Test 2"); | ||
} | ||
|
||
/** | ||
* Demonstrates a safe approach to concurrently reading the latest | ||
* warnings while periodically clearing them. | ||
* | ||
* One drawback of this approach is that it requires the reader to make it to the end of the | ||
* warning chain before clearing it, so long as your warning processing step is not very slow, | ||
* this should happen more or less instantaneously even if you receive a lot of warnings. | ||
*/ | ||
@Test | ||
public void testConcurrentWarningReadAndClear() | ||
throws SQLException, InterruptedException, ExecutionException, TimeoutException { | ||
final int iterations = 1000; | ||
final ExecutorService executor = Executors.newSingleThreadExecutor(); | ||
con.createStatement() | ||
.execute("CREATE OR REPLACE FUNCTION notify_loop() RETURNS VOID AS " | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Should the procedure be removed from the database after the test? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I have added function removal to the |
||
+ "$BODY$ " | ||
+ "BEGIN " | ||
+ "FOR i IN 1.. " + iterations + " LOOP " | ||
+ " RAISE NOTICE 'Warning %', i; " | ||
+ "END LOOP; " | ||
+ "END " | ||
+ "$BODY$ " | ||
+ "LANGUAGE plpgsql;"); | ||
con.createStatement().execute("SET SESSION client_min_messages = 'NOTICE'"); | ||
final PreparedStatement statement = con.prepareStatement("SELECT notify_loop()"); | ||
|
||
final Future warningReaderThread = executor.submit(new Callable<Object>() { | ||
@Override | ||
public Object call() throws SQLException, InterruptedException { | ||
SQLWarning lastProcessed = null; | ||
int warnings = 0; | ||
//For production code replace this with some condition that | ||
//ends after the statement finishes execution | ||
while (warnings < iterations) { | ||
SQLWarning warn = statement.getWarnings(); | ||
//if next linked warning has value use that, otherwise keep using latest head | ||
if (lastProcessed != null && lastProcessed.getNextWarning() != null) { | ||
warn = lastProcessed.getNextWarning(); | ||
} | ||
if (warn != null) { | ||
warnings++; | ||
//System.out.println("Processing " + warn.getMessage()); | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Please add assert for each received warning |
||
lastProcessed = warn; | ||
if (warn == statement.getWarnings()) { | ||
//System.out.println("Clearing warnings"); | ||
statement.clearWarnings(); | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. As far as I can understand, a warning can be lost in the following scenario:
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. When you say "suspended" are you referring to There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I was able to simulate your scenario by manually adding delay into the There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
I am afraid I do not see how the additional There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Well there are only two places a new warning can be added, onto the previous chain or added as the new head of the chain, There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks, I see now. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think it would be unlikely someone would replicate it without looking at this unit test. I considered other approaches like employing locks/synchronization or maybe even exposing a queue data structure, but I felt that those options would introduce negative performance implications for users who don’t need the feature, or in the case of exposing a queue, wouldn’t work with the standard JDBC API. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It's not obvious to me what happens in the loop here and when warnings are cleared, etc. Would it be possible to improve readability somehow? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Warnings are cleared if the warning just processed is the head of the warning chain, does that explanation help? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes, I can see this condition, but trying to understand how often it happens. will try more making notes on paper :) There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. How often it happens depends on a few factors, I tried to explain it a bit in the javadoc comment.
Overall it's fairly random, im not sure how to describe it in a useful way. |
||
} | ||
} else { | ||
//Not required for this test, but a good idea adding some delay for production code | ||
//to avoid high cpu usage while the query is running and no warnings are coming in. | ||
//Alternatively use JDK9's Thread.onSpinWait() | ||
Thread.sleep(10); | ||
} | ||
} | ||
//Ensure that we didn't double process the same warning. | ||
assertEquals(lastProcessed.getMessage(), "Warning " + iterations); | ||
return null; | ||
} | ||
}); | ||
statement.execute(); | ||
//If the reader doesn't return after 2 seconds, it failed. | ||
warningReaderThread.get(2, TimeUnit.SECONDS); | ||
} | ||
|
||
/** | ||
* The parser tries to break multiple statements into individual queries as required by the V3 | ||
* extended query protocol. It can be a little overzealous sometimes and this test ensures we keep | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As far as I understand, this might cause double-processing via the following scenario:
PgResultSet.this.warnings = this.getWarning();
write results in the same head being used.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please add a test for
ResultSet.getWarnings()
as well.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@vlsi I decided to revert the change to
ResultSet
because I couldn't actually come up with test a scenario where it made sense.I'm not sure it is even possible for postgres to raise warnings while a result set is reading from a cursor.