Format exception when using culture and partitionKey, difference between Windows and Linux

[mlankamp]

Describe the bug
We see different behavior between running queries on Linux versus Windows, when using culture Info and partition key/cross-partition key queries.

To Reproduce

using Microsoft.Azure.Cosmos.Fluent;
using Microsoft.Azure.Cosmos;
using Newtonsoft.Json;

namespace ConsoleApp1;

internal class Program
{
    static void Main(string[] args)
    {
        var client = new CosmosClientBuilder("---url---", "---key---").Build();
        var container = client.GetContainer("--database--", "--container--");

        Thread.CurrentThread.CurrentCulture = new System.Globalization.CultureInfo("en-US");

        // Retrieves data
        var noCultureWithPartitionKey = container.GetItemLinqQueryable<QueryObject>(true, null, new QueryRequestOptions() { PartitionKey = new PartitionKey("bc7df108-4c93-4534-ba32-167d77274511") })
            .Where(c => c.Area > 5.65252);
        Console.WriteLine(noCultureWithPartitionKey.Count());

        // Retrieves data, cross partition
        var noCultureNoPk = container.GetItemLinqQueryable<QueryObject>(true, null, new QueryRequestOptions() { PartitionKey = null })
            .Where(c => c.Area > 5.65252);
        Console.WriteLine(noCultureNoPk.Count());

        // Change the culture
        Thread.CurrentThread.CurrentCulture = new System.Globalization.CultureInfo("fr-FR");

        // Throws error
        try
        {
            var withCultureWithPartitionKey = container.GetItemLinqQueryable<QueryObject>(true, null, new QueryRequestOptions() { PartitionKey = new PartitionKey("bc7df108-4c93-4534-ba32-167d77274511") })
                .Where(c => c.Area > 5.65252);
            Console.WriteLine(withCultureWithPartitionKey.Count());
        }
        catch (Exception ex)
        {
            Console.WriteLine(ex.Message);
        }

        // Same query, cross partition
        var withCultureNoPk = container.GetItemLinqQueryable<QueryObject>(true, null, new QueryRequestOptions() { PartitionKey = null })
            .Where(c => c.Area > 5.65252);
        Console.WriteLine(withCultureNoPk.Count());
    }
}

class QueryObject{

    [JsonProperty("id")]
    public string Id{ get; set; }

    [JsonProperty("partitionKey")]
    public string PartitionKey { get; set; }

    [JsonProperty("area")]
    public double Area{ get; set; }
}

Results on Windows:

10
240193
10
240193

Results on Linux:

10
240193
Input string was not in a correct format.
240193

Expected behavior
Same behavior when running on Windows/Linux/Docker, independent on the thread culture.

Actual behavior
We are running a docker container with an webapi which uses request localization based on the accept language header. Customers from Norway, France are running into problems, because of the "Input string was not in a correct format." exception (all our queries are on a specific partition key).

Environment summary
SDK Version: 2.32.3
OS Version Windows vs Linux (Ubuntu) vs Official Docker images

Additional context
Add any other context about the problem here (for example, complete stack traces or logs).

[mlankamp]

Looks to be the same behaviour as #542, but only happens on Linux and not cross-partition

[ealsur]

@mlankamp can you confirm this happens only with LINQ? If you change the query to be a QueryDefinition with Parameters instead of the .Where(), does it work?

[mlankamp]

The same happens with a normal query, like "SELECT * FROM c WHERE c.area > 5.12345".

[ealsur]

@mlankamp Can you share the full stacktrace exception that happens on the non-LINQ scenario?

[mlankamp]

Using the following code:

var options = new QueryRequestOptions() { PartitionKey = new PartitionKey("bc7df108-4c93-4534-ba32-167d77274511") };
var withCultureWithPartitionKey = container.GetItemQueryIterator<QueryObject>("SELECT * FROM c WHERE c.area > 5.65252", null, options);
while (withCultureWithPartitionKey.HasMoreResults) {

    var response = await withCultureWithPartitionKey.ReadNextAsync();
    Console.WriteLine(response.Count);
}

gives the following error:

Exception thrown: 'System.FormatException' in System.Private.CoreLib.dll: 'Input string was not in a correct format.'
Stack trace:
   at System.Number.ThrowOverflowOrFormatException(ParsingStatus status, TypeCode type)
   at System.Double.Parse(String s)
   at Microsoft.Azure.Cosmos.Query.Core.Parser.CstToAstVisitor.GetNumber64ValueFromNode(IParseTree parseTree)
   at Microsoft.Azure.Cosmos.Query.Core.Parser.CstToAstVisitor.VisitLiteral(LiteralContext context)
   at sqlParser.LiteralContext.Accept[TResult](IParseTreeVisitor`1 visitor)
   at Antlr4.Runtime.Tree.AbstractParseTreeVisitor`1.Visit(IParseTree tree)
   at Microsoft.Azure.Cosmos.Query.Core.Parser.CstToAstVisitor.VisitLiteralScalarExpression(LiteralScalarExpressionContext context)
   at sqlParser.LiteralScalarExpressionContext.Accept[TResult](IParseTreeVisitor`1 visitor)
   at Antlr4.Runtime.Tree.AbstractParseTreeVisitor`1.Visit(IParseTree tree)
   at Microsoft.Azure.Cosmos.Query.Core.Parser.CstToAstVisitor.VisitUnary_scalar_expression(Unary_scalar_expressionContext context)
   at sqlParser.Unary_scalar_expressionContext.Accept[TResult](IParseTreeVisitor`1 visitor)
   at Antlr4.Runtime.Tree.AbstractParseTreeVisitor`1.Visit(IParseTree tree)
   at Microsoft.Azure.Cosmos.Query.Core.Parser.CstToAstVisitor.VisitBinary_scalar_expression(Binary_scalar_expressionContext context)
   at sqlParser.Binary_scalar_expressionContext.Accept[TResult](IParseTreeVisitor`1 visitor)
   at Antlr4.Runtime.Tree.AbstractParseTreeVisitor`1.Visit(IParseTree tree)
   at Microsoft.Azure.Cosmos.Query.Core.Parser.CstToAstVisitor.VisitBinary_scalar_expression(Binary_scalar_expressionContext context)
   at sqlParser.Binary_scalar_expressionContext.Accept[TResult](IParseTreeVisitor`1 visitor)
   at Antlr4.Runtime.Tree.AbstractParseTreeVisitor`1.Visit(IParseTree tree)
   at Microsoft.Azure.Cosmos.Query.Core.Parser.CstToAstVisitor.VisitLogical_scalar_expression(Logical_scalar_expressionContext context)
   at sqlParser.Logical_scalar_expressionContext.Accept[TResult](IParseTreeVisitor`1 visitor)
   at Antlr4.Runtime.Tree.AbstractParseTreeVisitor`1.VisitChildren(IRuleNode node)
   at sqlBaseVisitor`1.VisitLogicalScalarExpression(LogicalScalarExpressionContext context)
   at sqlParser.LogicalScalarExpressionContext.Accept[TResult](IParseTreeVisitor`1 visitor)
   at Antlr4.Runtime.Tree.AbstractParseTreeVisitor`1.Visit(IParseTree tree)
   at Microsoft.Azure.Cosmos.Query.Core.Parser.CstToAstVisitor.VisitWhere_clause(Where_clauseContext context)
   at sqlParser.Where_clauseContext.Accept[TResult](IParseTreeVisitor`1 visitor)
   at Antlr4.Runtime.Tree.AbstractParseTreeVisitor`1.Visit(IParseTree tree)
   at Microsoft.Azure.Cosmos.Query.Core.Parser.CstToAstVisitor.VisitSql_query(Sql_queryContext context)
   at sqlParser.Sql_queryContext.Accept[TResult](IParseTreeVisitor`1 visitor)
   at Antlr4.Runtime.Tree.AbstractParseTreeVisitor`1.Visit(IParseTree tree)
   at Microsoft.Azure.Cosmos.Query.Core.Parser.CstToAstVisitor.VisitProgram(ProgramContext context)
   at sqlParser.ProgramContext.Accept[TResult](IParseTreeVisitor`1 visitor)
   at Antlr4.Runtime.Tree.AbstractParseTreeVisitor`1.Visit(IParseTree tree)
   at Microsoft.Azure.Cosmos.Query.Core.Parser.SqlQueryParser.Monadic.Parse(String text)
   at Microsoft.Azure.Cosmos.Query.Core.Parser.SqlQueryParser.TryParse(String text, SqlQuery& sqlQuery)
   at Microsoft.Azure.Cosmos.Query.Core.ExecutionContext.CosmosQueryExecutionContextFactory.TryCreateCoreContextAsync(DocumentContainer documentContainer, CosmosQueryContext cosmosQueryContext, InputParameters inputParameters, ITrace trace, CancellationToken cancellationToken)
   at Microsoft.Azure.Cosmos.Query.Core.AsyncLazy`1.GetValueAsync(ITrace trace, CancellationToken cancellationToken)
   at Microsoft.Azure.Cosmos.Query.Core.Pipeline.LazyQueryPipelineStage.MoveNextAsync(ITrace trace)
   at Microsoft.Azure.Cosmos.Query.Core.Pipeline.NameCacheStaleRetryQueryPipelineStage.MoveNextAsync(ITrace trace)
   at Microsoft.Azure.Cosmos.Query.Core.Pipeline.CatchAllQueryPipelineStage.MoveNextAsync(ITrace trace)
   at Microsoft.Azure.Cosmos.Query.QueryIterator.ReadNextAsync(ITrace trace, CancellationToken cancellationToken)
   at Microsoft.Azure.Cosmos.FeedIteratorCore`1.ReadNextAsync(ITrace trace, CancellationToken cancellationToken)
   at Microsoft.Azure.Cosmos.ClientContextCore.RunWithDiagnosticsHelperAsync[TResult](String containerName, String databaseName, OperationType operationType, ITrace trace, Func`2 task, Func`2 openTelemetry, String operationName, RequestOptions requestOptions)
   at Microsoft.Azure.Cosmos.ClientContextCore.OperationHelperWithRootTraceAsync[TResult](String operationName, String containerName, String databaseName, OperationType operationType, RequestOptions requestOptions, Func`2 task, Func`2 openTelemetry, TraceComponent traceComponent, TraceLevel traceLevel)
 >   at ConsoleApp1.Program.<Main>d__0.MoveNext() in C:\source\repos\ConsoleApp1\Program.cs:line 25

[mlankamp]

Maybe

azure-cosmos-dotnet-v3/Microsoft.Azure.Cosmos/src/Query/Core/Parser/CstToAstVisitor.cs

Line 966 in f262f89

number64 = double.Parse(text);

should be:

number64 = double.Parse(text, CultureInfo.InvariantCulture);

[mlankamp]

Maybe this helps to find the problem. When I use the query, but make the partition part of the query it works:

var options = new QueryRequestOptions() { PartitionKey = null) };

var withCultureWithPartitionKey = container.GetItemQueryIterator<QueryObject>("SELECT * FROM c WHERE c.area > 5.65252 and c.partitionKey='bc7df108-4c93-4534-ba32-167d77274511'", null, options);
while (withCultureWithPartitionKey.HasMoreResults) {

    var response = await withCultureWithPartitionKey.ReadNextAsync();
    Console.WriteLine(response.Count);
}

This causes

azure-cosmos-dotnet-v3/Microsoft.Azure.Cosmos/src/Query/Core/Pipeline/CosmosQueryExecutionContextFactory.cs

Lines 191 to 192 in f262f89

	if (cosmosQueryContext.QueryClient.ByPassQueryParsing()
	&& inputParameters.PartitionKey.HasValue)

to not call the SqlQueryParser which doesn't result in the error

[ealsur]

Might be this line:

azure-cosmos-dotnet-v3/Microsoft.Azure.Cosmos/src/Query/Core/Parser/CstToAstVisitor.cs

Line 966 in e41eea5

number64 = double.Parse(text);

@neildsh Could you help confirm if this might be the culprit? The double.Parse not having invariant culture might be causing this exception on query parsing?

EDIT: I see this is repeated as it was also highlighted by @mlankamp

[neildsh]

This looks like the correct diagnosis. I'll get this issue assigned. Thanks @mlankamp

[mlankamp]

I can provide a PR (and unit-test also ) if you want