Skip to content

Commit

Permalink
add automatic SubQueries for performance in DQM
Browse files Browse the repository at this point in the history
  • Loading branch information
@olmobrutall
olmobrutall committed Mar 16, 2022
1 parent 8ab8940 commit b4922d0
Show file tree
Hide file tree
Showing 8 changed files with 1,754 additions and 1,487 deletions.
57 changes: 40 additions & 17 deletions Signum.Engine/DynamicQuery/AutoDynamicQuery.cs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,8 @@ public override ResultTable ExecuteQuery(QueryRequest request)

var result = query.TryPaginate(request.Pagination, request.SystemTime);

result = result.SelectManySubQueries();

if (inMemoryOrders != null)
{
result = result.OrderBy(inMemoryOrders);
Expand All @@ -44,7 +46,9 @@ public override async Task<ResultTable> ExecuteQueryAsync(QueryRequest request,

var result = await query.TryPaginateAsync(request.Pagination, request.SystemTime, token);

if(inMemoryOrders != null)
result = result.SelectManySubQueries();

if (inMemoryOrders != null)
{
result = result.OrderBy(inMemoryOrders);
}
Expand Down Expand Up @@ -92,29 +96,48 @@ private DQueryable<T> GetDQueryable(QueryRequest request, out List<Order>? inMem
if (!request.Columns.Where(c => c is _EntityColumn).Any())
request.Columns.Insert(0, new _EntityColumn(EntityColumnFactory().BuildColumnDescription(), QueryName));

var query = Query
.ToDQueryable(GetQueryDescription())
.SelectMany(request.Multiplications())
.Where(request.Filters);

if (request.Pagination is Pagination.All)
if (request.MultiplicationsInSubQueries())
{
var allColumns = request.Columns.Select(a => a.Token)
.Concat(request.Orders.Select(a => a.Token))
.Distinct()
.Select(t => new Column(t, null)).ToList();
var columnAndOrderTokens = request.Columns.Select(a => a.Token)
.Concat(request.Orders.Select(a => a.Token))
.Distinct()
.ToHashSet();

inMemoryOrders = request.Orders.ToList();
inMemoryOrders = request.Orders;

var query = Query
.ToDQueryable(GetQueryDescription())
.Where(request.Filters)
.SelectWithSubQueries(columnAndOrderTokens);

return query.Select(allColumns);
return query;
}
else
{
inMemoryOrders = null;
var query = Query
.ToDQueryable(GetQueryDescription())
.SelectMany(request.Multiplications())
.Where(request.Filters);

if (request.Pagination is Pagination.All)
{
var allColumns = request.Columns.Select(a => a.Token)
.Concat(request.Orders.Select(a => a.Token))
.Distinct()
.Select(t => new Column(t, null)).ToList();

inMemoryOrders = request.Orders.ToList();

return query
.OrderBy(request.Orders)
.Select(request.Columns);
return query.Select(allColumns);
}
else
{
inMemoryOrders = null;

return query
.OrderBy(request.Orders)
.Select(request.Columns);
}
}
}

Expand Down
Loading

1 comment on commit b4922d0

@olmobrutall
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Faster queries using @foreach on WordTemplate / EmailTemplate

WordTemplates and EmailTemplates are based in dynamic queries, this means that all the tokens defined in the template are going to be columns in a flat table.

Example:

With the original query:

sb.Include<CompanyEntity>()
   .WithQuery(() => a=> new {
    Entity = e, 
    e.Id,
    e.Name,
    e.CreationDate,
    e.Country,
    e.CompanyType
})

And the template

<div>
   @[Name]
  <ul>
    @foreach[Entity.Contracts.Element] as $c
      <li>@[$c.ContractNumber] (@[$c.State])</li>
    @endforeach
  </ul>
</div>

More or less produces que QueryRequest equivalent to this FindOptions

{
    queryName: CompanyEntity, 
    filter: [{ token : "Entity", opetation: "Equals", value:  currentCompany}],
    columns: [
       { token: "Name" },
       { token: "Entity.Contracts.Element.ContractNumber" },
       { token: "Entity.Contracts.Element.State" },
   ],
}

Until this change, for each collection Element token like the one in @foreach[Entity.Contract.Element], the query needed to to make a SelectMany (OUTER APPLY in SQL).

Database.Query<CompanyEntity>()
.Select(c =>  new {
    Entity = e, 
    e.Id,
    e.Name,
    e.CreationDate,
    e.Country,
    e.CompanyType
})
.SelectMany(a => a.Contracts.DefaultIfEmpty(), (a, c) => Tuple.Create(a, c.ToLite()))
.Where(t => t.Item1.Entity == currentCompany)
.Select(t => Tuple.Create(
 t.Item1.Name,
 t.Item2.ContractNumber,
 t.Item2.State,
))

If you have one or two @foreach this is not a problem, but once you have 4 or 5 @foreach, specially from unrelated collections, you get an explosion of rows due to the orthogonal combination of results.

To prevent this, now we try to replace the SelectMany by an automatic sub query.

Database.Query<CompanyEntity>()
.Select(c =>  new {
    Entity = e, 
    e.Id,
    e.Name,
    e.CreationDate,
    e.Country,
    e.CompanyType
})
.Where(a =>a.Entity == currentCompany)
.Where(t => t.Item1.Entity == currentCompany)
.Select(t => Tuple.Create(
 t.Name,
 t.Contracts.Select(c => Tuple.Create(c.ContractNumber, c.State)).ToList()
))
.ToList() //Continue In-Memory
.SelectMany(t => t.Item2.DefaultIfEmpty(), (t, c) => Tuple.Create(
t.Name, 
c.ContractNumber, 
c.State)

Note how the end result is the same flat table, but has been flattened in-memory.

This means that WordTemplates and EmailTemplates with many @foreach:

  • Instead of 1 huge query with many columns and many rows
  • Will make a few simpler and smaller queries with few columns and few rows.

Limitations

This optimization is made at the dynamic query level for any query that:

  • Contains some Columns and/or Orders with Element
  • Contains no Filters using Element
  • Pagination is All

This is a exceptional case for SearchControl, but a very typical case for WordTemplate / EmailTemplate.

Enjoy!

Please sign in to comment.