Custom Query Filters Test Project

A test project devoted to replacing Entity Framework Core's global query filters with a more flexible and controllable solution. This is built on top of ABP Framework and runs on .NET 5. P.S. It's a mess because it's a test 😛

A list of current issues can be found below. If you feel like helping out, please create a PR

The problem

Filters such as soft deletion can be configured on entites by using global query filters e.g. ModelBuilder.Entity<Post>.HasQueryFilter(p => !p.IsDeleted) which will ensure that only non-deleted posts are returned by appending the filter to every query.

There are times when you need to return deleted entities, and only way to achieve this (with the built-in query filter functionality) is to add a property to your DbContext which is evaluated at runtime. This causes databases providers to not index the query whilst also increasing the query complexity which can hurt performance.

A common example of when you might not want to have and entitiy automatically filtered is when it is a child of another entity.

class Blog
{
    public ICollection<Post> Posts { get; set; }
}

class Post
{
    public Blog Blog { get; set; }
}

Using the example above, if you had a Blog entity which contains a collection of Posts (ICollection<Post> Posts) and some of those posts are deleted, the Blog.Posts property will only contain the non-deleted Post entities due to gloabl query filters being applied to the Posts.

Similarly, if you want to return ALL Posts and you have the Blog as a property of the Post i.e. Post.Blog, only posts belonging to non-deleted blogs will be returned even if they themselves are not deleted!

Microsoft say this is to ensure referential integrity, but this is only applicable at a database level and will be enforced at that level anyway. It leads to unexpected results (which are hard to spot) and there are also bugs with the implementation (try running Count() on the IQueryable before returning the list - you'll get less results than count says you should have!). In my opinion, query filters are business rules, and sometimes you need to ignore those rules when manipulating data. Microsoft's current implementation is also too restrictive for any real-world application.

The goal is simple: Ignore the global query filters for specific entities by using ABP's DataFilters i.e. DataFilter.Disable<ISoftDelete<Blog>>() unless the IQueryable extension method IQueryable.IgnoreAbpQueryFilters() has been used, which stops the filters being applied on a per-query basis.

This solution should fix various issues which have been raised in the past - see linked issues. It could even be extended to address current query filter limitations, create dynamic filters and allow ABP to leverage EF6 compiled models.

I am also planning on integrating these changes to ABP's DataFilters to facilitate this project.

The method

1. Disable the use of global query filters (the ABP ModelBuilder.OnModelCreating() query filter generation code is bypassed, so no calls to Entity.UseQueryFilter())
2. Intercept the query compilation and append the appropriate data filters

Usage example

using (DataFilter.Enable<ISoftDelete())
using (DataFilter.Disable<ISoftDelete<Blog>>())
{
    // This should return a list of non-deleted posts with the 'Blog' populated
    // even if the Blog is marked as deleted.
    var postWithDeletedBlog = PostsRepository
        .Include(x => x.Blog)
        .ToListAsync();
}

Current issues

Replacing query filters sounds simpler than it actually is and there are many hurdles to overcome.

The following known issues (non-exhaustive) are present in the solution:

• ❌ Collection filtering doesn't work for nested includes. It will also return unexpected results on filtered collections when using the same DbContext due to navigation fixup when tracking entities.
• ❌ Lazy/Eager/Implicit loading isn't considered nor is IgnoreAutoIncludes, entity tracking and skip navigations etc.
• ❌ Filters shouldn't be applied if the navigation items are not going to be loaded. This relates the the point above.
• ❌ Different DB providers implement things differently - Only Relational EF provider is currently implemented.
• ✔️ Queries are cached so if filters change at runtime, they don't take effect.

Running the project

1. Ensure that MySQL is installed and that your appsettings.json is correct
2. Run the DbMigrations project to create the database and seed the blog/post data
3. Run the Web project to see the sample data. You can modify Pages/index.js to change which queries are run.

Execution flow:

• CustomAbpDbContext is an extension to AbpDbContext which overrides some functionality, adds the AbpGlobalFiltersOptionsExtension and replaces the IQueryTranslationPreprocessorFactory
  • CustomQueryTranslationPreprocessorFactory creates an instance of CustomQueryTranslationPreprocessor
    • An instance of AbpFilterAppendingExpressionVisitor is created by CustomQueryTranslationPreprocessor which then modifies the query in the Process() method before allowing the base provider to proccess the query
      1. This uses the AbpGlobalFiltersOptionsExtension to gain access to important contextual information such as the IDataFilter and CurrentTenant which are required to modify the query
      2. The CompiledQueryWithAbpFiltersCacheKey decides if the query needs to be processed again or a cached copy of the results can be returned. This looks to see if the DataFilters have changed etc. and if they have then the AbpFilterAppendingExpressionVisitor is executed to regenerate the query.

For more info about the ABP project, you can visit docs.abp.io.

Main project files

• Application
  • BlogAppService.cs
  • PostAppService.cs
• Domain
  • Data/AppDataSeedContributor.cs
  • Extensions/AbpQueryableExtensions.cs
  • AbpQueryFilterDemoConsts.cs <-- Change core configuration here
• Domain.Shared
  • IMultiTenantExtension.cs
  • ISoftDeleteExtension.cs
• EntityframeworkCore
  • Extensions/
    • AbpFilterAppendingExpressionVisitor.cs
    • AbpGlobalFiltersOptionsExtension.cs
    • CompiledQueryWithAbpFiltersCacheKeyGenerator.cs
    • CustomQueryTranslationPreprocessor.cs
    • Extensions.cs
  • CustomAbpDbContext.cs
  • Repositories/PostRepository.cs
• Web
  • Pages/Index.cshtml
  • Pages/Index.js
  • Logs/log.txt <-- View the compiled SQL queries here

Most of the logic is in AbpFilterAppendingExpressionVisitor so check that out first if you want to get stuck in.

Useful links

Docs

• https://docs.microsoft.com/en-us/ef/core/querying/filters/
• https://docs.microsoft.com/en-us/ef/core/querying/related-data/
• https://docs.microsoft.com/en-us/ef/core/dbcontext-configuration/
• https://docs.microsoft.com/en-us/ef/core/modeling/
• https://docs.microsoft.com/en-us/ef/core/providers/writing-a-provider/
• https://docs.microsoft.com/en-us/archive/blogs/mattwar/linq-building-an-iqueryable-provider-series/
• Getting started guide to Expression trees: https://tyrrrz.me/blog/expression-trees
• All available ExpressionTypes: https://docs.microsoft.com/en-us/dotnet/api/system.linq.expressions.expressiontype?view=net-5.0
• Explains what Expression.Quote is: https://stackoverflow.com/a/19148022/2634818
• Another Expression tree guide: https://docs.microsoft.com/en-us/dotnet/csharp/expression-trees-interpreting/

Source code

• Entry point: QueryCompilationContext.cs#L179-L210
• QueryTranslationPreprocessor.Process method: QueryTranslationPreprocessor.cs#L55-L69
• IQueryable IgnoreQueryFilters() parsing: EntityFrameworkQueryableExtensions.cs#L2369-L2398 and action: QueryableMethodNormalizingExpressionVisitor.cs#L217-L223
• IQueryable Include() parsing: QueryableMethodNormalizingExpressionVisitor.cs#L80-L118
• NavigationExpandingExpressionVisitor ApplyQueryFilter method: NavigationExpandingExpressionVisitor.cs#L1412-L1462

Linked issues

• dotnet/efcore#11691
• dotnet/efcore#21093 (comment)
• dotnet/efcore#8576
• abpframework/abp#6680
• abpframework/abp#7482
• abpframework/abp#1181
• abpframework/abp#5650