EF Core doesn't lazy-load virtual navigation properties

[bitcrazed]

Issue: EF7 may not be lazy-loading all entities in a graph of related entities, despite all entities' navigation properties being virtual:

I built a basic blog atop EF6. It behaves as expected and correctly models many-to-many relationships - in this case, the fact that posts may have zero or more authors.

I then rewrote this basic blog atop EF7.

This is where I found out that EF7 doesn't currently support modelling many-to-many relationships (see #1368) and requires one to manually create and handle the join entity itself.

So, I added the PostAuthor type to the app's model.

Aside: Many-to-many is going to be supported for RTW, right? This is a pretty essential feature

  +----------+          +----------+          +-------------+          +--------+
  |          | 1      n |          | 1      n |             | n      1 |        |
  |   Blog   | -------- |   Post   | -------- | PostAuthors | -------- | Author |
  |          |          |          |          |             |          |        |
  +----------+          +----------+          +-------------+          +--------+

When querying the model, I was found that EF wasn't populating the Blog's Posts collection requiring me to .Include(blog => b.Posts) and then didn't populate the Post's PostAuthors collection, requiring me to .ThenInclude(p=>p.PostAuthors). Imagine my surprise, then, when I also found that I had to .ThenInclude(pa=>pa.Authors). The net result? This:

 var blogs = ctx
        .Blogs
        .Include(b => b.Posts)
        .ThenInclude(p => p.PostAuthors)
        .ThenInclude(pa => pa.Author)
         .ToList();

My expectation is that since all the entities' navigation properties are virtual (Post, PostAuthor and Author ) that they'd be lazy-loaded on demand.

What am I doing wrong here?

TIA,

[HenryGarle]

The short answer is that EF7 does not currently support automatic lazy loading in the same way that EF6 did. I've seen it discussed in a few places and if anyone is better informed feel free to correct me but as far as I am aware it is fairly far down their priority list and if some posts I have read are to be believed, that's if they choose to implement it at all.

[divega]

@HenryGarle is right. There is no automatic lazy loading support in EF7, and we don't have a clear outlook on its priority. Lazy loading can make data access easier to deal with but it has pitfalls such as potentially producing very inefficient query patterns that cause many more database roundtrips than necessary. By comparison eager loading (i.e. the Include() methods), and using projections or multiple queries to load a graph of objects can be more cumbersome but give you finer control.

[bitcrazed]

Wow. I am surprised - while far from perfect, removing Lazy Loading is a pretty major change that I've not seen explicitly stated in any of the docs, nor in the ASPNET/Announcements repo, no mention of it in the Guiding Principles, etc. Can you point me at any docs where this is stated?

@divega: You state that EF7 doesn't support lazy loading, but in #3318 you state that "... and if lazy loading is enabled it will work the same for collection and reference properties." which indicates that LazyLoading can be enabled, no?

[divega]

Edited that last sentence to make it clearer that I was referring to EF6 behaviors.

[bitcrazed]

Thanks.

Assume there's currently no statement on the removal of Lazy Loading? Any chance the EF team could add Announcements and/or document breaking changes like these - it'd be a huge help to those of us trying to get started (and help others do so) on the new bits?

[natemcmaster]

@bitcrazed Thanks for the feedback. Since we are still pre-release, there is no documentation on what is a "breaking" change from EF6 to EF7. As EF7's features stabilize, we will write more docs. I've added dotnet/EntityFramework.Docs#40 to help capture this feedback.

[rowanmiller]

Agreed, we will have some "Should I be using EF7" documentation and the "What's new in EF7" and "What's removed in EF7" content will be a large part of that.

[bitcrazed]

@natemcmaster Many thanks.

I know it's early days for the EF team but when developing in the open like this, timely release of breaking changes vs. a previous version is pretty important.

I think that official guidance on issues like this (along with no automatic many-to-many support) will be crucially important for many who need to decide between using EF7 or EF<=6 for new projects that may be shipping in the next 6-8 months.

I'll be blogging on subjects like this as I find them anyhow, but an official position would be most useful.

[bitcrazed]

@rowanmiller For me, this is just another example why I think this project should have been called something other than EntityFramework - because it's not EF. While EF7 retains some vestiges of EF concepts, I think it's sufficiently (so very) different from EF that it'd be easier to explain what it is if it wasn't called EF!

[JeffGillin]

@divega, @rowanmiller
First off, great job so far with EF7. The team is doing an excellent job. I know a major re-write of an ORM can't be easy - to say the least.
Regarding Lazy Loading, I feel it is absolutely essential. Here's why.
A lot of us have domain logic coded into model classes and use the repository pattern to fetch entities. In this case, the domain model class expects the navigation property to have values that match what's in the database (whether they are loaded on-demand or eager loaded). Otherwise, for a particular operation in the domain model class, you are 'assuming' that the property was pre-loaded. This is a dangerous assumption. If it wasn't pre-loaded, then in the case of a Collection, it's Count will be 0, and your logic is assuming it's zero in the database - but it's not - it just wasn't loaded.
Thus, in order to work properly, you need to now coordinate each type of action in your application with a proper fetch strategy from your repository. I'm working with EF7 now, and whenever I load something from a repository, I pass in an enum value that represents a "eager loading strategy" enum parameter so that I can determine which navigation properties need to be included, and add then include those dynamically to the Linq-to-entities query. This works - but the necessary coordination between repository fetching and domain model action is definitely brittle. The lazy loading is a nice safe haven to ensure it will always be loaded when some logic is expecting it will be. EF6 had the ability to disable lazy loading. I'm going to suggest going the opposite direction with EF7. Make it a feature that is disabled by default but can be enabled via an option in the 'DbContextOptionsBuilder'.
One other note along the same lines, is that, with a clearly defined separation of layers (where domain model has no knowledge of EF), I'm finding it difficult to deal with collections in a way that works as I would expect. Let's say a have a method in a model class that removes some child entities based on some filter. If I remove an entity from a ICollection navigation property, I would expect that this would automatically set a 'deleted' state on that child entity, and a 'SaveChanges' on the tracked parent entity will know to delete that item from the child table.
Since we want the domain model to be persistence ignorant, we don't want any references to EF in the domain model assembly, and thus can't directly set a deleted state with EF methods. In this type of architecture, it's necessary that this removal is tracked (again - maybe an optional setting or something - but without it - my architecture, of keeping logic in domain model classes, no longer works and it's necessary to start putting some of this logic in the service(application) layer which is undesirable because I want to write automated tests against the domain model to test it's logic without having any db involved in the process.
Is this something that's possible to do? If so, is it something that is intended to be included at some point?

[rowanmiller]

@JeffGillin - here is the backlog item tracking Lazy Loading #3797

[SimonOrdo]

I'm using Identity and EF7. I have extended my ApplicationUser class with a navigation property like so:

public class ApplicationUser : IdentityUser
{
          public virtual PersonalConfigData MyConfig { get; set; }

          [ForeignKey("ID")]
          public long MyConfigID { get; set; }
}

In the account controller, the user is loaded via the Identity framework like so:

var user = await _userManager.FindByNameAsync(model.Username);

How would you get the additional data to load? I thought lazy loading would take care of this.

[rowanmiller]

Lazy loading is not implemented in EF7 (backlog item is #3797).

When you query, you can use the Include method to bring in related data:

context.Users.Include(u => u.PersonalConfigData).Where(u => u.Username == username).Single()

[SimonOrdo]

Understood @rowanmiller , but that does mean I have to query the entire object twice, in this case, right?

First time, the Identity _userManager does the query. Then, if it comes back gtg, I need to query the object AGAIN to load the nav property, if I'm understanding you correctly.

Is there also currently no way to hydrate specific properties on an object? Like "LoadProperty", for instance? Seems that without lazy loading, that would be a minimum requirement, considering I might not always have control over the initial query.

[Edit]

Nevermind. I guess I'm a bit ignorant on how objects are tracked. It looks like that line of code you sent adds the additional property data to the originally queried object, not just a newly returned one.

[rowanmiller]

Correct, or if you just query for the related object then EF will also join things up in-memory for you.

context.PersonalConfigData.Where(c => c.Id == user.MyConfigId).Single()

[dwdickens]

Thanks @rowanmiller this is very helpful.

I'm having trouble with using an include to bring in the related data.

var _gifts = _context.Gifts.Include(g => g.Image).Where(g => g.Registry.ApplicationUser.Id == usrId)

throws the error: Unable to cast object of type 'System.Linq.Expressions.FieldExpression' to type 'System.Linq.Expressions.ParameterExpression'

I don't understand this error - my code seems to be the same as that above.

If I remove the Include it works, but I then need to have: await _context.Images.ToListAsync(); to load the Images (which is bad because this loads all images, not just those for the selected gifts).

How can I just load the Images selected in _gifts?

The models are:

public class Gift
{
    public int GiftID { get; set; }
    public virtual Registry Registry { get; set; }
    public virtual Image Image { get; set; }

}

public class Image
{
    public int ImageID { get; set; }
    public byte[] Content { get; set; }
}

[NullVoxPopuli]

Hey, I asked this question on Stack Overflow: http://stackoverflow.com/questions/34790533/how-do-i-simplify-the-access-of-a-has-many-relationship-with-the-entity-framewor

It seems that implementing lazy loading would solve the root of my problem.

But If there was a dynamic work around to achieve something similar, I'd be up for that as well.

//This is what I have so far

        public object RelationshipFor(string relationship)
        {
            using (var db = User.DbContext())
            {
                var myTable = (DbSet<DatabaseModels.User>)db.Send(RelationshipName);
                var myInclude = myTable.Include(i => i.Send(relationship));
                var meWithRelationship = myInclude.First(MatchesIdFor);
                return meWithRelationship.Send(relationship);
            }
        }


        public bool MatchesIdFor(object item)
        {
            return (long)item.Send(IdColumn) == Id;
        }

but I get the error:

Unable to cast object of type 'System.Linq.Expressions.MethodCallExpressionN' to type 'System.Linq.Expressions.MemberExpression'.

[NullVoxPopuli]

Also, is there a way to dynamically cast an object of a DbSet?

[rowanmiller]

@dwdickens and @NullVoxPopuli the bug being tracked by this issue has been resolved. Please open new issues for the problems you are encountering.

[rowanmiller]

@dwdickens when you open a new issue can you include a full code listing (or project) that we can use to reproduce the issue. There is probably a specific pattern in your model that is causing the error to occur.

[makarti]

This code

categories .Include(c => c.Products).ThenInclude(p => p.Reviews.).ThenInclude(r => r.Author) .Include(c => c.Products).ThenInclude(p => p.Supplier);

in sql query genarate two joins "Products". Is it possible optimize this expression?

[rowanmiller]

@makarti we have #3419 to look at consolidating joins in scenario like this.

[christophedemey]

I am also missing lazy loading in EF7 👎

[gdoron]

@christophedemey , why?
This proxy-lazy loading pattern caused people to hit the Database multiple times and possibly N+1 times.
Now you must think before you query what relationships do you need, and you will never hit the DB multiple times.
That's not lazy loading, it's lazy programming... 😆
The lazy loading that is quite clever but not frequently used is "Lazy loading properties", which means that unless you eagerly include (pre-configured) properties, they will stay null.

It's helpful mainly when querying for entities that have lengthy string properties.
So unless you actually need that property, it will remain null, just like with Navigation Properties.

[dwdickens]

I appreciate the performance reasons for discouraging lazy loading.

Developing an app now with EF, it gets confusing if a bug is due to bad db data or a failure to properly load - null is ambiguous.

Why doesn't EF throw an error if you try to access data that hasn't been loaded?

On 7 Mar 2016, at 6:05 AM, Doron Grinzaig [email protected] wrote:

@christophedemey , why?
This proxy-lazy loading pattern caused people to hit the Database multiple times and possibly N+1 times.
Now you must think before you query what relationships do you need, and you will never hit the DB multiple times.
That's not lazy loading, it's lazy programming...
The lazy loading that is quite clever but not frequently used is "Lazy loading properties", which means that unless you eagerly include (pre-configured) properties, they will stay null.

It's helpful mainly when querying for entities that have lengthy string properties.
So unless you actually need that property, it will remain null, just like with Navigation Properties.

—
Reply to this email directly or view it on GitHub.

[christophedemey]

@gdoron if you put it this way you are definitly right !

[gdoron]

@dwdickens, you will get a null reference exception if you don't load the navigation property (and you didn't init the property in the ctor), and an empty list if you did, so it is already easy to understand.

I'm happy with the current implementing o navigation properties in EF core.

[willotheblessed]

@rowanmiller I had the same problem as he did, but my issue isn't solved yet. When I try to reference the "context" I am using ApplicationDBContext and though it has an option called "Users" That doesn't include a method called "Include"

I'm using ASP5, MVC6. So I either need a different method, or a way to get the right context so I can use this:

context.Users.Include(u=> u.PersonalConfigData).Where(u => u.Username == username).Single()

[rowanmiller]

@willotheblessed make sure you have a using for the Microsoft.Data.Entity namespace, Include is an extension method in that namespace.

[willotheblessed]

@rowanmiller

That did it! Thanks!

[SimonOrdo]

@rowanmiller Could I ask you a tangentially related question? I noticed VS doesn't offer hints to add missing using namespace statements for extension methods the way it does for other references (right click on red-lined code -> Quick Action).

It seems like with this version of ASP.NET there's more extension method stuff going on, so it's been much more noticeable lately.

Could you suggest an appropriate place for me to submit that feature request?

[rowanmiller]

@SimonOrdo you can submit VS ideas at http://connect.microsoft.com/VisualStudio

That said, quick actions does offer to add namespaces for extension methods for me:

[rahulsahay19]

Here is a scenario.
//Movie Class
public class Movie
{
public int Id { get; set; }
public string MovieName { get; set; }
public string DirectorName { get; set; }
public string ReleaseYear { get; set; }
public virtual ICollection Reviews { get; set; }
}

//MovieReview
public class MovieReview
{
public int Id { get; set; }
public string ReviewerName { get; set; }
public string ReviewerComments { get; set; }
public int ReviewerRating { get; set; }
public int MovieId { get; set; }
}

//MovieViewModel
public class MovieViewModel
{
public int Id { get; set; }
public string MovieName { get; set; }
public string DirectorName { get; set; }
public string ReleaseYear { get; set; }
public int NoOfReviews { get; set; }
}

//Now, with below Get query, I am expected to get NoOfReviews as well. But, its always returning 0 as shown below in the screenshot
// GET api/movies
[HttpGet("api/movies")]
public IQueryable Get()
{
var model = UOW.Movies.GetAll().OrderByDescending(m => m.Reviews.Count())
.Select(m => new MovieViewModel
{
Id = m.Id,
MovieName = m.MovieName,
DirectorName = m.DirectorName,
ReleaseYear = m.ReleaseYear,
NoOfReviews = m.Reviews.Count
});
return model;
}

[rahulsahay19]

here movie class is having public virtual ICollection Reviews { get; set; } . Don't know why it is not getting printed properly.

[rahulsahay19]

Attaching Movie class.

[gdoron]

@rahulsahay19

var model = UOW.Movies.Include(x=> x.Reviews).GetAll().OrderByDescending(m => m.Reviews.Count())

[rahulsahay19]

This is breaking with the following exception.

however, when i am writing the below piece
var model = UOW.Movies.GetAll().Include(x => x.Reviews).OrderByDescending(m => m.Reviews.Count);
it is giving below error.

[rahulsahay19]

My Mistake done. Here is the completed code

var model = UOW.Movies.GetAll().Include(x => x.Reviews).OrderByDescending(m => m.Reviews.Count)
.Select(m => new MovieViewModel
{
Id = m.Id,
MovieName = m.MovieName,
DirectorName = m.DirectorName,
ReleaseYear = m.ReleaseYear,
NoOfReviews = m.Reviews.Count
});

[daddydrac]

If using .NET Core and Entity Framework Core, and .Include(x => x.TableName ) is not returning relationships (from principal table to dependent table), or only returning one row of data, FIX HERE:

[Stackoverflow Answer]((https://stackoverflow.com/questions/43127957/include-not-working-in-net-core-returns-one-parent)

Also, in Startup.cs make sure you have this at the top:

using Microsoft.EntityFrameworkCore;
using Newtonsoft.Json;
using Project_Name_Here.Models;

*For older .NET you can use the [JsonIgnore] attribute as well.

[inlineHamed]

@joehoeller The Stackoverflow page is removed!

[gdoron]

@hmdhasani there you go.

[daddydrac]

The jerks at Stack removed it bc they have serious control issues, even though I answered the ques, see here: https://www.linkedin.com/pulse/include-working-net-core-only-returns-one-parent-entity-joe-hoeller-

[daddydrac]

Also, Im not sure that this will help, but try placing "app.UseResponseBuffering();" inside public void Configure located in Startup.cs.

[MartinDawson]

@joehoeller They closed it because that's not how you answer your own question. You first post the question and then answer it otherwise it's confusing to future readers. If you had actually read the guidelines for posting before posting you would have known this and not got upset.

[shpsyte]

@joehoeller not work form me

Iam using .NET Core 2.0 and Entity Framework Core, and .Include(x => x.TableName ) returning one row of data

i try:
in Startup.cs

services .AddMvc() .AddJsonOptions(options => { options.SerializerSettings.ReferenceLoopHandling = ReferenceLoopHandling.Ignore; });

Exists another alternative ?

[lukemason76]

The GroupEditModel is part of a CRUD for editing a Group.

First the object is declared and annotated
[BindProperty]
public Group Group { get; set; }

Then in OnGetAsync() it is filled.
Group = await _db.Groups
.Include(g => g.Tier)
.Include(g => g.GroupType)
.Include(g => g.Parent)
.Include(g => g.Organisation)
.FirstOrDefaultAsync(g => g.Id == id.Value);

Then the object comes back in the OnPostAsync(). Here the object is filled by the binder, but the Navigation properties are null. How do I get these filled by the binder, or fill them easily after that occurs?

[smitpatel]

@lukemason76 - Did you enable lazy loading proxies? EF Core does not lazy load virtual navigations by default. See https://docs.microsoft.com/en-us/ef/core/querying/related-data#lazy-loading

If you are still having issues after following the documentation, please file a new issue with detailed repro steps.

[kccarter76]

I am developer that has ran into this issue with EF7 and I have long since decided to just build implement DAL that gets around the issue.

this lazy loading issue was the driving decision to resurrect subsonic for me.

https://github.com/kccarter76/SubSonic-Core
is the DAL which is in beta
https://github.com/kccarter76/SubSonic-Integration-Db
is the integration DB created to enable unit integration tests against.

[ajcvickers]

@kccarter76 You realize that EF7 doesn't actually exist, right?