Skip to content

Commit

Permalink
mm/hugetlb: make pud_huge() and follow_huge_pud() aware of non-presen…
Browse files Browse the repository at this point in the history
…t pud entry

follow_pud_mask() does not support non-present pud entry now.  As long as
I tested on x86_64 server, follow_pud_mask() still simply returns
no_page_table() for non-present_pud_entry() due to pud_bad(), so no severe
user-visible effect should happen.  But generally we should call
follow_huge_pud() for non-present pud entry for 1GB hugetlb page.

Update pud_huge() and follow_huge_pud() to handle non-present pud entries.
The changes are similar to previous works for pud entries commit
e66f17f ("mm/hugetlb: take page table lock in follow_huge_pmd()") and
commit cbef847 ("mm/hugetlb: pmd_huge() returns true for non-present
hugepage").

Link: https://lkml.kernel.org/r/[email protected]
Signed-off-by: Naoya Horiguchi <[email protected]>
Reviewed-by: Miaohe Lin <[email protected]>
Reviewed-by: Mike Kravetz <[email protected]>
Cc: David Hildenbrand <[email protected]>
Cc: kernel test robot <[email protected]>
Cc: Liu Shixin <[email protected]>
Cc: Muchun Song <[email protected]>
Cc: Oscar Salvador <[email protected]>
Cc: Yang Shi <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
  • Loading branch information
nhoriguchi authored and akpm00 committed Aug 9, 2022
1 parent c053171 commit 3a194f3
Show file tree
Hide file tree
Showing 2 changed files with 37 additions and 3 deletions.
8 changes: 7 additions & 1 deletion arch/x86/mm/hugetlbpage.c
Original file line number Diff line number Diff line change
Expand Up @@ -30,9 +30,15 @@ int pmd_huge(pmd_t pmd)
(pmd_val(pmd) & (_PAGE_PRESENT|_PAGE_PSE)) != _PAGE_PRESENT;
}

/*
* pud_huge() returns 1 if @pud is hugetlb related entry, that is normal
* hugetlb entry or non-present (migration or hwpoisoned) hugetlb entry.
* Otherwise, returns 0.
*/
int pud_huge(pud_t pud)
{
return !!(pud_val(pud) & _PAGE_PSE);
return !pud_none(pud) &&
(pud_val(pud) & (_PAGE_PRESENT|_PAGE_PSE)) != _PAGE_PRESENT;
}

#ifdef CONFIG_HUGETLB_PAGE
Expand Down
32 changes: 30 additions & 2 deletions mm/hugetlb.c
Original file line number Diff line number Diff line change
Expand Up @@ -6985,10 +6985,38 @@ struct page * __weak
follow_huge_pud(struct mm_struct *mm, unsigned long address,
pud_t *pud, int flags)
{
if (flags & (FOLL_GET | FOLL_PIN))
struct page *page = NULL;
spinlock_t *ptl;
pte_t pte;

if (WARN_ON_ONCE(flags & FOLL_PIN))
return NULL;

return pte_page(*(pte_t *)pud) + ((address & ~PUD_MASK) >> PAGE_SHIFT);
retry:
ptl = huge_pte_lock(hstate_sizelog(PUD_SHIFT), mm, (pte_t *)pud);
if (!pud_huge(*pud))
goto out;
pte = huge_ptep_get((pte_t *)pud);
if (pte_present(pte)) {
page = pud_page(*pud) + ((address & ~PUD_MASK) >> PAGE_SHIFT);
if (WARN_ON_ONCE(!try_grab_page(page, flags))) {
page = NULL;
goto out;
}
} else {
if (is_hugetlb_entry_migration(pte)) {
spin_unlock(ptl);
__migration_entry_wait(mm, (pte_t *)pud, ptl);
goto retry;
}
/*
* hwpoisoned entry is treated as no_page_table in
* follow_page_mask().
*/
}
out:
spin_unlock(ptl);
return page;
}

struct page * __weak
Expand Down

0 comments on commit 3a194f3

Please sign in to comment.