LSK: BUG: failure at change_huge_pmd in huge_memory.c

Hi,
Could you please merge the following:
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=44b6dfc556811c0e6981329e29e1865fc7a8847d

and,
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=c2c93e5b7f3f42277ec25ff3746096abc0c0d0f7

Thanks,
--
Steve

Also, could you please merge this one:
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=9c7e535fcc1725fc2e2d4f0d9dd14137f0243e23

Cheers,
--
Steve

Same bug when using 32bit LTP on 64bit kernel:
https://validation.linaro.org/scheduler/job/127055/log_file#L_29_9531

Steve, please send followups for LSK bugs via e-mail (commenting on the bug is fine too) - unfortunately due to some hilarity with the way the LAVA lab is set up I get copies of all Launchpad bugmail for LAVA which gmail isn't capable of filtering out.

Milosz, any chance we could get the 32 bit CI set up?

Steve, the first commit you mention above (arm64: mm: Remove PTE_BIT_FUNC macro) is already present as is the second (arm64: mm: Introduce PTE_WRITE). Is there something else going on here?

@broonie
do you mean 64bit kernel + 32bit rootfs CI?

@mwasilew: Yes, please.

Hi Mark,
What about:
9c7e535fcc1725fc2e2d4f0d9dd14137f0243e23
"arm64: mm: Route pmd thp functions through pte equivalents"

If that doesn't work (or is already in), can you please send me the
git branch corresponding to the failure and I'll look at it when I'm
back from holiday (3rd June).

Cheers,
--
Steve

That's in there. There's a few bits and pieces I'm still catching up on but the git branch is the standard LSK - git://git.linaro.org/kernel/linux-linaro-stable.git linux-linaro-lsk

A bug has been found by Will in the way pmds are set in THP.

The following patch fixes some problems Will experienced in LTP:
http://lists.infradead.org/pipermail/linux-arm-kernel/2014-May/260476.html
This is in 3.15-rc8 as:
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=ceb218359de22e70980801d4fa04fffbfc44adb8

I see this is already in LSK, has this fixed the issue?

Cheers,
--
Steve

On 27 May 2014 16:27, Mark Brown <email address hidden> wrote:
> That's in there. There's a few bits and pieces I'm still catching up on
> but the git branch is the standard LSK - git://git.linaro.org/kernel
> /linux-linaro-stable.git linux-linaro-lsk
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1320782
>
> Title:
> LSK: BUG: failure at change_huge_pmd in huge_memory.c
>
> Status in Linaro Stable Kernel (LSK):
> New
>
> Bug description:
> While testing LTP test suite and memory stress tests the following bug
> found.
>
> Test case name: thp03
>
> Error log:
> --------------
> thp02 0 TINFO : mremap 0x7f90e01000 to 0x7f8fa01000
> thp02 1 TPASS : Still alive.
> [ 2525.441528] BUG: failure at /home/buildslave/workspace/linux-linaro-stable-lsk-pre-merge/mm/huge_memory.c:1490/change_huge_pmd()!
> [ 2525.455612] Kernel panic - not syncing: BUG!
> [ 2525.459900] CPU: 5 PID: 5959 Comm: thp03 Not tainted 3.10.39+ #1
> [ 2525.465924] Call trace:
> [ 2525.468388] [<ffffffc000087464>] dump_backtrace+0x0/0x12c
> [ 2525.473807] [<ffffffc0000875a0>] show_stack+0x10/0x1c
> [ 2525.478879] [<ffffffc0005f36c4>] dump_stack+0x1c/0x28
> [ 2525.483950] [<ffffffc0005ef0f4>] panic+0xe4/0x208
> [ 2525.488673] [<ffffffc00016fffc>] change_huge_pmd+0xf0/0xf4
> [ 2525.494178] [<ffffffc000156d7c>] change_protection+0x39c/0x49c
> [ 2525.500031] [<ffffffc000156f5c>] mprotect_fixup+0xe0/0x204
> [ 2525.505534] [<ffffffc000157180>] SyS_mprotect+0x100/0x210
> [ 2525.510953] CPU0: stopping
> [ 2525.513677] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.10.39+ #1
> [ 2525.519787] Call trace:
> [ 2525.522249] [<ffffffc000087464>] dump_backtrace+0x0/0x12c
> [ 2525.527667] [<ffffffc0000875a0>] show_stack+0x10/0x1c
> [ 2525.532737] [<ffffffc0005f36c4>] dump_stack+0x1c/0x28
> [ 2525.537808] [<ffffffc00008dea0>] handle_IPI+0x15c/0x174
> [ 2525.543050] [<ffffffc0000812cc>] gic_handle_irq+0x74/0x7c
> [ 2525.548464] Exception stack(0xffffffc000827df0 to 0xffffffc000827f10)
> ...
> Complete Error log attached to this bug and same can be found in this link.
> https://validation.linaro.org/scheduler/job/127011/log_file#L_27_9641
>
> Bug reported kernel code snippet
> -------------------------------------------------
> File: mm/huge_memory.c
>
> 1479 int change_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd,
> 1480 unsigned long addr, pgprot_t newprot, int prot_numa)
> 1481 {
> 1482 struct mm_struct *mm = vma->vm_mm;
> 1483 int ret = 0;
> 1484
> 1485 if (__pmd_trans_huge_lock(pmd, vma) == 1) {
> 1486 pmd_t entry;
> 1487 entry ...

Not visibly but the CI jobs seem to have stopped running unless I kick them by hand - I just did that, you should be able to see the results in LAVA soonish.

Didn't seem to have any impact.

Could you please revert:

2d89dd0 arm64: Introduce execute-only page access permissions

And give it another go? This appears to be skipping the write logic
and causing the problems to manifest in THP03.

cc: Catalin for info.

Cheers,
--
Steve

On Wed, Jun 04, 2014 at 09:34:06AM +0100, Steve Capper wrote:
> Could you please revert:
>
> 2d89dd0 arm64: Introduce execute-only page access permissions

Do you mean commit "arm64: Introduce execute-only page access
permissions" is in LSK? It's been in linux-next temporarily and I
reverted it in the arm64/for-next/core tree (and linux-next) for lack of
more intensive testing. But it really should not have been picked into
LSK since it's not even in mainline.

Thanks.

--
Catalin

Unfortunately due to the strong demand from users to get features in we're pretty much being forced to track -next rather than Linus' tree (as well as things like Lorenzo's cpuidle PSCI integration that haven't made it in).

I will have got around to picking up the revert if it's done as a revert (which it is) anyway as part of my regular catchups, I'll go pull this one a little sooner and see if that helps.

That revert doesn't seem to have made much difference unfortunately.

Okay,
So going through the logs, and testing on my board. The
change_huge_pmd problem has been solved.
There are some problems with libhugetlbfs unit tests, as they test fixes > 3.10.
I managed to get all the tests passing by cherry-picking:
07443a8 mm, hugetlb: return a reserved page to a reserved pool if failed
8312034 mm, hugetlb: grab a page_table_lock after page_cache_release
5944d01 mm, hugetlb: remove useless check about mapping type
8bb3f12 mm, hugetlb: fix subpool accounting handling
f522c3a mm, hugetlb: change variable name reservations to resv
4ef9184 mm, hugetlb: protect reserved pages when soft offlining a hugepage
af0ed73 mm, hugetlb: decrement reserve count if VM_NORESERVE alloc page cache
a63884e mm, hugetlb: remove decrement_hugepage_resv_vma()
72231b0 mm, hugetlb: add VM_NORESERVE check in vma_has_reserves()
37a2140 mm, hugetlb: do not use a page in page cache for cow optimization
c0d934b mm, hugetlb: remove redundant list_empty check in gather_surplus_pages()
b226102 mm, hugetlb: fix and clean-up node iteration code to alloc or free
81a6fca mm, hugetlb: clean-up alloc_huge_page()

This will require some thorough testing.

Cheers,
--
Steve

Do you have your cherry picks somewhere I can pull?

Hi Mark,
I've put them up here:
https://git.linaro.org/people/steve.capper/linux.git/shortlog/refs/heads/for-broonie/lsk-hugetlb

Cheers,
--
Steve

Great, thanks.

Ah, I see you applied everything in the opposite order to that listed - that makes things much easier! I've applied everything and thrown it at CI (should be pre-merge build 86), let's see what happens.

One thing if you're applying other people's commits: please remember to always add a Signed-off-by: - it's important for legal reasons. git cherry-pick and am both have a "--signoff" option for this.

Initial results on hardware show that it's made things happier - we now have more tests passing, though there is still one failure left. Only just saw this, not taken a look at all:

https://validation.linaro.org/dashboard/streams/private/team/linaro/ci-linux-linaro-stable-lsk-pre-merge/bundles/fb2729d88e75cd4153db4e529ee47f3f223245ef/

Still waiting for the models but I expect they are similar, most likely even better (they had one more test passing previous to this update). The still failing test is icache-hygene which has been passing on the models and failing on the hardware.

(Assuming the models are fine and other CI looks good I'll probably push these updates since they appear to be a substantial win all round.)

Yup, 90/90 on the model - thanks Steve!