[EHL] Integrated TSN controller (stmmac) driver support

Update Patch list to be backported:
(5.11)
676b7ec67d79 stmmac: intel: Enable HW descriptor prefetch by default
96874c619c20 net: stmmac: Add HW descriptor prefetch setting for DWMAC Core 5.20 onwards
5e6038b88a57 net: stmmac: fix TSO and TBS feature enabling during driver open
17cb00704c21 stmmac: intel: set TSO/TBS TX Queues default settings
622d13694b5f xdp: fix xdp_return_frame() kernel BUG throw for page_pool memory model

Patches still under discussion:
[net,V2,1/1] net: phy: fix invalid phy id when probe using C22
[net-next,v2,1/2] net: stmmac: enable 2.5Gbps link speed
[net-next,v2,2/2] net: pcs: configure xpcs 2.5G speed mode

Features yet to be enabled:
Wake-on-LAN support - Advanced Power Management (APM)
Magic Packet support: Expected timeline

Packet HW timestamping support via AF_XDP socket - TSN specific
Packet Transmit Time support via AF_XDP socket - TSN specific

The following patches have been landed in Ubuntu-intel-5.11.0-1009.9 but with different SHA.
We can skip them.

mainline ubuntu
8eb37ab7cc04 0ed5ee99d2b7 stmmac: intel: Fixes clock registration error seen for multiple interfaces
9a7b3950c7e1 3caea6cc6b79 net: stmmac: Fix VLAN filter delete timeout issue in Intel mGBE SGMII
879c348c35bb c61e714ff1c8 net: stmmac: fix incorrect DMA channel intr enable setting of EQoS v4.10
24877687b375 725d7f7a4e39 net: stmmac: fix CBS idleslope and sendslope calculation
d82c6c1aaccd 4f3320e619b4 net: phylink: Fix phylink_err() function name error in phylink_major_config
5e6038b88a57 e5f16595557e net: stmmac: fix TSO and TBS feature enabling during driver open
622d13694b5f 286fb6686ca7 xdp: fix xdp_return_frame() kernel BUG throw for page_pool memory model

Continue #2, and the following patches have been landed in Ubuntu-intel-5.11.0-1009.9 with the same SHA.

ab1c637cc6d8 stmmac: intel: Fix kernel crash due to wrong error path
29e0c2f39f98 stmmac: intel: Place object in the Makefile according to the order
ccacb703b0f8 stmmac: intel: Fix indentation to put on one line affected code
d5383b037664 stmmac: intel: Eliminate useless conditions and variables
52c1f7948454 stmmac: intel: Convert to use pci_alloc_irq_vectors() API
e578f043ffcf stmmac: intel: Remove unnecessary loop for PCI BARs
09f012e64e4b stmmac: intel: Fix clock handling on error and remove paths
801eb0501824 stmmac: intel: Fix kernel crash due to wrong error path
c89f44ff10fd net: stmmac: Add support for VLAN promiscuous mode
b9663b7ca6ff net: stmmac: Enable SERDES power up/down sequence
d63439f575dc net: stmmac: add EHL 2.5Gbps PCI info and PCI ID
67c08ac4140a net: stmmac: add EHL PSE0 & PSE1 1Gbps PCI info and PCI ID
58da0cfa6cf1 net: stmmac: create dwmac-intel.c to contain all Intel platform
ed64639bc1e0 net: stmmac: Add support for VLAN Rx filtering
3e07df430c2b net: stmmac: Get rid of custom STMMAC_DEVICE() macro
909c1dde67c4 net: stmmac: update pci platform data to use phy_interface
2f633d5820e4 net: stmmac: xgmac: fix missing IFF_MULTICAST checki in dwxgmac2_set_filter
2ba31cd93784 net: stmmac: fix missing IFF_MULTICAST check in dwmac4_set_filter
907a076881f1 net: stmmac: xgmac: fix incorrect XGMAC_VLAN_TAG register writting
9eeeb3c9de4e net: stmmac: fix incorrect GMAC_VLAN_TAG register writting in GMAC4+
c593642c8be0 treewide: Use sizeof_field() macro
8e5debed3901 net: stmmac: Use rtnl_lock/unlock on netif_set_real_num_rx_queues() call
785ff20bce2d stmmac: intel: Fix kernel panic on pci probe
388e201d41fa net: stmmac: Modify configuration method of EEE timers
b4c5f83ae3f3 stmmac: intel: Adding ref clock 1us tic for LPI cntr
ac322f86b56c net: stmmac: Fix clock handling on remove path
e0f9956a3862 net: stmmac: Add option for VLAN filter fail queue enable
d0ea5cbdc286 drivers/net/ethernet: clean up mis-targeted comments
7241c5a69747 net: stmmac: removed enabling eee in EEE set callback
aa042f60e496 net: stmmac: Add support to Ethtool get/set ring parameters
9f19306d1666 net: stmmac: use netif_tx_start|stop_all_queues() function
686cff3d7022 net: stmmac: Fix incorrect location to set real_num_rx|tx_queues
0366f7e06a6b net: stmmac: add ethtool support for get/set channels
9efc9b2b04c7 net: stmmac: Add dwmac-intel-plat for GBE driver
f317e2ea8c88 net: stmmac: set TxQ mode back to DCB after disabling CBS
7cfc4486e7ea stmmac: intel: Configure EHL PSE0 GbE and PSE1 GbE to 32 bits DMA addressing
8450e23f142f stmmac: intel: Add PCI IDs for TGL-H platform
523437d7b5ba net: stmmac: allow stmmac to probe for C45 PHY devices
bff6f1db91e3 stmmac: intel: change all EHL/TGL to auto detect phy addr
be1c7eae8c7d net: stmmac: Enable EEE HW LPI timer with auto SW/HW switching
b618c3270249 net: phy: mscc: fix excluded_middle.cocci warnings
043d2be2a782 net: phy: leds: Deduplicate link LED trigger registration

Hi @pragyan and @apoorv,

I have checked that the 99 patches[1] are all landed in the mainline kernel (ff1176468d36 Linux 5.14-rc3).

Can I expect to use the mainline kernel to make the TSN controller work (5.14-rc3 or above)?
So that I can use it as a reference kernel.

Thanks!

---
[1] After comparing all the patches mentioned in the bug description and comment #1, except the still under discussion patches.

Hi @shanelin,

Yes, you can use 5.14-rc3 has reference kernel.

Regarding the patches under discussion:
[net,V2,1/1] net: phy: fix invalid phy id when probe using C22
[net-next,v2,1/2] net: stmmac: enable 2.5Gbps link speed
[net-next,v2,2/2] net: pcs: configure xpcs 2.5G speed mode

These 3 have been accepted in mainline kernel. Please proceed to backport them as well

Regarding features that were yet to be enabled as mentioned in #1:
Wake-on-LAN support - Advanced Power Management (APM)
Magic Packet support: Expected timeline

The following patches have already landed upstream, please proceed to backport them:

622d13694b5f xdp: fix xdp_return_frame() kernel BUG throw for page_pool memory model
08041a9af98c net: phy: marvell10g: enable WoL for 88X3310 and 88E2110
6b28a86d6c0b net: stmmac: Terminate FPE workqueue in suspend
1dd53a61488d stmmac: intel: set PCI_D3hot in suspend
945beb755633 stmmac: intel: Enable PHY WOL option in EHL
5a9b876e9d76 net: stmmac: option to enable PHY WOL with PMT enabled
3c3ea630e87c stmmac: intel: fix wrong kernel-doc
fb9349c4163e stmmac: intel: move definitions to dwmac-intel header file
52e597d3e2e6 net: stmmac: Fix potential integer overflow
3e6dc7b65025 net: stmmac: Fix unused values warnings
1f7096f0fdb2 net: stmmac: Fix mixed enum type warning
46682cb86a37 net: stmmac: enable Intel mGbE 2.5Gbps link speed
597a68ce3216 net: stmmac: split xPCS setup from mdio register
ab00f3e051e8 net: stmmac: fix issue where clk is being unprepared twice
5ac712dcdfef net: stmmac: enable platform specific safety features
e80fe71b3ffe net: stmmac: Add callbacks for DWC xpcs Energy Efficient Ethernet
4c7a94286ef7 net: stmmac: Clear receive all(RA) bit when promiscuous mode is off
db7c691d7f4d net: stmmac: cleared __FPE_REMOVING bit in stmmac_fpe_start_wq()

#MaxLinear GPY PHY driver
7d901a1e878a net: phy: add Maxlinear GPY115/21x/24x driver
8b72b301b442 net: phy: add API to read 802.3-c45 IDs

Hi @sachinmokashi,

Could you share the TSN verification steps to know if the backport I did is correct?

Hi @shanelin,

Intel has shared the Testcases document "Intel_IOTG_Testcases_v1.pdf" with Canonical folks (<email address hidden>, <email address hidden>) which have TSN tests.

Can you please contact them for the testcases. Thanks.

Hi @sachinmokashi,

I got the document "Intel_IOTG_Testcases_v1.pdf", thanks!

Regarding comment #5 you mentioned:
[net,V2,1/1] net: phy: fix invalid phy id when probe using C22
[net-next,v2,1/2] net: stmmac: enable 2.5Gbps link speed
[net-next,v2,2/2] net: pcs: configure xpcs 2.5G speed mode

I can not found these patches in the mainline kernel, neither net, and net-next.

Could you please provide the commit ID?

Thanks!

I have backported the following patches, but not submit yet. (Please refer to the attachment)

ab1c637cc6d8 stmmac: intel: Fix kernel crash due to wrong error path (5.7)
...
...
622d13694b5f xdp: fix xdp_return_frame() kernel BUG throw for page_pool memory model

Hi @sachinmokashi,

After backporting the patches (the WoL has not been backported yet), I can see the stmmac related initialization dmesg. But it seems probed fail.

[ 3.175073] intel-eth-pci: probe of 0000:00:1e.4 failed with error -110

I have also tried the mainline kernel v5.14-rc3 [1] as I mentioned in comment #4, and got the same error message.

Could you please help to take a look at the dmesg and give the advice if I need to adjust the jumper or BIOS settings or any other additional steps?

Thank you!

---
[1] https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.14-rc3/

Hi @shanelin,

Regarding your comment #9, please ignore these patches as they were just interim

[net,V2,1/1] net: phy: fix invalid phy id when probe using C22
[net-next,v2,1/2] net: stmmac: enable 2.5Gbps link speed
[net-next,v2,2/2] net: pcs: configure xpcs 2.5G speed mode

Hi @shanelin,

For the comment #11, Please go through the attached Document released by Intel for its customers. You can go through the BIOS and TSN tabs of the sheet to make the necessary BIOS settings/ TSN configuration.

For the TSN tests, you can refer to the RDC document mentioned in this sheet:
TSN Ref SW Get Started Guide (ID# 616446)

We do not have access to ID#616446 yet, have reached out to Pragyan for help.

Got the #616446 document and have shared with Shane.

I have filed a new public bug for TSN enablement. (bug 1939511)

For the other WoL/magic packet/MaxLinear GPY PHY driver patches (comment #6), I will file another public bug later.

Test kernel for TSN (without WoL, etc): https://people.canonical.com/~jesse/linux-intel/tsn01/

Hi @sachinmokashi,

I got a new EHL hardware (rev 201).

One of the ethernet ports can work. (On the Top-Left port)
I tried to check the other two ports by "ifconfig -a", dmesg, and lspci [1], but they were still unavailable.

Is this behavior expected? (Only one ethernet port works)

---
lspci:

00:1d.1 Ethernet controller [0200]: Intel Corporation Device [8086:4ba0] (rev 11) (prog-if 18)
        Subsystem: Intel Corporation Device [8086:7270]
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 128
        Region 0: Memory at 6001300000 (64-bit, non-prefetchable) [size=256K]
        Capabilities: [80] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [90] Vendor Specific Information: Len=14 <?>
        Capabilities: [d0] MSI: Enable+ Count=32/32 Maskable+ 64bit+
                Address: 00000000fee00418 Data: 0000
                Masking: c3ff0000 Pending: 00000000
        Kernel driver in use: intel-eth-pci
        Kernel modules: dwmac_intel

Hi @shanelin,

Are all the patches as mentioned in comment #16 already backported? There would be dependencies on MaxLinear GPY PHY driver patches.

Also, for the port that works, are you able to get the IP and connect to network? This would help Doug to run Checkbox test suite

Hi @sachinmokashi,

Yes, the port that works can get the IP and connect to the network.

Is there any difference between rev 200 and 201 for the ethernet/PHY circuit?
Because the rev 200 hardware can not probe any ethernet interfaces, even I use the same kernel.

Thanks!

Hi @Shanelin,
There is no difference w.r.t ethernet/PHY circuit.
You can access the schematics from document id#607610

Successfully backported all the patches mentioned in the description, comments #1, #2, and #6

Please refer to the source code and changelog for more details.

Source code: https://git.launchpad.net/~shanelin/ubuntu/+source/linux-intel/+git/focal/log/?h=tsn02

Changelog: https://git.launchpad.net/~shanelin/ubuntu/+source/linux-intel/+git/focal/tree/debian.intel/changelog?h=tsn02

Test kernel for comment #22 (include all patches)

https://people.canonical.com/~shane/linux-intel/tsn02/

Status changed to 'Confirmed' because the bug affects multiple users.

From our testing on the rev201 board, the kernel in #23 does not work better:
1. The two non-functional ethernet ports are still non-functional
2. The ethernet port that used to work now can't get any IP

Our earlier question on reference kernel still remains. We tried 5.14-rc3 as suggested but we still only get one port working (can get IP).

From Chao Qin:
After confirm with TSN feature owner, it needs to rework the board to enable the third network interface.