ECMP tuning not effective when packet is received from a remote compute

We have the following set-up:

Two Vns: vn_right and vn_left
Two VM: VM_left and VM_right on a same compute (COMPUTE1)
A SI in scaled out with two Instances on another compute (COMPUTE2)

 VM_left ---vn_left -------[SI_scaled_out -- Instance1 -- ] ----- vn_right ---- VM_right
                                          -- Instance2 --
 [ COMPUTE1 ]============== [ COMPUTE 2 ]===================[ COMPUTE1 ]

Appropriate policy to send traffic through the SI for Rigth to/from Left communication.

From a a data plane perpective,

VM_Left ===== MPLSoUDP ====== [SI]====== ===== MPLSoUDP ====== CM_right

Testing showed that tuning ECMP settings has no effect on how traffic is send toward the scaled-out SI.

Example of per VN setting (both on right and left), where only destination IP is taken into account:

        "ecmp_hashing_include_fields": {
            "destination_ip": true, ############## only field set as true #####
            "destination_port": false,
            "hashing_configured": true,
            "ip_protocol": false,
            "source_ip": false,
            "source_port": false
        },

The problem also occurs when LB is modified on a PE port basis.

Having a same a set of flows with same IP SRC/dst and different ports, showed that traffic is still load balanced accross both instances of the scaled-out SI.

In the below trace: src = 5.5.5.3 / dst = 4.4.4.3

compute 1 vn left (Destination 4.4.4.3)

[root@compute1 qemu(keystone_admin)]# rt --get 4.4.4.3/32 --vrf 1
Match 4.4.4.3/32 in vRouter inet4 table 0/1/unicast

Flags: L=Label Valid, P=Proxy ARP, T=Trap ARP, F=Flood ARP
vRouter inet4 routing table 0/1/unicast
Destination PPL Flags Label Nexthop Stitched MAC(Index)
4.4.4.3/32 0 LP 40 20 -

[root@compute1 qemu(keystone_admin)]# nh --get 20
Id:20 Type:Tunnel Fmly: AF_INET Rid:0 Ref_cnt:194 Vrf:0
              Flags:Valid, MPLSoUDP,
              Oif:0 Len:14 Flags Valid, MPLSoUDP, Data:18 66 da 5e 5a c3 b8 2a 72 d6 8b 19 08 00
              Vrf:0 Sip:200.0.0.1 Dip:200.0.0.2

========== on compute 2 (packet received with MPLS label 40)

[root@compute2 ~]# mpls --get 40
MPLS Input Label Map

   Label NextHop
-------------------
      40 98
[root@compute2 ~]#

packets is popped/swapped toward the ECMP NH made up of both instances. In this case ECMP is not based on selected 5-tuples criterias.

[root@compute2 ~]# nh --get 98
Id:98 Type:Composite Fmly: AF_INET Rid:0 Ref_cnt:2 Vrf:2
              Flags:Valid, Policy, Ecmp,
              Sub NH(label): 116(57) 81(53)

We can check that IP flows will follow the same logic (e.g. NH of flow is the ECMP NH = 98)

Listing flows matching ([4.4.4.3]:*)

    Index Source:Port/Destination:Port Proto(V)
-----------------------------------------------------------------------------------
    23756<=>351916 5.5.5.3:0 17 (2->1)
                         4.4.4.3:0
(Gen: 10, K(nh):98, Action:F, Flags:, E:0, S(nh):20, Stats:32745/1506270, SPort 53135)