Bluez doesn't work well with Multipoint Audio Devices

Modern bluetooth devices allow to connect to multiple computers at once (aka "multi-point"). My headset, for example, can connect to my cell phone and my laptop at the same time. Connecting actually works fairly well with bluez.

The desired behaviour is the following: When I try to play audio on one device it should play on the bluetooth headphones unless the other device is already playing audio.

However, with my current setup this doesn't work well.

Steps to repproduce
1. I play some youtube video on my laptop
2. I stop the video (and even close the tab)
3. I start playing musing on phone

Expected:
4. I can hear the phone's music on my headphones

Actual:
4. I cannot hear the phones music until I turn bluetooth on my laptop of or switch the output device in the sound panel.

I am on Ubuntu 20.04, my headphones are the Bose 700, and my phone is just a regular Android device.