Google Play Music for Android shows the video inside the mobile app, so it doesn't launch the YouTube app. Google Play Music for iOS opens the YouTube app.
In the desktop site, the YouTube player is overlaid. Press Esc or click outside the player to hide it.
I've checked a long list of songs and the "start video" option is available for many popular songs that have music videos. You can also find YouTube videos when searching for artists and albums.