If you want to try WebM, the open video format released by Google, you first need a browser that supports it. For now, WebM is not supported in a stable version of a browser, but you can install a Firefox nightly build or an Opera build. I download Opera 10.54, which is more stable than the nightly builds of Firefox.
After installing the browser, go to YouTube's HTML5 experiment and click on "join HTML5 beta". To find videos that are available in the WebM format, use YouTube's search feature and append &webm=1 to the URL, like this: http://www.youtube.com/results?search_query=web&webm=1.
Click on a video and you'll notice that YouTube no longer uses Flash or H.264. There's even a messages that makes the video player more cluttered: "HTML5 * WEBM".
YouTube converted a small percentage of videos to WebM. A search for [web] returns 4,610 WebM videos, while millions of other videos aren't available yet in this format. Check Opera's blog post for more information about the WebM format, embedding WebM videos and more examples of WebM videos.