The Game recording feature in Steam works really well while using minimal resources, has good settings etc but exporting in H.265 seems to be universally broken. This forum is full of the same error im getting as well:

Failed to export, try again (2)

Exporting in H.264 under custom settings works but this is limited to 8bit SDR Rec. 709. If the clip captured anything with HDR on, it will be missing the metadata and have washed out colors and wrong brightness values.

Has exporting in H.265 ever worked for anyone? Cant seem to find a single instance of that.

There is a work around to extract the clips from the clip folder (mixed results):

Go to the Steam folder-> Userdata -> (User id) -> Gamerecordings -> Video -> (Video id)
Then open the session.mpd file in VLC player. In VLC in the playlist right click the clip -> Save -> Convert and set up the codec and video format you want, then destination and it will export in a video format.

To find the files easier you can also just go into the steam folder and in windows explorer search for "mpd". FFmpeg can export mpd files too.

Did anyone manage to get this to work properly through Steam?

Im on the 5080, newest driver, newest windows, hevc and gpu encoding enabled.