As it turns out in order for the music mod to work, you must have at least 2 songs for menu tracks and race tracks, otherwise the game will crash.

To start creating a music mod of your own, you will need two main tools:
BF3Pack Smart - googling "flatout uc bfs file unpacker" should show a video that has the program in description to download
Any music editing software, alternatively online converter (mp3 to ogg, wav to ogg, [Any file format your song has] to ogg).I personally use Audacity.

Something to convert your music to ogg is needed as that's more less the only sound format the game can read.

BF3Pack Smart is a tool that works with Flatout UC's BFS file system. It can pack folders into bfs files and unpack bfs files into folders. I recommend using bfs3pack_smart_gui as it's easier to use.
In order to pack a folder using gui, you open the program and drag the folder on the small window. It will then ask you where you want to save the bfs file. Similarly, you unpack a bfs file by draging it over the window and pointing where it should be unpacked.

Create a folder named "data". Inside that folder create another folder named "music". This is required as that's Flatout UC's filepath for track list. By setting up a new bfs file with these two folders, we can override base track list with the new one specified in our mod.

Now, inside the music folder we need to create at minimum two files:
playlist_ingame.bed
playlist_title.bed

You can create these by renaming a .txt file with .bed extension.

Unlike in Flatout 2, Flatout UC's playlist references a folder outside the bfs file. Specifically, it references data/songs in Flatout UC directory. As such, in the bfs you only create playlist files that contain which files the game should look for and what track information it should show.

Fortunately, just like in Flatout 2, both playlist files have exact same structure. The best way to explain it is by providing an example:
This code will make the game play songs from a six track playlist.

"Loop" brackets open at the beginning and end at the very end of the code. They indicate that it's a code for our filesystem.

"Count" states how many tracks the game should expect. 6 means that the playlist should have 6 tracks information provided.

The most important part is the track info itself. Each track you want to add follows following formula:

[which track added it is] = {
File = "data/songs/[filename.ogg]",
Artist = "[Name of the artist]",
Song = "[Name of the song]",
StartPos = 0,
},

In example above, I named all music files numers, from 01 to 22. That is not necessary - you can name the file anything you want, whether it is a number or the track name, as long as it is in the ogg format.

After you have prepared playlist files using code from previous section, you should now have something like this:

data -> music -> playlist_ingame.bed and playlist_title.bed

Use bf3pack_smart to pack the data folder. I recommend naming the bfs file "songs.bfs". Open Flatout UC directory and move the newly created bfs file in it.

Inside Flatout UC there is a "filesystem" file with no extention. Edit it with notepad and add a new line "song.bfs" at the bottom. Make sure you haven't created an extra empty line, easiest way to check is using arrow keys (if it goes down the line to an empty line then delete the line using backspace). Save and close the file.

This is the part which actually includes song files. Again, all songs you want to use need to be in .ogg file.

Assuming you haven't used bfs3pack before, "Data" folder inside Flatout UC directory should not contain much, but it definitely has a "Songs" folder. Open it. Inside the folder are default songs that play in the game. You can either remove them or keep them in, no difference. Add songs you specificied in playlist files here.

Assuming everything was done correctly, booting up the game should now show the new playlist as well as play the songs you added.

In case something goes wrong, here are some issues I could think of:

Booting up the game shows "Invalid archive version" - You used bfs2pack instead of bf3pack. bfs2pack creates bfs files for Flatout 2. Unfortunately it's not compatible with Flatout UC.

The game boots up but the default playlist plays/default song names show but no song plays - in both cases there probably is an error in the playlist files. If any mistake appears in playlist files, the game will immediately ignore it and default to regular playlist. No song playing means on top of that you removed the default songs. In either case, make sure the code is correct. If it is, make sure you are packing the data folder, inside of which there is the music folder, inside of which there are the playlist files.

The game boots up and the new song names show, but no sound is playing - the files are either not placed in the Songs folder or are the wrong format. The music files are not to be put in the bfs file (unlike Flatout 2). Make sure the music files are in the correct place.

Hopefully the guide explains well enough how to create a music mod of your own.

The tutorial skips replacing stunt music. If you wish to replace stunt songs as well, you create a playlist_stunt file with the exact same formula as other files. Thanks for reading!