On this map I have outlined (In color) The locations of the gems.
They are not exact but more of a "search here"

https://gtm.steamproxy.vip/ugc/1664606352303619999/AC1F769873F79C744F80B8AF5A949D8BB9987D83/?imw=256&&ima=fit&impolicy=Letterbox&imcolor=%23000000&letterbox=false

The Forest Gem/Earth Gem/Green Gem is the quickest to obtain and may be a requirement to obtain the other gems first (Except the ice gem)

First you must craft a sword out of three specific parts.

https://gtm.steamproxy.vip/ugc/1664606352303631193/5BB35E610FB3EC68A9C3D5209637B6509BCF3CEA/?imw=256&&ima=fit&impolicy=Letterbox&imcolor=%23000000&letterbox=false

-Can be obtained from Geoffrey's parts store (Not guaranteed to show up, may have to progress to the next day)

https://gtm.steamproxy.vip/ugc/1664606352303631409/A9785A2C115A551819BB4060FFD89213C2CD9424/?imw=256&&ima=fit&impolicy=Letterbox&imcolor=%23000000&letterbox=false

-Can be obtained from Geoffrey's parts store (Not guaranteed to show up, may have to progress to the next day)

https://gtm.steamproxy.vip/ugc/1664606352303631635/4F09625EDB76D34E405D89263E0CD325014B8DC6/?imw=256&&ima=fit&impolicy=Letterbox&imcolor=%23000000&letterbox=false

-Crafted with two copper ingots.

Then you want to assemble them. If done right the weapon will be unique in name.
https://gtm.steamproxy.vip/ugc/1664606352303630931/80B9A2C830740FA64625E97EBA5E1851247411B5/?imw=256&&ima=fit&impolicy=Letterbox&imcolor=%23000000&letterbox=false

Next, head to the left from the back of the Smithy, into the forest.
You may see the glowing deer. He's a dead giveaway of the Pedestal.
https://gtm.steamproxy.vip/ugc/1664606352303630409/4D3A2AD1710FCAB0397990B7CE7547C7830812ED/?imw=256&&ima=fit&impolicy=Letterbox&imcolor=%23000000&letterbox=false

With the sword, Rotate it so the tip points towards the slit in the ground and place it into the slot. (Alt and mouse movement to rotate)
https://gtm.steamproxy.vip/ugc/1664606352303630158/A8DF2880FDCE4FFB9359E305A3FE1C37311064A6/?imw=256&&ima=fit&impolicy=Letterbox&imcolor=%23000000&letterbox=false

If done correctly, it'll snap into place and you'll be given the green gem.

Next is the light gem.
This gem and the Dark gem require something to occur BEFORE they show up. I don't exactly know what but on a new game the light gem was not present.

Facing away from smithy, while out the back, it's on the far right side of the map nestled between two rocks. It will float in place (If it's there at all)

https://gtm.steamproxy.vip/ugc/1664606352303629839/0B706105474678470AA891C5E6FDB2752DCCBC0D/?imw=256&&ima=fit&impolicy=Letterbox&imcolor=%23000000&letterbox=false

The requirement MAY be for it to be night time (Untested) As this is when it appeared for me on my other save.

This Gem, like the light gem ALSO has some unknown pre-requisite for it to show up. (I assume you have to get the forest gem first)

This one spawns in the mine in the side of the mountain at the edge of the map far behind the Smithy. It spawns floating in the air above some purple glowing flowers in a tucked away small section of the cave behind a wall of rock.

https://gtm.steamproxy.vip/ugc/1664606352303630660/B6A792C3C4C516E2F6AFE2F20F5202A9587DBF9C/?imw=256&&ima=fit&impolicy=Letterbox&imcolor=%23000000&letterbox=false

By far the most tedious (At least in my opinion)
The Ice gem is far from the Smithy and has a puzzle.
Luckily I have found a very quick and doubly rewarding strategy for obtaining this trinket.

First, approach the bridge to the castle.
https://gtm.steamproxy.vip/ugc/1664606352303629454/93FC6B2DCE02ED93D3ACF199C1CFB2B5558B9587/?imw=256&&ima=fit&impolicy=Letterbox&imcolor=%23000000&letterbox=false

Next, approach the guard. And left click on him to bop him one.
https://gtm.steamproxy.vip/ugc/1664606352303629136/740AB177570544CD2E403B8BBB7D458E4D4F8671/?imw=256&&ima=fit&impolicy=Letterbox&imcolor=%23000000&letterbox=false

You will be arrested (Teleported without any consequence) Into a jail cell. Use the bed inside to sleep for one hour.
Upon sleeping for an hour, open the door and go to last cell on the opposite side for a chest containing some loot. The doors out are a one way teleporter, to get the loot out you will have to hold E to ghost the object and then keep it ghosted then head out the doors and un-ghost it with E by placing it on the ground.
https://gtm.steamproxy.vip/ugc/1664606352303628833/9DBA320B44DAD6A90D9DA5CB2F85C7612005B356/?imw=256&&ima=fit&impolicy=Letterbox&imcolor=%23000000&letterbox=false

Once you're out (With or without your loot) Head for the big archway to the left on the stone path.
You will find a fenced in area and a pink teleporter that will turn red as you approach.
https://gtm.steamproxy.vip/ugc/1664606352303628427/1C793AF37BE8AF38A9A1A84F425591C1B441E664/?imw=256&&ima=fit&impolicy=Letterbox&imcolor=%23000000&letterbox=false

Stand on it and wait, and it will summon floating cubes. These cubes are a puzzle. The puzzle is very simple. Click on two cubes, see what the symbols on each cube are, and then click more and match the cubes in groups of two, to the symbols that show on each when you click them.
https://gtm.steamproxy.vip/ugc/1664606352303627629/03943A7ECC6A83CCC967C90F20E2C790DD4B1A0A/?imw=256&&ima=fit&impolicy=Letterbox&imcolor=%23000000&letterbox=false

Repeat this 3 (4?) times until a chest appears
Claim your skill book and gemstone.
https://gtm.steamproxy.vip/ugc/1664606352303627986/6F262F98A2E970218D3E2E642171236187ABE0EC/?imw=256&&ima=fit&impolicy=Letterbox&imcolor=%23000000&letterbox=false

Some things are still not clear to me (Like the location of the fire gem)
Some small ideas if the gems are not showing up:
-Start with the forest gem and then do the ice gem. These two puzzles may cause the light and dark gems to appear.

-The light and dark gems may only show up during the night or day. (Untested)

-Getting jailed to reach the ice gem is not necessary. It's just much faster than walking and is the only way to obtain the gauntlet piece in the chest.

-The forest and ice gems do NOT require and pre-requisite actions. (Tested)

-The fire gem is not in the Smithy. (I have checked extensively)

-The sword parts to obtain the forest gem are RNG based and you may or may not be successful in getting them.

-This is my first guide and I have not figured out how to get the images to embed, so if anyone knows please tell me.

-If anyone has any idea where the fire gem is, tell me and I will update and edit the guide accordingly.

Small side note: An older post on the wiki says that some of the gems can be obtained randomely from dungeons.
If you want to try this RNG, craft a SECOND one of the swords made in the forest gem step (It's cheap to make if you can get the parts) sharpen the blade (Just the blade, not the whole assembled sword) and give it to the adventurer outside and it should boost them above the 40 gear quality required to depart.

Final note: Due to a bug, all the grips in Geoffrey's store do not seem to have a cost. This means you can buy ALL of them. However, you can ONLY buy them at all, if you also buy something that DOES have a copper value to it. So put a random guard piece into the purchase and watch your profits roll in. (I am unsure if it's just me who has this.)