I had experienced the same with my gem miners. It definately needs to be fixed, but not with a sledge hammer
However, I do think the sucess rate on a rarer gem would be higher then a common gem, atleast logically it would.
If you found a ruby after sorting through 100 tons of rock crushing each piece, would you take a HECK of a lot of time to remove it without damage?
If every pebble you picked up had atleast a single piece of garnet, how careful would you be about gem extraction? How many would you cast aside for easier to remove ones?
Anyway, thats the way things work in real life. In the game I see no problem with a small nerfing factor for when the stars allign and you find something really rare. Nothing is more depressing then destroying something that took you days to find.
Granted the way things are now do need to be fixed, and I would not be surprised if values were reversed. Just remember that rarity is as much a factor as success rate is.