UV unwrapping is a difficult topic. They can be both combinatorial algorithms or variational methods but in general they're optimization based, i.e. you setup an optimization problem and you solve it using some numerical optimization solver. I'm just going to give you some names and some libraries you can use eventually. I'll give you few names (both classic and more recent) and you can have a look yourself. ABF (Angle Based Flattening), LSCM (Least Square Conformal Mapping), ARAP (As rigid as possible), BFF (Boundary first flattening), Variational Surface Cutting, OptCuts.

For ARAP and LSCM you can use libigl. Boundary first flattening was developed by Keenan Crane few years ago. Variational surface cutting is an improvement over BFF. And I recently discovered OptCuts. The code is available and you can run them yourself.

You need to read through the papers to understand how they works if you wanted to implement them from scratch (not going to be easy, parametrization is a difficult subject). But if you want to just to run them just clone the repos and try them yourself. (libigl comes with tutorials, and two of these are about ARAP and LSCM). BFF, Variational Surface Cutting and OptCuts have a GUI based app so you can see the unwrapping and you can easily isolate the code if you need them in your own code.