Recharging (rechargeable) lithium-ion batteries is — unfortunately — not a straight-forward process.
Unless you cap current you will damage severely-discharged batteries during the initial charging phase. They react very poorly to being overcharged. Even small amounts of undervolting will translate to large amounts of capacity loss.
Added to that, there are a number of different rechargeable lithium-ion chemistries — each with distinct charge voltages.
While it is possible to use a very simple solar-powered circuit to charge a Li-ion battery, and you are unlikely to rupture a battery and set fire to anything (in the short run, at least), the highly uncontrolled nature of solar is such that there's no way you're going to charge the battery the way it wants to be charged and, as a result, you will without doubt have highly variable charge states and shorten the battery's lifespan.
Rechargeable lithium-ion batteries are still expensive. It's worth spending the (relatively small amount of) money on a purpose-built charger.
There are a number of battery chemistries that are very tolerant of 'poorly controlled' charging regimes (Ni-Fe comes to mind). If you want a simple charging system, then I would suggest looking at those. Li-ion, however, is not one of them.
If you want to test for yourself that the above is true, and since you have already purchased the PV panels, you could set up a recharging area that is fire-proof and dedicate a (clearly marked) set of batteries just for simple solar charging. Just be prepared to have them die quickly.
More details can be found here: