If that is a rechargeable lithium, something like https://www.ebay.co.uk/itm/1PCS-5V-Micro-USB-1A-18650-Lithium-Battery-Charging-V0Q3-Charger-To-Board/402292719717?hash=item5daa83c065:g:uX0AAOSwx79evOvj should do the job.
I've bought some from one supplier (if you look around Ebay there are some providing a pack of 5) and they produce the expected 5V output from the battery but haven't got around to connecting one to my Pi yet. For the cost though, you can't lose much!
My only comment about your battery is that it may go flat fairly quickly (Wikipediea says 300mA average so only a couple of hours) - that may of course not be an issue for you.
Also, check that your battery can take the 1A charge current that these modules produce. It can be modified, but you'll need to understand the datasheet and change components to reduce the charge current.
If we ignore Peukert's Law for the moment, each one of your batteries (3.7 volt 560 mAH lithium batteries) will run one RPi4b for at most 1 hour. Let's do the math:
RPi4B power consumption ~ 500mA (if lightly loaded)
Time = 560mAh/500mA = 1.12 hours
But this is unrealistically optimistic because:
You'll need a boost converter voltage regulator to get the voltage up to the 5V level required by RPi; its efficiency may be optimistically estimated at 80%.
The discharge rate needed (500mA from a 560MAh-rated battery) does not allow you to ignore Peukert's Law.
Just as a WAG, it be surprised if your battery would last 30 minutes.
A 560mAh 3.7V battery contains 2.07Wh of energy. Given you'll need a boost converter that might be 85% efficient, that leaves you with 1.76Wh of energy. A RPi operates at 5V, and the amount of current drawn depends on the model and what it is doing. Around 400mAh is typical for a Pi3B not doing much, so that's 2W (5V * 0.4A). So your battery is going to last just over 50 minutes.
I use Anker phone batteries for projects a lot. I'm sure there are better solutions, but not simpler ones. The big will keep my Pi3 running for at least a day or two. The smaller one might be more practical for you at just $20.