What is a passion project and is this something negative?
A "passion project" is normally one written primarily to serve the needs/wants of the person writing it. It's not as simple as being always negative - certainly there's many reasons to undertake them and they can also fulfill a successful secondary position of having wide appeal.
In this context however I think it's unavoidable to consider it a negative - I think you have to take that as a comment that they don't believe the book has the mainstream appeal you were aiming for. That doesn't mean they were right - and I don't think we're particularly in a place to determine whether they were or not.
Further complicating the picture is that it may not even be the real reason for the rejection - it could be, or it could be a generic platitude to cover up anything from as mundane as "I just didn't like it" to "I don't want to touch a book that covers religious persecution in a country with a reputation for taking extreme dislike to criticism".
Single data points don't generally tell you much and single rejections are no exception - remember Decca Records rejected the freaking Beatles. So if this is a book you remain interested in getting published then shop it around a bit more - see if anyone else is interested.