The main downside I see is updates. The only way to do an update in the Lucene ecosystem was to read the whole document, modify it, delete the original, and write back the contents as a new document. Some syntactic sugar has been added to Solr to make this easier. Lucene itself doesn't support per-field updates, though some stacked update feature appears to be being worked on.
Otherwise, for read-only data, you won't get a better general purposes way of slicing/dicing/analyzing data. We've done a lot of work for clients where they use Lucene (through Solr) just for the fact that every column is indexed, and its easy to lookup/filter/group/facet on anything. That's why tools like kibana (built on ElasticSearch, another service built on top of Lucene) are so powerful even though they have relatively little to do with fulltext search.