I did buy the X-E1 with kit lens and 35 mm f1.4 to supplement my Canon 5D MKII and 5D MKIII; so I sympathize with you.
However, I think you are giving up too soon on becoming a portrait photographer. I am retired but do enough portraits to pay for my equipment, ca $6,000 per your for a couple hours work per week. You just need a different group of subjects. I found that the most willing group of subjects are wannabe models. I was able to get plenty of subjects by advertising on Craigslist and Model Mayhem. All shoots were done on a TF basis, meaning that I didn't pay the models and they didn't pay me, but we both got nice pictures for our portfolios as you can see on my pbase site, www.pbase.com/jcassatt.