A reasonably priced, fast, fairly compact, good optical quality, 24 mm-ish prime lens for APS-C sensor. Is that too much to ask for?
Do I really have to switch to different brand (Fuji?) to get this kind of lens? frustrating.
Canon make a 24mm or 28mm f2.8 with IS and a 28mm f1.8 which isn't much different in size than the 85mm f1.8.
Is there a problem with any of those options?
These may be great for FF, but I don't find f/2.8 to be fast enough for APSC.
Then your problem isn't finding fast enough lenses. Your problem is that you are putting the wrong camera behind them. This might sound flippant, but I'm quite serious. Think this through with me.
Making very fast wide glass is (a) quite expensive, (b) heavy and bulky, and (c) unlikely to be commercially viable because most people who care enough about their images to be wanting roughly what you want will either have already made the jump to 135 format (something like a 6D or 5D II body), or else be planning to make the upgrade before too long. Either way, they are not going to be buying (say) an EF/S 24mm f/1.4 prime. They will be getting the exact same framing, perspective, and depth of field from a readily available, inexpensive ($550 US) 35mm f/2 prime.
Actually, the 35/2 on a 6D is a little bit wider than a 24mm lens on APS-C, and it's a bit faster too (more DOF control) - you only need f/2.2 to match DOF with an APS-C system at f/1.4). And, of course, it's way cleaner in low light.
There is your answer - full-frame and a 35/2. Or, if you can find the extra cash, the 35/1.4L is a sheer delight to use - albeit quite expensive, so not for everyone.
PS: I hasten to add that I am not one of those annoying Full-frame Is The Answer To Everything evangelists. I own and regularly use full-frame, APS-C and APS-H bodies, and love them all - but I use different tools for different tasks as they are best suited.What you are looking to do is a natural task for a full-frame body and a 35mm prime. Go that way if you can.