I bought both the E-PL1 with the kit zoom lens and the GF1 with the 20 mm pancake lens. After going back and forth for a while, I've decided to sell the GF1 body (not sure yet how to find a buyer) and keep the Olympus with both lenses. But lots of other people have come to different conclusions.
As I've said previously, IMO the Panasonic is more fun to shoot with and the Olympus produces better results. In terms of RAW image quality, both cameras will produce nearly identical results, as they share the same sensor. But the Olympus has a truly engine; the Panasonic does not. I am too impatient (and lack enough hard disk space) to shoot all my photos in RAW and then PP. Furthermore, I am not a skilled photographer, so the absence of IBIS on the GF1 (and OIS on the pancake lens) meant that a significant number of my GF1 shots came out blurry. The Olympus also has some excellent art filters.
That said, the GF1 has much better build quality, a much better screen, and is noticeably quicker--not so much in AF speed but in shot-to-shot capacity. In particular, the E-PL1 has a slight (but noticeably) delay in displaying just-taken photos on its LCD. The differences in AF speed is mostly attributable to lens speed (the Olympus zoom lenses trade off AF speed for light weight and the ability to collapse).
In terms of controls, I actually prefer the Olympus since you can access almost every setting via the Super Control Panel. But the lack of a dial is really annoying.