I have been looking for something like this for a long time. I chose the Arduino Uno because of its low price and ease of programming (and favorable reviews). I picked up an Arduino Uno starter kit from Amazon for about the same price as just the Uno board at Radioshack. The starter kit includes the Arduino Uno R3 board, a USB cable, a small breadboard for prototyping, a tray for holding the Uno and the breadboard, and some hookup wires.
You can also get Arduino "shields", which are purpose-specific boards that plug into the Arduino. Some shield examples are for ethernet, wireless, and motor control. I haven't tried any of the shields yet.
The Arduino software is java-based and therefore cross-platform. It works well on my Linux machine. Just load up your program and click the Upload button. The program will automatically compile and upload to the Arduino, which will then run it.
Finally, here is my Arduino driving an LCD module. It only took a few minutes to modify an example program to get it to display what I wanted.
I encourage anyone looking for this sort of thing to give the Arduino a try. Happy tinkering!