Go to the home screen on your device
Hold onto any icon for 3 seconds.
The icons should all start to wiggle. If you want to delete an icon, click the "X." To move an icon, drag it around.
You can only delete apps that you have installed. The other apps, besides "Newsstand" can be put into a folder. There is another Snapguide I made on how to do that.
Here, I have clicked the "X" on podcasts. You should get this message. If you delete the app, you can still re-download it from iCloud or when your device is plugged in.