Probably the cheapest way to stay warm when it's cold is with a wood stove... IF you can get out and get your own wood. Buying it may be cheaper than buying gas or oil, but again, it may not.
You can get wood for free or cheap but you have to work at it. Some places (newspapers, notably) get deliveries on wood pallets which they sometimes give away. It's cheaper to do that than to ship them back where they came from and they're there for the asking.
Besides that, keep an eye on your local Freecycle or Craigslist for trees that need to be removed or other free firewood for the picking/cleaning up. You might even have some in your own backyard.
Besides wood stoves? Kerosene can be inexpensive if you live in a small house, but there are dangers to using kerosene heaters indoors. Look for one that's made specifically for indoor heating and don't use lamp oil in it. Go to a gas station or farm supply store and ask for plain old fashioned kerosene. It will smell, yes. If you can't stand the smell, you can get kerosene that's been deodorized, but it will cost more.
Electric heaters can inexpensively heat a small space like one room of your house while you turn down the heat in the rest of it. Figure the cost by finding the cost per kilowatt hour on your electric bill, then multiplying times 1 1/2, which is 1500 watts, the maximum allowed by law. Multiply that by how many hours you use the heater to get the cost.
How else can you stay warm? Warm yourself and not the house.
Or go for some extremely frugal warming up tips.
Or see how I stayed warm when the power went out!
Don't let conventionality make you pay more in heating bills than you need to. If your bills are too high, look around, make some adjustments and do what you can to stay warm and keep as much of your money as you can.
