Windows and doors are the largest culprit in lost of energy efficiency. It may not be the cheapest option but will provide you with the largest impact.
You should decide if you want above ground or in-ground. If you decide to go with in-ground, you should do some research on vinyl liner compared with concrete. :-) Good Luck!