Building without Crafting is same as you go fishing without bait, you need to collect material and store it such as goal, tree, coal, oil, flour, meat etc to survive liven up your village, be aware of the weather save your supplies while the weather is perfect (summer) plant some fruit trees, wheat raise some animal pets (chicken, cow, sheep, goat) for your winter stock.
the prefect timing for plant and raise animal at the beginning of the dry season then you can harvest before winter.