The soil seems relatively sandy to me, which doesn't necessarily have to be a bad thing. How deep can you dig with a normal spade, and what is the soil like "inside"? High proportion of large stones? More clay? Or does it also have dark parts that suggest humus? In our case, the topsoil wasn't particularly good either. The proportion of stones and clay was quite high. If you want to get really good garden soil, rent a powerful rototiller, plow the soil once, collect all the stones, and work in a lot of compost. In our area, they have it at the local composting plant for a nominal fee for delivery (I paid €40 for 12 cubic meters). Distribute everything, till it all again with the rototiller, and the garden soil is perfect.