The evidence is based on archaeological findings, as well as (less directly and admittedly with some conjecture) on anthropological studies of modern hunter-gathering societies, e.g. the Bushmen of the Kalahari.
'Until recently, archaeologists had to resort to indirect tests, whose results (surprisingly) failed to support the progressivist view. Here’s one example of an indirect test: Are twentieth century hunter-gatherers really worse off than farmers? Scattered throughout the world, several dozen groups of so-called primitive people, like the Kalahari bushmen, continue to support themselves that way. It turns out that these people have plenty of leisure time, sleep a good deal, and work less hard than their farming neighbors. For instance, the average time devoted each week to obtaining food is only 12 to 19 hours for one group of Bushmen, 14 hours or less for the Hadza nomads of Tanzania. One Bushman, when asked why he hadn’t emulated neighboring tribes by adopting agriculture, replied, "Why should we, when there are so many mongongo nuts in the world?"
While farmers concentrate on high-carbohydrate crops like rice and potatoes, the mix of wild plants and animals in the diets of surviving hunter-gatherers provides more protein and a bettter balance of other nutrients. In one study, the Bushmen’s average daily food intake (during a month when food was plentiful) was 2,140 calories and 93 grams of protein, considerably greater than the recommended daily allowance for people of their size. It’s almost inconceivable that Bushmen, who eat 75 or so wild plants, could die of starvation the way hundreds of thousands of Irish farmers and their families did during the potato famine of the 1840s.'
I'm not advocating a sudden worldwide return to hunting and gathering; our planet's population is huge and grossly dependent, both nutritionally and economically, on the growing, buying, and selling of staple crops, such as rice. It does, however, make me wonder what would happen if I stopped eating rice, wheat, and other foods I would only find in agricultural societies, as my body, like that of all humans, is adapted for eating stuff I find or kill. I started looking into this with greater interest after running across Mark's Daily Apple, where Mark and his minions write about their modern "primal" lifestyle (which is, of course, closely related to the paleo diet, for those who have seen mention of that in recent years). This has led to my beginning to experiment more with cooking greens and baking with almond flour (the cookies were good, the bread is stellar, the pie was a disappointment) and I'm looking forward to trying and learning more; and who knows? Maybe I will see results.