Ask most people to name their biggest fear about growing old, and chances are they won’t say gray hair and wrinkles, but the devastating loss of their mental capacity. Alzheimer’s disease accounts for up to 80 percent of all dementias, striking as many as 5 million Americans.

While the disease has genetic underpinnings, it’s also associated with certain lifestyle factors, including diet, exercise and level of education. So what steps can you take to help prevent it? Some recent studies suggest that eating more fruit and vegetables and less saturated fat may be the ticket. Others point to folic acid or fish oil supplements as beneficial. Still others have found that drinking moderate amounts of alcohol confers some protection.

But an expert panel recently convened by the National Institutes of Health says there’s not enough evidence from any of these studies to warrant making lifestyle changes to lower your risk of Alzheimer’s.

“The primary limitation with most of these studies is the distinction between association and causality,” write the NIH experts in their “state of the science” paper published in June in the Annals of Internal Medicine. For instance, people with a higher level of education have a lower risk of Alzheimer’s, but that doesn’t mean going to grad school will protect you.

It could be that those individuals read more books and play more chess in their lifetime than other folks, which continually challenges their brains and has a disease-preventing effect.

Here’s what the scientists concluded about the current scientific evidence concerning various factors thought to either raise or lower your Alzheimer’s risk:

• Nutrition and dietary supplements: The scientific evidence is strongest for omega-3 fatty acids from fish or fish oil supplements, with several studies showing an association between higher levels of omega-3s and a lower risk of cognitive decline. Still, the expert panel says there’s not enough evidence to prove that taking supplements or eating more fish will protect you from Alzheimer’s.

The evidence is very limited for other nutrient supplements—like vitamins B, C, E, folate and beta-carotene— and for dietary changes like lowering saturated fat intake and increasing vegetable consumption.

• Medical conditions: High blood pressure has been the one condition most often associated with reduced brain function—particularly severe decline. While diabetes may increase the risk of Alzheimer’s somewhat, the evidence isn’t as strong that the two conditions are truly linked. Studies have been scant or inconclusive for other conditions like obesity, sleep apnea or traumatic brain injury.

• Psychological problems: Depression has been consistently associated with Alzheimer’s, but it’s tough to tell which one comes first. Depressed mood can be a sign of early dementia, and the studies conducted on mood disorders and Alzheimer’s haven’t been able to fully establish whether depression can actually lead to dementia, according to the Duke University researchers who wrote a paper also published in June in the Annals of Internal Medicine.

• Social connections and cognitive engagement: There is a “robust association” between losing a spouse and cognitive decline, the expert panel says—not exactly something we can control. How about brain games and other mentally challenging tasks? The evidence is “limited but inconsistent” that learning a new language or honing crossword puzzle skills, for example, actually protects against loss of memory or reasoning.

• Exercise and leisure activities: So far, there’s only preliminary evidence that staying active or participating in hobbies like gardening, painting or attending a social club helps preserve cognitive function. The evidence could grow stronger as researchers conduct more studies.

• Smoking and drinking habits: Smokers have a higher risk of losing brain function as they age— that much is clear from the scientific evidence. The studies are less certain when it comes to predicting former smokers’ Alzheimer’s risk or the risk for those who drink excessive amounts of alcohol.

• Genes: Most studies have demonstrated an increased rate of cognitive loss in elderly folks who carry the ApoE gene variation, especially on memory tasks and the ability to quickly identify objects and faces. The gene variation, though, doesn’t appear to affect all areas of brain function and may not completely explain the global decline that occurs with Alzheimer’s.