Type 2 diabetes mellitus (T2DM) is a risk factor of dementia. The effect of T2DM treatment quality on dementia risk, however, is unclear. 1,342 elderly individuals recruited via general practitioner registries (AgeCoDe cohort) were analyzed. This study analyzed the association between HbA1c level and the incidence of all-cause dementia (ACD) and of Alzheimer's disease dementia (referred to here as AD). HbA1c levels ≥6.5% were associated with 2.8-fold increased risk of incident ACD (p = 0.027) and for AD (p = 0.047). HbA1c levels ≥7% were associated with a five-fold increased risk of incident ACD (p = 0.001) and 4.7-fold increased risk of incident AD (p = 0.004). The T2DM diagnosis per se did not increase the risk of either ACD or AD. Higher levels of HbA1c are associated with increased risk of ACD and AD in an elderly population. T2DM diagnosis was not associated with increased risk if HbA1c levels were below 7%.