This is an old expression and the earliest citation is in John Gower's Middle English text Confessio Amantis, 1390:
Lo, how they feignen chalk for chese.
Tourist boards in several of the chalkland areas of the UK try to place the phrase's origin in their locality and allude to vague connections between chalk and the local cheese. None of these is convincing and they clearly owe more to marketing than to etymology. So, how did the phrase come about?
There must have been a time in the development of English when we had no standard phrase to express the idea that two things were 'as different as X and Y'. When someone coined such a phrase, and that someone may well have been Gower in 1390, clearly he needed candidates for the roles of X and Y. That doesn't sound difficult, after all most things are different from most other things.
"Maybe, 'as different as a cormorant and a lamp-post'", thinks our coiner, "or 'as different as floorboards and greengrocers'". "No, 'as different as chalk and cheese' sounds better". Why? For no better reason that the fact the 'chalk' and 'cheese' are short and snappy words that alliterate. The English language is packed full of phrases that contain pairs of rhyming or alliterating words - often just because the person who coined them liked the sound of them; for example, hocus-pocus, the bee's knees, riff-raff etc.