This use is of course related to the common slang use of the word to describe a person as eccentric, crazy, or just unreliable.
A system that is flaky is working, sort of - enough that you are tempted to try to use it - but fails frequently enough that the odds in favour of finishing what you start are low. Commonwealth hackish prefers dodgy.