Why are SF and Fantasy so often grouped together? Obviously, because they share readership and so are well placed together in book stores. And... heck... some of us write both! Still, there are very real differences.
Look, fantasy is the mother genre -- e.g. Gilgamesh, the Illad, Odyssey and most religions. Sci Fi is the brash offshoot. All literature has deep roots in fantasy, which in turn emerges from the font of our dreams.
Having said that, what is my definition of the separation? I think it is very basic, revolving around the notion of human improvability.
"Do you believe it is possible for children to learn from the mistakes of their parents?"
For all the courage and heroism shown by fantasy characters across 4000 years of great, compelling dramas -- NOTHING EVER CHANGES! Aragorn may be a better king than Sauron would have been. Hurray. Fine. But he's still a freaking king. And the palantir on his desk that lets him see faraway places and converse with viceroys across the realm is still reserved for the super elite. No way are we going to see mass-produced palantirs appearing on every peasant's tabletop from Rohan to the Shire. (The way our civilization plopped such a miracle on YOUR tabletop.) It never even occurs to Aragorn or Gandalf to give the poor the godlike powers they themselves get to wield... let alone provide them with libraries, running water, printing presses or the germ theory of disease. Only little Peregrin Took seems to get a glimmer of an idea in that direction. The only character who briefly ponders possibilities, and he's soon bullied out of it.
Fantasy has its attractions. Something about feudalism resonates, deep inside us. We fantacize about being the king or wizard. Heck it's in our genes. We are all descended from the harems of the guys who succeeded at that goal. The core thing about fantasy tales is that, after the adventure is done and the bad guys are defeated... the social order stays the same.
It may be the natural genre... but should we be proud of that?
Science fiction, in sharp contrast, considers the possibility of learning and change.
Not that children always choose to learn from their parent's mistakes! When they don't, when they are obstinately stupid and miss opportunities, you can get a sci fi tragedy... far more horrible than anything "tragic" in Aristotle's POETICS. Aristotle says tragedy is Oedepus writhing futilely against fate. A sci fi tragedy portrays people suffering, same as in older tragedies... but with this crucial difference -- things did not have to be this way. It wasn't "fate." We - or the characters - could've done better. There was, at some point, a chance to change our own destiny.
One type of tragedy makes you weep - hey, Oedepus is powerful stuff. But for millennia the deep moral lesson - the thing taught in all "campbellian myths" - is that resistance is futile. The overall situation, the rule of fate, remains the same.
The other type of tragedy - the new kind - is a cautionary tale that may change your decisions. It may alter destiny.
You can see why the absurd old farts who inhabit most lit departments hate science fiction. SF considers it possible that the eternal "verities" and relentless stupidities praised by Henry James might someday be obsolete! If we make kids who are better than us (our goal, duh?) then their Startrekkian heirs will still have problems. Why insist that our descendants have to fret over the same ones? Can't they assume the solutions we find, take them for granted, and move on to new, interesting issues of their own?
Isn't that what we did?
The implicit assumption in most fantasy is that the form of governance that ruled most human societies since the discovery of grain must always govern us. And when a fellow like Tim Powers resists that assumption, he is writing science fiction, whether or not there are pirates, or wizards or demons.
Anne McCaffrey says "Never call me a fantasy author! I write science fiction!" Indeed. Despite the dragons and lords and medieval craft and renaissance fair stuff... her characters have heard of flush toilets and universities and democracy...
...AND THEY WANT THOSE THINGS BACK! They want starships. And Anne is going to let them earn those things. They will get them back, and move on. And she is a science fiction author.