"Ambiguous ending" does not mean "peters out to a halt without resolving anything."
I occasionally ask a fellow writer what the point of her story is. It is best to ask this question from a safe distance, or at least while wearing protective body armour.
"Why does it have to have a point? I’m not out to change the world or anything. I hate moralizing drivel disguised as fiction!"
Honestly, so do I. But that’s not what I’m asking for. How many times have you said about a movie or book, "It just didn’t work," or "What a waste of time," or even, "That was pointless"? Maybe it was even a well-written book, but you were left feeling unsatisfied at the end of it.
The point of the story is the reward you give your reader for sticking it out for five hundred pages. In a character-driven story, the reward is in seeing the hero learn something, or maybe we learn something new about the hero that makes us see him in a different light (as in a flawed example, Hannibal by Thomas Harris). In a plot driven story, the point may come as a battle is won, a mystery solved, or a misunderstanding healed.
In the original Star Wars there were several points: Princess Leia was rescued and the plans were delivered; the Rebel Alliance destroyed the Death Star and saved the planet; Luke Skywalker became a fighter pilot like his father, tasting the adventure he'd always craved and learning the price of it; Han Solo embraced an ideal higher than me-first. Multiple storylines, multiple ends to wrap up, multiple points successfully made.
I call it closing the loop, and it was not taught in any Creative Writing class I ever took. In fact, plotting was actively discouraged by most of my teachers. My college creative writing professor, although knowledgeable and helpful in many ways, would freely admit he thought "plot" was a dirty word. He force-fed us the New Yorker and John Updike and all of their fuzzy-ended non-circular slice-of-life angst. He told me to my face that a genre writer was a hack writer, and I was "too good" to be writing that sort of "slick" fiction.
By "slick" he meant a fast read, a plot with twists and turns and slight of hand.
I learned the most about plotting, oddly enough, from three non-fiction undergrad classes: Freshman comp, which I took in high school; Business Writing 101, and Persuasive Writing In the Real World.
Here's what I learned from those three classes, although it took me thirteen years to distill and synthesize the knowledge: Story is an argument.
It's okay if your story is only intended for entertainment. It doesn't have to take a stand or try to change somebody's mind; in fact most readers will be put off by anything heavy-handed or preachy. But in order for it to be a lively and successful story, something has to move, something has to change from A to B or beyond, and if that change is believable, then you have done, in essence, what a defense attorney does to get his client off: he has built an argument as to why his version of the truth is the most believable.
The "point" of a story is easily illustrated in certain very simple fantasy or sci-fi plots, in which the characters have to find some kind of object or device in order to solve a problem or save the world. SFF writers and editors call such a device a "McGuffin." The appearance and function of the McGuffin don't really matter--once the characters locate it and employ it, the problem is solved and the story is over.
A potentially more sophisticated plotline is when the character wants something really bad, and strives hard to get it. There are multiple possible endings to this story: either he gets what he wants and he's happy, or he gets what he wants and it doesn't make him happy after all. Or the opposite happens--he doesn't get what he wants and he may be happy or sad about it. Those types of stories, where the hero fails to achieve his goals, usually place emphasis on the journey and the changes it works in our hero.
Your reader could potentially be satisfied with any of those choices, if you do your job as character advocate. You must ensure that everything in the story—character development, setting, scenes, mood, pacing, all of it—serves to convince the audience that your ending is the only right, proper, and believable ending.
Weaving all that stuff together and keeping it relevant is a complex and delicate task--like packing everything you need into a single suitcase. If you waste time in digression or seeming contradictions, the reader--your jury--may decide you are a bore, or a liar. However, you must anticipate the reader's arguments and questions, and address them, or the jury will claim you have "holes" in your plot.
At all times, you need to be lively and relevant and tantalizing, revealing only as much as will serve your purpose, like dragging a string in front of a cat’s nose: Come on. Just a little further. There’s a reward here. You know you want it.
I had a friend, a non-writer and a big fan of comic books and all things hackneyed, who recognized this intuitively. I spent the winter of 2003 in a long, deep, black ravine of writer’s depression: I was stuck, as a writer. I was considered a "good" writer, but I knew my work was missing something, that final relevance that could make it really enjoyable—and yes, marketable. But damned if I could see what it was.
After listening to me whine for three weeks, my friend cut me off very succinctly: “You just don’t want badly enough to make your readers happy.”
It stopped me in my tracks. I suspected he was right, but I was still hung up in 'slick-writing' guilt. I was convinced that if I wrote something with a tightly-plotted, wrapped-up ending, it would be predictable and trite. In a deep, quiet place in myself, my instincts were debating with my programming.
If I give the audience what they want, doesn’t that make the story predictable?
Not necessarily. You can use slight-of-hand—make them think you’re going one way, build expectation, then subvert it.
But how do I know that’s what they want? And what if they don’t believe it?
You have to tell them what they want. Like advertising. Besides, whose world is this? They only see what you want them to see.
But if the story is slick and tidy like that, doesn’t that make me a hack?
Are the characters intriguing?
Is the plot logical and reasonably original?
As far as I can tell.
Were the seeds of the conclusion present in the beginning of the story? Did you point the reader in the right direction at the start, and did you leave a cookie trail so the reader could see where they came from?
Then who the hell cares if you’re a hack? That’s a damn good story.