One of central agile programming ideas is to deliver frequently and get feedback early.
Here is a situation I have seen a few times more than I would have wanted: the team and the clients split the whole project into several phases that will be shipped every few weeks; early deliverables mostly provide the plumbing for the later ones, and have only enough user-interfacing functions for the plumbing to be tested functionally. The clients set up and play with each new release. They dutifully give their feedback and participate in testing each time. When the whole thing comes together on the end, it turns out that the system still requires a lot of changes to solve the problem that it was intended for — the iterations and customer feedback failed to provide the directions.
Unless the software is really being used, we cannot expect authoritative feedback.
So how do we avoid this problem?