If you recall, the earlier 5.4 L engine was plagued with a different spark plug problem -- it spit them out of the heads. The aluminum heads had three or four threads that tried to hold the steel plug. We remember how that worked out and it took some six years to develop this solution.
Ford just dared these new improved long-snouted plugs blow out. Oh no. They were in to stay. They were probably tested fairly well, but it's the combination of mileage, carbon buildup and carbon aging along with corrosion that makes these near-impossible to remove.
Another six or so years go by before Ford makes yet another improvement to solve that problem.
I believe most engineers do their work in good faith. They try their best to test their designs and to anticipate problems. Often they suffer from the "designers optimism" and underestimate the shortcomings of their work. Other times, the shortcomings are compromises dictated by economics. Sometimes the shortcomings just aren't apparent at testing or turn out to be more severe than expected. SImetimes they are known but ignored by an incompetent management.
Environments, materials and suppliers vary. These engines were designed to run on unleaded gasoline, later ethanol blends, with additional impurities were introduced into them. The engines work in sub-zero and torrid climates, and testing can't simulate every possible environment.
Then once a design is tested, modified, agreed on by hundreds of decision-makers from all works of corporate management, millions and often billions are invested in its tooling.
I'm sure the board-room crisis meeting is a regularly scheduled event. "Just how bad's the problem??" "... How many claims do we anticipate over the next three years?" "What will the warranty work cost us?" "What's the cost of retooling".
The job of designing, testing and manufacturing isn't an easy one. Correcting runaway shortcomings discovered late in the game is even harder.... Every engineer knows that drill too well.