XYBench, a benchmark of 8,115 real-world questions spanning programming and everyday tasks, tests AI assistants' ability to recognize and respond to users' misconceptions—such as asking how to thicken frosting when the real issue is warm cupcakes. Developed with researchers from Carnegie Mellon, NVIDIA, and Johns Hopkins, XYBench measures whether models identify underlying goals (X), address them effectively, and steer users away from flawed approaches (Y). Evaluations of models like Claude Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.5-9B, and DeepSeek-V4 Pro showed they frequently focused on the stated request rather than the actual goal, with limited success in misconception identification. Human reference answers outperformed all models, highlighting a gap in pragmatic reasoning. The benchmark uses automated and human evaluation to assess pragmatic solution, emphasis, and misconception identification, offering a tool for researchers to improve AI assistants’ ability to provide helpful advice even when users ask the wrong question.