Go to System Settings, Advanced, System Services.
Then in the bottom uncheck "Detecting RANDR (monitor) changes", then press Stop on the bottom right. The problem disappears immediately now.
You may read more about the bug here and here. Thanks to Electricroo for the fix. By the way, you know you have this bug, if DDC EDID probes ("(II) intel(0): Printing DDC gathered Modelines, (II) intel(0): EDID vendor") keep showing up in tail -f /var/log/Xorg.0.log and you know it's fixed once they disappear. If you don't have that message in your X log, your most likely don't have this issue.
Note that this will stop KDE from automatically detecting when you plug in an external monitor. But as the configuration doesn't yet work well anyway I think there's no detriment. And of course there will be other causes for latency, but this one was the only real problem for me.
Unfortunately it seems like the fix does not (currently) work for KDE 4.2. If you don't find the service in the list in KDE 4.1.x, this might be a good sign and show that it's already disabled. If you still find the messages in Xorg.0.log, the please post a comment and let's try to find a way around it. If there's no way to disable it, all you can do is file a bug for your distribution and refer to the here mentioned information.
The bug has already been reported for KDE 4.1 at the kde bugzilla. Please participate there if you see this problem in KDE 4.2. You might also help to get a fix by voting for it. Until that time you can probably use my previously posted dirty hack that works around the kde session management service (ksmserver) by starting the KDE4 environment manually - please let me know if this works for KDE 4.2 as well.
You can also try to manually disable all outputs you don't need in Xorg, but the instructions vary significantly between different graphics cards, setups, distributions etc. And of course that won't help you if you're using a dual screen setup. If you find instructions on how to do that, please let me know in the comments.
For KDE 4.2, check your config file:
grep polling ~/.local/screen-configurations.xmlIf it says polling false and after upgrading KDE to 4.2 you suddenly get the messages described above in Xorg.log, especially if the fix above worked for you before, then there's a bug somewhere in KDE. If you get polling true, you can try to reconfigure KDE to make it stop polling. (You can just edit the file and change true to false.)
Related: Automatically switch to connected External Display on Boot with XRandR shows you how to automatically set up your displays without just a single screen, so that you can disable xrandr.