It seems plasma leaks into x shared memory. This would explain why I had such a high value xrestop. After I restarted X it was down to only around 6 MB. It's funny but I don't even really notice these things anymore now that I have 2 GB RAM.
I wonder how many memory won't get noticed because developers have "too much" RAM in their developing systems...
By the way, if plasma constantly crashes for you and your settings get lost, you can just quit it with "kquitapp plasma", this saves its settings and then quits plasma.
Update: I've found a bug report on bugs.kde.org and it seems to be already fixed in SVN trunk.