Well, I say, that there is user experience design, or whatever you wish, I should call it.
First, look at this picture:
What the hell is this? This is a floor plan of an IKEA store. The perfectly designed user experience. Most of the users (visitors) go through this path, and have more or less the same experience, because people's behaviour and desires are easily predictable. When we talk about user experience, we talk about the majority of our users, not all users. Like in the IKEA, there are shortcuts for those visitors who want to go directly to the market area. The example could be Tesco or Wal-Mart as well, these stores are planned perfectly, too. Maybe you all know, if you're a supplier of Tesco, you have to pay, if you want your product on the shelf right before the people's eye.
Another great example is the iPhone. I'm not a Mac user, and I'm not a macfag. But I appreciate Steve Jobs as a UX designer: I think he was the first well-known of his kind. Everyone knows that he was mad about the small details, he wanted a perfect product - from the shape, through the touch, to the icons in the software. Everything was controlled through the design process. all because he wanted to reach the same experience for all the users.
I think UX design is about what I want to reach. Yes, some of my users will leave, some will make a shortcut, but if I do my job correctly, most of the visitors will do exactly what I want. That's the point. It's like a TV show, from viewer to viewer, the experience will different (favourite actor, opinion about their behaviour), but the biggest part of it will be the same for everyone. JR Ewing will always be the evil character and Bobby his counterpart - because this is how the experience is designed.
Yes, things like site speed affect the experience, but, for God's sake, if your user feels any problem with the speed, than something is technically wrong with your site. Load time should count to Google, but for your user.
Anyway, if you like, I'm a UI designer.