No layout does not matter for readers. Most of my readers on my blog read its contents by its RSS feed and I think almost nobody even noticed that I did a complete redesign (read: installation of a new template ;)). Same goes for my twitter page. Hardly anyone will visit it often and I think the layout of this page is the same since I started to tweet. So does layout really matters?
Yes it does
Yes layout matters for strangers and it matters if you offer another service than ‘content’ (a service in which user experience is important). If you offer just content (like I do via my blog and via Twitter) than layout only matters for people who do not know who you are. Layout helps strangers to identify if you are someone they want to spend time with (by for example reading your content, not necessarily by watching hours at your layout). If you have a layout which they cannot relate to, they are likely to not invest time in you. The reason why Scobleizer can claim that his basic layout has no impact on traffic, is because due to the fact people already know him and people can use other sources to identify him as someone who they want to spend time with.