My version of the robots.txt content after which site traffic increased by 20% in a week.
User-agent: *
Allow: /d/*
Allow: /t/*
Allow: /p/*
Allow: /tags
Disallow: /u/*
Disallow: /professional_achievements/*
Disallow: /masquerade/*
Disallow: /user_advcomments/*
Disallow: /user_subscriptions/*
Disallow: /?q=
Disallow: /users?q=
Disallow: /?page=
Disallow: /following
Host: ........
