Following solution only works with WordPress 2.5 or higher. If you need help for a lower version, let me know in the comment field and I'll try to help.

Number of posts

$num_posts = wp_count_posts( 'post' );
$num_posts = $num_posts->publish; //publish, draft

Number of pages

$num_pages = wp_count_posts( 'page' );
$num_pages = $num_pages->publish; //publish

Number of categories

$num_cats  = wp_count_terms('category');

Number of Tags

$num_tags  = wp_count_terms('post_tag');

Number of comments

$num_comm  = get_comment_count();
$num_comm  = $num_comm['approved']; //approved, awaiting_moderation, spam, tot

// Solution 2
$num_comm2 = wp_count_comments( );
$num_comm2 = $num_comm2->approved; //approved, moderated, spam, total_comments

Output

The above syntax shows simple how to get the values without including HTML. In another example I show you all results in HTML as a list. Everybody can adjust the format to integrate in their design.

<?php
$num_posts = wp_count_posts( 'post' );
$num_posts = $num_posts->publish; //publish, draft
$num_posts = sprintf( __ngettext( '%s Post', '%s Posts', $num_posts ), number_format_i18n( $num_posts ) );

$num_pages = wp_count_posts( 'page' );
$num_pages = $num_pages->publish; //publish
$num_pages = sprintf( __ngettext( '%s Page', '%s Pages', $num_pages ), number_format_i18n( $num_pages ) );

$num_cats  = wp_count_terms('category');
$num_tags  = wp_count_terms('post_tag');

$num_comm  = get_comment_count();
$num_comm  = $num_comm['approved']; //approved, awaiting_moderation, spam, total_comments
$num_comm  = sprintf( __ngettext( '%s Categorie', '%s Categories', $num_comm ), number_format_i18n( $num_comm ) );
$num_comm2 = wp_count_comments( );
$num_comm2 = $num_comm2->approved; //approved, moderated, spam, total_comments

echo '<ul>';
echo '<li>Posts: ' . $num_posts . '</li>';
echo '<li>Pages: ' . $num_pages . '</li>';
echo '<li>Categories: ' . $num_cats . '</li>';
echo '<li>Tags: ' . $num_tags . '</li>';
echo '<li>Comments: ' . $num_comm . '</li>';
echo '<li>Comments 2: ' . $num_comm2 . '</li>';
echo '</ul>';
?>

You can adjust the design of these lists. You can use classes or IDs. An example is in article „Total Amount of Comments“ (in german language), where you can see a button with the amount of comments.