Another tip so you can see the next scheduled date/time is to print out your hook via the wp_next_scheduled function. This is great for troubleshooting. WP seems to use GMT and my offset is -7 so once I printed out the actual date, I saw that the scheduler wasn't going to fire for another 7 hours.
echo date("d-M-Y h:i:s A", wp_next_scheduled(my_task_hook));

Also I think you have a mistake in your code above:
delete_action( 'my_task_hook', 'my_task_deactivate' );
should be
remove_action( 'my_task_hook', 'my_task_deactivate' );

The action will trigger when someone visits your WordPress site, if the scheduled time has passed.

This is an important nuance. In my case, I need to fetch data from an external site every hour. The first visitor after the scheduled time will see the old data. It will trigger the cron, and only then will the new data be available.

*/5 * * * * curl http://somesite.com/

Another way to put it:
When you set a cron to run every hour and no one visits your site for 3 days, will it run every hour during that time?