Hey I have that heatsink! It came on a slotket with a Celly I'm not using. It was really messy to clean that thing up when I moved it to a S7 rig...
What you're going to want to do is clean as much as possible with rubbing alcohol, and then you're going to need some wet dry sandpaper, a very flat surface, and a bit of time. That tape doesn't just scrape off. Actually maybe it does if you're patient enough, but you'll need some sort of scraping tool, and it will very likely gouge the surface. I recommend you lap the bottom of the heat sink with the wet dry.
Will the crap come off of the CPU? You probably don't want to lap it, the risks are considerably more pronounced.
A third alternative is to just get as much of this crap off as possible and just squirt some thermal compound on there and put it back together. That's probably the best option, especially since K6-2's don't get super hot.