As someone who also uses a Pentium 4 (Prescott 2.8 GHz, though in 32-bit & s478 flavor) on a daily basis for the web / daily driver, I find it hard to believe that you can watch 1080p @ 60 FPS and not get stutter. Even with MP4 / H.264 compression in MVK, I can't get 1080p @ 30 FPS without stutter. I've tested numerous other P4 systems (some Cedar Mill too) and they didn't fare much better compared to my P4. The last one I tested was a 3.4 GHz 651. That one was just barely able to handle the same 1080p 30 FPS clip I use for testing on all of my retro machines. As for browsing, same machine did considerably better than my 2.8 GHz P4, though I suspect much of that had to do with the fact that I was using latest version of Firefox at the time on the 651, whereas the 2.8 Presshot was using the Pale Moon -based version of Mypal.
In any case, my point is not to disprove your claim here. But having used various P4 systems, I think you will find the upgrade from 3 to 3.6-3.8 GHz somewhat marginal. Most you might see a difference in is offline video decoding (again, noting my experiments above with the same 1080p 30 FPS clip.) A discrete GPU with H.264 decoding should help off-load the CPU a little with online videos, but don't expect miracles. On for that matter, no need to go with a high-end or even mid-range GPU. Something like a low-end GT730 or Radeon HD7570 should be enough for this task.
And as the others have noted, this being an OEM (Dell) machine, you likely won't be able to do any overclocking. Though I shall note that I ran across a forum member in another forum that showed overcloking is possible on OEM machines with software like ThrottleStop. I remember reading a little bit on it at the time (since I have a lot of OEM machines at home), but gave up on the idea, since it wasn't a very neat solution from what I remember.
Lastly, in regard to the PSU: there's not that much of a difference in TDP between the various single-core Cedar Mill CPUs. So your cooling and PSU may be OK as-is. Though the bit you mentioned that your PSU is *not* the original OEM one worries me somewhat more - OEM PSUs in these Optiplex machines were usually very well-built and could provide the power stated on their label (it's just that too many of them suffered from bad caps after a decade.) On the other hand, a low-end / no-name non-OEM 250-300W PSU could be quite terrible in terms of build quality, and I would be worried about using such PSU on a P4 PC already.