Honestly unless you're writing your own physics and/or rendering engine you don't really need to have a deep knowledge of math or physics for writing a web game, since most engines already handle that for you. Back in the day it used to be libraries like Box2D and its ports, nowadays it's built in to frameworks like Phaser and so on.
There's nothing wrong with learning math and physics for building things yourself, or to get a deeper understanding of how things work (so that you can break the rules, as it were) but for most purposes you won't need that knowledge. And you can always look them up as and when you do.
It's the same with deep learning - you need to know the maths (calculus + statistics) behind it if you're trying to create new neural architectures, or if you're trying to improve existing ones. But if you're looking for off-the-shelf solutions, then most libraries can handle the hard parts and give you something that works reasonably well, even if you don't quite fully understand how it works. That's one of the advantages (and disadvantages) of abstraction.
And if you ask me, buying into the idea that you need to learn X before you can do Y when it's not needed just provides ways to duck out of doing Y directly - e.g. people saying they're not good at math so they shouldn't bother learning programming, etc. Not saying that applies to you, but it is a common refrain I heard a lot in the past.
Also, 3D-for-the web has come quite a ways since the last decade or so, they run fairly smoothly even on integrated graphics these days. Part of it comes down to mobile GPUs becoming far better, but part of it is also the better ecosystem around getting JS as close to bare metal where needed (e.g. with wasm, webgpu, etc.)