However pretty early in the lectures they jump right into taking partial derivatives of matrices for calculating gradient descent. I think by 'no particular math background' they mean 'assuming nothing more than single and basic multivariable calculus plus linear algebra'. If your linear algebra and/or multi-variable calculus are weak/fuzzy (as they are for me) I think those sections will take quite a while to really understand.