We discuss the numerical evaluation of single and double layer heat potentials in two dimensions on stationary and moving boundaries. One of the principal difficulties in designing high order methods concerns the local behavior of the heat kernel, which is both weakly singular in time and rapidly decaying in space. We show that standard quadrature schemes suffer from a poorly recognized form of inaccuracy, which we refer to as “geometrically-induced stiffness”, but that rules based on product integration of the full heat kernel in time are robust. When combined with previously developed fast algorithms for the evolution of the “history part” of layer potentials, diffusion processes in complex, moving geometries can be computed accurately and in nearly optimal time.