Lithopone is rather nontoxic, due to the insolubility of its components. It has been used in medicine as a radiocontrast agent. Lithopone is allowed to be in contact with foodstuffs in the US and Europe.
Overwhelmingly, research that’s relevant to human eating patterns shows us that E171 is safe when ingested normally through foods and drugs (1,2).