I find that sometimes files included in Python projects, for example Fortran files, have corrupted characters that are incorrect UTF-8 characters. Maybe it’s a case of bad OCR that also plagues LaTeX/BibTeX copy/paste references from journal websites. Thus, this method will also apply to BibTeX files.
I have created a Python script that recursively:
- finds such corrupt files
- converts from UTF-8 to ASCII
- removes the corrupted characters
- puts the output file to a temporary location
- end user can copy fixed file over original