Links will go missing

My textfile backups fail to have many URLs enumerated. Online backups with a small crawler were also a 'failure.' - They tended to get stuck in loops of articles that linked to each other.