Post-process duplication detection removed in favor of filtering out duplicates and redirects before processing