handbook/tools/Others/Wordlist/3.Remove-Duplication.md
2024-08-31 01:07:22 +02:00

25 lines
643 B
Markdown
Raw Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

## General
This tool can be very useful to cleanup a wordlist containing duplicate entries.Might be handy for CTF challenges where wordlists are intentionally made large and contain lots of duplicate entries.
## Command
Tool
```Terminal
>$ ./dup-rem.py input-file output-file {Optional arguments: asc, dsc}
```
- asc - dsc  ---> Sorts the output in a Ascending or descending order
## Other Possibility
Terminal
```
# Combine files together
cat file1.txt file2.txt file3.txt > combined_list.txt
# Sort and remove duplicate
sort combined_list.txt | uniq -u > cleaned_combined_list.txt