I installed repeatmasker locally in my bin directory: /uufs/chpc.utah.edu/common/home/u6007910/bin/RepeatMasker/ (http://www.repeatmasker.org/RMDownload.html). I used HMMER and Dfam as the default sequence search engine for repeatmasker.
To install:
perl configure
This will prompt for various paths, most specific ones are as follows:
I had to specify the path for TRF.pm, which is present in the RepeatMasker folder: /uufs/chpc.utah.edu/common/home/u6007910/bin/RepeatMasker
I had to specify the path for HMMER nhmmer program which is installed globally on the cluster: /uufs/chpc.utah.edu/sys/installdir/hmmer3/3.1b2/bin
I had to specify the path for Dfam.hmm which is present in the RepeatMasker/Libraries folder (or can be downloaded): /uufs/chpc.utah.edu/common/home/u6007910/bin/RepeatMasker/Libraries
Once, the program is installed, you can test run to get the help options: ./RepeatMasker -h
FILTERING ON SEPT 27, 2018
makeblastdb -in /uufs/chpc.utah.edu/common/home/gompert-group1/data/lycaeides/matt_transcriptome/txome_assembly/trinity_denovo/trinity_out_dir/bowtieassembly/Trinity.fasta -dbtype nucl
makeblastdb -in trinitydenovo.no_tes.fa -dbtype nucl
makeblastdb -in /uufs/chpc.utah.edu/common/home/gompert-group1/data/lycaeides/dovetail_melissa_genome/Annotation/repeatmodeller/RM_108478.FriJul201433492018/consensi.fa.classified -dbtype nucl
perl fastaqual_select.pl -f /uufs/chpc.utah.edu/common/home/gompert-group1/data/lycaeides/dovetail_melissa_genome/Annotation/repeatmodeller/RM_108478.FriJul201433492018/consensi.fa.classified -e <(awk '{print $1}' repeatmodlib_transcriptsnotes_blastn.out | sort | uniq) > repeatmodellib.no_prots.fa
perl fastaqual_select.pl -f trinitydenovo.no_tes.fa -e <(awk '{print $1}' repeatmodlibdb_transcriptsnotes_blastn.out | sort | uniq) > repeatmodellibdb.no_tes.fa
##numbers
646092 Trinity.fasta
2820 repeatmasklib_transcripts_blastn.out #blast transcriptome against repeatmasker repeat library
641980 trinitydenovo.no_tes.fa #removed the hits from the transcriptome
646092 Trinity.fasta
641980 trinitydenovo.no_tes.fa
280782 repeatmodlibdb_transcriptsnotes_blastn.out #blast transcriptome with no repeatmasker hits (From above) with the repeatmodeler library
401256 repeatmodellibdb.no_tes.fa #removed the hits from the trinitydenovo.no_tes.fa transcriptome
#blast repeatmodeler library against the transcriptome no tes file
7507 repeatmodlib_transcriptsnotes_blastn.out
#repeats in the transcriptome
196 repeatmodellib.no_prots.fa
### blasting transcriptome against repeatmodeler library
646092 Trinity.fasta
283412 repeatmodlib_transcripts_blastn.out #blast transcriptome againts repeatmodeler library
402974 trinitydenovo_repeatmod.no_tes.fa #removed hits from the transcriptome
7501 repeatmodlib.no_prots.fa #blast repeatmod library against transcriptome to get TEs
perl fastaqual_select.pl -f /uufs/chpc.utah.edu/common/home/gompert-group1/data/lycaeides/matt_transcriptome/txome_assembly/trinity_denovo/trinity_out_dir/bowtieassembly/Trinity.fasta -e <(awk '{print $1}' repeatmodlib_transcripts_blastn.out | sort | uniq) > trinitydenovo_repeatmod.no_tes.fa