How to hack a nucleotide substitution matrix for MUSCLE 
 
Create an amino acid substitution matrix in BLAST format
The ACGT rows and columns should have your desired nucleotide values, the other entries should be zero. Values can be integers or floating-point. In the following example, integers are used with a match score of +1 and a mismatch score of -2 (these are the BLASTN defaults).

Download an example matrix here.
 
Use the following options:

    -matrix <filename> -seqtype protein