How can I keep time memos without displaying them on invoices?

This can be done by modifying the invoice template and unchecking the option. For example, if you were using the Hourly Template, you could modify the template and uncheck the option to show Time Entry Memo. The time entry would still have the memo attached, without it being displayed on invoices.