The project memos show up on invoices if you have the rule Use project memo on invoices selected in Projects > Settings > Assignment. Remove this rule for the project to prevent the memo from showing up on its invoices. Note that the change only applies to the new invoices you create and not the existing ones that have the memos. You might also want to check out these articles:
CORE Help Center