No. You cannot restrict access to users per invoice. However, you can customize their security permissions and limit their access to the invoices of the projects they manage by un-checking the security Allow access to data of all employees under Invoices module in the Security > Edit Profile screen. You can also restrict them to only bill the projects they manage by un-checking the security option Allow billing of other manager's projects under the Billing module. Check Core Help for details on customizing security permissions.