VACUUM has no effect on read/write performance to that table. Never running VACUUM on a table will not make read/write performance to a Delta Lake table any slower.
If you run VACUUM very infrequently, your VACUUM runtimes themselves may be pretty high, so it is suggested to run VACUUM somewhat regularly. How often you should run VACUUM depends on your storage costs.