阻止通过htaccess直接访问所有文件

A security audit highlighted an issue where some files could be accessed directly eg www.domain.com/readme.txt

No biggy as there is nothing on the server that contains anything sensitive although this isn't going to cut it on the audit and we need to patch this.

Is there a way to block ALL files from being accessed directly unless though the website or specified (file extension)? Hoping it can be achieved via htaccess?

order allow,deny
<Files ~ ".(php|html|js|css)$">
   allow from all
</Files>
(Php, html, js and css are allowed to access)