I am developing a website crawler using golang. When i tried to crawl some websites, I am getting weird results. Root Url of some website returns script tag as shown below.
<script>window.location="index.php";</script>
And it redirects to index.php page. Why people are using this approach to redirect user to index page. Any security vulnerability with this approach? And also, how can i handle this situation in crawler?
Well, if you really want to hide the page by redirecting the user to another page, then you obviously cannot use this method, because anyone can turn javascript off and see the page, thus this can be a security risk. However, if you just simply want to redirect for some reason, this is fine.
As for you crawler, what you can do is search the source code with regex for redirections like that, but it can be very challenging to cover all cases.