{"id":531,"date":"2016-10-01T16:22:52","date_gmt":"2016-10-01T08:22:52","guid":{"rendered":"https:\/\/www.techcoil.com\/blog\/?p=531"},"modified":"2018-09-04T23:54:15","modified_gmt":"2018-09-04T15:54:15","slug":"the-nginx-configurations-that-i-had-created-to-serve-only-robots-txt-and-sitemap-index-xml-over-http","status":"publish","type":"post","link":"https:\/\/www.techcoil.com\/blog\/the-nginx-configurations-that-i-had-created-to-serve-only-robots-txt-and-sitemap-index-xml-over-http\/","title":{"rendered":"The Nginx configurations that I had created to serve only robots.txt and sitemap-index.xml over http"},"content":{"rendered":"

I had initially configured a set of Nginx rules that will redirect every single request made over http to https. However, that breaks my doing so breaks my reference on how to send HTTP GET request with Java without using any external libraries<\/a>. <\/p>\n

Since the robots.txt is not sensitive data, I had decided to allow it to be retrieved via http as well. And since my robots.txt points to sitemap-index.xml<\/code> via http, I also allowed sitemap-index.xml<\/code> to be retrieved via http as well. <\/p>\n

This post documents the changes that I had made to my Nginx configurations to serve only robots.txt<\/code> and sitemap-index.xml<\/code> over the http protocol.<\/p>\n

The Nginx configurations that redirects all http traffic to https<\/h2>\n

My initial Nginx configurations for redirecting all http traffic to https was as follows:<\/p>\n

\r\nserver {\r\n\r\n    listen 80;\r\n    server_name   techcoil.com www.techcoil.com;\r\n    return 301 https:\/\/www.techcoil.com$request_uri;    \r\n\r\n}\r\n<\/pre>\n
What this set of code does is straightforward; it tells Nginx to return a response with the 301 Moved Permanently<\/strong> status code whenever there are HTTP requests received from port 80 which are directed at the domains techcoil.com<\/code> or www.techcoil.com<\/code>.<\/p>\n
The resultant Nginx rules for allowing HTTP accesses to only robots.txt and sitemap-index.xml<\/h2>\n
The resultant Nginx rules for allowing HTTP accesses to only robots.txt and sitemap-index.xml is as follows:<\/p>\n
\r\nserver {\r\n\r\n        listen 80;\r\n        server_name   techcoil.com www.techcoil.com;\r\n        root    \/path\/to\/directory\/that\/contains\/robot.txt\/and\/sitemap-index.xml;\r\n\r\n        location \/robots.txt {\r\n\r\n        }\r\n\r\n        location \/sitemap-index.xml {\r\n\r\n        }\r\n\r\n        location \/ {\r\n                return 301 https:\/\/www.techcoil.com$request_uri;\r\n        }\r\n}\r\n<\/pre>\n
I included the root<\/code> command that will point Nginx to read files from the directory that contains robots.txt<\/code> and sitemap-index.xml<\/code>. <\/p>\n
I then added 3 new location blocks:<\/p>\n