Странное поведение nginx, характерное для запросов bingbot

мы нашли что-то странное в журналах nginx. Некоторые запросы только от bingbot идут непосредственно в бэкэнд, но если мы запускаем эти запросы от curl, они обрабатываются как обычно. Пожалуйста взгляните конфигурации.

I’ve got two servers, one (server1) if frontend and it’s proxying some requests to server2 and some requests process by himself.

Пожалуйста, посмотрите конфигурацию сервера 1:

server {   
  listen  80;# default_server;

  proxy_set_header        X-Real-IP       $remote_addr; 
  proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;   
  proxy_set_header        Host            $host;

  server_name myservername.com;

     location / {
         index  index.html;
         root /var/www/apps/delta/shared/public;

         if (-f $request_filename/index.html) {
           rewrite (.*) 

we have found something strange in nginx logs. Some requests only from bingbot goes directly to backend, but if we fire this requests from curl, they processed as usually. Please take a look of configuration.

I've got two servers, one (server1) if frontend and it's proxying some requests to server2 and some requests process by himself.

Please take a look Server 1 config:

server {   
  listen  80;# default_server;

  proxy_set_header        X-Real-IP       $remote_addr; 
  proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;   
  proxy_set_header        Host            $host;

  server_name myservername.com;

     location / {
         index  index.html;
         root /var/www/apps/delta/shared/public;

         if (-f $request_filename/index.html) {
           rewrite (.*) $1/index.html break;
         }

         if (-f $request_filename.html) {
           rewrite (.*) $1.html break;
         }

         if (!-f $request_filename) {
           proxy_pass http://cloud;
           break;
         }   
    }

    location /store/index/ {
      index  index.html; 
      root /var/www/apps/delta/shared/public;

      if (-f $request_filename/index.html) {
        rewrite (.*) $1/index.html break;
      }

      if (-f $request_filename.html) {
        rewrite (.*) $1.html break;
      }

      if (!-f $request_filename) {
        proxy_pass http://delta;
        break;
      }
   }

}

Config file for server2 is simple, process requests at upstream and nothing interesting.

As it must be with this configuration: nginx need to process requests started from '/store/index' like http://myservername.com/store/index/category/898/author/989 at upstream "delta", and he do it. BUT: for bing requests he ignores location /store/index and proxy'ing request to upstream cloud (server2)

Take a look:
Server1 log:

157.55.39.83 - - [04/May/2016:15:33:45 +0300] "GET /store/index/category/67/author/6 HTTP/1.1" 200 12150 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"

Server2 log, little bit custom log, x.x.x.x - my frontend ip:

x.x.x.x - - [04/May/2016:15:33:45 +0300]  "GET /store/index/category/67/author/6 HTTP/1.0" 200 12137 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" "myservername.com" 5.045

What we see:

  1. nginx proxy'ing request to server2
  2. bingbot request http 1.0
    version
  3. this is one proxed request, and not two from bingbot
    to server1, and same time to server2.
  4. after I make the same
    request from another server: curl -0 -v
    http://myservername.com/store/index/category/67/author/6 all is ok,
    only server1 process this request and nginx not proxy'ing this to
    server2.

I'm totally confused. Location is working like I need for usually user requests. And by unknown reason and only for bingbot requests it's proxy'ing them to server2. Please tell me maybe bingbot send some headers or what wrong in config? Thank's a lot!

/index.html break;
}

if (-f $request_filename.html) {
rewrite (.*)

we have found something strange in nginx logs. Some requests only from bingbot goes directly to backend, but if we fire this requests from curl, they processed as usually. Please take a look of configuration.

I've got two servers, one (server1) if frontend and it's proxying some requests to server2 and some requests process by himself.

Please take a look Server 1 config:

server {   
  listen  80;# default_server;

  proxy_set_header        X-Real-IP       $remote_addr; 
  proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;   
  proxy_set_header        Host            $host;

  server_name myservername.com;

     location / {
         index  index.html;
         root /var/www/apps/delta/shared/public;

         if (-f $request_filename/index.html) {
           rewrite (.*) $1/index.html break;
         }

         if (-f $request_filename.html) {
           rewrite (.*) $1.html break;
         }

         if (!-f $request_filename) {
           proxy_pass http://cloud;
           break;
         }   
    }

    location /store/index/ {
      index  index.html; 
      root /var/www/apps/delta/shared/public;

      if (-f $request_filename/index.html) {
        rewrite (.*) $1/index.html break;
      }

      if (-f $request_filename.html) {
        rewrite (.*) $1.html break;
      }

      if (!-f $request_filename) {
        proxy_pass http://delta;
        break;
      }
   }

}

Config file for server2 is simple, process requests at upstream and nothing interesting.

As it must be with this configuration: nginx need to process requests started from '/store/index' like http://myservername.com/store/index/category/898/author/989 at upstream "delta", and he do it. BUT: for bing requests he ignores location /store/index and proxy'ing request to upstream cloud (server2)

Take a look:
Server1 log:

157.55.39.83 - - [04/May/2016:15:33:45 +0300] "GET /store/index/category/67/author/6 HTTP/1.1" 200 12150 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"

Server2 log, little bit custom log, x.x.x.x - my frontend ip:

x.x.x.x - - [04/May/2016:15:33:45 +0300]  "GET /store/index/category/67/author/6 HTTP/1.0" 200 12137 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" "myservername.com" 5.045

What we see:

  1. nginx proxy'ing request to server2
  2. bingbot request http 1.0
    version
  3. this is one proxed request, and not two from bingbot
    to server1, and same time to server2.
  4. after I make the same
    request from another server: curl -0 -v
    http://myservername.com/store/index/category/67/author/6 all is ok,
    only server1 process this request and nginx not proxy'ing this to
    server2.

I'm totally confused. Location is working like I need for usually user requests. And by unknown reason and only for bingbot requests it's proxy'ing them to server2. Please tell me maybe bingbot send some headers or what wrong in config? Thank's a lot!

.html break;
}

if (!-f $request_filename) {
proxy_pass http://cloud;
break;
}
}

location /store/index/ {
index index.html;
root /var/www/apps/delta/shared/public;

if (-f $request_filename/index.html) {
rewrite (.*)

we have found something strange in nginx logs. Some requests only from bingbot goes directly to backend, but if we fire this requests from curl, they processed as usually. Please take a look of configuration.

I've got two servers, one (server1) if frontend and it's proxying some requests to server2 and some requests process by himself.

Please take a look Server 1 config:

server {   
  listen  80;# default_server;

  proxy_set_header        X-Real-IP       $remote_addr; 
  proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;   
  proxy_set_header        Host            $host;

  server_name myservername.com;

     location / {
         index  index.html;
         root /var/www/apps/delta/shared/public;

         if (-f $request_filename/index.html) {
           rewrite (.*) $1/index.html break;
         }

         if (-f $request_filename.html) {
           rewrite (.*) $1.html break;
         }

         if (!-f $request_filename) {
           proxy_pass http://cloud;
           break;
         }   
    }

    location /store/index/ {
      index  index.html; 
      root /var/www/apps/delta/shared/public;

      if (-f $request_filename/index.html) {
        rewrite (.*) $1/index.html break;
      }

      if (-f $request_filename.html) {
        rewrite (.*) $1.html break;
      }

      if (!-f $request_filename) {
        proxy_pass http://delta;
        break;
      }
   }

}

Config file for server2 is simple, process requests at upstream and nothing interesting.

As it must be with this configuration: nginx need to process requests started from '/store/index' like http://myservername.com/store/index/category/898/author/989 at upstream "delta", and he do it. BUT: for bing requests he ignores location /store/index and proxy'ing request to upstream cloud (server2)

Take a look:
Server1 log:

157.55.39.83 - - [04/May/2016:15:33:45 +0300] "GET /store/index/category/67/author/6 HTTP/1.1" 200 12150 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"

Server2 log, little bit custom log, x.x.x.x - my frontend ip:

x.x.x.x - - [04/May/2016:15:33:45 +0300]  "GET /store/index/category/67/author/6 HTTP/1.0" 200 12137 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" "myservername.com" 5.045

What we see:

  1. nginx proxy'ing request to server2
  2. bingbot request http 1.0
    version
  3. this is one proxed request, and not two from bingbot
    to server1, and same time to server2.
  4. after I make the same
    request from another server: curl -0 -v
    http://myservername.com/store/index/category/67/author/6 all is ok,
    only server1 process this request and nginx not proxy'ing this to
    server2.

I'm totally confused. Location is working like I need for usually user requests. And by unknown reason and only for bingbot requests it's proxy'ing them to server2. Please tell me maybe bingbot send some headers or what wrong in config? Thank's a lot!

/index.html break;
}

if (-f $request_filename.html) {
rewrite (.*)

we have found something strange in nginx logs. Some requests only from bingbot goes directly to backend, but if we fire this requests from curl, they processed as usually. Please take a look of configuration.

I've got two servers, one (server1) if frontend and it's proxying some requests to server2 and some requests process by himself.

Please take a look Server 1 config:

server {   
  listen  80;# default_server;

  proxy_set_header        X-Real-IP       $remote_addr; 
  proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;   
  proxy_set_header        Host            $host;

  server_name myservername.com;

     location / {
         index  index.html;
         root /var/www/apps/delta/shared/public;

         if (-f $request_filename/index.html) {
           rewrite (.*) $1/index.html break;
         }

         if (-f $request_filename.html) {
           rewrite (.*) $1.html break;
         }

         if (!-f $request_filename) {
           proxy_pass http://cloud;
           break;
         }   
    }

    location /store/index/ {
      index  index.html; 
      root /var/www/apps/delta/shared/public;

      if (-f $request_filename/index.html) {
        rewrite (.*) $1/index.html break;
      }

      if (-f $request_filename.html) {
        rewrite (.*) $1.html break;
      }

      if (!-f $request_filename) {
        proxy_pass http://delta;
        break;
      }
   }

}

Config file for server2 is simple, process requests at upstream and nothing interesting.

As it must be with this configuration: nginx need to process requests started from '/store/index' like http://myservername.com/store/index/category/898/author/989 at upstream "delta", and he do it. BUT: for bing requests he ignores location /store/index and proxy'ing request to upstream cloud (server2)

Take a look:
Server1 log:

157.55.39.83 - - [04/May/2016:15:33:45 +0300] "GET /store/index/category/67/author/6 HTTP/1.1" 200 12150 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"

Server2 log, little bit custom log, x.x.x.x - my frontend ip:

x.x.x.x - - [04/May/2016:15:33:45 +0300]  "GET /store/index/category/67/author/6 HTTP/1.0" 200 12137 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" "myservername.com" 5.045

What we see:

  1. nginx proxy'ing request to server2
  2. bingbot request http 1.0
    version
  3. this is one proxed request, and not two from bingbot
    to server1, and same time to server2.
  4. after I make the same
    request from another server: curl -0 -v
    http://myservername.com/store/index/category/67/author/6 all is ok,
    only server1 process this request and nginx not proxy'ing this to
    server2.

I'm totally confused. Location is working like I need for usually user requests. And by unknown reason and only for bingbot requests it's proxy'ing them to server2. Please tell me maybe bingbot send some headers or what wrong in config? Thank's a lot!

.html break;
}

if (!-f $request_filename) {
proxy_pass http://delta;
break;
}
}

}

Конфигурационный файл для server2 прост, обрабатывайте запросы в апстриме и ничего интересного.

Как это должно быть с этой конфигурацией: nginx нужно обрабатывать запросы, запущенные из ‘/ store / index’, как http://myservername.com/store/index/category/898/author/989 в верховьях «дельты», и он это делает. BUT: for Bing requests he ignores location / store / index and proxy ‘ in request to upstream cloud (server2)

Взглянуть:
Журнал Server1:

157.55.39.83 - - [04/May/2016:15:33:45 +0300] "GET /store/index/category/67/author/6 HTTP/1.1" 200 12150 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"

Журнал Server2, немного пользовательский журнал, x.x.x.X-my frontend ip:

x.x.x.x - - [04/May/2016:15:33:45 +0300]  "GET /store/index/category/67/author/6 HTTP/1.0" 200 12137 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" "myservername.com" 5.045

Что мы видим:

  1. запрос прокси-сервера nginx к server2
  2. bingbot запрос http 1.0
    версия
  3. это один проксированный запрос, а не два от bingbot
    к server1, и в то же время к server2.
  4. после того, как я сделать то же самое
    запрос с другого сервера: curl -0-v
    http://myservername.com/store/index/category/67/author/6 все в порядке,
    только server1 обработать этот запрос и nginx не прокси ‘ в
    server2.

Я совершенно запуталась. Расположение работает так, как мне нужно для обычно запросов пользователей. И по неизвестной причине и только для запросов bingbot это proxy ‘ ING их к server2. Пожалуйста, скажите мне, может быть, bingbot отправить некоторые заголовки или что неправильно в конфигурации? Спасибо большое!

1 ответ

  1. Кажется, что Bing запрос через. через https-версию-я редактирую мою конфигурацию https, и теперь все хорошо. Спасибо