Fuel for OpenStack

'ntpd' on controllers is configured to access itself as a remote peer

Bug #1481627 reported by Dennis Dmitriev on 2015-08-05

This bug affects 3 people

Affects		Status	Importance	Assigned to	Milestone
	Fuel for OpenStack	Fix Released	Critical	Oleksiy Molchanov	Fuel for OpenStack 7.0
	6.1.x	Fix Committed	High	Alexey Stupnikov	Fuel for OpenStack 6.1-updates

Bug Description

'ntpd' on all controllers is configured to access the vip__management IP as a remote peer.
Thus, the time on controllers and other remote nodes is never synchronized to external clock source.

root@node-1:~# ip netns exec vrouter ip r | grep vr-mgmt
10.109.12.0/24 dev vr-mgmt proto kernel scope link src 10.109.12.2

root@node-1:~# fgrep 'server' /etc/ntp.conf
server 10.109.12.2 iburst minpoll 3

Reproduced on swarm tests: http://jenkins-product.srt.mirantis.net:8080/view/7.0_swarm/job/7.0.system_test.ubuntu.cic_maintenance_mode/50/

ISO version: {u'build_id': u'2015-08-03_22-04-28', u'build_number': u'132', u'auth_required': True, u'fuel-ostf_sha': u'53109a99d923cccdf88c5cf5aba0af8050df47e3', u'fuel-library_sha': u'1cfd80a833ed27c777c950006a8d4e4080f81616', u'nailgun_sha': u'd1536c3a57459e205e39bc4d86d2b4efc6141c4e', u'openstack_version': u'2015.1.0-7.0', u'fuel-nailgun-agent_sha': u'1512b9af6b41cc95c4d891c593aeebe0faca5a63', u'fuel-agent_sha': u'1fe47720ba554818a0be707f2e16281791492d50', u'api': u'1.0', u'python-fuelclient_sha': u'4fe70fb5c0ce8905ae5908f63d45b45e89a99340', u'astute_sha': u'6d09f3fc7f69ac558095299211ebfd081fa54b8f', u'fuelmain_sha': u'7a374fbd1f5ebde943cb391a4f71b94888ce4a15', u'feature_groups': [u'mirantis'], u'release': u'7.0', u'release_versions': {u'2015.1.0-7.0': {u'VERSION': {u'build_id': u'2015-08-03_22-04-28', u'build_number': u'132', u'fuel-library_sha': u'1cfd80a833ed27c777c950006a8d4e4080f81616', u'nailgun_sha': u'd1536c3a57459e205e39bc4d86d2b4efc6141c4e', u'fuel-ostf_sha': u'53109a99d923cccdf88c5cf5aba0af8050df47e3', u'fuel-nailgun-agent_sha': u'1512b9af6b41cc95c4d891c593aeebe0faca5a63', u'fuel-agent_sha': u'1fe47720ba554818a0be707f2e16281791492d50', u'api': u'1.0', u'python-fuelclient_sha': u'4fe70fb5c0ce8905ae5908f63d45b45e89a99340', u'astute_sha': u'6d09f3fc7f69ac558095299211ebfd081fa54b8f', u'fuelmain_sha': u'7a374fbd1f5ebde943cb391a4f71b94888ce4a15', u'feature_groups': [u'mirantis'], u'release': u'7.0', u'openstack_version': u'2015.1.0-7.0', u'production': u'docker'}}}, u'production': u'docker'}

Tags:

Revision history for this message

Dennis Dmitriev (ddmitriev) wrote on 2015-08-05:

fail_error_auto_cic_maintenance_mode-fuel-snapshot-2015-08-05_04-29-44.tar.xz Edit (98.7 MiB, application/octet-stream)

Oleksiy Molchanov (omolchanov) on 2015-08-05

Changed in fuel:
status:	New → Confirmed

Oleksiy Molchanov (omolchanov) on 2015-08-05

Changed in fuel:
assignee:	Fuel Library Team (fuel-library) → Oleksiy Molchanov (omolchanov)

Revision history for this message

Oleksiy Molchanov (omolchanov) wrote on 2015-08-05:

Strange, the config was changed by task ntp-client.conf that run in post-deploy. But it is not supposed to be executed on controller nodes. Python team, please take a look.

Changed in fuel:
assignee:	Oleksiy Molchanov (omolchanov) → Fuel Python Team (fuel-python)

Igor Marnat (imarnat) on 2015-08-06

tags:

added: fuel-to-mos

Revision history for this message

Andrey Shestakov (ashestakov) wrote on 2015-08-06:

Works on build #140

node-1 - compute
node-2 - controller

root@node-1:~# fgrep 'server' /etc/ntp.conf
# Specify one or more NTP servers.
# Use servers from the NTP Pool Project. Approved by Ubuntu Technical Board
# Use Ubuntu's ntp server as a fallback.
# Note that "restrict" applies to both servers and clients, so a configuration
# up blocking replies from your own upstream servers.
# Local users may interrogate the ntp server more closely.
server 10.20.0.2 burst iburst

root@node-2:~# fgrep 'server' /etc/ntp.conf
# Set up servers for ntpd with next options:
# server - IP address or DNS name of upstream NTP server
# prefer - select preferrable server
server 0.fuel.pool.ntp.org iburst minpoll 3
server 1.fuel.pool.ntp.org iburst minpoll 3
server 2.fuel.pool.ntp.org iburst minpoll 3

Revision history for this message

Serge Kovaleff (serge-kovaleff) wrote on 2015-08-06:

@Fuel QA Team Please verify

What is the correct status for it if we didn't fix it?

Changed in fuel:
assignee:	Fuel Python Team (fuel-python) → Fuel QA Team (fuel-qa)

Nastya Urlapova (aurlapova) on 2015-08-06

Changed in fuel:
status:	Confirmed → Incomplete

Revision history for this message

Andrey Shestakov (ashestakov) wrote on 2015-08-06:

looks like fixed at
https://github.com/stackforge/fuel-library/commit/54c510d97db11b216fe2a91884c5c8e3ce02c24a

Revision history for this message

Dennis Dmitriev (ddmitriev) wrote on 2015-08-07:

Still reproduced on ISO with build_number: '139' build_id: 2015-08-05_21-24-26. Let's wait for tests on more fresh ISO.

Revision history for this message

Sergey Yudin (tsipa740) wrote on 2015-08-07:

I have verifyied this bug on

VERSION:
  feature_groups:
    - experimental
  production: "docker"
  release: "7.0"
  openstack_version: "2015.1.0-7.0"
  api: "1.0"
  build_number: "135"
  build_id: "2015-08-06_06-45-26"
  nailgun_sha: "c842770d56680d0e0cdf3573d329ce36d4fb2c64"
  python-fuelclient_sha: "28ddd022fd230fde0e88f2f3e4a6a6dddcb58abe"
  fuel-agent_sha: "1fe47720ba554818a0be707f2e16281791492d50"
  fuel-nailgun-agent_sha: "1512b9af6b41cc95c4d891c593aeebe0faca5a63"
  astute_sha: "e1d3a435e5df5b40cbfb1a3acf80b4176d15a2dc"
  fuel-library_sha: "347d51f391a6f2c621006c00e3416718140889ac"
  fuel-ostf_sha: "5cb828f6200e5ae0da63dd39d11fbf910237db95"
  fuelmain_sha: "858a4ee8336789c10c5c2189c3718f9a605d0eeb"

and it was fixed controller:
Last login: Fri Aug 7 17:08:29 2015 from 10.109.5.2
root@node-6:~# cat /etc/ntp.conf | grep server
server 10.109.6.1 iburst minpoll 3
root@node-6:~# cat /etc/astute.yaml | grep '10.109.6.1$'
        gateway: 10.109.6.1
  dns_list: 10.109.6.1
      gateway: 10.109.6.1
  ntp_list: 10.109.6.1

This bug was fixed by Dmitry Ilyin in https://review.openstack.org/#/c/197629/14/deployment/puppet/osnailyfacter/modular/ntp/tasks.yaml

Changed in fuel:
status:	Incomplete → Fix Committed

Revision history for this message

Dennis Dmitriev (ddmitriev) wrote on 2015-08-10:

Confirmed on ISO #150.

Reproduced on CI job: http://jenkins-product.srt.mirantis.net:8080/job/7.0.system_test.ubuntu.thread_3/54/console

root@node-1:~# cat /etc/astute.yaml |grep 'vrouter:' -A4
    vrouter:
      network_role: mgmt/vip
      ipaddr: 10.109.4.2
      node_roles:
      - controller
--
vrouter:
  network_role: mgmt/vip
  ipaddr: 10.109.4.2
  node_roles:
  - controller

root@node-1:~# cat /etc/ntp.conf | grep 'server'
# Set up servers for ntpd with next options:
# server - IP address or DNS name of upstream NTP server
# prefer - select preferrable server
server 10.109.4.2 iburst minpoll 3

root@node-2:~# cat /etc/ntp.conf | grep 'server'
# Set up servers for ntpd with next options:
# server - IP address or DNS name of upstream NTP server
# prefer - select preferrable server
server 10.109.4.2 iburst minpoll 3

root@node-3:~# cat /etc/ntp.conf | grep 'server'
# Set up servers for ntpd with next options:
# server - IP address or DNS name of upstream NTP server
# prefer - select preferrable server
server 10.109.4.2 iburst minpoll 3

Changed in fuel:
status:	Fix Committed → Confirmed
importance:	High → Critical
assignee:	Fuel QA Team (fuel-qa) → Fuel Library Team (fuel-library)

Dennis Dmitriev (ddmitriev) on 2015-08-10

tags:

added: swarm-blocker

Revision history for this message

Andrey Sledzinskiy (asledzinskiy) wrote on 2015-08-10:

Reproduced on swarm, logs are attached

Revision history for this message

Andrey Sledzinskiy (asledzinskiy) wrote on 2015-08-10:

#10

fail_error_check_ceph_cinder_cow-fuel-snapshot-2015-08-10_03-47-29.tar.xz Edit (35.0 MiB, application/octet-stream)

Bogdan Dobrelya (bogdando) on 2015-08-10

Changed in fuel:
assignee:	Fuel Library Team (fuel-library) → Bogdan Dobrelya (bogdando)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-08-10: Fix proposed to fuel-library (master)

#11

Fix proposed to branch: master
Review: https://review.openstack.org/211092

Changed in fuel:
status:	Confirmed → In Progress

Revision history for this message

Bogdan Dobrelya (bogdando) wrote on 2015-08-10:

#12

https://review.openstack.org/211092

Revision history for this message

Bogdan Dobrelya (bogdando) wrote on 2015-08-10:

#13

I believe the https://review.openstack.org/#/c/197629/14/deployment/puppet/osnailyfacter/modular/ntp/tasks.yaml didn't fix the issue as multiple roles still may be applied on controllers. And this executes ntp-client task at controllers.

Oleksiy Molchanov (omolchanov) on 2015-08-10

Changed in fuel:
assignee:	Bogdan Dobrelya (bogdando) → Oleksiy Molchanov (omolchanov)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-08-10: Change abandoned on fuel-library (master)

#14

Change abandoned by Bogdan Dobrelya (<email address hidden>) on branch: master
Review: https://review.openstack.org/211092
Reason: This would have made all nodes to rely on only hiera provided ntp servers, which are normally external ones. While on compute nodes we may want to not rely on external ntp servers

Revision history for this message

Timur Nurlygayanov (tnurlygayanov) wrote on 2015-08-10:

#15

Not reproduced on my environment with MOS 7.0 #132:

root@node-1:~# ip netns exec vrouter ip r | grep vr-mgmt
192.168.0.0/24 dev vr-mgmt proto kernel scope link src 192.168.0.1
root@node-1:~# fgrep 'server' /etc/ntp.conf
server 0.fuel.pool.ntp.org iburst minpoll 3
server 1.fuel.pool.ntp.org iburst minpoll 3
server 2.fuel.pool.ntp.org iburst minpoll 3

Changed in fuel:
status:	In Progress → Invalid

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-08-10: Fix proposed to fuel-library (master)

#16

Fix proposed to branch: master
Review: https://review.openstack.org/211194

Changed in fuel:
status:	Invalid → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-08-10: Fix merged to fuel-library (master)

#17

Reviewed: https://review.openstack.org/211194
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=a4b5e9bb4510563d5bc52ca356a92bb6bab44dfc
Submitter: Jenkins
Branch: master

commit a4b5e9bb4510563d5bc52ca356a92bb6bab44dfc
Author: Oleksiy Molchanov <email address hidden>
Date: Mon Aug 10 17:24:25 2015 +0300

Restrict ntp-client on controller nodes

NTP-client task shouldn't be run on controller multirole nodes

Change-Id: I54defe421ded251ad13b665b6fd1843600a517a7
Closes-Bug: 1481627

Changed in fuel:
status:	In Progress → Fix Committed

Revision history for this message

Oleksiy Molchanov (omolchanov) wrote on 2015-08-11:

#18

Should be merged to 6.1 with https://bugs.launchpad.net/fuel/6.1.x/+bug/1466090

Revision history for this message

Dennis Dmitriev (ddmitriev) wrote on 2015-08-12:

#19

Released on ISO #160:

[root@nailgun ~]# for i in {1..3}; do ssh node-$i cat /etc/ntp.conf|grep server|grep -v "^#"; done
Warning: Permanently added 'node-1' (RSA) to the list of known hosts.
server 10.109.1.1 iburst minpoll 3
Warning: Permanently added 'node-2' (RSA) to the list of known hosts.
server 10.109.1.1 iburst minpoll 3
Warning: Permanently added 'node-3' (RSA) to the list of known hosts.
server 10.109.1.1 iburst minpoll 3

root@node-1:~# cat /etc/astute.yaml |grep ntp
external_ntp:
ntp_list: 10.109.1.1

Changed in fuel:
status:	Fix Committed → Fix Released

Revision history for this message

Dennis Dmitriev (ddmitriev) wrote on 2015-08-12:

#21

api: '1.0'
astute_sha: e1d3a435e5df5b40cbfb1a3acf80b4176d15a2dc
auth_required: true
build_id: 2015-08-11_17-24-26
build_number: '160'
feature_groups:
- mirantis
fuel-agent_sha: 57145b1d8804389304cd04322ba0fb3dc9d30327
fuel-library_sha: ecdfcc0b23c4fbeea193e222e6eb5fb1f4b4e68f
fuel-nailgun-agent_sha: e01693992d7a0304d926b922b43f3b747c35964c
fuel-ostf_sha: c7f745431aa3c147f2491c865e029e0ffea91c47
fuelmain_sha: af2d875a47c5d08bb62943bed5ef1ae7d7ea3329
nailgun_sha: ca95dd5d79282d4613da4275205b67b6da408e2e
openstack_version: 2015.1.0-7.0
production: docker
python-fuelclient_sha: b297cee0c54b4ff787ca0ac518a54348b838d342
release: '7.0'
release_versions:
  2015.1.0-7.0:
    VERSION:
      api: '1.0'
      astute_sha: e1d3a435e5df5b40cbfb1a3acf80b4176d15a2dc
      build_id: 2015-08-11_17-24-26
      build_number: '160'
      feature_groups:
      - mirantis
      fuel-agent_sha: 57145b1d8804389304cd04322ba0fb3dc9d30327
      fuel-library_sha: ecdfcc0b23c4fbeea193e222e6eb5fb1f4b4e68f
      fuel-nailgun-agent_sha: e01693992d7a0304d926b922b43f3b747c35964c
      fuel-ostf_sha: c7f745431aa3c147f2491c865e029e0ffea91c47
      fuelmain_sha: af2d875a47c5d08bb62943bed5ef1ae7d7ea3329
      nailgun_sha: ca95dd5d79282d4613da4275205b67b6da408e2e
      openstack_version: 2015.1.0-7.0
      production: docker
      python-fuelclient_sha: b297cee0c54b4ff787ca0ac518a54348b838d342
      release: '7.0'