Discussion:
[lopsa-tech] VM as an NTP server
(too old to reply)
Jeremy Charles
2016-04-04 16:18:35 UTC
Permalink
I'm seeing all sort of documentation about how it's not a great idea to use a VM as an NTP server due to how sketchy time tracking is within a VM.

My supervisor directed me to try it anyway. He feels that our existing NTP servers are too old and need to be replaced, and he wants to replace them with VMs rather than physical servers.

I'm not seeing any difference in behavior between the two existing physical NTP servers and the VM that I set up to test as an NTP server.

Thoughts?

==
Jeremy Charles
Epic's Computer and Technology Services Division
***@epic.com
608-271-9000
Derek J. Balling
2016-04-04 16:36:38 UTC
Permalink
My experience is thus:

The misbehavior of NTP on a VM will happen "when you care most". When
the host is under resource constraints, clock cycles will be stolen from
some guests so that other more demanding guests can have them.

D
I’m seeing all sort of documentation about how it’s not a great idea
to use a VM as an NTP server due to how sketchy time tracking is
within a VM.
My supervisor directed me to try it anyway. He feels that our
existing NTP servers are too old and need to be replaced, and he wants
to replace them with VMs rather than physical servers.
I’m not seeing any difference in behavior between the two existing
physical NTP servers and the VM that I set up to test as an NTP server.
Thoughts?
==
Jeremy Charles
Epic’s Computer and Technology Services Division
608-271-9000
_______________________________________________
Tech mailing list
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
--
I prefer to use encrypted mail. My public key fingerprint is
FD6A 6990 F035 DE9E 3713 B4F1 661B 3AD6 D82A BBD0.

You can download it at http://www.megacity.org/gpg_dballing.txt

Learn how to encrypt your email with the E-Mail Self Defense
Guide: https://emailselfdefense.fsf.org/en/
Chris Snell
2016-04-04 16:52:23 UTC
Permalink
NTP as a client on VMs is sketchy at best. Make sure you follow the
documentation from both your hypervisor and OS vendors in order to get it to
behave as well as possible. I doubt I would trust a VM as an actual NTP
server, but it also likely depends on the sensitivity of the devices utilizing
those NTP servers. Certain protocols/software fail badly as soon as time is out
of sync even a few milliseconds (*cough* AD clients *cough*).

A lot of places offload NTP servers to their network devices. If you've got
sufficiently powerful physical networking hardware, you might try that.

- Chris
Post by Derek J. Balling
The misbehavior of NTP on a VM will happen "when you care most". When
the host is under resource constraints, clock cycles will be stolen from
some guests so that other more demanding guests can have them.
D
I?m seeing all sort of documentation about how it?s not a great idea
to use a VM as an NTP server due to how sketchy time tracking is
within a VM.
My supervisor directed me to try it anyway. He feels that our
existing NTP servers are too old and need to be replaced, and he wants
to replace them with VMs rather than physical servers.
I?m not seeing any difference in behavior between the two existing
physical NTP servers and the VM that I set up to test as an NTP server.
Thoughts?
==
Jeremy Charles
Epic?s Computer and Technology Services Division
608-271-9000
_______________________________________________
Tech mailing list
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
--
I prefer to use encrypted mail. My public key fingerprint is
FD6A 6990 F035 DE9E 3713 B4F1 661B 3AD6 D82A BBD0.
You can download it at http://www.megacity.org/gpg_dballing.txt
Learn how to encrypt your email with the E-Mail Self Defense
Guide: https://emailselfdefense.fsf.org/en/
"The greatest pleasure in life is doing what people say you cannot do."
- Walter Bagehot
_______________________________________________
Tech mailing list
***@lists.lopsa.org
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
Edward Ned Harvey (lopser)
2016-04-04 18:51:01 UTC
Permalink
On Behalf Of Chris Snell
Certain protocols/software fail badly as soon as time is
out
of sync even a few milliseconds (*cough* AD clients *cough*).
AD's tolerance is +/- 5 minutes by default.
_______________________________________________
Tech mailing list
***@lists.lopsa.org
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
Brian Ruppert
2016-04-04 17:05:40 UTC
Permalink
We're running VMs as NTP servers in several places, and our
experience has been that it works fine under normal circumstances, but
things get out of sync (at least temporarily) when a vMotion operation
occurs.

I haven't had time to do any significant research into this, but have
considered setting "Latency Sensitivity" to High in VMware, which should
ensure that the memory is reserved (so the balloon driver doesn't try to
clear cache or make things swap in the VM) and a PCPU is 100% allocated
for each VCPU on the VM (which should be OK since the NTP VMs only need
one VCPU, and we have many cores on a host). The idea here would be to
avoid things going badly if the host gets busy.

Several of the hosts which we run NTP servers on aren't running DRS
in fully automated mode, but if we did, I would likely tell DRS to try
to keep the NTP VMs on designated hosts to avoid them being shifted
around for load balancing.

VMware is the only VM environment I have deployed right now, so I
can't say how it works in other hypervisors.

-- Brian
I’m seeing all sort of documentation about how it’s not a great idea
to use a VM as an NTP server due to how sketchy time tracking is
within a VM.
My supervisor directed me to try it anyway. He feels that our
existing NTP servers are too old and need to be replaced, and he wants
to replace them with VMs rather than physical servers.
I’m not seeing any difference in behavior between the two existing
physical NTP servers and the VM that I set up to test as an NTP server.
Thoughts?
==
Jeremy Charles
Epic’s Computer and Technology Services Division
608-271-9000
_______________________________________________
Tech mailing list
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
Dan Ritter
2016-04-04 17:57:09 UTC
Permalink
Post by Jeremy Charles
I'm seeing all sort of documentation about how it's not a great idea to use a VM as an NTP server due to how sketchy time tracking is within a VM.
My supervisor directed me to try it anyway. He feels that our existing NTP servers are too old and need to be replaced, and he wants to replace them with VMs rather than physical servers.
I'm not seeing any difference in behavior between the two existing physical NTP servers and the VM that I set up to test as an NTP server.
Thoughts?
You will likely run into problems.

Note that NTP is an extremely easy task in most situations, and
dedicating two 1U boxes for general infrastructure (DNS, DHCP,
NTP, possibly TFTP for PXE) should be an easy sell in most
companies.

-dsr-
_______________________________________________
Tech mailing list
***@lists.lopsa.org
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
Tom Perrine
2016-04-04 18:15:37 UTC
Permalink
I've found Raspberry PI's to be OK for small-scale NTP servers, if not
doing much else. The clocks aren't fantastic, but they track higher level
stratum clocks quite well in my very limited experience.

Hmmm, the screenly folks do a turnkey SD card image for PIs for their app.
I wonder if it would be worthwhile to do something similar for making an
NTP server?
Post by Jeremy Charles
Post by Jeremy Charles
I'm seeing all sort of documentation about how it's not a great idea to
use a VM as an NTP server due to how sketchy time tracking is within a VM.
Post by Jeremy Charles
My supervisor directed me to try it anyway. He feels that our existing
NTP servers are too old and need to be replaced, and he wants to replace
them with VMs rather than physical servers.
Post by Jeremy Charles
I'm not seeing any difference in behavior between the two existing
physical NTP servers and the VM that I set up to test as an NTP server.
Post by Jeremy Charles
Thoughts?
You will likely run into problems.
Note that NTP is an extremely easy task in most situations, and
dedicating two 1U boxes for general infrastructure (DNS, DHCP,
NTP, possibly TFTP for PXE) should be an easy sell in most
companies.
-dsr-
_______________________________________________
Tech mailing list
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
Dan Ritter
2016-04-04 18:21:01 UTC
Permalink
Post by Tom Perrine
I've found Raspberry PI's to be OK for small-scale NTP servers, if not
doing much else. The clocks aren't fantastic, but they track higher level
stratum clocks quite well in my very limited experience.
Hmmm, the screenly folks do a turnkey SD card image for PIs for their app.
I wonder if it would be worthwhile to do something similar for making an
NTP server?
You'll want to start with a Debian stable image or similar; this
is production we're talking about. At that point:

sudo apt-get install ntp
or
sudo apt-get install openntpd

and dropping a new config into place is all that you absolutely
need. After that you will want local management changes -- chef
or puppet or ansible or what have you, a user and so forth.

How stable is Raspberry Pi hardware compared to a cheap Atom
server?

-dsr-
_______________________________________________
Tech mailing list
***@lists.lopsa.org
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
John Stoffel
2016-04-04 18:24:22 UTC
Permalink
Tom> I've found Raspberry PI's to be OK for small-scale NTP servers,
Tom> if not doing much else. The clocks aren't fantastic, but they
Tom> track higher level stratum clocks quite well in my very limited
Tom> experience.

Tom> Hmmm, the screenly folks do a turnkey SD card image for PIs for
Tom> their app. I wonder if it would be worthwhile to do something
Tom> similar for making an NTP server?

I wonder how well BeagleBone Blacks (BBB) would do instead of the PIs?
Faster CPU and more memory and better network hardware. It's not like
NTP takes a ton of resources anyway... you could setup a triplet at
multiple locations for not alot of money. And I think you can power
them over PoE... nope, not without a bunch of injectors and at that
point it's not worth it in my book.

But I'd still thing the BBB might be a better solution. Or even some
of the dedicated 1U rackable units. But getting bosses to buy these
things is a hard sell for something they don't always think of as
important or critical.

John
_______________________________________________
Tech mailing list
***@lists.lopsa.org
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
Matthew Butch
2016-04-04 18:23:16 UTC
Permalink
Post by Dan Ritter
Post by Jeremy Charles
I'm seeing all sort of documentation about how it's not a great idea to use a VM as an NTP server due to how sketchy time tracking is within a VM.
My supervisor directed me to try it anyway. He feels that our existing NTP servers are too old and need to be replaced, and he wants to replace them with VMs rather than physical servers.
I'm not seeing any difference in behavior between the two existing physical NTP servers and the VM that I set up to test as an NTP server.
Thoughts?
You will likely run into problems.
Note that NTP is an extremely easy task in most situations, and
dedicating two 1U boxes for general infrastructure (DNS, DHCP,
NTP, possibly TFTP for PXE) should be an easy sell in most
companies.
I’ve found most companies don’t understand NTP. Case in point, I worked for a company who not only was okay with VMs as NTP servers, but insisted that they not be hosted in our e-comm datacenters, but only all the way back at corporate.
_______________________________________________
Tech mailing list
***@lists.lopsa.org
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.or
Edward Ned Harvey (lopser)
2016-04-04 18:52:33 UTC
Permalink
On Behalf Of Dan Ritter
Note that NTP is an extremely easy task in most situations, and
dedicating two 1U boxes for general infrastructure (DNS, DHCP,
NTP, possibly TFTP for PXE) should be an easy sell in most
companies.
These are the cheapest I'm aware of. Around $300 to $500
http://baudlabs.com/low-cost-ntp-hardware-appliance/
_______________________________________________
Tech mailing list
***@lists.lopsa.org
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
Jeff Wasilko
2016-04-04 18:15:20 UTC
Permalink
You could consider an appliance approach...

on the cheap side:

http://www.bswusa.com/Site-Control-Broadcast-Tools-NTP-Server-Sentinel-P8700.aspx?gclid=Cj0KEQjwoYi4BRDF_PHHu6rI7NMBEiQAKZ-JuLzp0zvvbEe8pxNHVIExueJFtjUYir1BjggIv6B-rIEaAs_W8P8HAQ

On te more expensive side:

http://www.endruntechnologies.com/
https://www.meinbergglobal.com/
Post by Jeremy Charles
I'm seeing all sort of documentation about how it's not a great idea to use a VM as an NTP server due to how sketchy time tracking is within a VM.
My supervisor directed me to try it anyway. He feels that our existing NTP servers are too old and need to be replaced, and he wants to replace them with VMs rather than physical servers.
I'm not seeing any difference in behavior between the two existing physical NTP servers and the VM that I set up to test as an NTP server.
Thoughts?
==
Jeremy Charles
Epic's Computer and Technology Services Division
608-271-9000
_______________________________________________
Tech mailing list
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
_______________________________________________
Tech mailing list
***@lists.lopsa.org
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
Robert Hajime Lanning
2016-04-04 18:17:25 UTC
Permalink
I’m seeing all sort of documentation about how it’s not a great idea to
use a VM as an NTP server due to how sketchy time tracking is within a VM.
My supervisor directed me to try it anyway. He feels that our existing
NTP servers are too old and need to be replaced, and he wants to replace
them with VMs rather than physical servers.
I’m not seeing any difference in behavior between the two existing
physical NTP servers and the VM that I set up to test as an NTP server.
Thoughts?
The timer tick interrupt is virtualized and replicated across all VMs
via software. It is no longer a stable measure of time. NTPd's detection
of slew rate is no longer a constant. Under high loads you get random
time drifts.

How do you track real time passage in a virtual reality?

https://www.kernel.org/doc/Documentation/virtual/kvm/timekeeping.txt
"4.1) Interrupt clocking

One of the most immediate problems that occurs with legacy operating systems
is that the system timekeeping routines are often designed to keep track of
time by counting periodic interrupts. These interrupts may come from the PIT
or the RTC, but the problem is the same: the host virtualization engine may not
be able to deliver the proper number of interrupts per second, and so guest
time may fall behind. This is especially problematic if a high interrupt rate
is selected, such as 1000 HZ, which is unfortunately the default for many Linux
guests.

There are three approaches to solving this problem; first, it may be possible
to simply ignore it. Guests which have a separate time source for tracking
'wall clock' or 'real time' may not need any adjustment of their interrupts to
maintain proper time. If this is not sufficient, it may be necessary to inject
additional interrupts into the guest in order to increase the effective
interrupt rate. This approach leads to complications in extreme conditions,
where host load or guest lag is too much to compensate for, and thus another
solution to the problem has risen: the guest may need to become aware of lost
ticks and compensate for them internally. Although promising in theory, the
implementation of this policy in Linux has been extremely error prone, and a
number of buggy variants of lost tick compensation are distributed across
commonly used Linux systems.

Windows uses periodic RTC clocking as a means of keeping time internally, and
thus requires interrupt slewing to keep proper time. It does use a low enough
rate (ed: is it 18.2 Hz?) however that it has not yet been a problem in
practice."
--
Mr. Flibble
King of the Potato People
http://www.linkedin.com/in/RobertLanning
Jason Barbier
2016-04-04 18:55:47 UTC
Permalink
To be totally fair if you are to the point of considering putting your
NTP server in a VM it may be worth seeing if you even actually need one.
If its going to run well in a VM the public stratum servers are going to
work for what you need, if its not you're just going to be buying
hardware in 6 months - a year when your entire infra is falling down
around your ears due to time sync issues.

--
Jason Barbier | E: ***@serversave.us
GPG Key-ID: B5F75B47(http://kusuriya.devio.us/pubkey.asc)
I’m seeing all sort of documentation about how it’s not a great idea
to use a VM as an NTP server due to how sketchy time tracking is
within a VM.
My supervisor directed me to try it anyway.  He feels that our
existing NTP servers are too old and need to be replaced, and he wants
to replace them with VMs rather than physical servers.
I’m not seeing any difference in behavior between the two
existing physical NTP servers and the VM that I set up to test as
an NTP server.
Thoughts?
==
Jeremy Charles
Epic’s Computer and Technology Services Division
608-271-9000
_________________________________________________
Tech mailing list
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
Mike Robinson
2016-04-04 18:05:52 UTC
Permalink
Are your NTP servers just pulling from upstream time servers and serving to your fleet? If so, and given that you're only running 2 physical servers (should be 3, 5 or 7 depending on the writeup you read...), that will be fine as long as you do follow your hypervisor's best practices for NTP serving.

m.
I’m seeing all sort of documentation about how it’s not a great idea to use a VM as an NTP server due to how sketchy time tracking is within a VM.
My supervisor directed me to try it anyway. He feels that our existing NTP servers are too old and need to be replaced, and he wants to replace them with VMs rather than physical servers.
I’m not seeing any difference in behavior between the two existing physical NTP servers and the VM that I set up to test as an NTP server.
Thoughts?
==
Jeremy Charles
Epic’s Computer and Technology Services Division
608-271-9000
_______________________________________________
Tech mailing list
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech <https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech>
This list provided by the League of Professional System Administrators
http://lopsa.org/ <http://lopsa.org/>
Brian Mathis
2016-04-04 19:29:08 UTC
Permalink
Time syncing is one of the biggest problems VMs have. Unless you're able
to fully understand the NTP source code and and all of the intricacies of
clock syncing, you really aren't qualified to evaluate it. "I don't see
any issues", especially in the face of pretty much every Internet resource
telling you no to do it, doesn't cut it.

As others have said, many other hardware devices in your environment might
be able to provide the service, such as routers, switches, firewalls,
etc... You're much better off looking into something like that than just
crossing your fingers and ignoring the generally well-accepted advice of
others.

~ Brian Mathis
@orev
I’m seeing all sort of documentation about how it’s not a great idea to
use a VM as an NTP server due to how sketchy time tracking is within a VM.
My supervisor directed me to try it anyway. He feels that our existing
NTP servers are too old and need to be replaced, and he wants to replace
them with VMs rather than physical servers.
I’m not seeing any difference in behavior between the two existing
physical NTP servers and the VM that I set up to test as an NTP server.
Thoughts?
==
Jeremy Charles
Epic’s Computer and Technology Services Division
608-271-9000
_______________________________________________
Tech mailing list
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
Jeremy Charles
2016-04-04 19:42:10 UTC
Permalink
To clarify
 I’m of the strong opinion that using VMs as NTP servers is a bad idea. My supervisor is electing to disregard the overwhelming documentation that it’s a bad idea. He’s also disregarding his own lack of qualification for evaluating the issues at hand. He said, “Well, try it and see if it works.” So far, I’ve been unable to demonstrate any results that show him beyond a doubt that it won’t work out well.

Based on earlier feedback from this group, I’ve sent him a potential workaround of sticking an NTP server VM on each of four or more physical VM hosts. The idea is that if one or two physical hosts get busy, the NTP clients will be able to disregard the one or two NTP server VMs that are drifting. The obvious problem is that if a physical host fails and all remaining physical hosts get swamped at the same time, chaos will result as all of the NTP server VMs drift randomly at the same time.

I also said I would only proceed if he provides a written acknowledgement that he’s directing me to implement a system that I am recommending against. :-)

JC


From: ***@betteradmin.com [mailto:***@betteradmin.com] On Behalf Of Brian Mathis
Sent: Monday, April 4, 2016 2:29 PM
To: Jeremy Charles <***@epic.com>
Cc: ***@lists.lopsa.org
Subject: Re: [lopsa-tech] VM as an NTP server

Time syncing is one of the biggest problems VMs have. Unless you're able to fully understand the NTP source code and and all of the intricacies of clock syncing, you really aren't qualified to evaluate it. "I don't see any issues", especially in the face of pretty much every Internet resource telling you no to do it, doesn't cut it.
As others have said, many other hardware devices in your environment might be able to provide the service, such as routers, switches, firewalls, etc... You're much better off looking into something like that than just crossing your fingers and ignoring the generally well-accepted advice of others.

~ Brian Mathis
@orev

On Mon, Apr 4, 2016 at 12:18 PM, Jeremy Charles <***@epic.com<mailto:***@epic.com>> wrote:
I’m seeing all sort of documentation about how it’s not a great idea to use a VM as an NTP server due to how sketchy time tracking is within a VM.

My supervisor directed me to try it anyway. He feels that our existing NTP servers are too old and need to be replaced, and he wants to replace them with VMs rather than physical servers.

I’m not seeing any difference in behavior between the two existing physical NTP servers and the VM that I set up to test as an NTP server.

Thoughts?

==
Jeremy Charles
Epic’s Computer and Technology Services Division
***@epic.com<mailto:***@epic.com>
608-271-9000<tel:608-271-9000>


_______________________________________________
Tech mailing list
***@lists.lopsa.org<mailto:***@lists.lopsa.org>
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
Jeff Wasilko
2016-04-04 19:50:20 UTC
Permalink
Based on earlier feedback from this group, I???ve sent him a potential workaround of sticking an NTP server VM on each of four or more physical VM hosts. The idea is that if one or two physical hosts get busy, the NTP clients will be able to disregard the one or two NTP server VMs that are drifting. The obvious problem is that if a physical host fails and all remaining physical hosts get swamped at the same time, chaos will result as all of the NTP server VMs drift randomly at the same time.
If you're going to just wade into that without understanding how NTP actually detects a falseticker, good luck.

If you only have 4 NTP servers, you're not going to have much luck if you have 2 outliers.

IIRC you need 2n+1 servers to detect n bad servers
_______________________________________________
Tech mailing list
***@lists.lopsa.org
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
Tony Micah Lambert
2016-04-04 20:20:35 UTC
Permalink
On the whole, using a VM as an NTP server is not necessarily a bad
thing as long as you understand what time sync services your virtual
host imposes upon the guest. Most virtual host solutions offer a time
sync service to match the guest time to the host. For a NTP server,
you'll likely want to disable this so the NTP system can be an
authoritative source for all other devices (including its host).

This sort of configuration is commonly seen in the Windows world,
where domain controllers are NTP sources for the rest of the domain.
If you have a Windows DC that syncs from an upstream time source,
you'd have to perform the configuration I mentioned above.

Thanks,

Tony M Lambert
Send Tech mailing list submissions to
To subscribe or unsubscribe via the World Wide Web, visit
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
or, via email, send a message with subject or body 'help' to
You can reach the person managing the list at
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Tech digest..."
1. Re: VM as an NTP server (Mike Robinson)
2. Re: VM as an NTP server (Edward Ned Harvey (lopser))
3. Re: VM as an NTP server (Matthew Butch)
4. Re: VM as an NTP server (Edward Ned Harvey (lopser))
5. Re: VM as an NTP server (Brian Mathis)
----------------------------------------------------------------------
Message: 1
Date: Mon, 4 Apr 2016 13:05:52 -0500
Subject: Re: [lopsa-tech] VM as an NTP server
Content-Type: text/plain; charset="utf-8"
Are your NTP servers just pulling from upstream time servers and serving to your fleet? If so, and given that you're only running 2 physical servers (should be 3, 5 or 7 depending on the writeup you read...), that will be fine as long as you do follow your hypervisor's best practices for NTP serving.
m.
I?m seeing all sort of documentation about how it?s not a great idea to use a VM as an NTP server due to how sketchy time tracking is within a VM.
My supervisor directed me to try it anyway. He feels that our existing NTP servers are too old and need to be replaced, and he wants to replace them with VMs rather than physical servers.
I?m not seeing any difference in behavior between the two existing physical NTP servers and the VM that I set up to test as an NTP server.
Thoughts?
==
Jeremy Charles
Epic?s Computer and Technology Services Division
608-271-9000
_______________________________________________
Tech mailing list
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech <https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech>
This list provided by the League of Professional System Administrators
http://lopsa.org/ <http://lopsa.org/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lopsa.org/pipermail/tech/attachments/20160404/f5c4768d/attachment-0001.html>
------------------------------
Message: 2
Date: Mon, 4 Apr 2016 18:51:01 +0000
Subject: Re: [lopsa-tech] VM as an NTP server
Content-Type: text/plain; charset="us-ascii"
On Behalf Of Chris Snell
Certain protocols/software fail badly as soon as time is
out
of sync even a few milliseconds (*cough* AD clients *cough*).
AD's tolerance is +/- 5 minutes by default.
------------------------------
Message: 3
Date: Mon, 04 Apr 2016 14:23:16 -0400
Subject: Re: [lopsa-tech] VM as an NTP server
Content-Type: text/plain; charset=utf-8
Post by Jeremy Charles
I'm seeing all sort of documentation about how it's not a great idea to use a VM as an NTP server due to how sketchy time tracking is within a VM.
My supervisor directed me to try it anyway. He feels that our existing NTP servers are too old and need to be replaced, and he wants to replace them with VMs rather than physical servers.
I'm not seeing any difference in behavior between the two existing physical NTP servers and the VM that I set up to test as an NTP server.
Thoughts?
You will likely run into problems.
Note that NTP is an extremely easy task in most situations, and
dedicating two 1U boxes for general infrastructure (DNS, DHCP,
NTP, possibly TFTP for PXE) should be an easy sell in most
companies.
I?ve found most companies don?t understand NTP. Case in point, I worked for a company who not only was okay with VMs as NTP servers, but insisted that they not be hosted in our e-comm datacenters, but only all the way back at corporate.
------------------------------
Message: 4
Date: Mon, 4 Apr 2016 18:52:33 +0000
Subject: Re: [lopsa-tech] VM as an NTP server
Content-Type: text/plain; charset="us-ascii"
On Behalf Of Dan Ritter
Note that NTP is an extremely easy task in most situations, and
dedicating two 1U boxes for general infrastructure (DNS, DHCP,
NTP, possibly TFTP for PXE) should be an easy sell in most
companies.
These are the cheapest I'm aware of. Around $300 to $500
http://baudlabs.com/low-cost-ntp-hardware-appliance/
------------------------------
Message: 5
Date: Mon, 4 Apr 2016 15:29:08 -0400
Subject: Re: [lopsa-tech] VM as an NTP server
Content-Type: text/plain; charset="utf-8"
Time syncing is one of the biggest problems VMs have. Unless you're able
to fully understand the NTP source code and and all of the intricacies of
clock syncing, you really aren't qualified to evaluate it. "I don't see
any issues", especially in the face of pretty much every Internet resource
telling you no to do it, doesn't cut it.
As others have said, many other hardware devices in your environment might
be able to provide the service, such as routers, switches, firewalls,
etc... You're much better off looking into something like that than just
crossing your fingers and ignoring the generally well-accepted advice of
others.
~ Brian Mathis
@orev
I?m seeing all sort of documentation about how it?s not a great idea to
use a VM as an NTP server due to how sketchy time tracking is within a VM.
My supervisor directed me to try it anyway. He feels that our existing
NTP servers are too old and need to be replaced, and he wants to replace
them with VMs rather than physical servers.
I?m not seeing any difference in behavior between the two existing
physical NTP servers and the VM that I set up to test as an NTP server.
Thoughts?
==
Jeremy Charles
Epic?s Computer and Technology Services Division
608-271-9000
_______________________________________________
Tech mailing list
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lopsa.org/pipermail/tech/attachments/20160404/821d3d3d/attachment.html>
------------------------------
_______________________________________________
Tech mailing list
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
End of Tech Digest, Vol 125, Issue 5
************************************
_______________________________________________
Tech mailing list
***@lists.lopsa.org
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
Michael Ryder
2016-04-05 01:58:57 UTC
Permalink
Hello Jeremy

Admittedly, I don't know much about NTP other than it's purpose and some of
its abilities.

Without wanting to hijack this thread, I'm curious - are all of your NTP
clients, VMs themselves? If VMs are so bad at tracking time, then what
good are they for any reason? If an NTP client on a VM is able to track
time and keep it's guest OS in-sync, then why is an NTP server (in a VM)
any less accurate? Has anyone got a study they can reference?

Or...are your VMs part of a Windows AD domain? If so, you will probably
want to let your servers get their time sync from the domain controllers,
so as not to inflict Kerberos problems on yourself.

Or... are you sharing time for physical servers? If you have physical
servers, why not just install an NTP service onto one of them and be done
with it?

Mike
I’m seeing all sort of documentation about how it’s not a great idea to
use a VM as an NTP server due to how sketchy time tracking is within a VM.
My supervisor directed me to try it anyway. He feels that our existing
NTP servers are too old and need to be replaced, and he wants to replace
them with VMs rather than physical servers.
I’m not seeing any difference in behavior between the two existing
physical NTP servers and the VM that I set up to test as an NTP server.
Thoughts?
==
Jeremy Charles
Epic’s Computer and Technology Services Division
608-271-9000
_______________________________________________
Tech mailing list
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
Robert Hajime Lanning
2016-04-05 02:35:57 UTC
Permalink
If you use an unstable time source, the clients will detect it and deem
the server unsuitable (marked as a false ticker.)

At that point all your clients drift on their own.

So, ya, it is important the the servers be stable. The clients can be a
bit unstable and the ntpd on them will fight the clock to keep it inline
with the server.

I like using the option "tinker panic 0" on all my client VMs so they
don't just give up.
Post by Michael Ryder
Hello Jeremy
Admittedly, I don't know much about NTP other than it's purpose and some
of its abilities.
Without wanting to hijack this thread, I'm curious - are all of your NTP
clients, VMs themselves? If VMs are so bad at tracking time, then what
good are they for any reason? If an NTP client on a VM is able to track
time and keep it's guest OS in-sync, then why is an NTP server (in a VM)
any less accurate? Has anyone got a study they can reference?
Or...are your VMs part of a Windows AD domain? If so, you will probably
want to let your servers get their time sync from the domain
controllers, so as not to inflict Kerberos problems on yourself.
Or... are you sharing time for physical servers? If you have physical
servers, why not just install an NTP service onto one of them and be
done with it?
Mike
--
Mr. Flibble
King of the Potato People
http://www.linkedin.com/in/RobertLanning
_______________________________________________
Tech mailing list
***@lists.lopsa.org
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
Edward Ned Harvey (lopser)
2016-04-05 12:47:53 UTC
Permalink
It's clear that the industry best practice is to use a real time source, but there is a flip side: If you deviate from the industry best practice, how much do you risk? When instructed to deviate from best practice, should you be upset and insist that your boss put the request in writing, creating friction between yourself and him/her? Should you refuse to do it completely? Should you just roll with it?

If you have a specific need, then it's important to cater to that need, and you might have to insist against deviation from best practice. But if instead, all you have to worry about is AD remaining functional, and approximate correct timestamps on files and such, and users knowing the correct time to show up at meetings, then your need is much less strict.

I hear people on this list referring to "unstable time source" and "false ticker." This is a valid concern, sort-of. In reality, your guest machines are no better at tracking time than the VM time server is, so when there's any deviation between the client and your time server, it doesn't result in a false ticker. The way you get a false ticker is when you have configured several time servers, and some of them agree with each other by quorum, but there is an outlier. In that case, the clients still get the correct time, but the one outlier server needs to be corrected.

There is something to be said for anecdotal evidence, when there's a lot of it. I've seen many dozens of environments where AD is run in virtual servers, which get their time from the internet (non-local stratum 2), and AD serves time to the clients on the LAN without noticeable problems. Sure the time may drift a few seconds in either direction, but again - unless you have a specific need, that's good enough for general use.

Most environments don't have a dedicated stratum 1 time source, and most AD is run on VM's.

In those many dozen environments, I've seen many hundreds of VM's running as guests on peoples' laptops. Most common is the windows VM running inside someone's macbook. This introduces yet another level of VM time-fuzziness, and yet again, I've never seen it cause a problem, because the only requirements in those environments have been to keep AD running, and clients operational, and users showing up to meetings at the right time.

I have some further anecdotal evidence: I have actually seen a situation or two, where the AD server was no longer able to get time from the internet (due to firewall change - once hardware, and once software). So the AD time source slowly drifted off, and all the clients slowly followed. We discovered when a human noticed, "Why does my laptop say 4:05, when my phone says 4:13?" So then we restored the ability for AD to follow the internet, and AD slowly adjusted back to the right time, and all the clients slowly followed.

Nevermind stratum 1. That's a VM client, following a VM server, following a non-local stratum 2 time source. It's about as bad as you can get, but the problem was caused by firewall blockage, and the behavior of the system was about as ideal as you can get, once the firewall problems were corrected.

Unless you have a specific need, in this case, the risk of deviation from best practice is pretty low.

Further evidence: Despite trying to demonstrate a problem with this, to prove your boss wrong, you couldn't demonstrate a problem.
_______________________________________________
Tech mailing list
***@lists.lopsa.org
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
Brock, Anthony
2016-04-05 15:25:23 UTC
Permalink
Post by Edward Ned Harvey (lopser)
It's clear that the industry best practice is to use a real time source, but there is a flip side: If you deviate from the industry best practice, how much do you risk? When instructed to deviate from best practice, should you be upset and insist that your boss put the request in writing, creating friction between yourself and him/her? Should you refuse to do it completely? Should you just roll with it?
If you have a specific need, then it's important to cater to that need, and you might have to insist against deviation from best practice. But if instead, all you have to worry about is AD remaining functional, and approximate correct timestamps on files and such, and users knowing the correct time to show up at meetings, then your need is much less strict.
Edward,

Remember that "industry best practice" truly is "practice". These are generally accepted ways of doing things that have a high probability of working well. However, almost everyone needs to deviate from these practices in some manner due to local needs. The real question isn't "do I deviate from best practice?" but instead "how do I deal with the need to deviate from best practices in a manner that fits my organization, our tolerance for risk and the business culture?". For example, I've found it advisable to:

1. Research best practice
2. Identify where we will need to deviate and why
3. Document specifically why we need to deviate [this is a form of risk assessment]
4. Verify with my supervisors that thy agree with my risk assessment, modifying the assessment if needed

I would NEVER recommend refusing to do something unless you're also prepared to find a different employer. However, nothing prevents you from documenting your risk assessment and ensuring that affected parties are on the same page.

Tony
_______________________________________________
Tech mailing list
***@lists.lopsa.org
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
c***@surewest.net
2016-04-05 19:43:18 UTC
Permalink
TL;DR: Here be nitpicks maquerading as answers, humor
in wisdom's drag.
Post by Edward Ned Harvey (lopser)
It's clear that the industry best practice is to use a real
time source, but there is a flip side: If you deviate from the
industry best practice, how much do you risk?
Hi Edward,

From your posts here, I've got tons of respect for your opinions
and insights, some of which I've snagged for posterity. Below,
when I say "you" I'm really addressing the List -- who as a rule
are less subtle, less skilled, and more in need of a sharp push.
I'm saddened by the irrational optimism often expressed here.
Please bear with me.

| As the IT person, I never trust anything. Every piece of hardware,
| every software, every system, is expected to fail, and it's my
| job as IT person, to minimize harm caused by these failures. So
| right now, I'm transitioning from "never trust anything" to
| "holy crap, look at that Corillian Death Ray.
| -- Edward Ned Harvey, 2013-05-12

The biggest problems facing sysadmins in general are risk
related. A previous employer saw backups as a time-consuming
predator that was never going to have a pay-back. The problem
was not that they hated backups, but they couldn't see the
existential risk to the business they help reduce. The ops
manager's predecessor had a career-limiting incident with a
failed system for which there was no backup. And yet nothing was
learned.
Post by Edward Ned Harvey (lopser)
When instructed to
deviate from best practice, should you be upset and insist that
your boss put the request in writing, creating friction between
yourself and him/her? Should you refuse to do it completely?
Should you just roll with it?
There's an over-arching game being played, the rules are baked
into our primate behavior. If your boss is supportive and you
have a warm and reciprocal relationship, mention that you need
resources to keep the lights on and wait for them to make it
rain.

| When the customer has beaten upon you long enough, give him what
| he asks for, instead of what he needs. This is very strong
| medicine, and is normally only required once.
| --Vadim Vygonets, a.s.r

Otherwise, carefully transact Vadim's advice and discreetly
carry an umbrella. You can use the tip to prod the most suitable
orifice if funds are delayed. If your manager misunderstands
their role and think they're a problem-solver or maybe
a dominatrix, you've lost the game.

| "It's called a shovel," said the Senior Wrangler. "I've seen
| the gardeners use them. You stick the sharp end in the ground.
| Then it gets a bit technical." -- Terry Pratchett, "Reaper Man"
Post by Edward Ned Harvey (lopser)
If you have a specific need, then it's important to cater to
that need, and you might have to insist against deviation from
best practice.
Paul Culmsee and K. Awati wrote a book-length screed about
best practices that I highly recommend, ISBN 1938908406.
Spoiler alert: they don't work.
Post by Edward Ned Harvey (lopser)
But if instead, all you have to worry about is AD
remaining functional, and approximate correct timestamps on
files and such, and users knowing the correct time to show up at
meetings, then your need is much less strict.
Timestamp accuracy requirments are relative. AD's +/- 5 minutes
(based, I think on the Kerberos requirements for ticket expiry)
is the guard-rail of timekeeping. Maybe you didn't plunge off
the highway down the embankment but you're definitely not
driving within the lane. At the other end of the scale, when
systems have gone pear-shaped and 1 msec clock uncertainty has
sundered your ability to separate cause from effect, the cost
savings of having run a loose ship will melt away into the lake
of overtime, downtime, and lost sales. If you're lucky you can
force the bill onto the DBAs and developers who will mop up the
mess. Just don't sit with your backs to them.
Post by Edward Ned Harvey (lopser)
I hear people on this list referring to "unstable time source"
and "false ticker." This is a valid concern, sort-of. In
reality, your guest machines are no better at tracking time than
the VM time server is, so when there's any deviation between the
client and your time server, it doesn't result in a false
ticker. The way you get a false ticker is when you have
configured several time servers, and some of them agree with
each other by quorum, but there is an outlier. In that case, the
clients still get the correct time, but the one outlier server
needs to be corrected.
Break some canary server loose from the tyranny of accurate
timestamps and wait for the alarm. If alarms don't sound have
a talk with your PFY about the nature of systems, geometry,
causal chains, overtime, risk management, and so on, until one
day they come (metaphorically) running over waving a printout
saying "holy sh*t the clock on canary666 is way out of spec
how do we fix that?". Until that moment arrives keep 3 to 6
month's wages in highly liquid form. Single malt is good.
Post by Edward Ned Harvey (lopser)
There is something to be said for anecdotal evidence, when
there's a lot of it. I've seen many dozens of environments where
AD is run in virtual servers, which get their time from the
internet (non-local stratum 2), and AD serves time to the
clients on the LAN without noticeable problems. Sure the time
may drift a few seconds in either direction, but again - unless
you have a specific need, that's good enough for general use.
The key thing is to have very tight control over clocks within
the blast radius of your troubleshooting domain. One time the
above reasonable assumptions went wrong was when a junior SA
misconfigured a subdomain in a way that got hundreds of systems
pulling time from the first workstation to boot in the morning.

| The ticket was closed with 'Colonel Mustard, in the
| datacenter, with the keyboard.'
Post by Edward Ned Harvey (lopser)
Most environments don't have a dedicated stratum 1 time
source, and most AD is run on VM's.
Is this the right place to remind that VMs experience time-
travel? One moment it's 10 after two, and then there was a
snapshot revert, and suddenly it's twenty 'till. Oh my what is
the luckless bastard of an NTP client going to make of the giant
jumping timeslop? (Hint: answer varies depending on OS make,
model, and vintage. Some exceptions apply. Results not
guaranteed in all states. When in doubt ask your doctor).
Post by Edward Ned Harvey (lopser)
In those many dozen environments, I've seen many hundreds of
VM's running as guests on peoples' laptops. Most common is the
windows VM running inside someone's macbook. This introduces yet
another level of VM time-fuzziness, and yet again, I've never
seen it cause a problem, because the only requirements in those
environments have been to keep AD running, and clients
operational, and users showing up to meetings at the right time.
Right -- the real world, of users and desktops. System
administration of the back end is less forgiving.
Post by Edward Ned Harvey (lopser)
I have some further anecdotal evidence: I have actually seen a
situation or two, where the AD server was no longer able to get
time from the internet (due to firewall change - once hardware,
and once software). So the AD time source slowly drifted off,
and all the clients slowly followed. We discovered when a human
noticed, "Why does my laptop say 4:05, when my phone says 4:13?"
So then we restored the ability for AD to follow the internet,
and AD slowly adjusted back to the right time, and all the
clients slowly followed.
Nevermind stratum 1. That's a VM client, following a VM
server, following a non-local stratum 2 time source. It's about
as bad as you can get, but the problem was caused by firewall
blockage, and the behavior of the system was about as ideal as
you can get, once the firewall problems were corrected.
Unless you have a specific need, in this case, the risk of
deviation from best practice is pretty low.
Further evidence: Despite trying to demonstrate a problem with
this, to prove your boss wrong, you couldn't demonstrate a
problem.
--
Charles Polisher

_______________________________________________
Tech mailing list
***@lists.lopsa.org
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
Steven Miano
2016-04-05 21:20:16 UTC
Permalink
No one wants to pay for insurance.

Running a dozen Chrony or NTP servers in VMware for a month does not equate
to: "Hey look, if I don't pay for auto insurance for 30 days, everything is
fine; let's cancel all of our policies on the fleet of trucks".

If you only have to worry about Kerberos in your environment, pointing to
the swath of pools available in your region is likely just fine (drift in
the amount of your average downtime per year is likely not enough to cause
production stoppage).

However, if you have high number of transactions coming into databases, or
financial transactions getting processed with any desire for accountability
(think 1000 per second is millisecond accuracy, and 1MM per second is
nanosecond accuracy, and god help you if you are getting down to
microsecond and need to consider precision time protocols).

There is a need to reduce the amount of time the signal is coming off of an
antennae to the silicon if you have a need for accuracy.

My recommendation in an enterprise that has a datacenter (good):

3 x
https://www.meinbergglobal.com/english/products/pci-express-gps-clock.htm
(slap it in your preferred 1U dual power supplied server)

Better would be:

Three sites (Primary Datacenter, Secondary Datacenter, Tertiary Datacenter):

3 x The above.

A man with two watches will never know or be sure what time it is. 3 is
really a good start.....
Post by c***@surewest.net
TL;DR: Here be nitpicks maquerading as answers, humor
in wisdom's drag.
Post by Edward Ned Harvey (lopser)
It's clear that the industry best practice is to use a real
time source, but there is a flip side: If you deviate from the
industry best practice, how much do you risk?
Hi Edward,
From your posts here, I've got tons of respect for your opinions
and insights, some of which I've snagged for posterity. Below,
when I say "you" I'm really addressing the List -- who as a rule
are less subtle, less skilled, and more in need of a sharp push.
I'm saddened by the irrational optimism often expressed here.
Please bear with me.
| As the IT person, I never trust anything. Every piece of hardware,
| every software, every system, is expected to fail, and it's my
| job as IT person, to minimize harm caused by these failures. So
| right now, I'm transitioning from "never trust anything" to
| "holy crap, look at that Corillian Death Ray.
| -- Edward Ned Harvey, 2013-05-12
The biggest problems facing sysadmins in general are risk
related. A previous employer saw backups as a time-consuming
predator that was never going to have a pay-back. The problem
was not that they hated backups, but they couldn't see the
existential risk to the business they help reduce. The ops
manager's predecessor had a career-limiting incident with a
failed system for which there was no backup. And yet nothing was
learned.
Post by Edward Ned Harvey (lopser)
When instructed to
deviate from best practice, should you be upset and insist that
your boss put the request in writing, creating friction between
yourself and him/her? Should you refuse to do it completely?
Should you just roll with it?
There's an over-arching game being played, the rules are baked
into our primate behavior. If your boss is supportive and you
have a warm and reciprocal relationship, mention that you need
resources to keep the lights on and wait for them to make it
rain.
| When the customer has beaten upon you long enough, give him what
| he asks for, instead of what he needs. This is very strong
| medicine, and is normally only required once.
| --Vadim Vygonets, a.s.r
Otherwise, carefully transact Vadim's advice and discreetly
carry an umbrella. You can use the tip to prod the most suitable
orifice if funds are delayed. If your manager misunderstands
their role and think they're a problem-solver or maybe
a dominatrix, you've lost the game.
| "It's called a shovel," said the Senior Wrangler. "I've seen
| the gardeners use them. You stick the sharp end in the ground.
| Then it gets a bit technical." -- Terry Pratchett, "Reaper Man"
Post by Edward Ned Harvey (lopser)
If you have a specific need, then it's important to cater to
that need, and you might have to insist against deviation from
best practice.
Paul Culmsee and K. Awati wrote a book-length screed about
best practices that I highly recommend, ISBN 1938908406.
Spoiler alert: they don't work.
Post by Edward Ned Harvey (lopser)
But if instead, all you have to worry about is AD
remaining functional, and approximate correct timestamps on
files and such, and users knowing the correct time to show up at
meetings, then your need is much less strict.
Timestamp accuracy requirments are relative. AD's +/- 5 minutes
(based, I think on the Kerberos requirements for ticket expiry)
is the guard-rail of timekeeping. Maybe you didn't plunge off
the highway down the embankment but you're definitely not
driving within the lane. At the other end of the scale, when
systems have gone pear-shaped and 1 msec clock uncertainty has
sundered your ability to separate cause from effect, the cost
savings of having run a loose ship will melt away into the lake
of overtime, downtime, and lost sales. If you're lucky you can
force the bill onto the DBAs and developers who will mop up the
mess. Just don't sit with your backs to them.
Post by Edward Ned Harvey (lopser)
I hear people on this list referring to "unstable time source"
and "false ticker." This is a valid concern, sort-of. In
reality, your guest machines are no better at tracking time than
the VM time server is, so when there's any deviation between the
client and your time server, it doesn't result in a false
ticker. The way you get a false ticker is when you have
configured several time servers, and some of them agree with
each other by quorum, but there is an outlier. In that case, the
clients still get the correct time, but the one outlier server
needs to be corrected.
Break some canary server loose from the tyranny of accurate
timestamps and wait for the alarm. If alarms don't sound have
a talk with your PFY about the nature of systems, geometry,
causal chains, overtime, risk management, and so on, until one
day they come (metaphorically) running over waving a printout
saying "holy sh*t the clock on canary666 is way out of spec
how do we fix that?". Until that moment arrives keep 3 to 6
month's wages in highly liquid form. Single malt is good.
Post by Edward Ned Harvey (lopser)
There is something to be said for anecdotal evidence, when
there's a lot of it. I've seen many dozens of environments where
AD is run in virtual servers, which get their time from the
internet (non-local stratum 2), and AD serves time to the
clients on the LAN without noticeable problems. Sure the time
may drift a few seconds in either direction, but again - unless
you have a specific need, that's good enough for general use.
The key thing is to have very tight control over clocks within
the blast radius of your troubleshooting domain. One time the
above reasonable assumptions went wrong was when a junior SA
misconfigured a subdomain in a way that got hundreds of systems
pulling time from the first workstation to boot in the morning.
| The ticket was closed with 'Colonel Mustard, in the
| datacenter, with the keyboard.'
Post by Edward Ned Harvey (lopser)
Most environments don't have a dedicated stratum 1 time
source, and most AD is run on VM's.
Is this the right place to remind that VMs experience time-
travel? One moment it's 10 after two, and then there was a
snapshot revert, and suddenly it's twenty 'till. Oh my what is
the luckless bastard of an NTP client going to make of the giant
jumping timeslop? (Hint: answer varies depending on OS make,
model, and vintage. Some exceptions apply. Results not
guaranteed in all states. When in doubt ask your doctor).
Post by Edward Ned Harvey (lopser)
In those many dozen environments, I've seen many hundreds of
VM's running as guests on peoples' laptops. Most common is the
windows VM running inside someone's macbook. This introduces yet
another level of VM time-fuzziness, and yet again, I've never
seen it cause a problem, because the only requirements in those
environments have been to keep AD running, and clients
operational, and users showing up to meetings at the right time.
Right -- the real world, of users and desktops. System
administration of the back end is less forgiving.
Post by Edward Ned Harvey (lopser)
I have some further anecdotal evidence: I have actually seen a
situation or two, where the AD server was no longer able to get
time from the internet (due to firewall change - once hardware,
and once software). So the AD time source slowly drifted off,
and all the clients slowly followed. We discovered when a human
noticed, "Why does my laptop say 4:05, when my phone says 4:13?"
So then we restored the ability for AD to follow the internet,
and AD slowly adjusted back to the right time, and all the
clients slowly followed.
Nevermind stratum 1. That's a VM client, following a VM
server, following a non-local stratum 2 time source. It's about
as bad as you can get, but the problem was caused by firewall
blockage, and the behavior of the system was about as ideal as
you can get, once the firewall problems were corrected.
Unless you have a specific need, in this case, the risk of
deviation from best practice is pretty low.
Further evidence: Despite trying to demonstrate a problem with
this, to prove your boss wrong, you couldn't demonstrate a
problem.
--
Charles Polisher
_______________________________________________
Tech mailing list
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
--
Miano, Steven M.
http://stevenmiano.com
Craig Cook
2016-04-05 16:14:08 UTC
Permalink
Weather or not you end up running NTP using a VM is up to your site.

I would want to know if your monitoring system detects when your NTP time drifts though.

If your time does drift, monitoring alerts happen, you investigate. If it turns out to be a VM drift issue you then have evidence for your manager that you may want to manage NTP differently.


If you don't have monitoring, check mk raw is a great opensource tool that has NTP monitoring as one of it's many features.


If you don't trust the NTP time source for your check mk server (i.e. your internal NTP VM), make it use an upstream NTP server itself. You will quickly see time drift if it happens.



Craig
_______________________________________________
Tech mailing list
***@lists.lopsa.org
https://lists.lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
http://lopsa.org/
Continue reading on narkive:
Loading...