Discussion:
[lopsa-tech] Odd KVM problem
(too old to reply)
Adam Levin
2018-01-23 18:19:52 UTC
Permalink
Hey everyone,

It's been a while since I've played around at the server level, and I have
an odd issue that my google-fu is failing to resolve.

I've got an Ubuntu 14.04 LTS server that I'm playing on in a lab. I've
created 20 numbered user accounts, and I want each account to initiate a VM
build using virt-install. The install image is Fedora core, but that
shouldn't matter for the problem I'm having.

I'm getting an error when the virt-install command is run:
ERROR Guest name 'foo-01' is already in use.

Except it's not.

These VMs get created and destroyed over and over again. Every time we're
done with them, I do a virsh destroy and a virsh undefine. When I do a
virsh list, I get nothing -- the list is empty. I've also blown away the
old qcow2 disk image.

All of these are in the system URI context. The user accounts run
virt-install via sudo.

The full command being run is:
sudo /usr/bin/virt-install -v -n foo-## \
-r 2048 \
--vcpus=1 \
--os-type=linux \
--accelerate \
--nographics \
--disk path=/opt/VMDisks/user##/foo-##.qcow2,size=10 \
--extra-args "console=ttyS0" \
--location /opt/VMImages/Fedora-Server-dvd-x86_64-25-1.3.iso \
--network network=default,model=virtio,mac=52:54:00:00:00:##

The ## above is 01 through 20 for the 20 users.

Would anyone have any suggestions as to why I'm getting the error? ps
output doesn't report any VM processes lying around. I wonder if it's just
something stuck and I should restart libvirt?

The install log is not particularly helpful:
[Tue, 23 Jan 2018 09:19:01 virt-install 27904] DEBUG (cli:227) Launched
with command line:

/usr/bin/virt-install -v -n foo-01 -r 2048 --vcpus=1 --os-type=linux
--accelerate --nographics --disk
path=/opt/VMDisks/user01/foo-01.qcow2,size=10 --extra-args console=ttyS0
--location /opt/VMImages/Fedora-Server-dvd-x86_64-25-1.3.iso --network
network=default,model=virtio,mac=52:54:00:00:00:01

[Tue, 23 Jan 2018 09:19:01 virt-install 27904] DEBUG (cli:332) Requesting
libvirt URI default

[Tue, 23 Jan 2018 09:19:01 virt-install 27904] DEBUG (cli:334) Received
libvirt URI qemu:///system

[Tue, 23 Jan 2018 09:19:01 virt-install 27904] DEBUG (virt-install:258)
Requesting virt method 'hvm', hv type 'default'.

[Tue, 23 Jan 2018 09:19:01 virt-install 27904] DEBUG (virt-install:469)
Received virt method 'hvm'

[Tue, 23 Jan 2018 09:19:01 virt-install 27904] DEBUG (virt-install:470)
Hypervisor name is 'kvm'

[Tue, 23 Jan 2018 09:19:01 virt-install 27904] DEBUG (cli:959) --graphics
compat generated: none

[Tue, 23 Jan 2018 09:19:01 virt-install 27904] ERROR (cli:605) Guest name
'foo-01' is already in use.

Thanks for any help!
-Adam
--
This list provided by the League of Professional System Administrators
http://lopsa.org/
---
You received this message because you are subscribed to the Google Groups "LOPSA Tech Discussion list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tech+***@lopsa.org.
To post to this group, send email to ***@lopsa.org.
To view this discussion on the web visit https://groups.google.com/a/lopsa.org/d/msgid/tech/CAH9MNTXJFVcmDPEU5ECorXub7O1uVXbA95kGkJyEyGV7370rOw%40mail.gmail.com.
Paul Heinlein
2018-01-23 18:32:28 UTC
Permalink
Post by Adam Levin
Hey everyone,
It's been a while since I've played around at the server level, and
I have an odd issue that my google-fu is failing to resolve.
I've got an Ubuntu 14.04 LTS server that I'm playing on in a lab.
I've created 20 numbered user accounts, and I want each account to
initiate a VM build using virt-install. The install image is Fedora
core, but that shouldn't matter for the problem I'm having.
ERROR Guest name 'foo-01' is already in use.
Except it's not.
These VMs get created and destroyed over and over again. Every time
we're done with them, I do a virsh destroy and a virsh undefine.
When I do a virsh list, I get nothing -- the list is empty. I've
also blown away the old qcow2 disk image.
Does "virsh list --all" also return empty?

Does "ls /etc/libvirt/qemu/*.xml" return empty? (That's a RHEL/CentOS
path; it may be different in Ubuntu. I don't have a test machine
handy.)
--
Paul Heinlein
***@madboa.com
45°38' N, 122°6' W
--
This list provided by the League of Professional System Administrators
http://lopsa.org/
---
You received this message because you are subscribed to the Google Groups "LOPSA Tech Discussion list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tech+***@lopsa.org.
To post to this group, send email to ***@lopsa.org.
To view this discussion on the web visit https://groups.google.com/a/lopsa.org/d/msgid/tech/alpine.OSX.2.21.1801231027240.9681%40silver.madboa.com.
Adam Levin
2018-01-23 18:39:10 UTC
Permalink
Heh, no, "virsh list -- all" does *not* return empty, it reports all of the
VMs in a "shut off" state with no Id and the correct Name.

And /etc/libvirt/qemu is full of xml files, one per VM that *should* be
gone.

So I have two questions. First, why weren't they
deleted/destroyed/undefined/removed when I did the virsh destroy / virsh
undefine, and second, what should I do about it now? :)

Thanks for the pointers!

-Adam
Post by Adam Levin
Hey everyone,
It's been a while since I've played around at the server level, and I have
Post by Adam Levin
an odd issue that my google-fu is failing to resolve.
I've got an Ubuntu 14.04 LTS server that I'm playing on in a lab. I've
created 20 numbered user accounts, and I want each account to initiate a VM
build using virt-install. The install image is Fedora core, but that
shouldn't matter for the problem I'm having.
ERROR Guest name 'foo-01' is already in use.
Except it's not.
These VMs get created and destroyed over and over again. Every time
we're done with them, I do a virsh destroy and a virsh undefine. When I do
a virsh list, I get nothing -- the list is empty. I've also blown away the
old qcow2 disk image.
Does "virsh list --all" also return empty?
Does "ls /etc/libvirt/qemu/*.xml" return empty? (That's a RHEL/CentOS
path; it may be different in Ubuntu. I don't have a test machine handy.)
--
Paul Heinlein
45°38' N, 122°6' W
--
This list provided by the League of Professional System Administrators
http://lopsa.org/
---
You received this message because you are subscribed to the Google Groups "LOPSA Tech Discussion list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tech+***@lopsa.org.
To post to this group, send email to ***@lopsa.org.
To view this discussion on the web visit https://groups.google.com/a/lopsa.org/d/msgid/tech/CAH9MNTX8W98nVMDUT3ujAoy2z2ABgODOX61Dvib%3DGgOv%2BbnGFQ%40mail.gmail.com.
Adam Levin
2018-01-23 19:10:10 UTC
Permalink
Ah, so I *was* able to do a "virsh undefine" *again*, and that seems to
have cleaned up the virtual machines. I'm not sure why it didn't work the
first time, because it appears that's all I needed to do to get things
running again. Strange.

Thanks for the help.

-Adam
Post by Adam Levin
Heh, no, "virsh list -- all" does *not* return empty, it reports all of
the VMs in a "shut off" state with no Id and the correct Name.
And /etc/libvirt/qemu is full of xml files, one per VM that *should* be
gone.
So I have two questions. First, why weren't they
deleted/destroyed/undefined/removed when I did the virsh destroy / virsh
undefine, and second, what should I do about it now? :)
Thanks for the pointers!
-Adam
Post by Adam Levin
Hey everyone,
It's been a while since I've played around at the server level, and I
Post by Adam Levin
have an odd issue that my google-fu is failing to resolve.
I've got an Ubuntu 14.04 LTS server that I'm playing on in a lab. I've
created 20 numbered user accounts, and I want each account to initiate a VM
build using virt-install. The install image is Fedora core, but that
shouldn't matter for the problem I'm having.
ERROR Guest name 'foo-01' is already in use.
Except it's not.
These VMs get created and destroyed over and over again. Every time
we're done with them, I do a virsh destroy and a virsh undefine. When I do
a virsh list, I get nothing -- the list is empty. I've also blown away the
old qcow2 disk image.
Does "virsh list --all" also return empty?
Does "ls /etc/libvirt/qemu/*.xml" return empty? (That's a RHEL/CentOS
path; it may be different in Ubuntu. I don't have a test machine handy.)
--
Paul Heinlein
45°38' N, 122°6' W
--
This list provided by the League of Professional System Administrators
http://lopsa.org/
---
You received this message because you are subscribed to the Google Groups "LOPSA Tech Discussion list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tech+***@lopsa.org.
To post to this group, send email to ***@lopsa.org.
To view this discussion on the web visit https://groups.google.com/a/lopsa.org/d/msgid/tech/CAH9MNTX%3D67jShYR6gEz7cGB20tBce9vZJhquXua-r77h66GfRg%40mail.gmail.com.
Adam Levin
2018-01-23 19:33:07 UTC
Permalink
Ok, here's a curious followup.

It appears that if I try to install more than 9 VM guests at once, the
console hangs until one is done.

In other words, I launch e.g. 20 virt-install processes in 20 terminal
sessions. 9 of them work. 10 and 11 start to work, and then pause. 12-20
just don't even start -- they just hang.

However, if I finish one of the first 9, then the next one (10) starts
going. If I quit out of one of the first 9, the next one starts going.
It's like I only 9 virtual consoles at a time or something. Is there a
limit defined somewhere that I'm missing?

Thanks,
-Adam
Post by Adam Levin
Ah, so I *was* able to do a "virsh undefine" *again*, and that seems to
have cleaned up the virtual machines. I'm not sure why it didn't work the
first time, because it appears that's all I needed to do to get things
running again. Strange.
Thanks for the help.
-Adam
Post by Adam Levin
Heh, no, "virsh list -- all" does *not* return empty, it reports all of
the VMs in a "shut off" state with no Id and the correct Name.
And /etc/libvirt/qemu is full of xml files, one per VM that *should* be
gone.
So I have two questions. First, why weren't they
deleted/destroyed/undefined/removed when I did the virsh destroy / virsh
undefine, and second, what should I do about it now? :)
Thanks for the pointers!
-Adam
Post by Adam Levin
Hey everyone,
It's been a while since I've played around at the server level, and I
Post by Adam Levin
have an odd issue that my google-fu is failing to resolve.
I've got an Ubuntu 14.04 LTS server that I'm playing on in a lab. I've
created 20 numbered user accounts, and I want each account to initiate a VM
build using virt-install. The install image is Fedora core, but that
shouldn't matter for the problem I'm having.
ERROR Guest name 'foo-01' is already in use.
Except it's not.
These VMs get created and destroyed over and over again. Every time
we're done with them, I do a virsh destroy and a virsh undefine. When I do
a virsh list, I get nothing -- the list is empty. I've also blown away the
old qcow2 disk image.
Does "virsh list --all" also return empty?
Does "ls /etc/libvirt/qemu/*.xml" return empty? (That's a RHEL/CentOS
path; it may be different in Ubuntu. I don't have a test machine handy.)
--
Paul Heinlein
45°38' N, 122°6' W
--
This list provided by the League of Professional System Administrators
http://lopsa.org/
---
You received this message because you are subscribed to the Google Groups "LOPSA Tech Discussion list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tech+***@lopsa.org.
To post to this group, send email to ***@lopsa.org.
To view this discussion on the web visit https://groups.google.com/a/lopsa.org/d/msgid/tech/CAH9MNTXnVpxaGpD6dRYJ%2B4Max53488nkSB%2BagXPO7vJzLOq7Jw%40mail.gmail.com.
Guus Snijders
2018-01-23 21:58:56 UTC
Permalink
Op 23 jan. 2018 20:33 schreef "Adam Levin" <***@gmail.com>:

Ok, here's a curious followup.

It appears that if I try to install more than 9 VM guests at once, the
console hangs until one is done.


A first question should of course be: anything in the relevant logs?

A wild guess; is there a (default) limit on the number of simultaneous
virtio devices?


Mvg, Guus Snijders
--
This list provided by the League of Professional System Administrators
http://lopsa.org/
---
You received this message because you are subscribed to the Google Groups "LOPSA Tech Discussion list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tech+***@lopsa.org.
To post to this group, send email to ***@lopsa.org.
To view this discussion on the web visit https://groups.google.com/a/lopsa.org/d/msgid/tech/CAHntV0Bxw68hLJsjFMFUVnnGXDv15%2BkmAu7u%2BhyNJsZZkbxyfQ%40mail.gmail.com.
Adam Levin
2018-01-23 22:03:46 UTC
Permalink
That's what I'm thinking: that there's some default limit somewhere. I'm
hoping someone with more experience with this might shed some light,
because I'm searching for answers and haven't found anything helpful yet,
but I may not be searching for the correct google terms. :)

I'm not even sure which logs to check, but the curious bit is really that
it's not *failing* -- it's just hanging until I quit out of the installer,
and then the next window continues where it stopped.

Even more curious is that 1-9 start fine, 10 and 11 hang at the "creating
domain" line, and 12-20 hang earlier than that, at the point where I think
they're trying to mount the image. I *did* create more /dev/loop devices
because the mount of the ISO requires loopback mounts.

-Adam
Post by Adam Levin
Ok, here's a curious followup.
It appears that if I try to install more than 9 VM guests at once, the
console hangs until one is done.
A first question should of course be: anything in the relevant logs?
A wild guess; is there a (default) limit on the number of simultaneous
virtio devices?
Mvg, Guus Snijders
--
This list provided by the League of Professional System Administrators
http://lopsa.org/
---
You received this message because you are subscribed to the Google Groups "LOPSA Tech Discussion list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tech+***@lopsa.org.
To post to this group, send email to ***@lopsa.org.
To view this discussion on the web visit https://groups.google.com/a/lopsa.org/d/msgid/tech/CAH9MNTVDe9fcgWBH00Dg9PVMzc62rtPLWFLfHPPJ6_Ph_g8B9A%40mail.gmail.com.
Guus Snijders
2018-01-23 22:23:22 UTC
Permalink
Op 23 jan. 2018 23:03 schreef "Adam Levin" <***@gmail.com>:

That's what I'm thinking: that there's some default limit somewhere. I'm
hoping someone with more experience with this might shed some light,
because I'm searching for answers and haven't found anything helpful yet,
but I may not be searching for the correct google terms. :)

I'm not even sure which logs to check, but the curious bit is really that
it's not *failing* -- it's just hanging until I quit out of the installer,
and then the next window continues where it stopped.


Libvirt logging is described here:
https://libvirt.org/logging.html

IIUC, the logs are probably under /var/log/libvirt, but could also be in
the systemd journal.


Actually, that part is easy to test; try starting without any network
devices.

Also: has your host enough memory?
(I'm not sure how libvirt handles that situation, though. I'd expect it to
refuse to start instead of waiting.)


Mvg, Guus Snijders
--
This list provided by the League of Professional System Administrators
http://lopsa.org/
---
You received this message because you are subscribed to the Google Groups "LOPSA Tech Discussion list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tech+***@lopsa.org.
To post to this group, send email to ***@lopsa.org.
To view this discussion on the web visit https://groups.google.com/a/lopsa.org/d/msgid/tech/CAHntV0Adqt%3Dw27r0MU%2BZ9YLecQwjadSLye7UhyK8ExJVydQ6yA%40mail.gmail.com.
Guus Snijders
2018-01-23 22:34:00 UTC
Permalink
Op 23 jan. 2018 23:23 schreef "Guus Snijders" <***@gmail.com>:

Op 23 jan. 2018 23:03 schreef "Adam Levin" <***@gmail.com>:

That's what I'm thinking: that there's some default limit somewhere.


In addition to my previous message, a max # of devices is certainly
possible.

I found some documentation[1] that explicitly mentions this. Still, at
least something should be logged, so start with the logs.


1.
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/virtualization_administration_guide/chap-guest_virtual_machine_device_configuration


Mvg, Guus Snijders
--
This list provided by the League of Professional System Administrators
http://lopsa.org/
---
You received this message because you are subscribed to the Google Groups "LOPSA Tech Discussion list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tech+***@lopsa.org.
To post to this group, send email to ***@lopsa.org.
To view this discussion on the web visit https://groups.google.com/a/lopsa.org/d/msgid/tech/CAHntV0C6KthyMeXnJEi11V7w2pyhKMttJgwp5yBmMGcG0Tkdug%40mail.gmail.com.
Adam Levin
2018-01-24 01:57:04 UTC
Permalink
Thanks, Guus. The log files as default configured aren't helpful. I tried
info level, but there was no obvious information to help here. I tried
debug level, and it's entirely possible that there's something in there,
but there's so much data that it's hard to tell -- a tremendous amount of
polling takes place, and just scrolls on forever.

I'm not sure which device is maxing out, and there's nothing in the config
files to suggest what's up, since the options are mostly per-domain, and
this appears to be outside of the domain if it's indeed a limit.

The host has 128GB of memory, and I'm creating 20 2G VMs, so I don't think
there's an issue with memory, and top isn't reporting anything untoward.
No swapping in vmstat either.

-Adam
Post by Adam Levin
That's what I'm thinking: that there's some default limit somewhere.
In addition to my previous message, a max # of devices is certainly
possible.
I found some documentation[1] that explicitly mentions this. Still, at
least something should be logged, so start with the logs.
1. https://access.redhat.com/documentation/en-us/red_hat_
enterprise_linux/6/html/virtualization_administration_
guide/chap-guest_virtual_machine_device_configuration
Mvg, Guus Snijders
--
This list provided by the League of Professional System Administrators
http://lopsa.org/
---
You received this message because you are subscribed to the Google Groups "LOPSA Tech Discussion list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tech+***@lopsa.org.
To post to this group, send email to ***@lopsa.org.
To view this discussion on the web visit https://groups.google.com/a/lopsa.org/d/msgid/tech/CAH9MNTWSk5nnBs3dcZMzWdkWOJmKcO0cTHxBQos9cXXaj0zOZA%40mail.gmail.com.
c***@surewest.net
2018-01-24 07:11:44 UTC
Permalink
Hi Adam,

Maybe ask on the KVM mailing list? ***@vger.kernel.org

Best regards,
--
Chuck Polisher
--
This list provided by the League of Professional System Administrators
http://lopsa.org/
---
You received this message because you are subscribed to the Google Groups "LOPSA Tech Discussion list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tech+***@lopsa.org.
To post to this group, send email to ***@lopsa.org.
To view this discussion on the web visit https://groups.google.com/a/lopsa.org/d/msgid/tech/20180124071144.GD3629%40vinny.peecee3.com.
l***@gmail.com
2018-01-24 17:33:26 UTC
Permalink
Thanks for the pointer. That's a great idea!

Adam
Post by c***@surewest.net
Hi Adam,
Best regards,
--
Chuck Polisher
--
This list provided by the League of Professional System Administrators
http://lopsa.org/
---
You received this message because you are subscribed to the Google Groups "LOPSA Tech Discussion list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tech+***@lopsa.org.
To post to this group, send email to ***@lopsa.org.
To view this discussion on the web visit https://groups.google.com/a/lopsa.org/d/msgid/tech/598AD186-59D0-4826-BFB6-55B9D8807F14%40gmail.com.
Continue reading on narkive:
Loading...