VMWare Networking Issue

[Sidoh]

So I'm running into a strange issue with VMWare on my machine.  I have several VMs with different operating systems that are all having the same issue.  I'm pretty sure it's something wrong with the settings on the host, but I was hoping someone else had seen this before and might be able to expedite the troubleshooting procedure.

I'm using NAT networking.  Here's the problem:

I can access anything outside of the local subnet vmware set up for me (which is some 192.168.x.0/24 subnet), but I can't get to the host through that subnet and I can't get to the vm from the host through that subnet.  I have the same problem with host-only networking.

Anyone have any ideas?

[iago]

So let me just make sure I understand this -- you can get to the Internet but not to the host?

Can you tell us how it's set up (the IPs of a couple VMs, the IP of the host (for the VMWare adapter and the other adapter), and anything else you think is relevant?

Try doing a traceroute to an internet site, and seeing what the first hop is. Is it the IP address that you expected?

[nslay]

I'm not sure how this is dealt with on Windows, but on Unix you need to have VMWare use the tap device.  Then you have to bridge the tap device with your ethernet nic.  The ultimate effect is, VMWare now gets direct access to your network (e.g. it gets an IP address from your router).

EDIT:  While doing this, keep in mind WiFi cards likely don't work with bridging (unless in AP mode).

[Sidoh]

So let me just make sure I understand this -- you can get to the Internet but not to the host?

Can you tell us how it's set up (the IPs of a couple VMs, the IP of the host (for the VMWare adapter and the other adapter), and anything else you think is relevant?

Try doing a traceroute to an internet site, and seeing what the first hop is. Is it the IP address that you expected?

Yep, that's right.

I can't believe I left out this detail: it worked fine for several days.  After I restarted, it suddenly stopped.

Unfortunately, I'm having deeper network issues right not, but they should be resolved tomorrow.  I can't give you a traceroute or anything like that.

Another few notes that might help:
  * I get an IP address from vmware's DHCP server.  It's the IP address I expect (i.e., it's in the subnet I specified for NAT networking).
  * The IP address is: 192.168.211.128.
  * Here's a copypasta of ifconfig from the host:

eth0      Link encap:Ethernet  HWaddr 00:50:8D:98:AA:87  
          inet addr:129.82.62.56  Bcast:129.82.63.255  Mask:255.255.254.0
          inet6 addr: fe80::250:8dff:fe98:aa87/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:14574 errors:0 dropped:0 overruns:0 frame:0
          TX packets:7695 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:3400243 (3.2 MB)  TX bytes:563676 (550.4 KB)
          Interrupt:22 Base address:0x6000 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:3403 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3403 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:1918593 (1.8 MB)  TX bytes:1918593 (1.8 MB)

vmnet1    Link encap:Ethernet  HWaddr 00:50:56:C0:00:01          <===== This is the device for host only
          inet addr:192.168.84.1  Bcast:192.168.84.255  Mask:255.255.255.0
          inet6 addr: fe80::250:56ff:fec0:1/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:13 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

vmnet8    Link encap:Ethernet  HWaddr 00:50:56:C0:00:08          <===== This is the device for NAT
          inet addr:192.168.211.1  Bcast:192.168.211.255  Mask:255.255.255.0
          inet6 addr: fe80::250:56ff:fec0:8/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:45 errors:0 dropped:0 overruns:0 frame:0
          TX packets:12 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

  * Here's what nmap does when I map the /24:

chris@shadow:~$ nmap 192.168.211.0/24

Starting Nmap 4.20 ( http://insecure.org ) at 2008-04-13 11:47 MDT
Interesting ports on 192.168.211.1:
Not shown: 1693 closed ports
PORT     STATE SERVICE
22/tcp   open  ssh
111/tcp  open  rpcbind
902/tcp  open  iss-realsecure-sensor
5900/tcp open  vnc

Nmap finished: 256 IP addresses (1 host up) scanned in 2.449 seconds

I'm not sure how this is dealt with on Windows, but on Unix you need to have VMWare use the tap device.  Then you have to bridge the tap device with your ethernet nic.  The ultimate effect is, VMWare now gets direct access to your network (e.g. it gets an IP address from your router).

EDIT:  While doing this, keep in mind WiFi cards likely don't work with bridging (unless in AP mode).

Thanks, but I think that's "bridged" connection.  I'm using NAT (vmware has bridged, NAT and host only).  I don't want that effect.  I want the VM to be able to access the internet through the host, but I don't want it to appear on the network.  There are several reasons for this, but mainly it's because the housing network here doesn't like more than on MAC address appearing on the same port.  Your session gets nuked when that happens, I've discovered.

Right, I discovered that the "hard way" a few months ago.  This is on a plain old ethernet card, though.

[iago]

Can you run:

nmap -sP -PR 192.168.211.0/24

That'll do an ARP scan. Can you also check your ARP table: /sbin/arp

It's a really weird problem since, if you can get to the network, you are going through the host! Are you sure you don't have some other factor, like a firewall or a packetfilter or something that's blocking it?

[Sidoh]

Can you run:

nmap -sP -PR 192.168.211.0/24

That'll do an ARP scan. Can you also check your ARP table: /sbin/arp

It's a really weird problem since, if you can get to the network, you are going through the host! Are you sure you don't have some other factor, like a firewall or a packetfilter or something that's blocking it?

I assume you want this ran on the guest OS?  Give me until tomorrow and I'll paste it here for ya.  Sorry about this, but they killed my connection because I was making more than 133 connections/minute, so I'm having to tunnel everything through SSH.  nmap yelled at me when I ran it on the host OS:

chris@shadow:~$ nmap -sP -RP 192.168.211.0/24
Illegal Argument to -P, use -P0, -PI, -PB, -PE, -PM, -PP, -PA, -PU, -PT, or -PT80 (or whatever number you want for the TCP probe destination port)
QUITTING!

I know!  That immediately occurred to me as really strange.  I'm almost certain I don't have any sort of network filtering stuff running.

[nslay]

Right, the VMware runs on its own little NAT'd network...The guest operating systems can connect outbound, but cannot accept connections outside of their little NAT'd network.  The only way you can connect to them is to use the tap device.  You don't necessarily have to bridge them with your nic, but then you wouldn't be able to access the Internet from a guest operating system. 

[Sidoh]

Right, the VMware runs on its own little NAT'd network...The guest operating systems can connect outbound, but cannot accept connections outside of their little NAT'd network.  The only way you can connect to them is to use the tap device.  You don't necessarily have to bridge them with your nic, but then you wouldn't be able to access the Internet from a guest operating system. 

That makes sense, but vmware should manage that for me.  It certainly did for before I restarted.  Like I mentioned earlier, it worked great for a few days.  Then I restarted and that part stopped working.

Is there an easy way to create the tap manually?

[nslay]

Right, the VMware runs on its own little NAT'd network...The guest operating systems can connect outbound, but cannot accept connections outside of their little NAT'd network.  The only way you can connect to them is to use the tap device.  You don't necessarily have to bridge them with your nic, but then you wouldn't be able to access the Internet from a guest operating system. 

That makes sense, but vmware should manage that for me.  It certainly did for before I restarted.  Like I mentioned earlier, it worked great for a few days.  Then I restarted and that part stopped working.

Is there an easy way to create the tap manually?

tap devices should be created on the fly just from an open().  Check that tap is loaded in the kernel...then do something that invokes an open like "cat /dev/tap93" or is it "cat /dev/net/tap93"...I forget if Linux places them in /dev or /dev/net.  Keep in mind, my experiences with vmware are limited.  Here are my two reasons for thinking this is the problem:  Qemu has this problem, and FreeBSD's tap man page mentions special ioctls for vmware.

[iago]

I assume you want this ran on the guest OS?  Give me until tomorrow and I'll paste it here for ya.  Sorry about this, but they killed my connection because I was making more than 133 connections/minute, so I'm having to tunnel everything through SSH.  nmap yelled at me when I ran it on the host OS:

chris@shadow:~$ nmap -sP -RP 192.168.211.0/24
Illegal Argument to -P, use -P0, -PI, -PB, -PE, -PM, -PP, -PA, -PU, -PT, or -PT80 (or whatever number you want for the TCP probe destination port)
QUITTING!

I know!  That immediately occurred to me as really strange.  I'm almost certain I don't have any sort of network filtering stuff running.

Yes, on the guest.

That problem is either because you specified the parameter backwards (-RP?), or because you're using an older version of nmap. I'm guessing it's because you got the parameter wrong.

[Sidoh]

Sorry.  Multitasking clearly makes me dyslexic!  That or inattentive.

This may not be useful, but here's what nmap spits out on the host OS:

chris@shadow:~$ sudo nmap -sP -PR 192.168.211.0/24
[sudo] password for chris:

Starting Nmap 4.20 ( http://insecure.org ) at 2008-04-13 13:34 MDT
Host 192.168.211.1 appears to be up.
Host 192.168.211.128 appears to be up.
MAC Address: 00:0C:29:E5:8A:AB (VMware)
Host 192.168.211.254 appears to be up.
MAC Address: 00:50:56:F5:E4:D5 (VMWare)
Nmap finished: 256 IP addresses (3 hosts up) scanned in 5.366 seconds
chris@shadow:~$ ping 192.168.211.128
PING 192.168.211.128 (192.168.211.128) 56(84) bytes of data.

--- 192.168.211.128 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 999ms

nslay, the only thing in /dev/net is tun.  Here's what I get when I cat it:

chris@shadow:~$ sudo cat /dev/net/tun
cat: /dev/net/tun: File descriptor in bad state

But I guess that's probably expected.

[iago]

Sorry.  Multitasking clearly makes me dyslexic!  That or inattentive.

This may not be useful, but here's what nmap spits out on the host OS:

chris@shadow:~$ sudo nmap -sP -PR 192.168.211.0/24
[sudo] password for chris:

Starting Nmap 4.20 ( http://insecure.org ) at 2008-04-13 13:34 MDT
Host 192.168.211.1 appears to be up.
Host 192.168.211.128 appears to be up.
MAC Address: 00:0C:29:E5:8A:AB (VMware)
Host 192.168.211.254 appears to be up.
MAC Address: 00:50:56:F5:E4:D5 (VMWare)
Nmap finished: 256 IP addresses (3 hosts up) scanned in 5.366 seconds
chris@shadow:~$ ping 192.168.211.128
PING 192.168.211.128 (192.168.211.128) 56(84) bytes of data.

--- 192.168.211.128 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 999ms

Right, so that means that you can see the host. Is the host Windows or Linux? And are you sure there's no firewall turned on?

Can you install hping3 on the guest and run:
hping3 -S -p XXXX 192.168.211.1

Where "XXXX" is a port that you know is open.

Can you also see if you can connect to the host by its external ip (not the VMware one)?

[Sidoh]

Right, so that means that you can see the host. Is the host Windows or Linux? And are you sure there's no firewall turned on?

Can you install hping3 on the guest and run:
hping3 -S -p XXXX 192.168.211.1

Where "XXXX" is a port that you know is open.

Can you also see if you can connect to the host by its external ip (not the VMware one)?

The host is Linux.  I'm nearly positive I don't have a firewall or anything like that turned on.  Again, it worked perfectly until I restarted (which was because my computer was acting a bit strangely).

I can't do anything right now that requires me to install or download software for the reason I mentioned earlier.  hping3 doesn't come with Ubuntu by default, so I can't use that, but I'll certainly give it a try tomorrow when the network decides to stop being an asshole.

Yes, I can connect to the host by its external IP, but that's no surprise since I can make use of other Internet functions.

[Sidoh]

Okay, sorry for the delay here.

I tried hping3.  100% packet loss.

nmap had some more surprising results. (By the way, looks like you typed -RP in the command you gave me, which is probably why I got it backwards .  It says 192.168.211.1 "appears to be up".

[iago]

I tried hping3.  100% packet loss.

ok

nmap had some more surprising results. (By the way, looks like you typed -RP in the command you gave me, which is probably why I got it backwards .  It says 192.168.211.1 "appears to be up".

I don't know what you're on, but it says -PR everywhere. I tested the command before I posted it here.

Try doing a traceroute outside your network.. I don't expect it to be useful, but it's worth a try.