I have been running into issues on Windows servers while running JMeter tests, but also when I have been tuning akka as described in another blog of mine http://www.baselogic.com/blog/development/java-javaee-j2ee/akka-actor-thread-utilization-and-optimization/ and after more digging, I have finally found the issue.
The issues have always been java.net.BindException: Address already in use: connect and I have been thinking this was some IO issue related to Disk or CPU but never had conclusive evidence and never found the smoking gun until now.
So today I was running JMeter on a windows machine attempting to send 250 concurrent requests to an Apache server for a load test and could not reach 250 concurrent threads on the machine where JMeter was running.
After searching, I found that this problem is Windows based. There is a article at http://support.microsoft.com/kb/q196271, which says Windows XP only will make outbound TCP/IP connections using ports 1024-5000, and takes up to 4 minutes to recycle them. Therefore, if you do a lot of connections in a short amount of time, you can easily eat that port range up if you are running load tests.
I found reference to this issue on Microsoft’s website: http://support.microsoft.com/kb/q196271
The default maximum number of ephemeral TCP ports is 5000 in the products that are included in the “Applies to” section. A new parameter has been added in these products. To increase the maximum number of ephemeral ports, follow these steps:
- Start Registry Editor.
- Locate the following subkey in the registry, and then click Parameters: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
- On the Edit menu, click New, and then add the following registry entry:
Value Name: MaxUserPort
Value Type: DWORD
Value data: 65534
Valid Range: 5000-65534 (decimal)
Default: 0x1388 (5000 decimal)
Description: This parameter controls the maximum port number that is used when a program requests any available user port from the system. Typically, ephemeral (short-lived) ports are allocated between the values of 1024 and 5000 inclusive. After the release of security bulletin MS08-037, the behavior of Windows Server 2003 was changed to more closely match that of Windows Server 2008 and Windows Vista. For more information about Microsoft security bulletin MS08-037, click the following article numbers to view the articles in the Microsoft Knowledge Base:
951746 MS08-037: Description of the security update for DNS in Windows Server 2008, in Windows Server 2003, and in Windows 2000 Server (DNS server-side): July 8, 2008
951748 MS08-037: Description of the security update for DNS in Windows Server 2003, in Windows XP, and in Windows 2000 Server (client side): July 8, 2008
953230 MS08-037: Vulnerabilities in DNS could allow spoofing
Exit Registry Editor, and then restart the computer.
This seemed like it was simple, but I was not able to find the correct editable parameter in my regedit.
Here is what I did to add this parameter:
- Right mouse clicked on Parameters.
- Create a new DWORD value with name MaxUserPort
- Right mouse click on MaxUserPort .
- then enter 65534 for the Value Data field.
- Under the radio button selection for Base, select the Decimal radio button.
- Restart the machine.
Once I did this and te server restarted, I was able to run a JMeter test that created 1,000 concurrent requests to an Apache server for 5 minutes with zero errors.
My next step will be to load test the Akka based application against these new settings, as see how many threads I can now support.