A comparison of the security and usability of personal firewalls

Security and usability of personal firewalls

March-August 2006

Authors: Almut Herzog, Kristian Köpsén, Malin Nilsson

Abstract:
Effective security of a personal firewall depends on
(1) the rule granularity and the implementation of the rule enforcement, and
(2) the correctness and granularity of user decisions at the time of an alert.
A misconfigured or loosely configured firewall may be more dangerous than no firewall at all because of the user's false sense of security. This study assesses effective security of 13 personal firewalls by comparing possible granularity of rules as well as the usability of rule set-up and its influence on security.

In order to evaluate usability, we have submitted each firewall to use cases that require user decisions and cause rule creation. In order to evaluate the firewalls' security, we analysed the created rules. In addition, we ran a port scan and replaced a legitimate, network-enabled application with another program to assess the firewalls' behaviour in misuse cases. We have conducted a cognitive walkthrough paying special attention to user guidance and user decision support.

We conclude that a stronger emphasis on user guidance, on conveying the design of the personal firewall application, on the principle of least privilege and on implications of default settings would greatly enhance both usability and security of personal firewalls.

The full paper as accepted for publication in IFIP Sec 2007.

An early project report with a description of the use cases and Nielsen's guidelines, contains 10 firewalls.

Links to the detailed evaluation of the different firewalls including screenshots.
BlackICE PC Protection 3.6
Comodo Personal Firewall
F-Secure Internet Security 2006 6.12-90
LavaSoft Personal Firewall 1.0
McAfee Personal Firewall Plus 7.0
Microsoft Windows XP Firewall
NetVeda Safety Net 3.61.0002
Norman Personal Firewall 1.42
Norton Personal Firewall 2006
Sunbelt Kerio Personal Firewall 4.3.268.0
Tiny Desktop Firewall 2005 (build 6.5.126)
ZoneAlarm 6.1.744.001
(Not updated) Agnitum Outpost Firewall Pro 3.5
(Not tried again) VisNetic Firewall 2.3

Screenshots of outgoing connection alerts for all firewalls

BlackICE PC Protection 3.6 (Trial)

General: The Blackice firewall is different from the other firewalls. After installation it looks for all applications on the local computer and adds them to its rule base. By default, all applications on the local computer are allowed to run and to communicate. Main interface.

Firewall Definition: "A hardware or software barrier that restricts access in and out of a network. Firewalls are most often used to separate an internal LAN or WAN from the Internet. A gateway can serve as a firewall between two or more networks."

UDP Definition: "A connectionless protocol that, like TCP, runs on top of IP networks. Unlike TCP, UDP provides very few error recovery services, offering instead a direct way to send and receive datagrams (packets) over an IP network. UDP is used primarily for broadcasting messages over a network or for time-sensitive network transmissions like streaming media."

Installation: After unpacking Blackice starts by scanning the system for creating an application baseline. The applications says it will take 20 minutes; on our system it terminated after 10 but that is still inconvenient.

Outgoing: There is no alert when WinSCP connects to the network. All applications are by default allowed to do anything. One can disallow all connections for WinSCP but cannot specify which host or on which port or only outgoing. If one disallows network connections for WinSCP, an application error---not a firewall alert---is created saying that there was a network error.

Granularity: Can block and even terminate outgoing applications. Allows inbound traffic only if rule exists.

Incoming: Incoming connections are by default not allowed. If one wants to change settings, this menu from the main interface must be used. From "Edit BlackICE Settings..." one reaches a tabbed interface where the default for inbound connections is given (Values unclear and not described in the help system).
Allowing incoming connections is done from the "Advanced Firewall Settings" menu item in the main interface. There one can see and modify existing rules and create new rules.

Log: In another tab of the main interface one can see network events that were blocked.

Help: Rudimentary but surprisingly good glossary.

Comodo Personal Firewall 2.0.0.0 Free

General: From the system tray icon the user can access several applications. One of them is the main control interface for the firewall, while another is called "Launch Pad", and claims to be a unified control panel for all installed Comodo products. When just using the firewall, the presence of this is not very useful, and it makes the system tray menu more complicated, since each application has its own sub-menu.
The general firewall interface has three main tabs: Summary, Security and Activity, and most settings are done through the middle one. The tabs have a few sub-categories each, but the total number of views is not very large. This makes finding your way around the interface quite easy. There is an advanced category under Security, where most complicated settings are hidden, making it easy for novice users to avoid.
There are, however, some inconsistencies relating to the naming of views. For example, all lists of rules are referred to as "Monitor" in the menu, but as "Control Rules" in the respective view headings.

Firewall definition: "Broadly speaking, a computer firewall is a software program that prevents unauthorized access to or from a private network. Firewalls are tools that can be used to enhance the security of computers connected to a network, such as LAN or the Internet. They are an integral part of a comprehensive security framework for your network. "
UDP definition: Not found

Installation: The first install screen worth mentioning is the Automatic/Manual configuration choice. After rebooting, there a rather large number of alerts queues up, which is likely to be annoying to the user who has just installed the firewall. A novice user will surely not feel comfortable. Most of the alerts seem to come from the same applications: Comodo's own support applications, such as an update manager.
There is also a dialog box prompting the user to activate the product with a code that has been received in an e-mail to be able to keep the firewall for more than 30 days. This is a complicated procedure for a software that claims to be "free".

Stealth: The firewall blocks all incoming ports by default, and does not even alert when connection attempts are denied. Connections are dropped rather than denied, which leads to stealth.

Outgoing connections:

There are two alerts that pop up when one tries to access the Internet with WinSCP. The first asks for the outgoing connection; the second asks whether the application should be allowed to "act as a server". Comodo is the only firewall that needs permissions for the back data connections on high ports.
The choices for a user are to deny or allow once or to remember the answer. "Once" means that the answer is kept for the life time of the application but the rule is not visible in the application rule view.
In the alert, there is also a "More Details" button that tells the user about the path and creator of the application.
The alert also contains an advice line, but we never got any advice.
At the top of the box, there is a security alert level indicator, showing the criticality of the alert.
Once the "server" permission has been given, the more usual question concerning ordinary Internet access appear. The two rules that are created when we choose to remember the decisions appear as rules in the Application Monitor view. In this view, rules can also be edited and added manually.

Shortly after installation, Comodo asks permission for itself to connect to the Internet. It tells the user that it is a "safe" application.
Comodo does not create a rule for Internet Explorer. It gets automatic access without any explicit application rule. The same goes for the Windows system processes lsass.exe, svchost.exe or userinit.exe.

The default for Comodo is to give applications like a web browser full incoming and outgoing permissions. It is not possible to create a rule including port and host from the alert. One must do this from the Application Monitor interface.

Granularity:

One can achieve almost full granularity for applications but the default is to allow everything incoming and outgoing to all applications even though the alert shows port and host.
It is not possible to allow a client host to only talk to Cerberus.

Incoming requests: When first starting the FTP server application, the firewall pops up an alert, asking if the server should be allowed to accept incoming connections (and because Cerberus contacts a host to ask for its IP address also outgoing). Creating this rule, is not enough to respond to connections. The firewall also contains a portion called the "Network Monitor", where port and host filters are configured. Here one has to allow the FTP client to connect to the local host. Setting up the rule is not straightforward as Comodo uses the terms Source and Remote, instead of Source and Destination.

Fooling: The attempt to fool the firewall did not succeed. There was an alert asking if the fake program should be able to act as a server. The hint field said that the signature of the software had changed, and that unless the user had recently reinstalled or upgraded Firefox, it could be a trojan. We find this hint rather helpful and believe that a novice user, unless they are confused by the "act as a server" phrase, should be able to understand and make the decision to deny access.
When changing back to the real Firefox, its original rules had been overwritten with "block", which meant that there was a new alert, claiming that its singature had changed again. Understaning the alert is relatively easy, but it is an extra nuisance and complication because the rules are overwritten. Since the firewall can tell the difference between the two Firefoxes, it ought to be able to keep the old rules.

Popup, error messages: The firewall has only one type of pop-up alert, as described above. General blocked connections and other events that the user can not affect, are not shown as alert, but put straight into the log. In the "Advanced" view, grouped together with many other settings, there is a tick-box where pop-ups can be turned on or off. If pop-ups are not shown, the default is to deny connections.

Activation/deactivation: The firewall can easily be turned off or set to block everything in two different ways. There is a security level slider in the Summary view, that can be set to "Block All", "Custom" or "Allow all". The same options also appear in the menu accessed from the system tray icon.

Help and documentation: There is a help button at the top of the interface at all times. Clicking there brings up the help interface, sometimes at a page related to the active view, but sometimes at a basic help page. It is a bit disappointing that only some views get context sensitive help, because there seems to be good help pages for several views where the help button leads to the basic help page. However, it is not very difficult to find the appropriate help page anyway, since its tree structure menu is organised excactly like the views of the interface, using the same names. This conformes well to Nielsen's tenth rule.

Log: There is only one log view in the interface, which can be filtered only by time. The main list shows severity, category, description and time. To get more details, there is a details field at the bottom showing more information on a logged item, but that window can only show 4 rows at a time, and can not be resized.
Only denied connections can be logged, and the only way that logging can be turned on or off is by right clicking in the log and checking or un-checking one of four categories to be logged.

Resources: Uses quite a lot of memory.

Trusted zone: The firewall can handle trusted zones, which can be added in one of the main views of the interface. These can be hosts, IP ranges or subnets, and show up as network monitor rules. Accordning to the help, trusted zones are given full access, except for some small restrictions relating to DoS prevention.

Advanced filtering: The firewall has a number of functions called "Application Behaviour Analysis" that monitor process injections, window message, parent applikation leaks, DLL injections, and DNS queries. Details on these monitors is not readily available.

F-secure Internet Security 2006 6.12-90 30 days trial version

General: F-secure Internet Security is a security suite not only containing a firewall, but also Spam Filter, Virus Protection and Parental Control. The personal firewall module is actually called Internet Shield, and the word firewall refers only to the packet filtering rules.
The Internet Shield has four different sets of rules: firewall, program control, intrusion prevention and dial-up control. During our evaluation, we did not encounter the functionalities of the intrusion prevention at all. Under each of these headings, and a log heading, there are a few tabs with settings and lists of rules.
In general, the interface windows are small and not possible to resize. This is a problem, because rules, logs and information quite often do not fit, which leads to extensive scrolling. It also makes it difficult to get a feeling for a rule, or a set of rules, because one can not see them in their entirety.
The firewall has several modes: Block All, Office, High, Normal, Custom, Allow All. What these setting mean is not well explained, and the help does not provide more.
For the evaluation, we first installed a Swedish language version of the suite, because we believed it might be easier for some users to manage their computer security in their own language. This was not too successful, since a lot of the content still was in English, e.g. the names and explanations of rules. Instead, the duality of languages caused more confusion since concepts had different names in Swedish and English.
Apart from a main interface that is the first the user sees at startup there is a parallel interface where all configuration and logging is done. The main interface only shows an overview of which features are turned on or off. The buttons at the left of the main interface match the headings in the tree structure of the configuration interface. This causes a lot of redundancy and the two interfaces might just as well have been one.

Not quite stable. At one installation, the setting up of the FTP client did and did not work despite the existence of the needed rules. At another installation, removing an application rule made that no further application rule would be automatically created for that app (unless manually).

Firewall definition: UDP definition: Even though there is a glossary with some security and firewall terms, the terms firewall and UDP do not appear. Neither is there an introductory page in the help that says what a firewall is. The definitions in the glossary are non-technical. Here is the definition of DNS: "The domain name system (DNS) is the way that Internet domain names are located and translated into Internet Protocol addresses. A domain name is a meaningful and easy-to-remember name for an Internet address. The Internet address www.example.com is an example of a DNS name.".

Installation: During install, a number of questions are asked relating to the default set up. The users are for example asked to verify their standard web browser, e-mail system etc. Many of the other questions pertain more to the extended functionalities of the suite, than to the firewall module.
After rebooting, the suite starts its main interface, letting the user have a look at the configuration.

Stealth: The firewall has several modes: Block All, Office, High, Normal, Custom, Allow All. Depending on these the firewall reacts differently to incoming connection attempts. In normal mode, it remains silent and does not even log the incoming attempts. In custom mode, however, most of the attempts generate a pop-up alert, as well as a log item. Regardless of mode, stealth is achieved, and packets are dropped unless otherwise stated. There are no specific alerts from the intrusion prevention system when we perform a port scan.

Outgoing connections: When running WinSCP the user is prompted with a dialog box (eng., swe.), asking whether to allow or deny the connection and whether to remember the answer. The Swedish help text is unhelpful: "What shall I do? --- You should only allow connections for programs that are needed to connect to the Internet." Might this be a translator's mistake? Yes. The English version says: "We recommend you allow connections only for application that you need to access the Internet." Not very helpful but at least correct.
To make the decision into a rule the user has to choose not to show a dialog box for the program again.
The dialog box is well-structured and easy to understand, it contains parts both for novice and advanced users.

Granularity: By default, there are prompts for outgoing connections and when the application tries to listen on a port. Answering "Allow" creates a rule that allows all outgoing and all listening, depending on the connection attempt. This can be made more specific but it cannot be specified by the user but must happen through the collection of alerts. The window is a mystery. The behaviour is not as expected: When choosing "Prompt" and anything else than "Application only", a prompt appeared every time, even if the same connection had already been allowed previously and should be remembered.

Incoming requests: Getting the FTP server up and running is straightforward. When starting the server, the firewall alerts and asks whether the server can be trusted to receive incoming connections. By default, any host can connect to the server. Why this is so is unclear. There is no such visible rule.
We experimented with setting up firewall rules to allow only a specific host to connect to Cerberus. We needed to create a rule that allows our host to connect and that denies all other hosts.

Fooling: Fooling the firewall with a replaced firefox.exe does not work. The firewall gives an alert saying that the program has been modified, and provides substantial supporting text. We believe that the tips provided should be enough to enable a novice user to make the right decision. When running the original Firefox again, the firewall does not alert. However, if we did make a rule for the fake firefox, the original one could not get online without an alert.

Popup, error messages: The number of pop-ups depend on the mode that is used. In normal mode, there are only pop-ups for applications that do not already have a rule, and these are rather understandable. In custom mode, one can choose to see all alerts. These alert windows have a number of rather unclear buttons at the bottom. The Clear button (Swedish: "Rensa") for example, actually removes the alert from the log completely, and the Rule... button ("Regel...") does not show what rule caused the alert as one might think, but brings up a dialog box to create a new one. There is also a "Show All"-button (Swedish: "Visa Alla"), which should really be called "Show log" (eng, swe).
The connection alerts can be turned on and off in the log view, or in one of the alert dialog boxes.

Activation/deactivation: From the system tray icon, there are shortcuts for allowing all or no traffic through. In many of the views of the interface, there are "activate" tickboxes, making it possible to deactivate parts of the firewall separately. Also, two of the firewall modes are the "allow all" and "block all" options, which means that there is a large number of ways to perform these actions.
In the system tray menu, there is also an option called deactivate, where the user can either "deactivate and maintain the security level" or "deactivate and allow all traffic". The help specifies that these options are supposed to be used temporarily during installs or memory sensitive operations, but we still find the concept of deactivating the firewall while still being protected rather paradoxical.

Help and documentation: There are help buttons in most views of the program, and these bring up the help system at a relevant page. This works fine in accordance with Nielsen's tenth rule. However, at some pages the terms in the help do not match the ones used in the interface. And sometimes, the help is not so helpful, e.g. when trying to explain the different security levels---Block all, Office, High, Normal, Custom, Allow all---to the user.

Log: There are different logs for different events: a packet log, an action log and an alert log. The action log is just a text file (with all allowed connections), while the others are opened in a log viewer. The packet log logs all packets for a given time or until a certain log file size is reached. Despite these logs, we are missing a log for all events that the firewall interfered with. The action log and the packet log quickly grow large. The alert log contains all rejected inbound connection attempts, but only if the firewall is in e.g. Custom mode. In Normal mode the alert log is empty.

Resources: A large number of small processes make up the F-secure Internet Security Suite. We can not distinguish those that relate to the firewall or Internet Shield modules, but in total they add up to about 71 MB, which is substantial. CPU usage is also up to 40% at times.

Trusted zone: Under Internet Shield/Firewall/Settings there is an option called Reliable network card. There is not supporting information nearby which explains what this means. When selecting a network interface here, all connections from that interface will be allowed, and the firewall help states that it is potentially very dangerous to configure this in a wrong way. However, if a user does not read the help, they may very easily misinterpret the function, and believe that they should select their network card from the list, because they trust it. The bad wording, that indicates that it is the card that should be trusted rather than the traffic coming through it, is a serious mistake. This is obviously not in coherence with Nielsen's fourth rule, and can potentially nullify the entire firewall. The dialog box that pops up to alert the user of the dangers of claiming that an interface is reliable is not clear enough: Its text warns that changing the setting will "affect" the security of the firewall, but does not say how.

Advanced filtering: In the firewall settings tab, there are some options relating to alerting about erroneous packet fragments or too short fragments. These are the only functionalities that inspect packets at a more advanced level than port and destination that we could find in F-secure Internet Security.

LavaSoft Personal firewall 1.0.543.5722. Eval 10 days

General: The LavaSoft firewall's main interface (with main menu) resembles that of a network monitor. It shows current open ports and active applications. As these change rapidly the interface is a bit hectic.
There are some default rules that allow e.g. DNS resolving for all applications.

Installation: Compared to Outpost, from which it is a spinoff, the installation process is easier. Almost all of the difficult questions are gone, the user only has to choose automatic configuration or run a wizard. The wizard is fairly straightforward and doesn't contain many questions at all.

Stealth: (Same as Outpost) The ports of the computer are blocked and stealthy by default. By default, there are no alerts informing the user when packets are blocked. Advanced users can specify packet filtering rules in detail. However, this has to be done in a modal dialog box, which does not allow the user to do anything else while it is active. If one for example wants to add a rule for an IP number found in the log, one can not look it up without aborting the rule creation.
The firewall contains an attack detection plugin, that discovered our port scans, and denied the remote host all access for a few minutes.

Outgoing connections: When trying to access the Internet with WinSCP, an alert is generated. The user can choose to "Allow all activities for this application", to "Stop all activities for this application", to "Create rules using preset Telnet Client/Custom", to "Allow Once" or "Block Once". Choosing the default of creating a rule using preset Telnet Client creates, creates two rules: One for outgoing on the Telnet port and one for outgoing on the SSH port. The DNS request is granted by default through the global rules.

Granularity: Full application granularity.

Incoming requests: When starting the FTP server, one gets an outgoing alert when Cerberus contacts a host to determine its IP address. As the socket connection is on the http port, LavaSoft suggests to create custom rules for a web browser (which is actually too much for this application). There is no alert when Cerberus listens on FTP port.
When the FTP client contact the host, there is an alert, but granting the preset rules LavaSoft has chosen for the Cerberus FTP server is not enough. The connection is not visible for the server or the client after allowing it in the firewall. This depends on the created default rule for incoming FTP, where stateful inspection was chosen. Unselecting that option makes the rule work fine.
When the FTP server opens a data connection to the FTP client, there is an alert from which one should generate a custom rule which is suggested like this, but should be modified not to contain the ephemeral port to also work for future connections.
Allowing only incoming FTP connections from our trusted host, is possible. The ftp rule must be changed as shown. If other clients connect, they cause an alert. From that alert one can easily create a rule that blocks all FTP clients and this rule is automatically placed at the end of the rule set, thus allowing our trusted host but blocking all others without causing alerts.

Fooling: Same as Outpost but without the helpful hint.

Popup, error messages: The LavaSoft firewall behaves in virtually the same way as Outpost, but the "Automatic Rules Creation" did not show up after installation. That might however depend on other factors rather than the differences between LavaSoft and Outpost.

Activation/deactivation: Same as Outpost.

Help and documentation: There is an icon with a book and a magnifying glass. If the user first clicks it and then selects something else, this leads to context sensitive help. The choice of this icon instead of the one in Outpost, with an arrow and a question mark, is rather odd since that one has a clearer meaning.
There is the same problem with accesssing the help from "options" as in Outpost firewall.

Log: There is a tidy log viewer.

Resources: Same as Outpost.

Trusted zone: Same as Outpost.

Advanced filtering: Same as Outpost.

McAfee Personal Firewall Plus 7.0

General: Maybe the best of the commercial firewalls. Simple. Somewhat difficult to find the rules. Only allows full access or nothing. Contains a lot of fun things like tracing events on a map and analysing the current state.

Firewall Definition: "Hardware and/or software designed to keep unauthorized outsiders from tampering with a computer system or network. That system can be a standalone computer, a small LAN or a company-wide network or WAN with thousands of users. Person Firewall is a software firewall effective in protecting standalone computers and small networks."

UDP: "User Datagram Protocol. UDP converts data messages generated by an application into packets to be sent via IP."

Installation: The installation process is short but cannot not be cancelled. No restart is required.
The first presentation of the firewall is a prompt for a Windows system component of which the firewall says that it does not know this standard Windows component!
The main interface is for all McAfee security products. The firewall interface looks like this. This is a screen shot of "View the Firewall Summary" which contains recent events.

Stealth: By default.

Outgoing: McAfee prompts for outgoing connections. It allows the choices: Grant Access, Grant Access Once, Block All Access, and Help Me Choose. Why is there no Block Once? Grant Access and Block All Access create rules; Help Me Choose opens the help page and is not all that helpful (a long text starting with the explanation of the above options). The window contains the path to the application and a small info pane where McAfee states whether it knows the application or not. The ones that we tried---WinSCP, Cerberus FTP and SSH---were unknown. Firefox and Internet Explorer are allowed to connect without a prompt and appropriate rules created (if there is an Internet connection to check the app when the app starts).
Grant Access creates a rule that allows full access. This can be manually changed to "outbound access only" but not more fine-grained. One wonders why not Firefox and Internet Explorer get "outbound access only" by default.
In the list of Internet applications, one can "get more information" about each application. Choosing this option Internet Explorer opens and displays a web page from hackerwatch.org about this application: Firefox, LSASS system process, Cerberus, Cerberus client when choosing "get more information" from the inbound events part.

Granularity:

For outgoing connections one can either "allow full access", which means that the application can both listen and receive, or one can allow "outbound access only", which is actually enough for web browsers but is not chosen as default by McAfee.
Incoming connections can be handled by either trusting (or denying) IP addresses or to open a port. One cannot specify that only a specific host is allowed to interact with Cerberus FTP only.

Incoming: The firewall creates an alert when the FTP server starts where one chooses Grant Access and gives the server full rights to send and receive. Connection attempts from FTP clients fail until one opens the FTP service port for all applications or until one declares a client host to be trusted (not only for FTP but for everything). One receives guidance for how to allow a client connection directly in the summary view or finds it in the inbound events part, which gives good advice on how to allow client connections.
Inbound events can be traced; this is fun and one might actually spend an evening enjoying the trace of incoming stray packets.

Fooling: The changed Firefox does not fool McAfee. It prompts that the software has been modified.

Popup, error messages: The default is to only alert on outgoing connection attempts. One can change this in a setting so that all alerts create a popup. Then, an incoming connection attempt looks like this.

Activation/deactivation: From the system tray.

Help and documentation: The interface is quite wordy as is and often supplies good help without the need to open the help. If one does so, one is pointed to a fitting page in the help system, which looks like the standard Windows help system. The help system is not overly creative and does not contain screen shots. But it is okay.

Log: Allowed connections are never logged. Only denied connection attempts appear in the log.

Resources: ?

Trusted zone: One can define trusted hosts in a list, and these hosts are fully trusted. There is no possibility to define "shades of grey". The subnet is suggested as one trusted zone but is not automatically enabled (good!).

Advanced filtering: None, but includes an intrusion detection tools that looks for certain intrustion patterns.

Microsoft Windows Firewall Built in

General: When evaluating the Windows Firewall, we used a Swedish language version. Afterwards we also made some comparisons with an English version, and found that all our conclusions were valid for that version as well.
In the Windows Firewall, the user does not make rules for allowing or blocking connections. Instead there is a list of "exceptions" from the general principle of blocking all incoming traffic. Windows Firewall is managed by a rather small interface, consisting of only three tabs. The first tab is for turning the firewall on and off, and also contains a button for a mode to ignore exceptions. The second tab relates to the exception list, and the third contains miscellaneous settings.

Installation: There is no specific installation process for the Windows XP firewall. Instead, it appears either when installing Windows, or through Windows Update. It is usually activated through the Windows Control Panel, where there is a Windows Firewall item, or through the Windows Security Center, which complains when no firewall is installed.

Stealth: Stealth was achieved by using the Windows Firewall. All connection attempts to closed ports timed out.

Outgoing connections: The Windows Firewall only blocks incoming connection attempts, and does not prevent applications from accessing the network. Because of this, this test is not relevant. WinSCP can use the network freely, and the Windows Firewall does not interfere.
Outbound protection is perhaps not a core functionality of a personal firewall, but we believe that the absence of such a feature should be pointed out more clearly. Users who have worked with any of the other personal firewalls in this study will probably expect outbound protection, and might not realise the more limited scope of Windows Firewall.

Incoming requests: Setting up the Cerberus FTP server is (too) easy because Cerberus changes the firewall as part of its behaviour. This can only be disallowed by bringing the firewall into "Don't allow exceptions" mode. Otherwise, a rule for Cerberus appears automatically, without any alert, warning or question for the user to answer. The automatic rule is coarse, opening the FTP ports for any computer on the Internet. The rules in the "Exceptions" tab have a tick-box for activation or deactivation. When un-ticking the box for Cerberus, and restarting the server, the tick re-appears anyway (reappear). This is because Cerberus changes the exception list itself. Normally, the firewall alerts when the program first starts, and asks whether one wants to make an exception for it (eng., swed.). That an application can change the firewall policy defies the existence of the firewall, but this is probably possible for all firewalls whenever the user is logged in as administrator.
Exceptions can be created both for applications and for ports, but not in combination. Both types of exceptions can be limited to certain hosts or host ranges. It was straightforward to only allow one specific host for FTP traffic.

Fooling: The test is not relevant, since Windows Firewall never blocks outgoing traffic.

Popup, error messages: The only kind of pop-ups that can appear are the ones asking if an application should be allowed to receive connections. These can be turned off with a button in the interface.

Activation/deactivation: In the main tab of the interface, there are radio buttons to turn the firewall on and off. However, the selection is not applied until the configuration interface is closed. There is also no "Apply" button. It is quite easy to move through the interface without realising that no changes have happened. In some views, there are explanatory texts describing the status of the firewall in words. But when these do not match the settings, the users are likely to get confused, before realising that they need to restart the interface to apply their changes.

Help and documentation: The help for the firewall is handled in the general Windows help system. This makes it hard to navigate through since the relevant pages are mixed with all other Windows help, and there is no specific menu for the firewall. The help can be accessed through links in the interface with slightly odd names, that usually do not contain the word "help". There also seem to be some translation issues. One page, for example, refers to the "Browse" button, but the button is actually called "Save as", which does not make sense in the context.

Log: Logged events are saved in a text file and can not be viewed through the firewall interface. The user can choose to log dropped packets and/or successful connections. The size of the log can obviously grow quite fast, and since there are no filters or sorting functionalities, an external tool would probably be needed for better understanding.

Resources: Difficult to tell, since the firewall is integrated with the operating system.

Trusted zone: There is no trusted zone concept in the firewall. There is a similar concept in the Windows Internet settings, but that applies only to certain services.

Advanced filtering: /

NetVeda Safety Net 3.61.0002 Free

General: The firewall has one general interface with a number of views. These are either accessed through a tree stucture on the left, or through buttons along the top of the interface. The views in the tree commonly have several tabs with different settings and functionalities. Since the software also contains parental control and other similar features, these also show up in the tree structure, but are separate from the firewall.

Firewall/UDP Definitions: None

Installation: There are no policy-related questions or decisions before the mandatory reboot. After the reboot the user is greeted with the main interface and an alert about the lsass and then the svchost process.

Stealth: The default set-up of the firewall puts the computer in stealth mode for our port scan attempts. There are options called "hide computer on Internet", but even with these unselected, the connection attempts time out.

Outgoing connections: When attempting to access the Internet with WinSCP, an alert appears, letting the user choose between allowing or blocking the connection, always, once or for this run. The alert has no help function, so if an alert asks about an application that the user does not recognise, it may be difficult to know the appropriate answer. Rules created in this way appear in the "Trusted Applications" view. It is not clearly shown that the list only relates to outbound connections, and even in the help, this is not explicitly stated.

Granularity: All or nothing. At first sight there is only an overview of which are the trusted applications.

From the main interface, item Security leads to a tab that is called Advanced Internet Firewall. There it looks as if one could create further rules, but currently one can only allow or deny traffic during certain hours to or from a specific port. It is not possible to edit the port. One must pick from a very limited list. The combo boxes Application, Local Network and Remote Network allow no other choices than <All> and no editing either.

One can create rules that deny all incoming traffic and then open up the FTP port again. But then one must also open many other ports because the fully trusted applications do not work any longer as port 80 now is closed. As one cannot type in the port and the SSH port 22 is not in the list, WinSCP can no longer be made to work with this special setting.

Also when removing the DenyAll other traffic rule, pressing Apply, closing windows, applications and restarting the firewall would not allow traffic again. One must restart the computer.

Incoming requests: Without any modifications of the default from the installation, starting Cerberus causes an outgoing alert (because it first contacts a server that tells it its IP address). Choosing "Always", starts the server (no more alert about listening) and also allows connections from all hosts.

Fooling: When trying to fool the firewall by replacing firefox.exe, we did not succeed. The firewall seems to base its application rules not only on the path and name of the application, but also on properties such as Company, Version, Date Modified etc. This means that it recognises the fake Firefox as a different application. It is even possible to make different rules for the two firefoxes, despite them having the same path.

Popup, error messages: There are two different kinds of alerts from NetVeda. Firstly, there are those that prompt the user for a desicion about allowing or denying a connection and secondly, there are floating windows that inform the user that a connection attempt was blocked. There is an options-menu on these boxes, but that only relates to what alerts will be shown in the future, not to the connection attempt.

Activation/deactivation: In all views of the interface there is a set of buttons along the top of the window. Three of these relate to activating or deactivating the firewall: "Allow all", "Block all" and "Enforce". The latter button apparently means that the defined rules are applied, although a different choice of words might have been easier to understand. Another issue with these buttons is that when in "Block All"-mode, the "Allow All" button is inactive. To get to that mode, one needs to first change to "Enforce"-mode, to be able to click the "Allow All"-button. The problem is the same in reverse.

Help and documentation: There are no context sensitive help buttons in any view of the application. Instead, the F1 button, or selecting help from the menu, always brings up the default help page. That page, however, is not a help index page, but the page with help for the default view of the program. Apart from that, the help system has the standard Windows help appearance.

Log: The logs, called "activity reports" remain empty while one has not chosen which network adapter is the Internet adapter.
There are three separate reports: application, network and alert. These are slightly different, and have different amounts of filter and sorting options. They are generally easy to understand, and contain quite a lot of information (example 1, example 2).

Resources: Normal

Trusted zone: The firewall does not seem to have the concept of a Trusted Zone, and all settings apply to the Internet as a whole, or to a LAN, which needs to be connected on a different interface.

Advanced filtering: The firewall is primarily a packet filtering firewall, and does not seem to inspect packets on any higher level.

Norman Personal Firewall 1.42 30 days trial version

General: Norman firewall is a very technical firewall which works well when the user has notions of ports and hosts. It allows advanced rule creation in its standard dialog, which through this becomes tiringly complicated and hard to get rid of. Norman is also the only firewall that does not create automatic rules for Windows services.
The Norman personal firewall has one main interface, but the log and help have separate interfaces. In general it gives a feeling of not being well thought through due to many inconsistencies in the design.
In the main interface, when pressing the Security icon, there are five different icons showing different default principles, however their names are not very descriptive and one has to resort to the help system to find out the differences between them. This clearly violates Nielsen's fourth rule. In some of the menus there are ways to set rules for different types of active content. These appear in three different interface views, but in each view at least one of them is missing, and changes made in one of the views don't always affect the other ones. This makes it difficult for the user to see the status of the firewall, and is not coherent with Nielsen's first rule.

Firewall Definition: The help is not so good that there is a glossary or indexes. Therefore we quote the Getting Started section which tells a bit about the product "With an Internet connection and the Norman Personal Firewall, you can feel free to do about anything in the Web safely. You will find more information to help you use the Internet and network safer with this product in the Help Contents."

UDP Definition: Not contained.

Installation: During installation one can choose "auto setup" or to configure the firewall manually. The configuration is done through a "course", which in the auto configuration mode is finished at once and the user is not even introduced to what the course is about. The couse is, if manual configuration is chosen, divided into several parts: setting a password for parental control, controlling active content in the browser (is this reasonable functionality for a personal firewall?), setting an arbitrary security level with levels that are not self-explanatory and a somewhat dangerous default choice, identifying the default browser and e-mail client, the possibility to set early rules for other applications (we have not explored this).
Within these parts there is a "back" button, but when a part is done there is no going back, which makes a mistake in the last step in each part unrecoverable. This is annoying and violates the third of Nielsen's rules.
Help is not accessible during the manual configuration.

Stealth: The default setting is to ask the user about every incoming request. If the user chooses to create a rule for the request (and there is no way to just say No, this will also trigger rule creation), a four step dialog is waiting as shown in the next section. The first time, it is nice and helpful, but after a few pop-ups it becomes irritatingly slow.
This happens even if one chooses the auto configuration mode during installation. When port scanning the computer we got a pop-up for each port, and that is more than anyone can handle. The popups can be disabled with the rather cryptically named "stop assistant" button but then also outgoing requests are silently blocked.
The firewall has a separate port scan detection module. But it is located behind the portblocking, so if ports are blocked it will not recognise a port scan.

Outgoing connections: It is easy and fairly straightforward to create a tight, application-based rules for outgoing traffic. The process is a longish and needs 4 windows. It might be possible to combine this in one window.
During testing, when the firewall was running in the background, an alert suddenly appeared. The firewall itself was trying to access the Internet, and recommended that it should be blocked because it was likely to be a trojan. The firewall does not recognise itself.

Granularity: Allows full granularity for both incoming and outgoing connections. Incoming connections to a specific application are supported.

Incoming requests: The Norman personal firewall does not warn when the Cerberus server starts listening but alerts on the incoming request from the FTP client. It then alerts also on the outgoing request of the Cerberus server and suggests to create a rule for any port to our FTP client host. This was straight-forward and also limits by default which hosts can connect.
We tried to create a manual rule allowing our host only to connect to Cerberus@port21 (and not to port 21 in general). We did not succeed with this but found other problems. In the "Rules" view there is a list of existing rules. When selecting one of them and clicking the info-button, a visualisation of the rule appears. This visualisation, however, has a few GUI design flaws. At first one might think that the colour of the arrow has something to do with whether a connection attempt will be blocked or not, but actually the colour as well as the direction of the arrow indicate if it is an incoming or outgoing request that the rule handles. In the list of existing rules, there are left and right arrows indicating if the rule handles incoming or outgoing connections. However there is no standardisation on whether left or right is in or out. In the visualisation of a rule, the arrows are in the opposite direction.
In the "Rules" view there is, when modifying a rule, a strange slide widget. The designers of the firewall seem aware that this is a unconventional design and provide the user with a short description of how to enter the different sub categories.
Being a Windows computer, our PC should be able to receive NetBIOS query packets on ports 137, 138, 139. These ports are denied by default. Opening them one by one, keeping 139 closed, lead to shaky network connections.

Fooling: Norman Personal Firewall detects that Firefox has changed and warns for this. The alert was clear enough in explaining why the alert was there, and the effects of the options. After choosing "No", running the real Firefox works fine.

Popup, error messages: Pop-ups appearing in the bottom right corner disappear after a short while or are replaced by newer alerts (after a while), usually before the user is done reading them, and there is no going back. It is extremely tiresome that one cannot step between current alerts. If one's computer is subjected to some connection attempts, and one is waiting for a certain connection attempt to create a rule from (as with setting up the FTP server), one has difficulties finding the right connection attempt in a timely manner.
At the pop-up there is a "details" button, which does not follow the conventions. It is a mouse-over item and not clickable, though it looks just like an ordinary button. The only way to get rid of the pop-up is with the "stop assistant"-button, which, as mentioned earlier, is somewhat hidden.
Alert windows are big and floating and one cannot turn them off easily. If one chooses to have no pop-ups then there are no popups even for outgoing connections.

Activation/deactivation: Norman Personal Firewall has no function to stop all network traffic. Turning the firewall on and off is done inside the program and not remembered after restarting the computer. If one turns off the popups by disabling the assistant, this is revoked after restart of the computer.

Help and documentation: Most of the views in the program have help buttons that take the user to relevant help pages. However, the help can use terms and names that are different from those used in the program itself. This lack of consistency makes it harder for the user to understand what the help is about.
Some of the help topics are meaningless, e.g. the one about the security custom level view, which is void of meaning.

Log: The log files are accessible from an icon on the main window. The log files are an excel sheet-like list, that can neither be sorted nor filtered, and there is no way to choose what should be logged. Furthermore the direction of the connections is shown by arrows here as well as in the rule list, and there is still no way to tell if left or right is in or out.
There are threee different log sheets, for Intrusions, Port scanning and Applications.
The numbers in the "time to live"-column are presented in hexadecimal, while numbers in other columns are decimal. This is inconsistent and confusing.

Resources: Normal

Trusted zone: IP numbers can be added to a trusted host list. Unfortunately there is no information on what rights you give these hosts.

Advanced filtering: We can not find any indications that Norman Personal Firewall would use any advanced methods for packet filtering.

Norton Personal Firewall 2006, 9,1.0.33, 15 days trialware

General: There is one single status interface that shows which Norton components are running. For changing the settings of the firewall, the user first has to select the personal firewall option and then click "configure". This opens up a different interface containing five tabs. For users only running the personal firewall the status interface is unnecessary, but obviously useful when running several Norton products.
It is a nuisance that the firewall constantly warns that the Norton Antivirus is not installed. All the red warnings are due to there not being any virus protection installed. This is misleading, exaggerates security threats and make users believe that their systems are insecure.
Worst default behaviour (learn = allow all) but full granularity of rules possible.
Windows services do not appear in the list of trusted application and can thus not be controlled by this firewall.

Installation: During the installation of Norton Personal Firewall, the user never needs to make any policy decisions. Immediately after the reboot, the program starts downloading upgrades and updates, and then requires another reboot before it is ready to protect. To do this in one step would be preferable.

Stealth: Stealth on all ports is achieved by default. All connection attemps time out.

Outgoing connections:

By default, the firewall is set up in learning mode, which has the effect that all applications are allowed to connect. When WinSCP attempted to access the network, the firewall only says that it has learnt this new application.
Some applications like Firefox and Internet Explorer are known to Norton and even if learning mode is switched off, these applications only create a small notice float information alert when they connect to the network. The created default rules for Firefox are sensible: Http, ftp client and DNS traffic are allowed to all hosts. The default rules for IE comprise more, among other things a rule that allow IE to act as an FTP server.
If switching off the learning and letting WinSCP connect, one gets an alert when WinSCP contacts a DNS server.

Choosing the recommende option "Always allow connections from this program on all ports" creates a rule that allows WinSCP full incoming and outgoing permissions! And naturally, no more alerts occur.
Choosing the "Manually create a firewall rule" option, opens a wizard that steps through five tabs of configuration, which creates quite a tight rule allowing outgoing connections to any host (!) (the remote address is not accessible or set as default!) but only on port 53 (default). After this step, one gets a Port 22 alert, which strangely offers more action alternatives than the DNS alert.

Again choosing the recommended "Allow always" action results in a rule that deletes the previous port 53 rule and replaces it with a rule that again gives full incoming and outgoing permissions.
Instead choosing to again manually create a rule results in the desired two rules that allow connections to any host on ports 53 and 22. In the first interface where the existence of the two rules is shown, no difference between the two can be seen!

Granularity: Norton allows full granularity on the application level but it is extremely difficult to review the specific rules for an application. Learning mode learns very specific details but to verify/review these details is difficult. In order to see what is generated for the FTP server, one has to go to the application tab, mark Cerberus and press Modify..., choose Manually Configure..., see all rules for this application. The text in rules does not say it all; some lines contain specific ports but the text says only "Communications: Specific" (Example). However, the learning mode also creates a fall through rule at the end that gives full incoming and outgoing permissions. Thus to allow only one host to access the FTP server, one has to remove the general fall through rule for Cerberus and modify the inbound rule so that only our host can connect.

Incoming requests: By default, everything is allowed. The firewall creates an information alert as above, stating that Cerberus has started. There is no alert when the client connects.
If one knows how to access the rules (described above in Granularity), one can use the learnt rules to create a more fitting rule (also described above).
When switching off learning mode, an alert is created when Cerberus contacts the DNS server (sv.). When choosing OK in this alert with the text saying "Always allow connections from this program on all ports (recommended)", one not only trusts Cerberus with all outgoing connections but also for incoming connections. Thus, full in/out permissions are given.
If choosing to configure the rule manually, one is walked through an 8-step wizard (with the same windows as shown for fall through rule) for setting up the rule. The next alert comes from Cerberus' http connection (sv.). The suggested answer is "Always allow (recommended)", and choosing this means again that Cerberus is completely trusted for both incoming and outgoing connections.
If choosing to configure also this rule manually and after completing another 8-step wizard, one is presented with an alert for listening (sv). If choosing the manual set-up one can immediately limit so that only the trusted host can connect. However, the IP address of the trusted host, though present in the details of the alert, is not accessible inside the wizard for creating a rule that only allows this host.

Fooling: When trying to run a fake Firefox, there are alerts. However, the text in them does not hint at the problem. Instead, they are general DNS access alerts. A user would probably be quite confused by this, because they would think that they should already have appropriate rules for Firefox configured. It seems that the the firewall is not fooled, but its inability to give the user the right message makes it quite possible that the user makes the wrong decision.
When running the original Firefox again, everything works fine.

Popup, error messages: In general, the Norton Personal Firewall gives very few pop-ups. When first running the program, there are some information pop-ups related to automatic updates, but they do not return during the rest of the evaluation. Denied connection attempts are never shown as alerts, and all pop-ups really require a user decision.

Activation/deactivation: The main interface contains options to turn off the different components of the firewall system. This can even be done for a limited time. In the same view, there is also a "block traffic"-button. When using this, a help message appears claiming that the "allow traffic"-option will allow all Internet traffic, when actually it only allows traffic according to the rules.

Help and documentation: With most views there is a "more info" option in the upper right corner. Selecting this opens the help interface with help for the active view. However, the left menu in the help interface is showing the index instead of the hierarchical tree structure which would give easier access to related information and misses the chance to give the user feedback about the structure of the application. Just showing the contents instead of the index the application would be more helpful.

Log: The log runs in a separate interface called "Log viewer" that has a tree structure at the left containing 10 different categories of the log. There are also descriptions of the content of these categories. The log window is resizeable, which makes it easy to see what you want, and when an item is selected there is a field at the bottom of the interface where the details are presented. This reduces the number of dialog boxes and new windows, which is obviously an advantage. There is quite a lot of information for each item, altough it differs quite significantly between the categories. There are no filters, which means that some of the categories build up rather immense lists quite fast. Both allowed and blocked connections/attempts are logged.

Resources: Slightly above normal, both in terms of CPU usage and memory

Trusted zone: The interface of the firewall has a tab called "Home Networking Zones" where Trusted and Restricted zones can be added, as hosts, ranges or subnets. There is supporting text explaining what access you grant to the trusted zone. We believe that the risk of making errors here is very small. It is possible to define trusted zones differently for different locations.

Advanced filtering: Norton does not seem to contain filtering functions at any higher level than hosts and ports.

Sunbelt Kerio Personal Firewall 4.3.268.0 Eval - 30 days

General: Everything related to the firewall is handled through a single interface that can be accessed from the system tray icon. It is a tab-based interface with main categories along the left side, and sub-categories at the top. It is not very clear which categories have higher priority, although it is not something that is difficult to learn. Advanced settings are clearly separated from the ordinary settings, so the inexperienced user can easily ignore the advanced settings.

Installation: The installation process consists of a sequence of dialogs, prompting the user for information. In one of these, the user is shown a long list of previous versions of the firewall, and what changes were made in each version. This breaks the flow of the process, and would take quite some time to read. We doubt that many users are interested in the version history at this step, and believe that this is an example of a breach of Nielsen's eighth rule, stating that there should be no irrelevant or rarely needed information in dialogs. Also, since the only available option is to continue with the next step of the install, an interested user can not leave the window open to read it at a later time.
Once the dialog sequence is done, the user is asked to reboot the computer. The firewall indicates that it is up and running with an icon in the system tray.
During the install process, the user gets a choice between simple and advanced mode, and is told that the mode can be changed at any time. There are, however, no concepts of simple or advanced mode once the firewall is installed, and the choice only seems to affect the initial set-up of rules. We chose to install using the default choice: Advanced.

Stealth: By default, the firewall is stealthy when it comes to incoming ports, i.e. any packets sent to a non-open port will be dropped. When port scanning, the NIPS-module of the firewall detects this and blocks the remote host.
The packet filtering can be configured in a view where the user can define and modify rules based on remote and local ports and hosts, by selecting options in drop down menus.

Outgoing connections:
Even though we installed with the advanced option, the firewall would not prompt for new applications, for attempts to start the FTP server, nor for the denied attempt to connect to the running FTP server. We had to change the default setting for how to behave for "Any other application...", to ask for incoming and outgoing connection attempts (see last line). The default was to deny all incoming and to allow all outgoing.
When asking is enabled, an alert box appears where the user can deny or permit the connection, once or as a rule. This creates a rule that allows (in the case of WinSCP) a rule where WinSCP has full permission for outgoing connections.

Incoming requests: One must configure the firewall to ask for incoming connections for a specific application. If this is done, one receives an alert when an FTP-client tries to connect. Permitting and remembering this rule, allows any incoming connection to the FTP server. To restrict the permission to one host, one must choose "Create an advanced filter rule" at the bottom of the alert and then "Advance Filter Rule..." which contains partly filled in information for the FTP connection. Also the current host IP address is pre-filled but not in the first window but only after trying to add the remote host manually.

Granularity: Full. One can choose three actions (Permit, Ask User, Deny) for the four different choices: Incoming connection from Internet, Outgoing connection to Internet, Incoming connection to trusted area, outgoing connection to trusted area. One can always create an additional advanced packet filter rule for an application.

Fooling: Kerio Personal Firewall was very easily fooled by replacing a trusted program. The fake firefox.exe was able to access the Internet through any port. The firewall seems to only base its decisions on the path of the application, not on any other characteristics.

Popup, error messages: No pop-ups except for the alert popups when the policy is set to "Ask the user".

In a previous version, 4.2.x there was a popup in the bottom right corner of the screen, which only contains information, and gives the user no option to interfere. These could not be recreated in the current version.

Activation/deactivation: The user can easily turn the firewall on and off by clicking on the system tray icon, which contains an option to block all traffic.

Help and documentation: In most views of the program there is a help button. Depending on where the user is different pages of the help system are displayed. The help is a separate application containing, among other things, an index and a seach function, all along the lines of Nielsen's tenth rule.

Log: For each rule there is an option to log traffic. The log events are shown in a different view of the program. There are different tabs for different categories of rules. However, the log can neither be sorted nor filtered, which makes it hard to use once it has grown beyond a few items. In a setting, one can choose to also log connection attempts to closed ports. This results in a quickly growing log view with no reasonable information at hand: no source host or port, no destination port. Right-clicking does not provide more.

Resources: Normal

Trusted zone: All rules can be configured differently for a trusted zone and the Internet. There is a "Trusted area"-tab in the program where the user can easily define the IP ranges and hosts that should belong to that zone.

Advanced filtering: The firewall contains a few modules labelled as Intrusion Prevention Systems. According to the describing texts, they are based on recognising known patterns and signatures of different types of attacks. They do not, however, seem to inspect packets in any greater detail than what the main firewall filter does.

Tiny Desktop Firewall 2005 (build 6.5.126) 30 day trial

General: Not only a network firewall but also a monitor for local applications, which allows monitoring for file access, process spawning and termination, shutdown, clipboard access, registry access, time-controlled access etc. Main window from system tray with opened tree to the left (normally, all are closed).
We have a hard time finding the application rules, e.g. Cerberus was allowed to create two registry keys (found here) and should run with Default Security, WinSCP should run with Default Security (being in the list of applications denotes this). One is looking for a view that shows permissions per application/principal, not per resource/object.

Installation: During installation one can choose between Custom or Full installation. We choose the full installation. A restart is required. After restart one sees a registration window with trial expiry details. After a short while, the registration window is superimposed by a main window. The window on top is a floating window that is always on top. So one closes it quickly to get back to work. The window below is an activity monitor (connections, running programs) that shows active applications.

Stealth: By default, there is no stealth of closed ports. The computer answers to ping and replies to port scanning with connection refused messages. For unknown reasons, Tiny has placed all our network interfaces into a safe zone, which by default are not put in stealth mode. When we first looked at this part of the interface, at least the infrared port, direct parallel and one Miniport were in the dangerous zone. But this changed subsequently without our intervention.

Outgoing: When starting WinSCP, even before it shows a window, one is prompted whether to run WinSCP at all. The answer options are not self-explanatory so the help page looks like this. Some words are marked in blue but they are not links. There is no default choice and the window is a float that one cannot get rid of. It is only partially possible to open other windows while the float is there. The Advanced... window from the dialog contains options for setting up a fine-grained policy. The pre-existing groups as shown in the Advanced Enrollment Settings window are not explained and there is no help button available in this window. The default choice to "Proceed to the enrollment dialog" just closes the window and returns to the simpler setup. The new group button allows one to enter a new group name and thus a new entry in the list but does not associate any permissions to it.
First choice We run WinSCP with the option of "This Time Only" + "Run with Default Security": The application starts and prompts with an alert for the outgoing connection. The option "apply permanently" is not enabled. This is the help page, which contains text that is not unproblematic: (1) Not all LAN-addresses need be from the IP range that is given in the instructions. (2) "Modify This Rule" is not enabled. (3) "Trust It" does not exist in the interface, but in the help text. When we choose "Apply for this connection attempt only", WinSCP proceeds without further notice and allows file transfer.
Second choice: We run WinSCP with the option of "Always" + "Run with Default Security": The application starts and prompts with an alert for the outgoing connection. Now, "Apply Permanently" is enabled and we choose this option. WinSCP proceeds without further notice and allows file transfer. It has created a tight rule (Window with default rules enabled, which is not default) that allows connections only to our SCP destination host on port 22 (remote).

Incoming: We start Cerberus with the setting "Always" + "Run with Default Security". After that initial alert, there are two alerts because Cerberus wants to set a registry key. The help text says that if an application is setting its own keys, that is fine, but the alert window does not show the complete registry key path so one cannot know whether Cerberus changes a system key or a Cerberus key (in the event log we can see that it is a Cerberus key). For each registry alert, there is a second user prompt with unclear meaning and no help can be gotten from that dialog window.
There are also alerts when Cerberus tries to acquire the local IP address through http connection to a web server. As Cerberus can try a number of diverse IP addresses, it is sensible to create a rule that allows Cerberus http traffic to any host. This can be done from the alert by pressing the "Modify this Rule"-link which leads to this advanced window.
There is no alert when Cerberus starts listening. The alert comes when the client tries to connect and creates a strict rule. The data back connection does not cause an alert or rule but is silently allowed.

Granularity: All network rules can be found in the main interface, part: Network Protection-->Rules. The rules have technical, specific names which unfortunately do not update automatically if one changes a rule in this view. The default is to create a tight rule (but fortunately not with ephemeral port numbers). A drawback is that e.g. a web browser will prompt for every new site unless one "modifies this rule" to allow connections to any host. IE has a default rule that allows it to connect to any host. Also the Windows processes do *not* cause alerts.

Fooling: Tiny is not fooled but the user may easily be. The alert looks very much like the genuine warning dialog for Firefox, even though Tiny claims to guard Firefox with checksum (but not path). Tiny does not make clear that it has a rule for firefox.exe from this destination (which maybe it hasn't but just checks checksums) but let's the user believe that this is a completely new application.

Popup, error messages: By default, Tiny warns for many new application that start (not MS Office, IE, and other trusted applications but warns for Adobe Acrobat Reader 6.0, Firefox) and wants to know if it really should start.

Activation/deactivation: From the system tray.

Help and documentation: The outgoing section shows some pointers to not so good help pages. The help reachable from alerts is plain text and therefore somewhat difficult to read. One wonders if one must read the whole text, which appears cumbersome.
The general help system, as accessible from the main GUI of Tiny, is a web resource located at tinysoftware.com.

Log: The event log is accessible from the Tiny Main interface.

Resources: Not checked.

Trusted zone: Zones are defined by the setting of a network interface being "safe" or "dangerous".

Advanced filtering: Application filter as described in the General part.

ZoneAlarm 6.5.722.000 Free

General: The free version of ZoneAlarm, that we tested, is a limited version of the commercially available ZoneAlarm Pro. The program promotes the Pro version in many situations, e.g. refers to options that are not available.
ZoneAlarm is based on a single interface with six categories in a menu at the left, each having a few tabs. This can be accessed from a system tray icon, which also shows little red and green bars for incoming and outgoing network traffic whenever there is any.

The system tray icon menu also contains lock, stop and shutdown options, and some promotion for ZoneAlarm Pro. In the main interface each tab contains explanations and hints. These are easy to hide but seem to be quite useful for explaining the basics for inexperienced users. The interface is easily resizeable, which is necessary to get some overview in some of the views.

Firewall Definition: (not in the glossary) "In buildings, a firewall is a barrier that prevents a fire from spreading. In computers, the concepts is similar. There are a variety of 'fires' out there on the Internet---hacker activity, viruses, worms, and so forth. A firewall is a system that stops these attempts to damage your computer."

UDP Definition: "UDP (User Datagram Protocol) A connection-less protocol that runs on top of IP networks and is used primarily for broadcasting messages over a network."

Installation: The installation process is friendly, consisting of a series of dialog boxes guiding the user through the process. There are no particularly difficult questions. The only one worth mentioning relates to whether the default browser should be allowed network access by default. Despite allowing this, the browser did not show up as trusted once the firewall was up and running.

Stealth: The firewall prevents the attempted port scan. All connection attempts time out, and there is an alert for each port, telling the user that a connection had been blocked. There seems to be no way to specify port and host based firewall rules in the free version of ZoneAlarm. The help system alludes to a button called "Custom" in the main firewall view, but since there is no such button, we conclude that port filtering options are only available in the Pro version. The lack of such basic firewall functionality reduces the security obtained through ZoneAlarm quite significantly.

Outgoing connections: Connecting to the Internet with WinSCP was fairly straightforward. An alert appears with details of the application, such as destination and that it was the first attempt. There are options to allow or deny the connection, and a tick box for remembering the decision (Free ZoneAlarm, ZonePro). Doing this added the application rule to the Program Control list. (in Zone Pro)
The list allows four different settings for each application, relating to inbound and outbound connections for a trusted zone, and for the Internet. Each can be set to "Allow", "Block" or "Ask". There are some dependecies between these settings, e.g. blocking access to the trusted zone implies blocking all other traffic. The ways that one setting affects the others is stated in the help system, but this can still be confusing as the users might not understand why one setting changes when they modify another. The program access rules can not be configured to relate to specific ports or hosts.

Granularity: In Free ZoneAlarm, it is full permissions for outgoing and/or incoming or nothing. In ZoneAlarm Pro, fine-grained permissions on the application are possible but one must always create a negative rule also. It is not enough to create an expert rule that allows something specific, one must also create a rule that denies everything.

Incoming requests: Setting up an FTP server was rather easy compared to other firewalls. There was an alert for the outgoing DNS connection and one for the subsequent listening connection, where the application could be allowed to act as a server. Once this was done, connections could be made and files could be transferred.
It is not possible to set up the server so that only our trusted host can connect. The granularity of ports and hosts is not supported.
ZonePro: Here it is possible to set up Cerberus so that only our trusted host can connect. This is done through the Options... button in the program control list. which can show Expert rules that are created through this interface.

Fooling: ZoneAlarm recognises that the fake Firefox is different, and gives an alert saying it has been modified. Choosing to deny the connection, and to remember that, did actually not affect the rules for the original firefox. However, no new rule for the fake Firefox appeared in the list. Despite that, later attempts to connect to the Internet with the fake Firefox were stopped. Since this rule is not shown anywhere, users can not change it, and if they mistakenly denied access to e.g. an updated version of firefox, it would be very difficult to fix. This does not go well with Nielsen's third rule.

Popup, error messages: By default the firewall only alerts on outgoing connection attempts of previously unknown application. One can switch on to be alerted even on incoming events that are blocked by default.

Activation/deactivation: There are two different ways to block Internet traffic, a stop button and a padlock. There are subtle differences. The Internet lock, represented by the padlock, is possible to bypass by giving applications special pass-lock permissions. This seems to be a functionality of ZoneAlarm Pro only, and we could not determine any other differences, because of the lack of a product specific help system.

Help and documentation: A major obstacle with ZoneAlarms help system is that it is not a specific help system for ZoneAlarm, but rather for the whole suite of ZoneAlarm products, such as ZoneAlarm Pro, ZoneAlarm Antivirus etc. This means that when looking at help for a function, it does not match what the program looks like at all, and often refers to functions and options that are not there. This makes using the help rather useless since one can never know if it is possible to perform the listed actions, and not even if the functionality is available. Only a few pages of the help system explicitly state that it relates to the other programs of the suite, and a user is quite likely to miss this, and be confused. This violates Nielsen's tenth rule.
There are help buttons in many of the views of the interface, and they lead to the relevant help pages. However, since the help relates to ZoneAlarm Pro, the examples shown do not match.

Log: There is a log viewer integrated in the main interface of the firewall application. It has a rather large number of columns, and is sortable by each of them. There are actually two different kinds of logs, referred to as "Firewall" and "Program", and the user can switch between these through a quite small drop down menu at the top. The "Firewall" log basically corresponds to denied connections that would cause alerts (if enabled) while the "Program" log lists all (allowed and denied) connections made by the applications in the program list. There is no way for the user to decide what is logged and not, or to filter the logs. If users are looking for a log item going back a day or two, they have to deal with a long list. Also, only 999 items can be viewed which means log information may disappear before the users want it to. Older items can be inspected in a more unstructured text file, but it would of course have been more convenient to be able to use the log viewer and a filter to find what one wants.

Resources: Normal

Trusted zone: The concept of a trusted zone is quite central in ZoneAlarm. All program control rules have the possibility for different settings for the Internet and the trusted zone. It is easy to add and remove hosts and IP addresses in the trusted zone.

Advanced filtering: Since the firewall does not even provide port and host based filtering, the issue is rather irrelevant.

Agnitum Outpost Firewall PRO 3.5 30 days trial version (Only part of the pre-study evaluation)

General: When running in the background, the firewall is only represented by an icon in the system tray. It has a menu, from which one can get to the different tools for the firewall, or change its mode. There is in fact no common interface for the firewall, but three or four separate tools: one showing current events, one for the log, one dialog box like mode where settings are changed, and one help application. Since these follow different design paradigms, the ways to perform tasks can differ considerably. In general, the large number of modes, menus and interfaces can make it quite difficult for a user to know what is going on.

Installation: The install process consists of a series of dialogs prompting the user for an install directory, accepting a license agreement etc. The user can choose between automatic or wizard configuration, where the first option configures the firewall in a default way, and the latter will ask the user for a number of decisions. This allows inexperienced users to avoid advanced settings.

In another dialog the user is asked two completely different questions grouped together. A question concerning settings for well known software is paired with a check-box for joining the manufacturer's quality feedback program, giving the firewall software permission to send information about network-allowed applications to the software provider (spyware warning!). A novice user may disregard this privacy violating second part of the dialog, much to the advantage of Agnitum Outpost.
There is also a dialog where the user is asked to select advanced or normal security. The text in the dialog recommends the user to select the normal security mode, but the advanced option, which is labelled to be "for advanced users", is the pre-selected option. This design makes it likely for users to make the error of choosing an undesired option. Also the text for the two options contains wording that is not likely to be understood by security laymen: What is a leak test? What is hidden process control, process memory control or component control? An informed decision is not likely.
After the install and reboot, a dialog describing "What's new" appears, and at the same time as a message in the corner tells the user that there has been "Automatic Rules Creation". The latter message is visible only for a short time, and users are likely to be frustrated if they do not have time to read it.

Stealth: The ports of the computer are blocked and stealthy by default. By default, there are no alerts informing the user when packets are blocked. Advanced users can specify packet filtering rules in detail. However, this has to be done in a modal dialog box, which does not allow the user to do anything else while it is active. If one for example wants to add a rule for an IP number found in the log, one can not look it up without aborting the rule creation. This is an unnecessary limit of the user's freedom. The firewall contains an attack detection plugin, that discovered our port scans, and denied the remote host all access for a few minutes.

Outgoing connections: When trying to get online with any unknown software, an alert is shown at the centre of the screen asking the user to allow or block the connection. There are also options to create permanent rules for the application. In this dialog there is also a "smart advisor"-button that shows hints to help the user decide on whether to allow or block a connection. However, this hint

Incoming requests: Configuring the firewall to let incoming FTP connections through is not an easy task. Since Outpost does not alert the user when a connection attempt is made to a port, there is never any alert box to derive a permanent rule from. Instead, the user has to somehow figure out what an FTP connection request looks like to be able to create a rule to let it in. Since the rejected connection attempts are logged, you might think that it is a good idea to create rules from there. In fact, there is a rule creation option when right-clicking on a logged item. But sadly these rules do not, for some unknown reason, become permanent. This is of course a rather serious flaw, since the users believe they have modified the settings of the firewall. Instead, to get the FTP connections through, a filtering rule has to be created with the standard rule creation function.

Fooling: When trying to run the fake Firefox, Outpost detects this and gives an alert. Apart from that the header did not relate to the name of the actual trusted program, the information was quite clear. The text in the smart advisor was particularily explanatory. Of the three possible decisions only one, "block", was easy to understand. The phrasing of the first option (to accept and trust the change) could definitely have been better, and we had serious trouble finding out what the second option ("Make changed component shared") would actually do.
When running the original Firefox after having blocked the fake one, Firefox worked fine. The firewall seems to treat the modified version separately in an appropriate way.

Popup and error messages: After re-starting the computer with Outpost installed, a pop-up labelled "Update" appears asking whether to overwrite a more recent configuration file or not. This is disturbing, since we never asked for any updates and do not know why this configuration file needs to be overwritten. This is a typical unuseful dialog where the user has not enough information to answer in a knowledgeable way.
Apart from this, the only time the firewall interrupts the user while running in the background is when a program makes an outbound connection, and there is no rule for that program.

Activation/deactivation: The system tray icon has a small menu where the firewall can be turned on and off, as well as set into different pre-defined modes. There is also an option to stop all network traffic.

Help and documentation: The firewall has built-in help. In most views, there is a button with a question mark. If the user clicks on this, and then on some other object, context related help will appear. This matches the spirit of Nielsen's tenth rule, though the two step process of first selecting the help tool and then clicking on the appropriate object is perhaps a bit complicated. Chances are that a novice user will not understand it and have to resort to searching the help system every time they need it.
The user does not have access to the context sensitive help icon when inside "options" interface. There are help pages for the options, but there is no easy way to access them.

Log: The logs that the firewall produces are shown in a separate "log viewer". This viewer has a tree structure in the left panel, giving access to logs for the different firewall functionalities, as well as filtered versions of these logs. For each item, it is also shown which firewall rule caused the desicion.
Both accepted and rejected connections are logged but there is no way to turn logging on or off. However, the viewer has quite advanced filtering functions. These custom filters also show up in the tree, together with the pre-defined ones. In fact, since both types of filters are treated in the same way, pre-defined filters can be deleted just as easily as custom ones. The risk of accidentally doing this is reduced by a confirmation dialog box, just as Nielsen's fifth rule recommends.
A more serious problem with the filters is that there is no "Not" or "Everything But" options. Because of this, there is no easy way to get rid of a majority of similar items in a log, to be able to see the odd ones.

Resources: Uses quite a lot of the CPU when processing.

Trusted zone: Outpost has no functionality for different trusted zones or levels. It is possible to specify hosts or IP ranges as trusted in the firewall settings. However, this means completely trusting them, and that the firewall will let absolutely everything to or from these hosts through.

Advanced filtering: When making firewall rules, there is an option to "Activate stateful inspection" of packets. The help system claims that the option: "Turns on 'Stateful inspection' for this application (After an application connects to a remote server, all incoming data from that server to the port opened by the application will be allowed or blocked according to the specified setting)." We can not quite determine what this actually means, or what "the specified setting" refers to. contains rather vague text, that can be interpreted as that you can safely allow any connections that keep popping up and bothering you. For security, this is a bad recommendation.

VisNetic Firewall 2.3 Free 30 day trial (Could not be installed at the pre-study, was dropped from main study)

General: We did not manage to get any network traffic working while VisNetic Firewall was installed, despite many attempts and reinstallations. This means that we could not properly evaluate it, and have very little to say about it.

Installation: In the install process of the trial version the user is suddenly prompted with a choice between server or workstation. The user is also asked to select one of five configuration options such as importing old rules or creating new rules through a wizard. It also asks whether the user wants the firewall to automatically start when the computer starts.
The setup also asks if the user wants network traffic allowed or blocked when the firewall is not running, i.e. during startup, and hints that DHCP will not work if you select "block". Despite selecting "allow", the computer could not get an Internet connection after the mandatory reboot. The problem seemed to relate to DHCP and IP address allocation, and the computer always failed to get a working IP number while the firewall was installed.

Selected screenshots for all firewalls

Firewall Product	Outgoing Alert	Incoming Alert
BlackICE	There are no alerts.
Comodo
F-Secure
Lavasoft
McAfee		N/A There is never any alert window. But the main interface shows that there was an incoming event and guides the user how to allow the client connection.
MS Win XP	N/A. Win XP does not control outgoing connections.
NetVeda		Same.
Norman	and a four-step wizard	and a four-step wizard
Norton
Sunbelt
Tiny
Free ZoneAlarm
ZoneAlarm Pro