I just wanted to take a moment and drop everyone a note about the new Joebox code (12.0), and some notes on known issues and usage guidelines. Those of you using the Joebox (about 100 of you) have had a mixed experience during this first year. The Joebox has been a learning experience for us and we have effectively re-designed much of the MECnet implementation to meet the needs of the MSLN project. This week we expect to wrap-up upgrades to the new code base. This upgrade has much better performance and stability, but also makes input and configuration errors more apparent. Here is a summary of know issues with the new code base. While they represent significant problems, they are limited to configuration error and can be avoided. Firewall Engine: Because the new firewall engine is dynamic and only adds or removes rules that have changed, it is possible for the engine to become in an indeterminate state if a bad rule is inserted and jumble the ordering of rules. An example of a bad rule is a rule that while accepted by the UI is syntactically or semantically incorrect. For example: Creating a rule with a protocol that has no knowledge of port numbers (such as IP, GRE, ICMP, or All) and defining a port. This results in a rule being generated that attempts to set a port for a protocol that doesn't understand ports and the rule is rejected by the kernel. Another example would be creating a rule with multiple ports and using the wrong delimiter (e.g. a – instead of a : ). This rule would be syntactically incorrect and rejected but the current UI doesn’t know the difference. Sometimes the source of a bad rule is less obvious. Each Group members field should contain a list of IP addresses, or IP networks (written in CIDR notation). These group lists are used to generate firewall policy (or more specifically firewall rules). On some Joeboxes we've encountered bad rules being formed because of typos in the group member field. Another tricky one is the use of hostnames. When a rule is created using a hostname instead of IP address, it attempts to resolve the hostname to its IP address and create a rule based on that. Hostnames that point to more than one IP address ultimately result in multiple rules being created (but the firewall engine thinks it is one rule). We have discouraged use of hostnames for rules in the past, and with the new firewall engine it should be considered a best practice to avoid them. The firewall engine also gets hung up on duplicate rules (rules that get generated with identical syntax). These rules get inserted fine, but when the engine deletes a rule it currently doesn't reference the rule index. This causes all matching rules to be deleted and throws the firewall engine into an indeterminate state. We have been working with MECnet to do stronger input checking on firewall rules, but we seem to find new cases that haven’t handled each time we have one fixed. The reason this has such a big impact with the firewall engine is because it maintains its own database of what rules are in the system. This database is used to determine what position a new rule should be inserted at or which rule index to remove. When the rules in this database become out of sync with the rules applied on the system, the engine falls into an indeterminate state and may begin trying to remove a rule at index 8 when it really wants index 7, with each change the firewall becomes further out of sync until restarted. We have provided MECnet with feedback on a few ways they could handle this and gracefully recover. In the meantime, we are currently testing a minor UI update that will add a "reinitialize firewall" button to the firewall options page. This button will force a complete flush and re-insert of all rules (effectively a forced restart). Keep in mind that services on the Joebox make frequent updates to firewall policy. In the past, this meant flushing all rules and reinserting them. This was found to be the cause of frequent service interruption. The move to a dynamic engine is a significant improvement but has presented MECnet with implementation challenges that they continue to work through. In its current state it is well tested and working provided that rules defined are correct. It does have the consequence of being unforgiving for mistakes though, so make changes with due caution. MECguard Filter: MECguard was quite broken when trying to build policy for more than one group in previous releases. We also discovered an outstanding bug with certain RTF keywords (like "soucy") not being applied because of the matching algorithm used. Thankfully the new MECguard is working correctly (as far as we know). Group ordering is respected. URL lists (Group and Global Allow and Deny) are verified to be working. RTF is verified to be working. Logging has been improved. That said, we still see questions on MECguard "not working". This is often because of incorrect syntax. In theory, incorrect syntax should be disregarded, however we have reason to believe that it can sometimes cause the database to be corrupted (unverified). Because of this we need to pay careful attention to input in MECguard like we do with the Firewall engine. How MECguard does matching: MECguard looks at URL lists (the group-level or global Allow List and Deny List) line-by line until a match is made or the list is exhausted. For each line, a string-in-string comparison is made. There are no wildcard or special characters (such as *.something.com) so rules with these will be ignored. The protocol is also omitted from the request string which comparisons are made against so entries that include http:// will also be ignored. Example: When you go to http://www.facebook.com/login.php in your browser, MECguard will set the requested string to www.facebook.com/login.php (no http://). It then checks URL lists for the presence of each line within this request string. So an entry for facebook.com would match while an entry for www.facebook.com/login.php?user would not (because this line includes the extra ?user at the end). The thing to watch out for is creating entries that might be too broad. For example an entry for "." will apply to everything (because almost every request will have at least one ".") and an entry for "ebook.com" would also apply to "facebook.com" because the string ebook.com is contained within facebook.com. Order of evaluation: 1. Group-level Allow List 2. Global Allow List 3. Global Block List 4. Group-level Block List 5. Soft-Allow List 6. Group Filter Category Lists 7. Known RTF URLs (previously blocked by RTF) 8. Real Time Filter (HTTP only) Keep in mind that for HTTPS (MECguard SSL) only the hostname of the website is seen by MECguard. So if you want to block http://encrypted.google.com/ you can only match "encrypted.google.com" (hostname only). Note that the Force MECguard SSL group-level checkbox will simply re-direct non-proxy HTTPS requests (TCP 443) to a block page explaining that HTTPS has been disabled unless a proxy is used. MECguard SSL will function regardless of if this is checked or not. MECguard SSL does require that browsers are configured to point to the Joebox for HTTPS, though. This can be automated by using WPAD (the Joebox hosts a WPAD script at "http://192.0.0.1/wpad.dat") and setting up hosts to resolve wpad.domainofhost to 192.0.0.1. For sites using NAT and a local DHCP serves on the Joebox (very simple setup) the Joebox DNS resolver includes an entry for "wpad.local". This will make any host that uses local as its domain (the default configuration) to discover the proxy server if using the Joebox for DNS. Note that many corporate networks today opt to block all traffic for hosts except HTTP, HTTPS, DNS, and ICMP to limit the use of unauthorized applications. Since HTTPS filtering is proxy-based (and not transparent) blocking non-proxy access the Internet is the only way to prevent proficient users from using off-network proxy services or tunnels that function on different ports. I would also like to note that MECguard is generally targeted at schools that need more control over HTTP filtering. We are currently testing OpenDNS as a second option for schools that don't require as much configuration or do not have a Joebox. OpenDNS can be used instead of MECguard for Joebox users if they wish (simply disable MECguard). SMTP Filtering: SMTP filtering is something the MSLN project has historically done though the use of access lists on routers. SMTP (TCP port 25) is continuously exploited to turn compromised machines into SPAM bots which ultimately result in getting MSLN IP address space on block lists. Not having SMTP filtering active on the Joebox by default at launch was a mistake. The previous update introduced SMTP filtering with a grace period to enable it and create rules to allow legitimate SMTP traffic through. The new update enables it by default. Traffic caught by SMTP filtering will appear as "SMTP" in the firewall log. In the firewall options page, there is an exception list that follows the same format of the group member list (one entry per line, either an IP address or network in CIDR notation). Hosts in this list will not be filtered for outgoing SMTP (this is the appropriate place to put the address of your mail server). SMTP filter is only applied to outgoing SMTP traffic. Incoming traffic still requires that your firewall policy is set to allow this (e.g. it will not block with a Low policy level, and you will need an open port rule to your mail server for Medium or High policy levels). If you have multiple networks and are using a Low policy level, you will need to explicitly create an open ports rule to allow SMTP traffic to your mail server to avoid internal clients on a different network being filtered to the local server. If you have a Medium or High policy level, you likely already have such a rule in place. This seems to have caused some controversy but SMTP filtering has been our policy in the past and will continue to be our policy. The difference with the Joebox is that you now have control to create exceptions for SMTP filtering when needed whereas traditionally we have not provided the option to do so. Common Configuration Issues: Here are some common configuration issues we've seen throughout the upgrade process. System Security "Users allowed to administer this Joebox" left as "0/0" (Everyone). Part of this upgrade process has been to restrict the access list of IP addresses and networks allowed to administer the Joebox if it hasn't been set to something appropriate. While the web UI does require user authorization, it doesn't protect the Joebox against zero-day remote code execution, brute force, or other types of directed attacks from the outside. If your firewall is compromised, then your network is compromised. As we perform these upgrades, we have been restricting this list to internal IP networks for each site if you still have the factory default of "0/0”. We strongly encourage you to go a step further and restrict Joebox management to a select few IP addresses used by you and your IT staff rather than your entire network, however. Port Forward v One-to-One NAT We see a lot of One-to-One NAT rules that aren't necessary and could easily be replace with a Port Forward rule. Because most Joeboxes are dual-homed between Portland and Orono any NAT rule created needs to be created twice (one for each egress interface). This includes user-generated One-to-One NAT rules. When creating Port Forward rules, a Port Forward will automatically create a rule to allow traffic to a host. A One-to-One rule will only create the translation and you will need to manually create open port rules for the host. So for a simple web server you can create a single Port Forward rule or you can create a One-to-One NAT rule, a NAT rule for the return traffic for each egress interface, and an Open Port rule for the web traffic (4 rules instead of 1). One-to-one NAT is required only for very specific applications (usually legacy applications like H.323 without the use of a Gatekeeper) and for 99.9% of users doesn't add any benefit. On the Joebox, using one-to-one NAT when a Port Forward will do the trick simply adds more opportunity for user-error in defining rules. You will want to weigh both options when building your policy. Either one is fine, but one-to-one is often not necessary (and commonly miss-configured). Firewall rules that have URL (or something completely bogus) in the IP/Hostname field: Every now and then we come across a firewall rule for "Destination: http://www.facebook.com/" The Joebox has no idea what you're talking about. It wants an IP or Hostname and not a URL. An even odder one we have seen is something like "Destination: PC in the Kitchen". Mind you, this wasn't a group, it was an IP/Hostname (nor did any such group exist). It is currently a design limitation of the Joebox for it to not be able to figure out what IP that the "PC in the Kitchen" has. As a general rule of thumb if you aren't comfortable with making a configuration change on the Joebox, give us a call. We generally don't have hold times and our students are getting quite good at Joebox configuration. For all the doom and gloom that this email feels like, there are a lot of positive changes to the Joebox in the updated code: Kernel tuning to accommodate larger networks. Re-written firewall policy engine to make the Joebox more flexible. Dynamic firewall engine that doesn't drop connections intermittently the day. MECguard that actually respects group order and applies filtering correctly. In reviewing Joebox problems over the last year there have been a few issues that were legitimate bugs or design problems with the Joebox itself. The vast majority, however, have been configuration issues or limitations of internal networking (e.g. slowness complaints). If your Joebox isn't working correctly or you think there is a Joebox problem be sure to open a support ticket with us. We do review them even after they're closed and use them to drive development priorities. If we don't know you're unhappy, we can't do anything to fix the problem. -- Ray Soucy Epic Communications Specialist Phone: +1 (207) 561-3526 Networkmaine, a Unit of the University of Maine System http://www.networkmaine.net/