I just wanted to take a moment and drop everyone a note about the new
Joebox code (12.0), and some notes on known issues and usage
guidelines.
Those of you using the Joebox (about 100 of you) have had a mixed
experience during this first year.
The Joebox has been a learning experience for us and we have
effectively re-designed much of the MECnet implementation to meet the
needs of the MSLN project.
This week we expect to wrap-up upgrades to the new code base. This
upgrade has much better performance and stability, but also makes
input and configuration errors more apparent.
Here is a summary of know issues with the new code base. While they
represent significant problems, they are limited to configuration
error and can be avoided.
Firewall Engine:
Because the new firewall engine is dynamic and only adds or removes
rules that have changed, it is possible for the engine to become in an
indeterminate state if a bad rule is inserted and jumble the ordering
of rules.
An example of a bad rule is a rule that while accepted by the UI is
syntactically or semantically incorrect.
For example: Creating a rule with a protocol that has no knowledge of
port numbers (such as IP, GRE, ICMP, or All) and defining a port.
This results in a rule being generated that attempts to set a port for
a protocol that doesn't understand ports and the rule is rejected by
the kernel.
Another example would be creating a rule with multiple ports and using
the wrong delimiter (e.g. a – instead of a : ). This rule would be
syntactically incorrect and rejected but the current UI doesn’t know
the difference.
Sometimes the source of a bad rule is less obvious. Each Group
members field should contain a list of IP addresses, or IP networks
(written in CIDR notation). These group lists are used to generate
firewall policy (or more specifically firewall rules). On some
Joeboxes we've encountered bad rules being formed because of typos in
the group member field.
Another tricky one is the use of hostnames. When a rule is created
using a hostname instead of IP address, it attempts to resolve the
hostname to its IP address and create a rule based on that. Hostnames
that point to more than one IP address ultimately result in multiple
rules being created (but the firewall engine thinks it is one rule).
We have discouraged use of hostnames for rules in the past, and with
the new firewall engine it should be considered a best practice to
avoid them.
The firewall engine also gets hung up on duplicate rules (rules that
get generated with identical syntax). These rules get inserted fine,
but when the engine deletes a rule it currently doesn't reference the
rule index. This causes all matching rules to be deleted and throws
the firewall engine into an indeterminate state.
We have been working with MECnet to do stronger input checking on
firewall rules, but we seem to find new cases that haven’t handled
each time we have one fixed.
The reason this has such a big impact with the firewall engine is
because it maintains its own database of what rules are in the system.
This database is used to determine what position a new rule should be
inserted at or which rule index to remove. When the rules in this
database become out of sync with the rules applied on the system, the
engine falls into an indeterminate state and may begin trying to
remove a rule at index 8 when it really wants index 7, with each
change the firewall becomes further out of sync until restarted.
We have provided MECnet with feedback on a few ways they could handle
this and gracefully recover. In the meantime, we are currently
testing a minor UI update that will add a "reinitialize firewall"
button to the firewall options page. This button will force a
complete flush and re-insert of all rules (effectively a forced
restart).
Keep in mind that services on the Joebox make frequent updates to
firewall policy. In the past, this meant flushing all rules and
reinserting them. This was found to be the cause of frequent service
interruption. The move to a dynamic engine is a significant
improvement but has presented MECnet with implementation challenges
that they continue to work through. In its current state it is well
tested and working provided that rules defined are correct. It does
have the consequence of being unforgiving for mistakes though, so make
changes with due caution.
MECguard Filter:
MECguard was quite broken when trying to build policy for more than
one group in previous releases. We also discovered an outstanding bug
with certain RTF keywords (like "soucy") not being applied because of
the matching algorithm used.
Thankfully the new MECguard is working correctly (as far as we know).
Group ordering is respected. URL lists (Group and Global Allow and
Deny) are verified to be working. RTF is verified to be working.
Logging has been improved.
That said, we still see questions on MECguard "not working". This is
often because of incorrect syntax. In theory, incorrect syntax should
be disregarded, however we have reason to believe that it can
sometimes cause the database to be corrupted (unverified). Because of
this we need to pay careful attention to input in MECguard like we do
with the Firewall engine.
How MECguard does matching:
MECguard looks at URL lists (the group-level or global Allow List and
Deny List) line-by line until a match is made or the list is
exhausted.
For each line, a string-in-string comparison is made. There are no
wildcard or special characters (such as *.something.com) so rules with
these will be ignored.
The protocol is also omitted from the request string which comparisons
are made against so entries that include http:// will also be ignored.
Example: When you go to http://www.facebook.com/login.php in your
browser, MECguard will set the requested string to
www.facebook.com/login.php (no http://). It then checks URL lists for
the presence of each line within this request string. So an entry for
facebook.com would match while an entry for
www.facebook.com/login.php?user would not (because this line includes
the extra ?user at the end).
The thing to watch out for is creating entries that might be too
broad. For example an entry for "." will apply to everything (because
almost every request will have at least one ".") and an entry for
"ebook.com" would also apply to "facebook.com" because the string
ebook.com is contained within facebook.com.
Order of evaluation:
1. Group-level Allow List
2. Global Allow List
3. Global Block List
4. Group-level Block List
5. Soft-Allow List
6. Group Filter Category Lists
7. Known RTF URLs (previously blocked by RTF)
8. Real Time Filter (HTTP only)
Keep in mind that for HTTPS (MECguard SSL) only the hostname of the
website is seen by MECguard. So if you want to block
http://encrypted.google.com/ you can only match "encrypted.google.com"
(hostname only).
Note that the Force MECguard SSL group-level checkbox will simply
re-direct non-proxy HTTPS requests (TCP 443) to a block page
explaining that HTTPS has been disabled unless a proxy is used.
MECguard SSL will function regardless of if this is checked or not.
MECguard SSL does require that browsers are configured to point to the
Joebox for HTTPS, though. This can be automated by using WPAD (the
Joebox hosts a WPAD script at "http://192.0.0.1/wpad.dat") and setting
up hosts to resolve wpad.domainofhost to 192.0.0.1. For sites using
NAT and a local DHCP serves on the Joebox (very simple setup) the
Joebox DNS resolver includes an entry for "wpad.local". This will
make any host that uses local as its domain (the default
configuration) to discover the proxy server if using the Joebox for
DNS.
Note that many corporate networks today opt to block all traffic for
hosts except HTTP, HTTPS, DNS, and ICMP to limit the use of
unauthorized applications. Since HTTPS filtering is proxy-based (and
not transparent) blocking non-proxy access the Internet is the only
way to prevent proficient users from using off-network proxy services
or tunnels that function on different ports.
I would also like to note that MECguard is generally targeted at
schools that need more control over HTTP filtering. We are currently
testing OpenDNS as a second option for schools that don't require as
much configuration or do not have a Joebox. OpenDNS can be used
instead of MECguard for Joebox users if they wish (simply disable
MECguard).
SMTP Filtering:
SMTP filtering is something the MSLN project has historically done
though the use of access lists on routers. SMTP (TCP port 25) is
continuously exploited to turn compromised machines into SPAM bots
which ultimately result in getting MSLN IP address space on block
lists. Not having SMTP filtering active on the Joebox by default at
launch was a mistake. The previous update introduced SMTP filtering
with a grace period to enable it and create rules to allow legitimate
SMTP traffic through. The new update enables it by default.
Traffic caught by SMTP filtering will appear as "SMTP" in the firewall log.
In the firewall options page, there is an exception list that follows
the same format of the group member list (one entry per line, either
an IP address or network in CIDR notation). Hosts in this list will
not be filtered for outgoing SMTP (this is the appropriate place to
put the address of your mail server).
SMTP filter is only applied to outgoing SMTP traffic. Incoming
traffic still requires that your firewall policy is set to allow this
(e.g. it will not block with a Low policy level, and you will need an
open port rule to your mail server for Medium or High policy levels).
If you have multiple networks and are using a Low policy level, you
will need to explicitly create an open ports rule to allow SMTP
traffic to your mail server to avoid internal clients on a different
network being filtered to the local server. If you have a Medium or
High policy level, you likely already have such a rule in place.
This seems to have caused some controversy but SMTP filtering has been
our policy in the past and will continue to be our policy. The
difference with the Joebox is that you now have control to create
exceptions for SMTP filtering when needed whereas traditionally we
have not provided the option to do so.
Common Configuration Issues:
Here are some common configuration issues we've seen throughout the
upgrade process.
System Security "Users allowed to administer this Joebox" left as
"0/0" (Everyone). Part of this upgrade process has been to restrict
the access list of IP addresses and networks allowed to administer the
Joebox if it hasn't been set to something appropriate. While the web
UI does require user authorization, it doesn't protect the Joebox
against zero-day remote code execution, brute force, or other types of
directed attacks from the outside. If your firewall is compromised,
then your network is compromised.
As we perform these upgrades, we have been restricting this list to
internal IP networks for each site if you still have the factory
default of "0/0”. We strongly encourage you to go a step further and
restrict Joebox management to a select few IP addresses used by you
and your IT staff rather than your entire network, however.
Port Forward v One-to-One NAT
We see a lot of One-to-One NAT rules that aren't necessary and could
easily be replace with a Port Forward rule. Because most Joeboxes are
dual-homed between Portland and Orono any NAT rule created needs to be
created twice (one for each egress interface). This includes
user-generated One-to-One NAT rules.
When creating Port Forward rules, a Port Forward will automatically
create a rule to allow traffic to a host. A One-to-One rule will only
create the translation and you will need to manually create open port
rules for the host.
So for a simple web server you can create a single Port Forward rule
or you can create a One-to-One NAT rule, a NAT rule for the return
traffic for each egress interface, and an Open Port rule for the web
traffic (4 rules instead of 1).
One-to-one NAT is required only for very specific applications
(usually legacy applications like H.323 without the use of a
Gatekeeper) and for 99.9% of users doesn't add any benefit. On the
Joebox, using one-to-one NAT when a Port Forward will do the trick
simply adds more opportunity for user-error in defining rules. You
will want to weigh both options when building your policy. Either one
is fine, but one-to-one is often not necessary (and commonly
miss-configured).
Firewall rules that have URL (or something completely bogus) in the
IP/Hostname field:
Every now and then we come across a firewall rule for "Destination:
http://www.facebook.com/" The Joebox has no idea what you're talking
about. It wants an IP or Hostname and not a URL.
An even odder one we have seen is something like "Destination: PC in
the Kitchen". Mind you, this wasn't a group, it was an IP/Hostname
(nor did any such group exist). It is currently a design limitation
of the Joebox for it to not be able to figure out what IP that the "PC
in the Kitchen" has.
As a general rule of thumb if you aren't comfortable with making a
configuration change on the Joebox, give us a call. We generally
don't have hold times and our students are getting quite good at
Joebox configuration.
For all the doom and gloom that this email feels like, there are a lot
of positive changes to the Joebox in the updated code:
Kernel tuning to accommodate larger networks.
Re-written firewall policy engine to make the Joebox more flexible.
Dynamic firewall engine that doesn't drop connections intermittently the day.
MECguard that actually respects group order and applies filtering correctly.
In reviewing Joebox problems over the last year there have been a few
issues that were legitimate bugs or design problems with the Joebox
itself. The vast majority, however, have been configuration issues or
limitations of internal networking (e.g. slowness complaints).
If your Joebox isn't working correctly or you think there is a Joebox
problem be sure to open a support ticket with us. We do review them
even after they're closed and use them to drive development
priorities. If we don't know you're unhappy, we can't do anything to
fix the problem.
--
Ray Soucy
Epic Communications Specialist
Phone: +1 (207) 561-3526
Networkmaine, a Unit of the University of Maine System
http://www.networkmaine.net/
|
