January 10, 2005

Pro and Con

A brief comparison of the pros and cons of hosting at a professional hosting company vs. hosting at home:

Hosting Company: You might have a newbie trip over the cables and knock your server offline.

Home: You might have a two-year old playing "trapped" decide to change the output of your variable-voltage power supply, thus destroying your switch and casting you into the depths of "offline".

Guess which one happened to us?

Posted by Jeff at 10:33 AM | Link Cosmos | Comments (0)

December 18, 2004

Eliminating Spam

Steven Green points to this John Dvorak article, and asks: "can some of you smart network-type people tell me if any of these ideas [for reducing spam technologically] are doable without needing an entirely new email system and software"?

I'm not going to go into detail on all of Dvorak's proposals, but I will say that he misses the point, for an entirely good reason. The Internet was built as a survivable network, meant in fact to survive nuclear strikes on some of its nodes and still keep going. It was also designed to assume security: all of the users would have to be authenticated by a node before obtaining access to the network, and all of the nodes were trusted. This was an excellent architecture in 1988 (when I first got onto the Internet), because all of the users were accountable for their actions - to their employer, or their university, or their government agency, etc. As a result, abuse was a minor problem (mostly incoming freshmen at colleges) and easily dealt with. But the system dealt with abuse - the human part of the system, not the network part. You could actually be shamed off of the Internet back then. Removing the human enforcer of netiquette and good practice was like giving the government the power to raise almost unlimited revenue from a very small proportion of the population: corruption and abuse exploded. Open source routing - where everyone passed traffic by the shortest route, so that my traffic could go right through IBM's internal network if that was the fastest way to the destination - was the first to go, replaced by a few backbone networks, with the multiply-connected large private networks protected by layers of firewalls and large IT staffs. (This increased the brittleness of the network at the same time that it increased security.)

So how do we fix this? We cannot build a safe, secure and reliable system on top of an inherently insecure, self-regulating and untrustworthy network. We would have to build a new network from the ground up. And to do that, we'd have to scrap everything except the physical layer - the NICs, wires, routers, bridges, modems and so forth would be all that's left. The intelligence of the network would have to change, some of it drastically.

The first problem to solve is design: how do you create a secure, trusted network, which at the same time still allows connections from anywhere to anywhere, using any protocol, as the default? How do you do this without either excluding people or forcing a central controlling agency (brittle, arbitrary and powerhungry as any bureaucracy) or limiting the ability of people to use the network in reasonable ways? How do you manage traffic in such a way that the network can be flexible, without overwhelming small companies who are multiply-connected by passing external traffic through their networks? How do you provide authentication and authorization, in other words, to a global network using only local resources?

It turns out that if you are willing to start with a blank page, it's not that difficult. The major issue to be solved is trust: how do I know whom I'm talking to? Since you don't want a brittle network, that will fall apart when something happens to a small number of nodes, the only trust model that works is to authenticate yourself to some local authentication source. For example, an ISP or a company would provide a directory which lists all of their users, and contains the information necessary to trust that user. Each node, then, has to trust its neighboring nodes. (This is already the case today, in that you cannot establish a physical connection to another node without being their customer or their provider, or entering into an agreement to do so.) Each node would need to cryptologically authenticate itself to its neighbors, and vice versa, and each user would have to authenticate themselves to their node. No node would pass traffic that did not include its partner node's connection key in the message portion of the packet, and no node would alter that information.

On the good side, this lets any traffic be traced back to its source. You cannot fake traffic as being from a node that you are not from, because the network will not pass on the message unless it has authenticated the upstream source - that is, you - and your information is in the packet header. Therefore, you could put preceding node information in the header, but it would be bogus and the trace would effectively stop being verifiable at you, the sender. On the bad side, this would dramatically increase the amount of traffic on the Internet (by increasing the size of any given packet) and would slow down all traffic, bandwidth being equal, because of the overhead of nodes authenticating to each other and the larger byte-count of a given data set.

Let's say, though, that we were to use that or some similar measure of trust to guarantee that the network was trusted. Now, we still have a whole host of problems to solve, because the IP spec would have to be rewritten at a pretty fundamental level. This means that the NICs would need updated firmware, or for cheap cards that have their firmware burned into the hardware, the whole NIC would have to be replaced. Then, you would have to rewrite the network stack to take account of the new protocol. You'd have to rewrite all of the protocols like UDP, FTP, SMTP, POP, NNTP, LDAP, SSH and the like. Some of these would be huge efforts, while others would need few or no changes. The OS network stacks would have to be rewritten for each OS, and some applications would need to be rewritten as well, if they deal with the network information at a low level.

All in all, it would be a huge load of work, and not likely to get done as an organized effort. The better way to do it would be to set up such a network privately, amongst friends as it were, and expand that to their friends, and their friends, and so on, and so on... And if you were to gateway to the global internet, you'd lose a lot of the benefits right there.

So in practical terms, I don't think it's going to happen.

Posted by Jeff at 05:12 PM | Link Cosmos | Comments (0)

November 25, 2004

Fun with PostgreSQL

During the Summer, when we were in Chicago, my server's hard drive died (quite literally the day we left to return to Chicago after two weeks at home). I run it out of my house, and there was no competent tech support with a key to the house. I ended up flying back down for a day to get the server going again, but in the process messed up a few things, including the ability to automatically download security patches.

This morning, I decided to fix that by upgrading the server's OS, then running the update service on the new OS. One small problem: turns out that using RPMs to install PostgreSQL (the database I use for, among other things, this blog) is a bad idea: the upgrade process requires you to dump the database under the old version, potentially hand edit the dump file without any instructions on how to do so (!!!), install the new version, then reimport your data. The RPM package does not dump the data before deinstalling the old version (!!!).

Fortunately, there are backups and other aids to fixing the resulting mess, but that's why comments were down all day. They're working now.

Posted by Jeff at 11:21 PM | Link Cosmos | Comments (0)

October 05, 2004

Heh

John Hawkins has posted a basic guide to computer maintenance. Since he explicitly refused to consider Mac maintenance in this, I'll do it for him.

1) Get a Mac
2) If you have a problem, and you can't solve it by quitting and restarting apps, reboot
3) In the remarkable situation that you still have a problem, reinstall the OS. Yes, over the existing OS. No, it won't kill your programs; Windows does that, but not Mac. It should take about 20 minutes.
4) Oh, and his tips on spam are fine. Those work for Mac users, too.

Posted by Jeff at 11:42 AM | Link Cosmos | Comments (1)

September 30, 2004

Things that Make me Drool

My area of expertise is directory services, provisioning, identity management and enterprise systems design. That is to say, I specialize in putting together large computer and network infrastructures to host applications, which can efficiently and accurately control who is accessing the applications and how. One of the critical tools in doing identity management and provisioning is directory services, and the best general-purpose directory for small to medium (<5 million objects) applications is the SunONE Directory. Developed from the Netscape Directory Server, SunONE (in any of its many rebrandings: SunONE, SunONE Java System, iPlanet, etc) is a fast, easy-to-administer (especially critical in large environments) directory with a very good implementation of the RFCs.

The open source competitor to SunONE is OpenLDAP. It is closer to Innosoft's product, which was developed out of the same base code from UMich that was used as the basis for OpenLDAP. It is not very fast, quite difficult to administer in enterprise environments, and lacks a robust replication mechanism. (Anyone who wants to debate me on slurpd, please feel free. I have a passionate hatred for it, and a good rant is fun.) In other words, the open source alternative is fine for a hobbyist, but I wouldn't hire someone recommending OpenLDAP for a professional environment.

And that is why this is so cool. This is the same code base used for SunONE, and RedHat will release it open source. Even if the code has not advanced beyond its 4.x releases, it will have the self-storage of configuration, the admin console, the web admin tools, the replication system and the basic architecture that make this code such a powerful product. The major thing that is lacking is a true multi-master model of replication (this is also lacking in SunONE, which last time I looked still allowed only two masters, and required them to be in the same data center - not a winning plan for disaster recovery).

It may very well be the case that within a few years, the open source directory alternatives will outstrip all but the high-end X.500 implementations (eTrust comes to mind), making directories much more affordable in the enterprise, and allowing the efficiencies that directories enable to be pushed further down the food chain. I look forward to that.

Posted by Jeff at 03:46 PM | Link Cosmos | Comments (0)

September 27, 2004

Hell, Yeah!

As a long-term system administrator (now systems architect), I recognize a kindred administrator soul when I see one. (hat tip: Accidental Verbosity) Yes, it's like that.

Posted by Jeff at 04:47 PM | Link Cosmos | Comments (0)

August 16, 2004

What I Do (Almost)

Interestingly, a project very closely related to mine has made the "news" in the New York Post. The article discusses FedLine Advantage, which is an expansion and modernization of the old FedLine DOS which has been in place for a long time, and the article is about as hysterical as I would expect from the media. Witness:

"If a security breach strikes the very heart of the financial world and money stops moving around, then our financial system will literally start to collapse and chaos will ensue."

Um, yeah. It's not exactly a secret that if "money stops moving around, then our financial system will literally start to collapse". Of course, no evidence is presented as to why that might happen, or even how. Or for that matter, how that differs from the current system, in which the same statement applies.

What I work on is the access and identity management system that protects this infrastructure, so I have a pretty fair understanding of the security measures in place. It's nowhere near as horrible as painted in the article, or in the comments to this Slashdot post. Believe it or not, these things do get thought out; if you have the ability to transfer millions of dollars, the Fed knows who you are and how you got that authority, and you won't be getting in unless you have the proper physical components and knowledge. And we'll know about it. And no, I'm not going to tell you how to get those things.

I don't know why people just assume that the Fed is incompetent about this kind of thing. They sometimes make bad decisions for bad reasons, like any organization, but it's not like they don't know what is at stake and what it takes to protect the infrastructure.

Posted by Jeff at 04:00 PM | Link Cosmos | Comments (2)

June 08, 2004

iDrool

Apple can be accused of many things, but one of them is not being conventional. This is a fantastic idea.

Posted by Jeff at 02:59 PM | Link Cosmos | Comments (0)

May 27, 2004

Drooling at High Speed

Sorry, Steph, but we can't leave Keller until Verizon rolls out elsewhere.

I'd still like to know the bitrates, costs and whether the service will allow symmetrical uploads and downloads. (this is as close as I've found). But I'm already drooling in anticipation.

Posted by Jeff at 04:00 PM | Link Cosmos | Comments (2)

April 20, 2004

Software Information Bleg

We use Macs. We love our Macs. You can have our Macs when you pry them from our cold, dead fingers.

But there is one problem with Macs: it's devilishly hard to find specialized software for them. In particular, my current problem is weather software. I want something like StormPredator, but I'd prefer not to run something under emulation. (We like Macs, but our Macs are not current, fast machines capable of running the Microsoft (ie, bloated and less useful than when it was Connectix) version of VirtualPC) well.

So here's the question: do any of you know of a Mac-native (preferably MacOS X native) application which can show current radar and storm tracking data, with path prediction and arrival ETAs? I'm less concerned about cost than about feature sets, since I can always save up if I have to.

Posted by Jeff at 11:25 PM | Link Cosmos | Comments (0)

April 16, 2004

Development and Production

When I was involved in systems administration, the bane of my existence was people who could not seem to see the difference between development systems and production systems. For example, when you have a patch to apply to a production system, you don't just run out and do it; you test it, verify it, try to break it, then put it in at the least busy time for your system or during a scheduled outage window if you can get one. Otherwise, you are likely to cause damage and lost money.

Now that I'm an application development architect (for the moment, anyway), the bane of my existence is people who cannot seem to see the difference between development and production systems. For example, when you are halfway through a development project, and you are migrating code from your raw development environment to your first (of two) test environment, and you have a problem that arises, the correct answer is not PANIC. The correct way to behave is not to force others to panic, and cause the developers to be up until midnight and working all weekend. Because when you have a real problem, at a critical junction, you will have worn out your development team, presuming they are still working for you.

Actually, it strikes me that this kind of attitude affects a lot of areas of life. Look for instance at the situation in Iraq: how would we know if the situation were really falling apart, since any setback is billed as a catastrophe? Or how would we know if a true fascist dictator were rising in America: would we listen when he was compared to Hitler, just like President Bush has been? A constant state of panic is debilitating.

I think that there is a single underlying cause: a misplaced standard of perfection. There are cases where the standard really has to be perfection, or as close as you can get: I'd hate to see an engineering plan for safety locks on nuclear weapons that had any lesser standard. But when that truly is the standard, you prepare for it, give yourself time and iterations to achieve it. Otherwise, panic all you want to, but I don't plan on being up at midnight to coddle you.

Not that I'm bitter.

Posted by Jeff at 05:48 PM | Link Cosmos | Comments (0)

March 13, 2004

The Rot Within

The Internet as it now exists is dying. The rot isn't very visible yet, because it is at the very heart of what makes the Internet function: the TCP/IP protocols underlying the whole network and the consolidation of backbone providers. Some of the symptoms are the rapidly-shrinking pool of available addresses, combined with slow adoption of IPv6 (which would expand the address pool dramatically); the prevalence of spam in virtually every medium; inability to easily find the resources you want the first time you need them; lawsuits over ownership of domain names; denial of service attacks and other types of remote hacking; and occasional outages across a large span of the network.

TCP/IP was designed for survivability. If a network node went down, the damage would be detected and routed around. In other words, the Internet as originally conceived was designed to survive a nuclear attack gracefully. Today, it cannot survive a backhoe accident gracefully.

The Internet works by INTERconnecting individual NETworks. Thus, my network at my house talks to Verizon's network via a permanent connection. I don't have any backup. If that connection (my DSL line) fails, I'm limited to my local network until the connection is restored. This is a risk I accept because the financial cost of redundant connections is higher than their worth to me. For a business, though, being disconnected is simply unacceptable; they connect at multiple points to multiple backbone providers (Verizon, Sprint, AT&T, etc), sometimes through geographically distinct locations (IBM, for example, connects in many many cities to many many different providers). If one of their connections fails, the other takes all of the traffic and routes it accordingly...theoretically.

In practice, this only works a little for most companies, for two reasons. First, most companies have consolidated their network connections down to a bare minimum. They have done this for cost, yes, but mostly they have done so because the more connections you have, the more difficult it is to seal off your network. Originally, all networks would route any traffic through them. But as attacks became more prevalent, and as free riders used up bandwidth they weren't paying for, companies increasingly sealed off their networks so that only their own traffic would go in and out. The connection points had to be protected, and monitored, and that was difficult if you had too many connection points. And, too, there was the problem of other companies also sealing off their networks, so you had to connect only to backbone providers or to intermediaries who specialize in connecting networks (ISPs).

The net effect of this was to make the backbone providers much more important, because there was no longer a web of interconnections between companies or organizations that were geographically near each other, or had shared business between them. (These days, it is not unusual for different units of the same company have no interconnection except through the backbone providers.) But the backbone providers have powerful incentives to consolidate. They have to share several locations to all come together and talk, in order to work (Google on MAE-East and MAE-West some time). This means that those interconnection points for backbone providers have a lot of high-capacity lines running in to them. And because the number of lines is finite, and the demand is large, most of those lines are in some way shared between different providers for some or all of their distance. As a result, it is very possible for a backhoe within a few hundred miles of one of these major interconnects to take down large parts of the network.

This problem cannot be fixed unless you fix the underlying issues with TCP/IP. TCP/IP was not designed with security or predictability in mind: all the packets are unencrypted; you don't know much about the packet other than it is coming from an adjacent node to which you are connected, and where it is going (the router then compares the destination with its table of where to send things, and sends the packet down the appropriate line); and it's not predictable - delivery of packets you send cannot be guaranteed.

There are ways to fix this, which would guarantee that you could establish dependable connections, still capable of routing around damage, where you know where the traffic is coming from in an unspoofable way. The only problem is that you have to replace all of the equipment on which the Internet runs (down to the the network cards in your home computer, potentially), and rewrite all or parts of the various applications and operating systems which understand the current ISO network model and the TCP/IP protocols.

I'm going off into gobbledygook land for a minute here, so if you're not a computer person, feel free to either Google the terms I've emphasized, or skip the next couple of paragraphs.

Here are some existing parts that could be brought together to make this work: the address scheme could be provided by OIDs. If every network device had an inalterable OID, and every user also had non-repudiatable (learn about PKI) identity, and every packet had the entire chain of OIDs from initiation to where the packet currently sits contained in it with the appropriate cryptographic signatures to allow the detection of forgery, any virus initiators or crackers could be tracked down and could not reasonably deny their culpability.

Ah, but what about routing these addresses, since OIDs are not tied to physical organizations? Well, neither are IP addresses. IP addresses are issued byIANA to one of APNIC, ARIN, LACTIC or RIPE, which then issue them further down. For example, my home network IP comes from IANA to ARIN to Verizon to me. Each issuer keeps track of those blocks that have been issued, and of how they route. IANA has an OID, and it could establish a branch for addresses (if it hasn't already) and thus the routing continuity is available.

How would you find a name under this scheme? DNS would need huge modifications. But it could be done using LDAP servers, with a modified DNS front ending them, using a cascading system of references not unlike what DNS does. This also would provide other benefits, like being able to instantly tie an address (not just for an end point, but for any device you talk to) to its owner. This has a further advantage, because you can tie down more than just addresses to names. You can also tie names to services and names to people, so that you could (for example) say something like, "send this email to Jeff Medcalf from the DFW area, wherever he is now" and the network could do it transparently. Or you could say something like "tell me where the nearest grocery store is", and if the grocery store registered its physical address and you registered your current location, the network could tell you.

OK, so at this point, you have a way of verifying where something came from when it hits your network, but the traffic is still going in the clear, so passwords and credit card numbers are still vulnerable unless you are using SSH or SSL or something similar. Well, heck, as long as we're replacing all of the underlying infrastructure, let's say than any two nodes can be configured to talk IPSec (or, more correctly, an appropriate analog). That is, all nodes will encrypt the traffic so that it can only be decrypted by the node it's sent to. Remember how I said above "every packet had the entire chain of OIDs from initiation to where the packet currently sits contained in it with the appropriate cryptographic signatures to allow the detection of forgery"? Well, it so happens that capability is tied up with the capability to encrypt traffic from node to node.

Now that you know where something is coming from, and can guarantee that it can't be forged without detection, you can establish webs of trust. For example, you could trust those networks you connect to to send you traffic, and also trust any traffic they pass on from outside. A company would want to cut off the web of trust at anonymizers (which would allow you to hide your ultimate identity even in this scheme), and everyone would cut off their web of trust at domains hosting spammers or virus writers.

With this security problem solved, you can now turn to traffic problems that prevent adjacent networks from connecting. Why would someone pay for the bandwidth they need, when they can connect to their neighbor's higher-bandwidth system cheaper and leech off of them? Well, as long as we're redesigning the core protocols, why not include traffic shaping as a core feature? If you know where packets are coming from, and you have a registry (remember the LDAP servers above?) tracking all of the networks you are connecting to, it would be pretty easy for routers to centrally update how much traffic is coming from where. You could then sign enforcable agreements with your neighboring network as to how much traffic each of you will allow to be routed through your network. At that point you've reestablished the ability to mesh networks and have thus addressed the fundamental routing weakness by deemphasizing the backbone providers.

One last thing, since we're redesigning the core protocol. There should be two connection types that you can request: shared or dedicated. A shared connection would be similar to IP now - it just routes each packet individually. But a dedicated connection would attempt to establish a persistent channel to the end point. If each device along the way was configured to allow this, and if you were in the web of trust of each of these devices, you could maintain a persistent connection that could guarantee deliverability (and even to some extent predicability of time of delivery) at the cost of overall network efficiency.

OK, so with a lot of fleshing out, this kind of scheme would allow you to solve the underlying weaknesses of the network. As a nice side effect, a lot of the services currently built on top of the network would become unnecessary, or would become easier. The only problem, as I said, is that you'd have to replace the Internet and all that runs on it. It turns out that DARPA is considering just that. (Link via SlashDot)

Posted by Jeff at 09:28 AM | Link Cosmos | Comments (2)

March 01, 2004

Tivoli Access Manager and Oblix NetPoint do not Play Together

It appears that Kaiser Permanente might be following down the track of GM and the Federal Reserve in implementing a split authentication/authorization solution, using Tivoli Access Manager for controlling access to the network, and Oblix NetPoint for user management. GM did this first, and made it work, to about the same level that the Bureau of Indian Affairs can be said to work (that is to say, badly, and hardly as a service to its users). However, the implementation is prone to crashing, not suitable for high-availability, requires bastarization of your directory server (this is an Access Manager fault in general - you have to redefine inetOrgPerson if you have any custom attributes, which frequently breaks other apps that actually expect you to follow the standards), and has serious performance issues (again, this is a characteristic largely of Access Manager, though PDLink adds its own issues).

The Federal Reserve is currently implementing this (well, reimplementing; the first two attempts were not exactly stellar examples of IT success), and the issues to overcome are enormous. If you are thinking of putting in an access and identity management solution, please, please take my advice and do not hybridize Access Manager and NetPoint. Just pick one and use that (I would lean towards Oblix, but there could be various characteristics of your environment which would mitigate against that).

Posted by Jeff at 02:48 PM | Link Cosmos | Comments (0)

February 18, 2004

Software Design Errors

As a sometime software designer, it annoys me no end when I see software that has really fundamental errors. One of the fundamental characteristics of software is that applications (as opposed to, say, daemons) are run by users. If the application can be run by users, it can be run by more than one user. On a multi-user operating system, an application could theoretically be run by several users at the same time.

This leads to an interesting problem: how do you keep separate what's happening to an application when different people are using it? For example, if two users are running an email program how do you differentiate between the mailboxes of each user?

This is not a new problem, and was solved many, many years ago. (UNIX is 35 years old, and was not the first multi-user OS.) But still, application designers can forget this fundamental truth: more than one person might want to use the same application, and might want to use it at the same time. The basic design strategy to overcome this is to separate an application into up to three parts: the core, which doesn't change from machine to machine or user to user; the system-specific configuration which is different based on the configuration of the machine the software is installed on, but is shared by all users of that machine; and the user-specific configuration, which is specific to a particular user.

With that in mind, here are some MacOS X apps which are particularly annoying offenders. (Why MacOS X? It's what I use at home, so I'm seeing these issues. I suspect that Windows will have the same problem the moment that they come up with something like Exposť.)

Age of Empires II - When you install AoE II, it apparently creates some files in your user space. When you attempt to run the game as another user, the game just dies. If the program needs to store information on a per-user basis, it should check for that information at run time, and create it (with appropriate defaults) if necessary. Crashes are not user-friendly.

Eudora - Eudora creates a directory for each user, with their prefs, mail files and so forth. However, there is apparently some file which cannot be shared between users. If one user is running the application, no other user on the machine can do so. Eudora will crash in generally less than a minute for any user that was not the first to start running it. Crashes are not user-friendly.

AOL Client - Cannot be run by multiple users. It simply exits out with a non-useful message (that is, the error message displayed does not tell you what the problem actually was, that another user was already running AOL). It's arguable that AOL should not be run by multiple users at the same time, since they would be contending for hardware, though I have a hard time believing that it would be difficult to allow for this, simply by running a daemon (launched with the client the first time only) to schedule requests to the network ports in question. Error messages which don't identify the problem are not user-friendly.

Those are a few I've noticed. I'm sure that there are many more. The lesson here is that if you are doing software design, you should avoid cheap shortcuts like assuming that you have the system to yourself, and the user has your application to himself. Otherwise, you look stupid when other software designers run your apps.

Posted by Jeff at 12:40 AM | Link Cosmos | Comments (0)

February 17, 2004

Fibre is Good for You

Soon, very soon, Verizon will apparently be installing fibre to the home in Keller. Drool...they should be wrapping up just in time for me to be in Keller to switch my server over to fibre (if they offer a static-IP service, that is, and aren't stupid about it).

Then I'll be able to waste time much more efficiently.

Posted by Jeff at 10:59 PM | Link Cosmos | Comments (1)

February 03, 2004

Identity Management Architecture

It's odd that I write so little about computers, given that shepherding large, complex computer systems from concept to operational reality is what I do for a living. For a taste of the project I'm on right now, Windley's Enterprise Computing Blog has a recent entry on Identity Management Architectures. My current project happens to be the implementation of such an architecture.

Posted by Jeff at 07:18 PM | Link Cosmos | Comments (0)

January 04, 2004

Security????

This is brazen.

Posted by Jeff at 09:39 PM | Link Cosmos | Comments (0)

December 11, 2003

Internet Tectonics

It appears that the UN wants to take control of the Internet. (And they are already acting predictably anti-freedom about it.) Clearly, this would presage a major shift in the way that the Internet works, and I'm inclined to think that it won't be a change for the better.

But it's also irrelevant, in a way.

You see, when I got on the Internet in 1988, there wasn't a world wide web. We kept notes of the useful FTP sites where we could get software and information, and later there was the short-lived gopher, but communication similar to what the web provides was then accomodated through email lists and network news. (Network news is now pretty useless - more chatterbox than information or entertainment conduit - but mailing lists are still with us.) This was not long after the introduction of DNS - which replaced unwieldy lists of host-address mappings in text files - and was also before the consolidation of Internet backbones and the widespread use of firewalls.

I said all of that as preface to this statement: the Internet is not magic, and because of that, it is not static.

Let's say that the powers that be decide to censor the Internet, or to tax it heavily. For that matter, let's say that spam levels keep increasing at the present rate. In that case, the Internet as we know it would become less and less useful. What would happen? Would we just accept it?

I have no doubt that many would, because to them the Internet is magic. But in reality, the Internet is a set of INTERconnected NETworks. In other words, I have a network in my home, and it connects to Verizon's network. (I can't justify having a separate connection to second network, so my network connects to only one other.) Verizon's network connects to many, many networks, because Verizon's network is a backbone on the Internet. (That is to say, it exists primarily to connect other networks, as opposed to my home network, which exists to connect my computers to each other).

I don't have to connect to Verizon. I could connect to several backbone providers, singly or in combination, if that was the best deal for me. I could, in fact, buy a dedicated network connection to, say, my friend Nathan's network, another to my parents' house, another to my wife's parents' house, another to amazon.com, and so on. Clearly, this is less efficient than connecting to a backbone, which connects to other backbones, which in turn are connected to the leaf networks of interest to me. It also requires a higher level of skill at each of those end points than connecting via backbones and local providers (which local providers offer assistance to their users as part of the fee).

But if it was sufficiently onerous to me to use the public Internet, I could create a private internet, connecting those networks that are of use to me, provided that those networks agree to have me connect to them. (It would have to be pretty onerous to get me to shell out for that many dedicated data lines.)

Assuming that I was able to raise the capital, and was not otherwise employed, I could in fact start a backbone network, and set my own rules. I could use the existing protocols and equipment and software, or I could set up my own. (For example, the IPv4 addressing scheme is too limited, email has no built-in method for verifying the sender, there is no standard on-the-wire encryption, there is only a primitive concept of trust (you trust the router on the other end of the line; that's it), and so forth - any or all of which could be fixed, at some cost.) Presumably, in order to get people to attach to my backbone, I'd have to offer something that they can't get from the public Internet, and it would have to be something that they need enough to put up with having two connections (one to me and one to the public Internet), or only connecting to me, or connecting to me and relying on some translator somewhere to bridge their traffic to the public Internet as needed (with associated performance hit and loss of function).

So let's say I were able to come up with a series of changes to the basic foundations of the Internet that were a compelling alternative to the public Internet, and one part of that package would be that every end-to-end connection were encrypted to hide the content. In that case, it would not be possible to censor individual content (even the protocol in use could be hidden, with proper design). Governments or other organizations would have a choice: block all traffic to/from the new internet, or allow it. (There would be another large advantage as well: cracking computers remotely would become much, much harder, and in turn open source routing would become reasonable again.)

I suspect that, should the Internet get too disconnected, censored, overloaded or otherwise not useful, some smart person somewhere will come up with the details to make this kind of situation work.

The Internet doesn't really route around censorship as damage now, nor could it survive the nuclear war it was designed to survive, but sufficient censorship or disruption could give rise to a new internet that would survive those circumstances.

Posted by Jeff at 01:01 AM | Link Cosmos | Comments (1)

August 13, 2003

Technical Support

I am not going to tell you about my current technical support woes, because you've heard it all before. I will just say this instead: if you are, say, Microsoft; and if you buy a company, say Connectix, with good products like, say, Virtual PC; and if you basically hang up on the customers when they call support - not even having the decency to hang up in person, but letting the automatic system hang up for you; and if you right before hanging up tell them to go to your support website; and if you then make it impossible for customers to get support on the purchased product over the web because the product doesn't have one of your product numbers; and if you provide no mechanism for customers to say "Hey, I'm trying to get support but I can't even get to the point where I can try to get support"; then you can forget about getting any money out of me to update that excellent product, which I otherwise might have done at some point, when I start using it again regularly.

I hate companies which have the easiest-to-use sales systems in the world, combined with the most impossible-to-use (and expensive if you don't want it to be impossible) support systems. It's basically telling your customer "Thanks for the money, but don't bother us when we've screwed up." And it's not a way to get my money.

The desktop computer industry contains an overabundance of such companies.

UPDATE: OK, the reason that I use VirtualPC is to run Project and Visio, which don't have Mac versions. When you're doing IT work, people expect you to have Office, Visio and Project, and are stunned when you don't. When you are doing this on contract, it's not good for business to tell your clients to pick another format to send you. VirtualPC has the added bonus that you can play PC games on the airplane, and run the occasional Windows-only program you just need to use once - like the console for Checkpoint firewall.

I am most likely about to start doing contract work again, and so would have most likely upgraded VirtualPC were it not for the lousy support I've gotten. So instead, I'll be buying ConceptDraw to replace Visio, and FastTrack Schedule to replace MS Project. (I currently own both Visio and Project.) So MS loses a potential upgrade sale of $100 (plus future upgrades of VirtualPC, Visio and Project) and two other companies get a total of $550 plus future upgrade revenue. All because of an inability to get minimal service on a product.

Posted by Jeff at 05:20 PM | Link Cosmos | Comments (4)

July 17, 2003

Eliminating Spam

Steven Green points to this John Dvorak article, and asks: "can some of you smart network-type people tell me if any of these ideas [for reducing spam technologically] are doable without needing an entirely new email system and software"?

I'm not going to go into detail on all of Dvorak's proposals, but I will say that he misses the point, for an entirely good reason. The Internet was built as a survivable network, meant in fact to survive nuclear strikes on some of its nodes and still keep going. It was also designed to assume security: all of the users would have to be authenticated by a node before obtaining access to the network, and all of the nodes were trusted. This was an excellent architecture in 1988 (when I first got onto the Internet), because all of the users were accountable for their actions - to their employer, or their university, or their government agency, etc. As a result, abuse was a minor problem (mostly incoming freshmen at colleges) and easily dealt with. But the system dealt with abuse - the human part of the system, not the network part. You could actually be shamed off of the Internet back then. Removing the human enforcer of netiquette and good practice was like giving the government the power to raise almost unlimited revenue from a very small proportion of the population: corruption and abuse exploded. Open source routing - where everyone passed traffic by the shortest route, so that my traffic could go right through IBM's internal network if that was the fastest way to the destination - was the first to go, replaced by a few backbone networks, with the multiply-connected large private networks protected by layers of firewalls and large IT staffs. (This increased the brittleness of the network at the same time that it increased security.)

So how do we fix this? We cannot build a safe, secure and reliable system on top of an inherently insecure, self-regulating and untrustworthy network. We would have to build a new network from the ground up. And to do that, we'd have to scrap everything except the physical layer - the NICs, wires, routers, bridges, modems and so forth would be all that's left. The intelligence of the network would have to change, some of it drastically.

The first problem to solve is design: how do you create a secure, trusted network, which at the same time still allows connections from anywhere to anywhere, using any protocol, as the default? How do you do this without either excluding people or forcing a central controlling agency (brittle, arbitrary and powerhungry as any bureaucracy) or limiting the ability of people to use the network in reasonable ways? How do you manage traffic in such a way that the network can be flexible, without overwhelming small companies who are multiply-connected by passing external traffic through their networks? How do you provide authentication and authorization, in other words, to a global network using only local resources?

It turns out that if you are willing to start with a blank page, it's not that difficult. The major issue to be solved is trust: how do I know whom I'm talking to? Since you don't want a brittle network, that will fall apart when something happens to a small number of nodes, the only trust model that works is to authenticate yourself to some local authentication source. For example, an ISP or a company would provide a directory which lists all of their users, and contains the information necessary to trust that user. Each node, then, has to trust its neighboring nodes. (This is already the case today, in that you cannot establish a physical connection to another node without being their customer or their provider, or entering into an agreement to do so.) Each node would need to cryptologically authenticate itself to its neighbors, and vice versa, and each user would have to authenticate themselves to their node. No node would pass traffic that did not include its partner node's connection key in the message portion of the packet, and no node would alter that information.

On the good side, this lets any traffic be traced back to its source. You cannot fake traffic as being from a node that you are not from, because the network will not pass on the message unless it has authenticated the upstream source - that is, you - and your information is in the packet header. Therefore, you could put preceding node information in the header, but it would be bogus and the trace would effectively stop being verifiable at you, the sender. On the bad side, this would dramatically increase the amount of traffic on the Internet (by increasing the size of any given packet) and would slow down all traffic, bandwidth being equal, because of the overhead of nodes authenticating to each other and the larger byte-count of a given data set.

Let's say, though, that we were to use that or some similar measure of trust to guarantee that the network was trusted. Now, we still have a whole host of problems to solve, because the IP spec would have to be rewritten at a pretty fundamental level. This means that the NICs would need updated firmware, or for cheap cards that have their firmware burned into the hardware, the whole NIC would have to be replaced. Then, you would have to rewrite the network stack to take account of the new protocol. You'd have to rewrite all of the protocols like UDP, FTP, SMTP, POP, NNTP, LDAP, SSH and the like. Some of these would be huge efforts, while others would need few or no changes. The OS network stacks would have to be rewritten for each OS, and some applications would need to be rewritten as well, if they deal with the network information at a low level.

All in all, it would be a huge load of work, and not likely to get done as an organized effort. The better way to do it would be to set up such a network privately, amongst friends as it were, and expand that to their friends, and their friends, and so on, and so on... And if you were to gateway to the global internet, you'd lose a lot of the benefits right there.

So in practical terms, I don't think it's going to happen.

Posted by Jeff at 01:36 PM | Link Cosmos | Comments (1)

July 02, 2003

Can I Just Have the Wire, Please?

Aubrey's cable modem problems reminded me of something that's bugged me for years. Why is it so difficult to find high-speed Internet connectivity without non-network services?

Internet service breaks down into two parts, network and service. The network piece includes physical connectivity, addressing, routing, and (optionally) naming services (DNS and reverse DNS). The service piece includes email hosting, web hosting, file service hosting (FTP in particular) and so on - in other words, all of the things that you do that need a server (including, say, hosting a weblog). Note that for web browsing, instant messaging and the like, there is no need to have a service - that comes for free with the network, because the applications involved have every piece running on your system.

Now, it's certainly true that most people will need both network and service pieces. Even someone who only wants to surf the web and do email needs a service provider for email. However, there are numerous service providers perfectly willing to provide these services (frequently for free or at very low cost) to anyone able to connect to them.

What I cannot find is an abundance of companies willing to provide a high-speed network connection - and nothing else required - for a reasonable price. If you want a high-speed connection, you can lease a T1 or better (for about $1200 per month with no restrictions, with fractional connections not being much cheaper - and in this case, for example, no faster than a good modem), or you can get some form of DSL or cable modem.

The problem is, getting a good DSL or cable modem connection is still expensive for a home user. I pay about $100 per month for mine, to get a commercial DSL connection at 768K down/384K up with 5 static IP addresses and no other services. The reality of it is, the marginal cost to the provider (Verizon, in my case) is almost certainly less than $5 per month. (I have worked with setting up services for several ISPs, and the large cost is in providing the services, not the network.) I could chop that in half, keep the data rate, add email and web (and, I'm sure, other) services, and lose the static IP addresses. At that point, I'd have to go out an pay additional costs to run this weblog, because Verizon's home user plans don't allow for what it would take to get MT to run, and would have significantly less control over my other server-based services (which I currently provide for myself).

Certainly there is a shortage of IPv4 addresses, but they are available, and the big providers (including mine) have them. Most people would be fine with a pure network connection with dynamic addresses, and a service plan from either their provider (more convenient) or some third party (likely better service and cheaper). For those few who want static addresses, they could be charged more. Still, the connectivity charge in a major city for high-speed service should be on the order to $20 to $30 at most, presuming that the user is getting less than or equal to 16 (raw, 14 usable) addresses, and that includes a hefty profit. Add another $10 to $20 per month for services from a third party (or $30 to $50 per month for services from the line provider) and you'd still be able to save a bundle, while the providers would still make a profit.

So why is it so difficult to get just a high-speed connection, with static addresses and no services? (OK, I suspect that it's an outgrowth of some FCC regulations creating a effective-monopoly or near-monopoly environment, but even with that it seems to me that someone would want to take the money they could get just providing Internet connectivity.)

Posted by Jeff at 11:21 AM | Link Cosmos | Comments (1)

June 04, 2003

Making up Numbers

It goes without saying that groups like the BSA and RIAA just make up the numbers they use to try to influence public policy. Actually, it doesn't go without saying, but it's been said better elsewhere. Slashdot has an item today on BSA's latest made-up numbers:

JakiChan writes "According to this story on Yahoo! news the BSA commissioned a study that decided that 39% of all business software is pirated, down from 40%. The decline is attributed to the BSA's enforcement techniques. 'The piracy rate was calculated by comparing the researchers' estimates on demand with data on actual software sales.'" In other words, some guys sat in a room and decided that people probably wanted to buy ten copies of software, but only five were sold, so the piracy rate must therefore be 50%. By a similar process we can calculate that 99% of all ocean-front homes are pirated.

Posted by Jeff at 10:32 AM | Link Cosmos | Comments (0)

April 17, 2003

Death to Clippy

Aubrey Turner points to a really funny collection of images.

UPDATE: Turned off comments, because this was basically an entry point for spam posts, for some reason.

Posted by Jeff at 03:04 PM | Link Cosmos | Comments (0)

March 26, 2003

Theft? Uh, No

Steven Den Beste is apparently having problems with images hosted on his site being linked to by other sites, rather than copied and hosted from the other sites. In response, he set up his server so that people getting images from his site via a referral not from within his site will instead get this image:

(and yes, I'm aware of the irony of referring you to his site for the image).

I take issue with the image. The word "theft" is much over-used these days, but I expected more from Den Beste, who generally thinks things through fairly well. (OK, except about computers, and maybe this is where the cognitive dissonance comes from.) If I call you on the phone, I am theoretically depriving you of the use of your phone line for other purposes for the time you are talking to me. But it's not theft, because in reality you have the opportunity to not answer. In effect, what Steven is doing is telling his server not to answer the line if a request comes in for a big image. Fair enough. But it's rudeness, not theft, to link to another person's large images when they don't want you to do so.

Posted by Jeff at 01:05 PM | Link Cosmos | Comments (10)

Corporate IT and Outsourcing

It's worse than this makes it out to be. First, my credentials, because I don't have sources to cite: I've worked in IT for 12 years, almost the entire time in enterprise-level jobs. I've worked as a consultant, a manager, and an admin in various for-profit companies.

OK, that said, the state of IT is worse than Bigwig's article makes out, because he only considers outsourcing as it compares to internal IT departments. Internal IT departments are themselves very inefficient. For example, I worked on a project once which spent a year and millions of dollars to build a production environment that was ill-conceived to begin with. When it was finally working and doing what it was supposed to do (for more money and in more time than was actually necessary, but at least it worked), it was immediately taken out of production because the new VP decided to do things differently. This is more typical than not. It has been said that 90% of IT projects fail, and as far as I can tell, that is true.

So why do big IT projects fail? Generally, they are political, which means that cancelling them is also a political, rather than a business decision. They tend to be thought of by the corporate sponsors in business terms without regard to technical considerations, and by the implementors in technical terms without regards to business considerations. There is a strong desire to shave short term pennies by spending long term dollars.

Another example: I once worked on a call and problem management system for doing technical support of software products. The support contracts were fiendishly complicated. Within a few seconds, the person on the phone had to be able to tell the caller if he could or could not place a support call as that person from that company for that product on that computer. This was in the mid-to-late '90s, so the computing power of the enterprise-class machines we had then is somewhat less than the laptop I'm typing this on. But the product we developed for internal use worked, worldwide and for many products and fast enough. Because it worked so well, we were asked to expand it out into the other software support lines within the company. Well, one group had a problem-management system from the time when buying a computer meant that you had lifetime support for free, and it ran on mainframes and cost some $40M every year to run. It was so heavily invested, and had so many applications written against it, that it was deemed too important to get rid of. But their application couldn't do the entitlement piece of support (are you contractually entitled to get support?), so it was decided that their mainframe-based problem management system would be used with our UNIX-based client server entitlement system (the problem management part of our product was to be left to wither and die). To do this, a client had to be developed, taking a year and a half, to talk to both systems. From a user-interface standpoint, the mainframe-based call management system was so bad (and the graphical client basically was a screen scraper for that app) that the support personnel had to increase in order to field the same number of calls. The entire app slowed down, because of the delay of going to two systems instead of one. With the amount of money it took to run that system for one year, we could have finished development of our system (it was rewritten from scratch to work for the whole company, and to eliminate accumulated cruft) and supported it for a decade with constant development. This is common, though: use the more expensive system in worse ways, because we already have it, rather than replacing it with something better. Note: this was considered a great success for the IT group of which we had by then become a part.

To put this in non-IT terms, imagine if you had a car which carried 2 people, got 8 miles to the gallon, and cost $40000 per year in maintenance. Imagine further that you could replace this with a car costing $20000, which carried 6 people comfortably and got 35 miles to the gallon. No-brainer, right? Not if you are in corporate IT, because maintenance comes from a different budget than acquisitions, and it is almost impossible to repurpose funds. (I feel for government heads of department; I really do.)

So what works? Generally, the products I've seen developed by corporate IT which work are those which are developed under a single management chain, generally by a small group of really good people using no formal development methodology, where they are trying to solve real user problems, as opposed to big-picture problems (like, optimize our supply-chain management). These products can grow over time to encompass a larger number of problems for more groups. They tend to go astray when they get big enough and important enough to become politically useful to other management chains. The fight for control is done over requirements, and it results in the biggest fish in the fight getting bigger, and the product getting more expensive and less useful over time.

It's not just me who sees it this way. Most of the IT people I've ever known (and I know a lot of them) see it similarly. (Two programmers at a company I know of were experts in a product. They were transferred to a group with a bigger-fish manager, and put to work on something completely outside of their expertise. The new manager didn't need their skills; he just wanted the headcounts to increase his importance.)

So why don't companies work this way, in general? Well, for two reasons. First, most people in charge of most companies have no clue of how to tie IT projects to business needs. Second, even when they do know how to do this, there is a tendency for projects which are efficient, useful and well-managed to become so important that the power-seeking managers with pointy hair and no clue end up taking them over to stroke their egos, then making bad decisions (if the takeover process was not itself fatal).

Not that I'm bitter, or anything. Time to go catch up on Dilbert.

Posted by Jeff at 12:33 AM | Link Cosmos | Comments (0)