Existing programming languages seem (mostly) up to the task of ordinary software development. The large variety of existing languages even allows you to choose exactly the features that you wish to use. So why would I want to create a new language?
Let's look more closely at the question of language choice. In the same way a logician would choose a logic system with which to explore certain statements about truth, you choose a programming language that is suited towards expressing certain concepts elegantly. You probably wouldn't use assembly language to explore concepts in artificial intelligence. But it's not just a question of "high level" versus "low level" languages. You probably wouldn't use Prolog as a scripting language. You also probably wouldn't use a spreadsheet's macro language to implement threads in an OS. But what's the deep reason why not? Because of what's missing in each of these languages, not because of what's present.
The classic choice of high level versus low level usually is based on speed, and possibly on interfacing to existing systems. If it were fast enough, I know a lot of people that would use Smalltalk to write interrupt handlers (actually, a famous Tektronics oscilloscope ran Smalltalk internally). If the obstacles of speed and the ability to interface with existing low level interfaces are removed, we should be able to do away with these error prone, difficult to use languages, or at least greatly reduce their frequency of use. Not that there's anything tricky about individual assembly language instructions (or concept), it's just that the number of instructions required to do something complex is usually too high to be practical, and because there is almost no assistance from the compiler (assembler) to catch a large class of common errors. For example, using a register prior to assigning it a value is an error which might be very, very hard to find.
Pretty much every high level language that has been written in the last 10-15 years has had the ability to interface with assembly code (or at least C). So this is much less of a barrier than it used to be... or is it? Very often there are serious restrictions when interfacing between different languages. For example, all Smalltalk systems I'm familiar with can call C code (in platform specific dynamic libraries), and they can even be called back from the C code, but there are seriously restrictive caveats. For example, the thread model available at the C level usually interferes with garbage collection. The Smalltalk implementation is not free to move objects around in memory if there is a chance a C routine is pointing to some Smalltalk object. The alternatives are to (1) prohibit multiple threads, (2) prohibit garbage collection during calls to C, or (3) prohibit C from pointing directly to Smalltalk objects. None of these solutions is particularly palatable. An alternative would be to allow C to be garbage collected as well, but this has consequences of its own. Its lack of serious type semantics prevents any kind of copying garbage collection, so conservative non-copying collection seems to be the limit of what can be done. The level of fragmentation this induces is unacceptable for a Smalltalk system, so there doesn't seem to be any common ground.
Solution: Why not simply remove the motivation for writing part of the system in C?
Goal 1: The language should be high level but as efficient as C, to avoid any motivation to implement parts of systems in a low level language that might introduce impedance mismatch.
This has far reaching effects. For example, an operating system could be written in a high level language without fear that it would be too slow to be worth the extra reliability and maintainability due to its implementation language. Oberon is an example of this, and Taligent was a step in the right direction, but I fear both had more to do with the availability of "fast enough" hardware than with efficient high level languages.
Let's get back to the premise that high versus low level is not the only criterion for language choice. One other criterion that's useful is how easy or difficult it is to use (or implement) a specific language feature. In Smalltalk I find much of the total development effort in large systems tends to be (1) user interface, and (2) database interface. Let's ignore (1) for now. Relational, and even object databases, require a lot of care to use, as vendor limitations on transaction boundaries (nestable, concurrent) and imperfect implementations of proxies and stubs (==, class, instVarAt: are wrong) leads to compromised (or just plain incorrect) code. This is not related to high level versus low level, because it's even harder to interface to databases in low level languages. A usual technique in low level languages is to introduce new syntax and preprocess all source files prior to compilation. This is not always a bad solution, but this usually introduces new keywords and restrictions. Since it's important to have some level of checking of what's going on by the compiler - the preprocessor in this case - the preprocessor must be able to parse the language in question. Not only that, but it must be able to correctly interpret all the current compiler's nonstandard directives (at least to ignore the ones that don't matter), and all the other preprocessors' code. Persistence isn't the only concept that might require preprocessing, after all. Instead of having all these preprocessors trying to make up for deficiencies of the syntax, why not have the syntax flexible enough to describe pretty much any concept seamlessly enough to make it look like it was always there?
Goal 2: The language should be capable of being extended seamlessly.
A way to ensure this goal is to have all syntax be expressed as extensions of a tiny kernel syntax. That makes sure that new syntax extensions will have the same power of expression as the rest of the syntax. If it was enough for the main syntax it should usually be enough for the extensions.
To this end, the grammatic scheme I chose was a tiny core of syntax (blocks, assignment, etc). User-defined operations could be any sequence of words, symbols, and placeholders for arguments. This scheme can allow expressions that are ambiguous. If I tried to prevent ambiguity at the syntax level, I believe I would be stuck with a basic grammar too weak to express even my core library. So I shunned LR(1) and its ilk, and allowed the most difficult-to-parse grammar possible from day one. Problems using such a difficult grammar are solved by other means, such as precedence declarations and pruning via type checking. The basic idea is that even though parts of a statement may be inherently ambiguous, each statement as a whole must have only one possible parsing that satisfies the precedence rules and type system (otherwise it will fail to compile). Since the chosen grammar framework is the least efficient grammar to parse (naively), a lot of effort went into producing an efficient parser. That's my job as a compiler writer, though, so that Avail programmers don't need to worry about it.
In order to satisfy goal 1 (efficiency), I had to find an advantage I could leverage. I couldn't just translate the code into C-quality machine code, because I wanted type safety, pointer safety, garbage collection, and clean semantics. If I just translated it directly I would lose reflection and some other things, but worst of all, I would introduce a bad gradient for inducing quality: The programmer would have an incentive to do C level optimizations by hand. Optimizations like loop hoisting, where an expression is lifted out of a loop, are often the kinds of things compilers can do, but you can't rely on the compiler to do it correctly. Anyone who has programmed in C++ for more than a year knows how many flaws are present in C++ compilers, due to the difficulty of these optimizations. The optimizations are not the hard part - it's identifying when it's safe for the compiler to use them. So the leverage I was looking for seemed to have something to do with clean semantics. Since I wanted portability (almost always a desirable goal for any kind of system, sometimes more important than other times), I had to take advantage of something present in the "object level" view, or at least somewhere above the "raw-storage" level (where most of C's optimizations happen). Since many of the optimization errors I had seen seemed to be errors of ordering of side-effects, I concluded that the solution was to use immutable objects.
Code manipulating immutable objects is much easier to optimize, due to the absence of side effects. I looked at functional programming (after bumping into the field for the first time), and decided that what they were doing was Good, but there seemed to be equally Good reasons why most functional languages were extended to allow side effects. So instead of designing Avail as a functional language that was later extended with side-effects, I chose a hybrid approach that admitted, from the beginning, the need for side effects. But I restricted them to a single data type, the variable (also called a container elsewhere in this documentation).
Unlike a "const" in C++, which is a declaration that someone with a reference to an object will not modify the object, I chose a stronger approach that says that nobody can modify it. At that point I realized immutable objects no longer required identity - two objects that are completely indistinguishable may as well be considered one object. All of a sudden I had clean comparison semantics: Immutables compare by value, and variables compare by identity. I could also hash things cleanly: Immutables hash based on their contained values, and variables hash to a random number permanently associated with that variable. Since two objects that were equal will always be equal, and since the hash value for an object could never change, I was suddenly able to implement sets and maps as primitive types. That meant the semantics of a set could be taken advantage of by the optimizer. Sets containing only constants could be constructed at compile time. Operations on such sets could sometimes even be "folded" (executed once, at compile time). This was exactly the leverage I was looking for:
Goal 3: The language should support high level optimizations.
Quite a few high level optimizations are possible. For example, starting with a map m, if you construct a map m' that is just like m but with k->v in it, and you look up k in m', you should get v. That's easy to believe (and true), but there are plenty of opportunities for coding it incorrectly in a complex optimizer. It's also likely to be accidentally omitted when writing an optimizer, so it would be nice if a user could safely add this rule to the system. So I introduced a new goal:
Goal 4: Optimization rules can be added by programmers, as long as they provide a proof of correctness that can be mechanically verified.
I haven't built much infrastructure for this goal yet, but eventually there will be a collection of axioms and theorems describing the primitive data types.
Earlier I said that much of the development time in Smalltalk is directly concerned with the user interface. Part of the problem is the implementation of the Observer pattern within the Model-View-Controller paradigm (it really only concerns the Model and the View). In order for a domain object to be directly used as a model for a view, it must broadcast changes to its state for all dependent objects (views) to intercept and deal with (refreshing the display). To trigger such a broadcast, every write to an instance variable must have extra code near it to trigger this change propagation. Since this adds overhead, Smalltalk doesn't do this automatically, so programmers have to go out of their way to deal with this. Worse, instances of some general purpose classes (for example, collections) are often used as models, and are expected to broadcast changes. Since the built-in collection types generally don't broadcast these changes, the basic views fail quite often. I've been describing VisualWorks Smalltalk, but VisualAge has basically the same problem. Smalltalk/V didn't even use the Observer pattern if I recall, and I don't have enough experience with any other Smalltalks to comment on them at this time. I doubt the situation is much better, however.
Since all objects in Avail except variables are immutable, this problem can be isolated to updates of variables. That steers us directly towards a solution in which variables have the responsibility for implementing the observer pattern. I generalized this to solve a few unrelated problems and came up with the following:
Goal 5: Any variable can have read-daemons and write-daemons attached to it. The read-daemons execute prior to retrieving a value from a variable, and the write-daemons execute just prior to storing a new value in a variable.
Besides supporting the observer pattern precisely, this generalization allows clean watchpoints and aspect adaptors, and even adds some support for an eventual animation-based user interface. And it only has a cost if it is used.