Grr now works under Windows! Another proof of GNUstep's versatility.

Screenshot attached, with the upcoming native WindowsUXTheme. Note the scrollbars, checkboxes, native in-window menus, native window-decorations for windows and panels...

For a fun Friday hack, this blog entry describes a program to generate a regular expression to recognize all the valid strings of integers in a given base, that is, a regular expression (regex) which accepts all in-range strings but rejects strings that would cause an overflow if converted.

This sort of regular expression could be used to verify a string won't cause a NumberFormatException when processed by one of the many text → number methods in the Java platform, such as Integer.parseInt(String s, int base). However, using a regular expression may be more expensive than attempting the conversion and catching the exception on invalid input; the regular expression discussed is possibly of more academic interest than practical significance!

An analogous regular expression for floating-point strings has been published in the javadoc for several releases.

First, is the set of strings in a given base that will convert to a 32-bit or 64-bit integer a regular language, a language that can be recognized by a regular expression? Yes, because ignoring sequences of leading zeros for a moment, there are only a finite number of strings that can be converted to integer values, one string for each of the 2³² or 2⁶⁴ values in the int or long types, respectively. All finite languages are regular, strings of zero of more leading zeros are regular, and concatenations of regular languages are regular. Putting those facts together, the entire set of convertible strings forms a regular language. This reasoning is not dependent on the base so it holds for all integer bases, including bases 2 through 36 supported by Java's libraries.

A core regular expression of billions of strings offered as alternatives ("1|2|3|...|2147483647") while fine from a mathematical perspective, would be too slow and awkward to use. Fortunately, the pattern of valid strings has sufficient structure to yield reasonably short and manageable regular expressions.

To start we will only consider decimal strings; additionally, we will assume only ASCII digits need to be recognized. Integer values range from

-2147483648

to

 2147483647

These strings both have 10 digits. Putting aside leading zeros for the moment, all strings of 9 or fewer digits are valid, with or without a leading minus sign. A regex to recognize strings with 9 or fewer digits is trivial; less obvious is how to specify the "ragged" 10-digit nonzero strings which are in range.

The core regular expression covering non-ragged inputs is

"(-)?"+			// optional leading minus sign
"0*"+			// leading zeros
"(\p{Digit}{1,9})"	// numbers with 1 to 9 digits

For the 10-digit strings, if the leading digit is less than the maximum digit, all other digits can have any value:

1(\p{Digit}{9})

If the first digit is at it's maximum, then if the second digit is less than its maximum, the third and subsequent digits can have any value:

20\p{Digit}{8}

Likewise, if the first and second digits are at their maximum, then if the third digit is less than its maximum, the fourth and subsequent digits can have any value. Continuing this pattern for all the digit positions:

1(\p{Digit}{9})|
20(\p{Digit}{8})|
21[0-3](\p{Digit}{7})|
214[0-6](\p{Digit}{6})|
2147[0-3](\p{Digit}{5})|
21474[0-7](\p{Digit}{4})|
214748[0-2](\p{Digit}{3})|
2147483[0-5](\p{Digit}{2})|
21474836[0-3](\p{Digit}{1})|
214748364[0-7]

This regular expression can be ORed with the regex for strings of 1 to 9 digits listed above. Finally, a separate case for the most negative value must be added:

-0*2147483648

All together, with some newlines for readability this yields:

((-)?0*((\p{Digit}){1,9})
|([0-1]\p{Digit}{9})
|(20\p{Digit}{8})
|(21[0-3]\p{Digit}{7})
|(214[0-6]\p{Digit}{6})
|(2147[0-3]\p{Digit}{5})
|(21474[0-7]\p{Digit}{4})
|(214748[0-2]\p{Digit}{3})
|(2147483[0-5]\p{Digit}{2})
|(21474836[0-3]\p{Digit}{1})
|(214748364[0-7])
)|
(-0*2147483648)

The ragged pattern for nonzero strings with maximum possible length will exist for bases that aren't powers of 2, for the Java libraries that includes all conversion radixes other than 2, 4, 8, 16, and 32. For powers of two, the pattern is more regular. For example, in base 16 the values range from

-80000000

to

7fffffff

so for base 16 the regular expression can simply be

((-)?0*(\p{XDigit}{1,7}|
	[0-7](\p{XDigit}{7}))|
(-0*80000000)

Generalizing the regular expression generation algorithm to any given base:

1. Create a regex for an optional leading minus sign ("-") and zero or more leading zeros, "0*."

2. For the base in question, generate the strings for MIN_VALUE and MAX_VALUE.

3. Find n, the length of the MAX_VALUE string. Signed strings of that base with (n-1) or fewer characters are all valid inputs; create a regex recognizing (n-1) or fewer digits.

4. Create a regex to recognize valid strings of length n.

5. Combine the above regular expressions.

6. To the above, concatenate a separate regex for the most negative value,

   -(0*)MIN_VALUE_STRING

Taken together, this regex will recognize exactly those strings convertible to the base in question.

Untested code implementing this algorithm is available for your viewing pleasure. Any further debugging, tuning, and enhancements are left as "an exercise for the reader," happy hacking!

 Last week Apple released their latest product destined to change the world (the iPad).  At least that's what they want us to believe.  Perhaps the biggest controversy over the thing is the lack of Flash capability.  However this being java.net I have to wonder out loud, where is Java capability, and more importantly why isn't as much controversy being raised over Java being missing? But I think we all can enumerate some reasons for both being missing.  And it's worth it for the Java community to ponder this issue.

A couple weeks ago I attended a meeting of the Silicon Valley Web JUG (yes: Java User Group).  (The Future of the Web According to Dion Almaer and Ben Galbraith)  A very interesting meeting with a great overview of advances in HTML5 with an eye on the great possibilities it holds.

The interesting thing is they began the evening with a question:  How many of you are interested in JavaFX?  A meeting of 100+ geeks in Silicon Valley who are associated with a Java User Group, you'd think a few of them would be interested in JavaFX.  One person raised their hand.  I think that says a LOT.

Their presentation said a LOT about why Java and Flash both are missing from the iPad and iPod, and why we shouldn't care about that functionality gap, and indeed should feel liberated at their absence.

The key is web components built using standardized web technologies (a.k.a. The Open Web).  That's HTML, XML, HTTP, JavaScript and that ilk.  Flash, not being standardized by anybody, is not part of the Open Web.  Despite Java having a standards body behind it and being treated/delivered by Sun as an Open Standard, it was never accepted by the tech/web community as part of the Open Web.  And.. uh.. JavaFX.. sheesh, that's nowhere near being treated/delivered by Sun as any kind of Open Standard, instead it's being treated as a proprietary product where Sun is the big gorilla.  Oh, wait, that's Oracle now.  Sigh.  In any case the Open Web should most certainly ignore JavaFX just like it calls for Flash to be eschewed.

By being based on Open Standards the Open Web has tooling available from many organizations and a rich ecosystem of experience and adoption.  

An issue traditionally with HTML+JavaScript was speed.  Javascript has historically been an interpreted only language and anybody who tried Java 1.0 knows how glacial that can be.  Lately some Javascript implementations have been developing JIT and bytecode interpretation capabilities that eerily echo the development of Java virtual machines.  According to Ben Galbraith some of those teams are staffed with former Java VM developers.  In any case it means faster HTML+JavaScript execution with more capability to provide a rich GUI experience using just HTML+JavaScript.  (FWIW I'm typing this in Google's Chrome browser)

This ain't the HTML+JavaScript of yesteryear.  This is a brave new world.  It seems in retrospect the promise of Flash and Java was speedier UI experience than the HTML+JavaScript of yesteryear, and that if the HTML+JavaScript of the future is good enough, then both Flash and Java will be rendered irrelevant.  Yes, sure, of course both Adobe and Sun have over a decade of virtual machine implementation experience.  But neither can achieve the tight browser integration that JavaScript can.

HTML vs. Flash: Can a turf war be avoided? That's an interesting article covering the current stance where Adobe is saying "HEY WAIT A MINNIT" about the lack of Flash in the iPad.  e.g. "We are now on the verge of delivering Flash Player 10.1 for smartphones with all but one of the top manufacturers," Lynch said, specifically mentioning the Nexus One as one such device and adding that the software also works on tablets, Netbooks, and Net-enabled TVs. "Flash in the browser provides a competitive advantage to these devices because it will enable their customers to browse the whole Web...We are ready to enable Flash in the browser on these devices if and when Apple chooses to allow that for its users, but to date we have not had the required cooperation from Apple to make this happen."  I happen to know for certain that Sun could say the very same thing about Java on iPhone/iPod/iPad.  

Where is the truth between these possible states:-

• The Web only has components standardized by standards bodies
• All the tools and components are completely open source under OSI approved licenses
• There is a mix of open standardized components, semi-open proprietary components and completely closed components (todays situation) 
• Every web site has its own incompatible standard (the fate we fortunately avoided several years ago)

??

(tangential digression) 

Open Source != Open Standard.  For a long time the hue and cry was for Sun to Open Source Java, and that would ensue a brave new era of wonderful harmony across the planet or some such.  In practice Sun didn't quite open source Java, instead it created an Open Source project on a specific implementation of Java (OpenJDK) but "Java" (in my mind) was explicitly not open sourced.  As a result while the resulting situation was much better than before the wonderful era of harmony did not ensue.

In any case an Open Standard can be delivered by closed source software so long as it obeys the standard.  An Open Standard still allows wide use of the software and a huge amount of freedom in lots of practical forms of freedom.  But Open Standards don't allow things like forking that Open Source explicitly allows.  

A small project I worked on during JDK 7 milestones 05 and 06 was the introduction of a java.util.Objects class to serve as a home for static utility methods operating on general objects (6797535, 6889858, 6891113). Those utilities include null-safe or null-tolerant methods for comparing two objects, computing the hash code of an object, and returning a string for an object, operations generally relating to the methods defined on java.lang.Object.

The code to implement each of these methods is very short, so short it is tempting to not write tests when adding such methods to a code base. But the methods they aren't so simple that mistakes cannot be made; replacing such helper methods with a common, tested version from the JDK would be a fine refactoring.

The current set of public methods in java.util.Objects is:

• static boolean equals(Object a, Object b)

• static boolean deepEquals(Object a, Object b)

• static <T> int compare(T a, T b, Comparator<? super T> c)

• static int hashCode(Object o)

• static int hash(Object... values)

• static String toString(Object o)

• static String toString(Object o, String nullDefault)

• static <T> T nonNull(T obj)

• static <T> T nonNull(T obj, String message)

The first two methods define two equivalence relations over object references. Unlike the equals methods on Object, the equals(Object a, Object b) method handles null values. That is, true is returned if both arguments are null or if the first argument is non-null and a.equals(b) returns true. A method with this functionality is an especially common utility method to write, there are several versions of it in the JDK, so I expect the two-argument equals will be one of the most heavily used methods in the Objects class.

The second equivalence relation is defined by the deepEquals method. The equals and deepEquals relations can differ for object arrays; see for the javadoc for details. Equality implies deep equality, but the converse is not true. For example, in the program below arrays c and d are deep-equals but not equals.

public class Test {
   public static void main(String... args) {
       Object common = "A string in common.";
       Object[] a = {common};
       Object[] b = {common};
       Object[] c = {a};
       Object[] d = {b};
       // c and d are deepEquals, but not equals
   }
}

A third equivalence relation is the object identity relation defined by the == operator on references, but since that is already built into the language, no library support is needed. Identity equality implies equals equality and deepEquals equality.

Next, Objects includes a null-tolerant Comparator-style method which first compares for object identity using == before calling the provided Comparator. While Comparable classes aren't as widely available as the methods inherited from java.lang.Object, Comparable is a very useful and frequently implemented interface.

Objects has two hash-related methods. The first is a null-handling hash method which assigns null a zero hash code and the second is a utility method for implementing a reasonable hash function for a class just by passing in the right list of values.

The toString methods provide null handling support, in case of a null argument either returning "null" or the provided default string.

Finally, there are two methods to more conveniently handle null checks, intended to be useful when validating method and constructor parameters.

Taken together, the methods in Objects should lessen the pain and tedium of null handling until more systematic approaches are used.

The Objects API was shaped by discussion in various threads on core-libs-dev in September and October 2009. Several other bugs were also fixed as a result of those discussions, one adding a set of compare methods for primitive types (6582946) and another to consistently define the hash codes of the wrapper classes (4245470).

Thanks to Kelly, the new component delivery model for jaxp and jax-ws is now available in both JDK 7, as of build 72 of milestone 5, and OpenJDK 6, coming in build 18 (6856630).

As described previously, the JDK build no longer tracks a copy of the jaxp and jax-ws sources under version control. Instead source bundles from the upstream teams are used. The jaxp.properties file in the jaxp repository contains the default URL from which the source bundle is downloaded as well as the expected checksum for that file. The analogous setup is used for jax-ws in its repository. To avoid downloading another copy of a bundle or to try out an alternate bundle, several variables can be set in the ant build of one of the repositories. For jaxp,

jaxp-repo-directory$ ant -f build.xml \
-Ddrops.master.copy.base=path-to-drop-directory \
-Djaxp_src.bundle.name=name-of-source-zip-bundle-in-drop-directory \
-Djaxp_src.bundle.md5.checksum=md5-of-source-zip-bundle

If changes local to the JDK are needed, patches can be applied from the new patches directory in the two repositories. For example, patches are a mechanism that could be used to deploy security fixes until a new source bundle with those fixes was externally available.

With this new delivery model, I look forward to low-overhead and coordinated updates to jaxp and jax-ws in OpenJDK 6 and JDK 7.

A possible future consolidation would fold the build logic in the now vestigial jaxp and jax-ws repositories into the main jdk repository.

Java developers are familiar with dynamic linking. Class files are a kind of intermediate format with symbolic references. At runtime, a class loader will load, link, and initialize new types as needed. Typically the full classpath a class loader uses for searching will have several logically distinct sub components, including the boot classpath, endorsed standards, extension directories, and the user-specified classpath. The manifest of a jar file can also contain Class-Path entries. Together, these paths delineate the boundaries of "jar hell."

For many years, modern Unix systems have also supported dynamic linking for C programs. Instead of a classpath, there is a runpath of locations to look to for resolving symbolic references. Like the classpath, the full runpath has multiple components, including a default component for system facilities (analogous to boot classpath), a component stored in a shared object (analogous to jar file Class-Path entries), as well as an end-user specified component (analogous to the -classpath command line option or CLASSPATH environment variable). The details of linking on Solaris are well explained in Sun's Linker and Libraries Guide. Other contemporary Unix platforms like Linux and MacOS have similar facilities, although the details of the various commands differ.

One of the tasks the JDK's launcher has handled is setting a suitable runpath for the JVM and platform libraries. Historically a runpath was needed to link in the desired JVM, such as client or server, and other system libraries. The client JVM and the server JVM are separate shared objects which support the same set of interfaces; by interpreting the command line flags the launcher selects which JVM to link in. Operationally, the linking is initiated by the Unix dlopen library call. So that the caller of the java command did not need to set LD_LIBRARY_PATH, after selecting the JVM to run the launcher would modify the LD_LIBRARY_PATH environment variable by prepending the path to the JVM shared object (and paths to other directories with JDK native system libraries). However, the runtime linker only reads the value of LD_LIBRARY_PATH when a process starts. Therefore, to have the new value take effect, the launcher would call an exec-family system call to start the process anew. Such re-execing to set LD_LIBRARY_PATH is not recommended practice on Unix systems.

The re-execing to set LD_LIBRARY_PATH had a number of unpleasant consequences in the launcher code. There is only a narrow path to pass information between the exec parent and the exec child, such as by modifying environment variables, which is generally discouraged. To decide whether or not an exec was needed, the launcher checked whether the prefix of LD_LIBRARY_PATH had the expected value; if it did, no exec was done for that purpose and infinite exec loops were avoided. Presetting LD_LIBRARY_PATH to the right value before calling java could thus be used to suppress the exec. There were also complications with correctly supporting multiple LD_LIBRARY_PATH variables on Solaris¹ and handling suid java executions on Linux.²

The proper way to accommodate such dependencies is not to set LD_LIBRARY_PATH but rather to use the runtime linker facilities analogous to jar file Class-Path entries; the facility is the $ORIGIN dynamic string token for the runtime linker. As the name implies, $ORIGIN is expanded to the path directory of the object file in question; thus relative paths to other directories can be specified. Therefore, as long as the directory structure of the JDK and JRE are known, $ORIGIN can be used to record any necessary dependencies.

For some time, the JDK build has actually used $ORIGIN in creating its native libraries. Therefore, it may have been the case that LD_LIBRARY_PATH was not actually needed. However, verifying that LD_LIBRARY_PATH was not actually needed would require building an exec-free JDK on all supported Unix platforms and running tests that exercise the all libraries in the directories no longer added to LD_LIBRARY_PATH. The engineering for Kumar's purge of execing for LD_LIBRARY_PATH was generally straightforward: deleting the the LD_LIBRARY_PATH-related code in the Unix java_md.c file and doing builds on all platforms. Most of the effort of getting this fix back involved running tests to verify everything still worked. The testing revealed an unneeded, troublesome symlink that was removed at the same time LD_LIBRARY_PATH usage was purged.

While the launcher no longer execs to set the LD_LIBRARY_PATH, there are still cases where an exec will occur for other reasons. If the java command is requested to change data models using the -d32 or -d64 flag, that is, a 32-bit java command is asked to run a 64-bit JVM or vice versa, an exec is needed to effect the change. Also, multiple JRE support, where a different version is requested via the -version:foo flag, will also cause an exec if a different Java version needs to be run. However, before Kumar's fix the common case was that the launcher would exec once; now the common case is that the launcher will exec zero times.³

I'm very happy this messy use of LD_LIBRARY_PATH has finally been removed in JDK 7. The removal makes the launcher code both simpler and more maintainable. Unless your use of java relies on the number of execs that occur, the change should be largely transparent, other than startup being marginally faster. One situation to be aware of is launching a LD_LIBRARY_PATH-free JDK 7 java command from a JDK 6 or earlier java process. If the LD_LIBRARY_PATH variable of the older JDK is not cleared, it can affect the liking of the JDK 7 process.

¹ Since Solaris 7, that OS line has supported three LD_LIBRARY_PATH variables:

• LD_LIBRARY_PATH_32: if set, overrides LD_LIBRARY_PATH for 32-bit processes.

• LD_LIBRARY_PATH_64: if set, overrides LD_LIBRARY_PATH for 64-bit processes.

• LD_LIBRARY_PATH: used by both 32-bit and 64-bit processes is not overridden by a data model specific variable.

On Solaris, back in JDK 1.4.2 I fixed the launcher to properly take into account all three variables (4731671); on re-exec the data model specific environment variable is unset and LD_LIBRARY_PATH contains the old data model specific value prepended with the JDK system paths. Tests to verify all this used to live in and around test/tools/launcher/SolarisDataModel.sh, but they have thankfully been deleted as they are no longer relevant.

² For suid or sgid binaries, LD_LIBRARY_PATH is handled differently to avoid security problems. While the Solaris runtime linker applies more scrutiny to LD_LIBRARY_PATH in this case, on Linux glibc sets LD_LIBRARY_PATH to the empty string. Since the empty string will not contain the expected JDK system directories, the prefix-checking logic detected this case to avoid an infinite exec loop (4745674). Running java suid or sgid isn't necessarily recommended, but it is possible. To actually resolve linking dependencies for such binaries, OS-specific configuration may be needed to add JDK directories to the set of trusted paths.

³ Before my batch of launcher fixes in JDK 1.4.2, the number of execs was even more varied. Specifying a different data model would exec twice, once to change the data model and again to set the LD_LIBRARY_PATH for that data model. From JDK 1.4.2 until the purge of LD_LIBRARY_PATH, the launcher used a single exec to set the LD_LIBRARY_PATH to the target data model (4492822).

Scrollbars now draw properly on Windows using the native WinUXTheme, sizing and placing is correct.

In the screenshot you can see Graphos running with the current version of the native windows theme.

This really pisses me off. Read it here, especially the last comment:

”Oracle states: Project Kenai, however, will be discontinued for public use. Oracle will continue to use it internally and look for ways that our customers can take advantage of it.

To me that means: Kenai will be continued but you‘ll have to pay for it.”

Good, very fair Oracle, very fair for all the projects currently hosted there. And it‘s not even a week it started. Perhaps the EU had reasons to be worried about. What‘s next? NetBeans?

Update: Oh... My... God!

Update 2: as Dmitri kindly points out in a comment below, the blog from Danese seems to contain wrong or misleading informations. This link explain the situation. I thought I would add the link the main post because this page is referenced in many blogs, and I don‘t mean to fight gratuitously Oracle just because it‘s fun, or because I liked Sun but I don‘t like the fact that Sun is no more: most of the people working there are the same and I have the honour to know most of them personally. I think though that moves like closing kenai are a very very bad sign of what the future will reserve for us.

The views expressed on this blog are my own and do not necessarily reflect the views of my employer, my friends, or the whole universe for that matter.

I've modified ikvmc to use IKVM.Reflection and largely rewritten ikvmstub to directly work with the ikvm internals instead of using the java reflection API. Both ikvmc and ikvmstub can now process assemblies independent from the .NET runtime they run on. This opens up the possibility to start investigating the possibility of Silverlight support.

Changes:

• Drag-n-drop fix by Nat.
• Fixed regression introduced in previous development snapshot, related to field accessors.
• Removed caching of inner classes.
• Fix for bug #2908683.
• Various AWT fixes by Volker.
• Changed JNI to use standard caller ID mechanism.
• Various JNI optimizations.
• Fixed http://gcc.gnu.org/bugzilla/show_bug.cgi?id=41696
• Fixed exception sorter bug exposed by recent Mono sorting change.
• Fixed Thread.getAllStackTraces() to resume threads that it suspends.
• Integrated new IKVM.Reflection implementation.
• Added AllowMultiple = true to RemappedClassAttribute.
• Fixed atomic update helper nested types to be invisible from Java.
• Removed support for "ikvm.stubgen.serialver" property that is no longer needed now that ikvmstub doesn't use the runtime to generate stubs.
• Removed pre-generated stub jars from cvs and modified build process to generate them during the build.
• Removed "constant" instance field support (which was only used by ikvmstub and doesn't make any sense anyway).
• Removed ReflectionOnly support from runtime. Now that ikvmstub no longer requires it, there's no good reason to allow Java code to see ReflectionOnly types.

Binaries available here: ikvmbin-0.43.3681.zip

One of the more subtle aspects of javac syntax trees is that every tree node has position information associated with it. This information is used to identify the location of errors with the source text, and is used by IDEs when refactoring or reformatting code. Ensuring the information is accurate is tricky, and with a number of projects ongoing to update the Java language, and hence the trees used by the compiler, the time has come for some better test support in this area.

The new test is called TreePosTest, and can either be run by the jtreg test harness or as a standalone utility. It can read one or more files or directories given on the command line, looking for source files. It reads each file in turn, ignoring those with syntax errors (there's a lot of those in the javac test directories!) For each file, it scans the tree, checking invariants for the position information for every tree node. Any errors that are found are reported.

Each tree node has three nominal positions, identified as character offsets from the beginning of the file. (Older versions of javac used line-number/char-on-line coordinates, packed into a single integer.) The three positions are the start of the node within the source text, the end of the node within the source text, and a position used to identify that node -- typically the first non-white character in the source text that is unique to that node. The last of these is stored directly in every tree node. The start position is always available and is recursively computed by TreeInfo.getStartPos. The end position requires a table to maintained on the side; for performance reasons, this table is not normally enabled; it must be enabled if end positions are going to be required. When enabled, the end position for a node is recursively computed by TreeInfo.getEndPos, using the endPosTable. (Certain nodes also store an end position directly, when such a position may be required for an error message.)

Given these three positions, we can identify various invariants.

• For any node: start = pos = end
• Any node must be enclosed in its parent node: parent.start = start && end = parent.end
• The position of a parent node should not be within any of its childen: parent.pos = start || end = parent.pos

The first surprise was that the test program found a number of faults within its own source text. Ooops. Running the test program over the source files in the langtools test/ directory, it found 6000 errors in over 2000 files. More ooops. Fortunately, many of those errors are repetitions, but what started as a proactive exercise to test new compiler code was turning out to have more payoff than expected.

I don't know about you, but I tend not to think in characters positions very easily, and error messages like the following leave a little to be desired:

test/tools/javac/treepostests/TreePosTest.java: encl.start = start: encl:WILDCARD[start:9232,pos:9232,end:9246] this:TYPEBOUNDKIND[start:9224,pos:9224,end:9231]
? extends 

Here is another example. Here, it becomes clear that the position for the parent AND node is incorrectly within the expression on the right hand side of the &&, instead of at the && operator. In fact, this is an instance of a previously reported bug, 6654037.

Issues

Enum declarations

These are desugared right in the parser into equivalent declarations of static final fields within the enclosing class. The question then becomes, what position information should be recorded for these desugared nodes. On the one hand, one might argue to use the "invalid position" constant, NOPOS, since these nodes do not directly correspond to source text, but on the other hand, it is important to record a position in case of errors. (See 6472751.)

Array declarations

Array declarations are complicated by support for legacy syntax, that allows constructions like:

int[] array[];
int[] f()[] { return null; }

Annotations

A number of issues have been observed with the positions recorded for annotations, but which have not yet been fully investigated.

Summary

Graphos (GAP Vector editor) got undo support. Add/remove shapes (that worked already) and proper undo of moving, inspecting and editing shapes.

This required implementing shallow vs. deep copy of the objects, but the deep copy needs to be smart about which references are indeed duplicated and which not... A bit of fun programming!

We are pleased to announce the release of IcedTea6 1.7!

The IcedTea project provides a harness to build the source code from
OpenJDK6 using Free Software build tools. It also includes the only
Free Java plugin and Web Start implementation, and support for
additional architectures over and above x86, x86_64 and SPARC via the
Zero assembler port.

What’s New?

• Updated to OpenJDK6 b17.
• Alpha version of the new IcedTea NPRuntime based plugin with support for Firefox >= 3.5, Chromium, and other browsers that support NPRuntime (use –enable-npplugin to build it). For xulrunner >= 1.9.2 (used by Firefox >= 3.6), the new plugin is required and the build will automatically enable it if the old plugin is requested.
• Support added for building with HotSpot 16 using –with-hotspot-build=hs16. This is the same as was released in the proprietary JDK6 update 18.
• Zero port updated to match the version submitted to OpenJDK as closely as possible.
• libjpeg7, libjpeg8, libpng 1.4 and libXext >= 1.1.0 supported.
• Added JNI call tracing using systemtap version 1.0+ when configuring with –enable-systemtap. See tapset/hotspot_jni.stp.
• Add support for building the Zero assembler port on Hitachi SH.

The tarball can be downloaded from:

• http://icedtea.classpath.org/download/source/icedtea6-1.7.tar.gz

The following people helped with the 1.7 release:

Lillian Angel, Gary Benson, Deepak Bhole, Andrew Haley, Andrew John Hughes, Nobuhiro Iwamatsu, Matthias Klose, Martin Matejovic, Edward Nevill, Xerxes Rånby, Robert Schuster,Jon VanAlten, Mark Wielaard and Man Lung Wong.

We would also like to thank the bug reporters and testers!

To get started:

$ tar xzf icedtea6-1.7.tar.gz
$ cd icedtea6-1.7

Full build requirements and instructions are in INSTALL:

$ ./configure [--enable-visualvm --with-openjdk --enable-pulse-java
--enable-systemtap ...]
$ make

I started out today with a simple goal; "Let's build OpenJDK for OMAP this morning." I said to myself.  So I begin with a fresh build tree, pulling in the build recipes bit by bit.  I start to see unresolved depencency issues.  "No problem, I know where to get that." I mutter under my breath.  Then I hit another, and another.  "Rhino?  Really?!" I mutter as my mood darkens.  "SNOBOL-native?!"* In fact over an hour later and I'm still in this dependency hell.  Since it's not the most interesting work, my brain had time to wonder why this problem exists.  "It's because Java is one big monolithic bowl of jello."  my internal voice spoke up.  I don't even like jello.  Yes, of course this is why there are so many dependencies.  The dark side of having so much functionality in the run-time is so much complexity and tight coupling in the build-time.  "What could possibly be the solution?" I wondered again...

"Java Modularity!" Of course, it's so simple!  If the JDK was broken up into semantic chunks, the build-time complexity could be greatly reduced if you only have to build what you need.  "But does a simple Java build really matter?" I pondered.  After all, only a select unlucky few ever have to actually build the damn thing.  Most of us just install binary packages and get on with it.  Well, I have no metrics, and cannot be sure, but it seems to me that a big cost of build complexity is adoption and inclusion of Java in other open source projects.  If I am a distro maintainer (or a machine maintainer), or I have a project that depends on JDK, and the build is difficult or breaks often, I may move on to greener pastures.  Complexity is risk and cost.  So, a simple, sane, and universal Java build system would put more jvms on more machines, distros, and platforms.  This is a worthy goal! I really hope the latest push for Java modularity makes some traction in OpenJDK sources soon!

* Haha, I kid!

Thanks to Marcin's tireless efforts the Ångström distribution is now running on BUG:

.-------. | | .-. | | |-----.-----.-----.| | .----..-----.-----. | | | __ | ---'| '--.| .-'| | | | | | | | |--- || --'| | | ' | | | | '---'---'--'--'--. |-----''----''--' '-----'-'-'-' -' | '---' The Angstrom Distribution bug ttymxc4 Angstrom 2009.X-stable bug ttymxc4

This is important because it allows us to merge the Java and Web Service goodness we have developed on Poky Linux with the state-of-the-art OpenJDK work provided in OpenEmbedded and Jalimo communities.  I imagine a day coming soon when we will not have to implenent our own string splitting functions!  Stay tuned for more exciting developments...

Few days ago I read an interesting news that speculated about the "upcoming" Playstation 4 hardware. The news was about the decision to use Larrabee instead of the Cell BE + nVidia setup that we currently have on the PS3.

Of course, at this point anything is just that: a speculation, and in fact, the news was already rejected as invalid.

The reason why the information has cought my attention is not the Larrabee itself (btw, when the PS4 will be out, there will be probably a second or third generation Larrabee anyway, and who knows how they will codename it), but it‘s the different approach that such a system would require. In fact, it‘s quite some time that there is some new interest around ray tracing algorithm, which can give a much much more photorealistic rendering than current generation raster GPU.

What‘s the difference so?

I don‘t want to dig too much into the topic, as it‘s quite complex although very very interesting, but I‘ll try to give some light on what is a ray tracer and the difference with the current raster approach. People involved in the subject will notice some mistakes, but my purpose is to make it easy to read for my girl friend, who has no clues about what a ray tracer or a raster GPU are, so please be nice with me in the comments :)

Light, exactly… The human eye sees the world sorrounding us by intercepting and interpreting light particles. A light particle, whose name is “photon“, starts from a light source and traverse the environment interacting with the object‘s surfaces that it finds through its path. This interaction may lead to change the photon trajectory, may cause a change of the photon colour, may even “kill” the photon, by absorbing it completely.

Those interaction have names: absorption, reflection, refraction and fluorescence, with the latest being the most rare condition.

This happens all the time any time we look at something, we grab light rays from a light sources and interprete them in our brain.

Of course, doing the same with a computer would require an amount of computation that is not even thinkable. A ray tracer is a special algorithm that tricks those interaction by reconstructing backward the path of the light; in other words, the photon path is tracked from the human eye (in fact, the camera that views the scene) to the light source. Because is still possible to get an indefinite amount of those interactions, especially when simulating light diffusion, for example in ambient that contains fog, there are a series of thresholds that are applied to the ray tracer, whose purpose is to define how much to keep track of a single ray (that is, when to stop following its path and its interaction with the objects surfaces).

Because the ray tracer, as the name imply, tracks ray of light, the results are often (although not necessarily) photorealistics.

The raster approach is different, in the sense that no light is tracked, instead the scene is rendered on a pixel by pixel basis. A Raster based GPU perform its rendering using a pipeline, that is, an assembly line where each step contribute to decided what will happen in the next step. It‘s true that in both ray tracing and raster approach a model of the scene is created using triangles. Those always form the base building blocks of any 3D world, but in the raster approach each triangle is considered a standalone entity. That is, the various stages in the pipeline define rules that determine how and when a given triangle is visible on screen, but in any single pass through it only the pixels that are contained in a geometric primitive (a triangle) are considered at any given time (there are special cases, for example gradient rendering, where the fragment shader, the part of the pipeline responsible to promote a pixel candidate, or fragment, to a pixel on screen, may request values for the gradient from adjacent pixels).

The ray tracer is more realistic because, as said, by tracking light allows for very precise representation of the scene, considered as a whole; but this comes at a cost. So far, the computational cost of the ray tracing approach was simply too high to be possible to implement for real time graphics. But it scales well with the number of processors in a system, that means that more processing units are added to a system, faster is the algorithm, and with the current tendency to parallel programming, ray tracing techniques start to be feasible.

I‘m not sure if we‘ll see this in the PS4, because a ray tracing GPU would mean no raster one, developers would lose some of the portability that they have today. I also don‘t think that performance are quite there, but of course we have to see what will be the state of things when the console hardware will be defined, they could always come up with some sort of hybrid approach to take the best of both worlds, for example.

If you want to have fun on speculations (that are quite fun, for sure), track this site, it already has all the details for the new hardware, including model charts :) and even already some details for the PS5 ;)

As a side news, seems that the PS3 was finally hacked. This is very interesting, indeed. If confirmed, I‘m sure it will help sales of the console ;)

Laterna Magica v0.2 is out! New features are more keyboard short-cuts, JPEG saving and better in-window menus support (NSWindow95Style). The latter feature delayed release another bit of this long-due release, but it works great.

The attached screenshot shows LaternaMagica running on windows using the experimental WinUXTheme which draws several controls native including in-window menus. Note how the menu is attached to the main application window but not to the image window.

LaternaMagica home page

One surprisingly tricky piece of the Java platform is the launcher, the set of C code that uses the JNI invocation API to get the JVM started and begin running the main class. While conceptually simple, the launcher is complicated by straddling the boundary between the host system and the JVM, often wrestling with native platform issues like thread configuration that need to be managed before starting the JVM. The launcher's tasks include selecting which VM to run (client, server, etc.) and running in the requested data model, 32-bit or 64-bit.

In the jdk Mercurial repository, the source code of the launcher is primarily composed of:

• src/share/bin/java.{c, h}

• Other files in the src/share/bin directory, including the Java launcher infrastructure utilties, jli_util.{c, h}.

• src/solaris/bin/java_md.{c, h} (covers both Solaris and Linux using #defines)

• src/windows/bin/java_md.{c, h} (covers various and sundry versions of MS Windows)

The launcher code is used to build the executables in the JDK bin directory; every invocation of java and commands like javac first executes through the launcher. Consequently mistakes in the launcher can cause severe build problems and cross-platform testing is especially important. On doing numerical work, Prof. Kahan has advised "The best you can hope for is that if you do your job very well, no one will notice" and working on the launcher has a similar flavor; you don't want your work to get noticed!

From JDK 1.4.2 through JDK 6 I was the lead launcher maintainer, amongst other responsibilities. When I first took over launcher maintenance, I introduced regression tests and performed several rounds of refactoring. Over time, my launcher activities shifting to reviewing and advising others on their launcher-related projects including:

• In JDK 5, multiple JRE support (java -version:foo to invoke Java version foo from some other version) and ergonomics to provide better out-of-the-box tuned performance.

• In JDK 6, the splashscreen functionality and classpath wildcards.

Since JDK 6 first shipped, I've happily handed over the reigns of primary launcher care and feeding to Kumar, while still being involved in code reviews and writing the occasional blog entry. Kumar moved common functinality of the launcher into a shared library to make the source more robust and speed up the builds.

In the near future I'll be writing about a long-standing launcher flaw concerning the Unix exec system call and LD_LIBRARY_PATH Kumar has fixed in JDK 7.

In November 2008 I introduced IKVM.Reflection.Emit, today I'm introducing IKVM.Reflection. It superseded IKVM.Reflection.Emit and also includes the ability to read managed assemblies. In addition, I've also added many other features that aren't directly needed for ikvmc, but are useful for other applications. Almost the complete reflection API has now been implemented and there are several API extensions to support managed PE features that reflection doesn't support (well).

Why?

When I started on IKVM.Reflection.Emit, it wasn't at all clear to me that it would be possible to re-implement the System.Reflection.Emit namespace without also re-implementing the System.Reflection namespace, but it turned out it was. I did run into a few snags, such as the inability to subclass Module and Assembly (this was fixed in .NET 4.0) and a couple of Mono bugs, but on the whole IKVM.Reflection.Emit was very successful. So why then re-implement the System.Reflection namespace as well? The main reasons are Silverlight and .NET 4.0. For ikvmc to be able to target versions of the runtime different from the one it is currently running on, it is necessary to avoid using System.Reflection, because System.Reflection can only ever work with the mscorlib version of the current runtime.

ikvmc & ikvmstub

In the coming time, I plan on integrating IKVM.Reflection into ikvmc (most of the work for this has already been done, if you've been following the ikvm-commit list, you may have seen some changes go in and wondered why they are necessary) and ikvmstub (I haven't started on this yet). Currently, ikvmstub uses java reflection to expose members of the .NET types in an assembly. I chose this because it was the easiest way to make sure that what ikvmstub generated matched the ikvm runtime behavior (because it simply used the ikvm runtime to do the mapping). There are two downsides to this approach. The first is the same as mentioned above with ikvmc, you can only generate mscorlib stubs for the runtime you're currently running on. The second is more philosophical, it introduces a cycle in the build process. To build the IKVM.OpenJDK.*.dll assemblies, you need mscorlib.jar and System.jar, but to generate these stubs you need a compiled class library. To solve both these issues, I plan to rewrite ikvmstub to work directly on the internal ikvm runtime representations (with conditional compilation, like ikvmc does).

Features

This list is not exhaustive, but here are some interesting features of IKVM.Reflection (that are not in System.Reflection):

• There is no AppDomain global state. Everything is contained in an instance of the IKVM.Reflection.Universe class.
• No thread safety. If you want thread safety, you'll have to lock the universe object during every operation.
• You can choose what version of mscorlib to load in the universe (using Universe.LoadMscorlib()) and by implementing a Universe.AssemblyResolve handler you can decide the framework assembly unification policy.
• Support for querying and emitting .NET 2.0 style declarative security.
• Support for defining unmanaged resources from a byte array (in .NET, AssemblyBuilder.DefineUnmanagedResources(byte[]) is broken, only the overload that accepts a filename works).
• The ability to read and emit .NET 1.1, .NET 2.0 and .NET 4.0 assemblies (while running on, for example, .NET 2.0).
• Full support for vararg calling convention.
• Support for reading field RVA data (e.g. for the fields that are used by the C# compiler to initialize arrays).
• The ability to enable/disable "exception block assistance", or get "clever" assistance.
• Support for querying methodimpl mappings.
• Support for reference, pointer and array types with custom modifiers (this CLR feature is used by C++/CLI).

Missing Features

Some things are still missing. The most notable being the Emit differences. The emit code was based on the IKVM.Reflection.Emit code and likewise still lacks some of the querying support (for baked types), although the new code is much better than the code in IKVM.Reflection.Emit.dll in this respect.

Here's a list of methods that can still throw a NotImplementedException:

• FieldBuilder.__GetDataFromRVA()
• ModuleBuilder.ResolveType()
• ModuleBuilder.ResolveMethod()
• ModuleBuilder.ResolveField()
• ModuleBuilder.ResolveMember()
• ModuleBuilder.ResolveString()
• ModuleBuilder.__ResolveOptionalParameterTypes()
• ModuleBuilder.GetArrayMethod()
• GenericTypeParameterBuilder.BaseType
• GenericTypeParameterBuilder.__GetDeclaredInterfaces()
• GenericTypeParameterBuilder.GetGenericParameterConstraints()
• GenericTypeParameterBuilder.GenericParameterAttributes
• TypeBuilder.CreateType() (when invoked a second time)
• TypeBuilder.__GetDeclaredFields()
• TypeBuilder.__GetDeclaredEvents()
• TypeBuilder.__GetDeclaredProperties()
• ISymbolDocumentWriter.SetCheckSum()
• ISymbolDocumentWriter.SetSource()
• Most methods in ISymbolWriter
• ManifestResourceInfo.ResourceLocation (for resources located in another assembly)
• ManifestResourceInfo.ReferencedAssembly
• ManifestResourceInfo.FileName
• MethodBase.GetMethodBody() (if the method data contains unexpected sections)
• Type.IsAssignableFrom()

Missing members:

• Module.GetSignerCertificate()
• Type.GUID
• ParameterBuilder.GetToken()
• PropertyBuilder.PropertyToken
• MethodBuilder.Signature
• ConstructorBuilder.Signature
• MethodBuilder.SetSymCustomAttribute()
• ModuleBuilder.SetSymCustomAttribute()
• ConstructorBuilder.SetSymCustomAttribute()
• ModuleBuilder.DefineResource()
• AssemblyBuilder.ModuleResolve
• Assembly.GetManifestResourceStream()
• Assembly.GetSatelliteAssembly()
• Assembly.GetFile()
• Assembly.GetFiles()
• MethodBuilder.SetMarshal() (obsolete, use MarshalAsAttribute instead)
• ParameterBuilder.SetMarshal() (obsolete, use MarshalAsAttribute instead)
• FieldBuilder.SetMarshal() (obsolete, use MarshalAsAttribute instead)
• ModuleBuilder.DefineUnmanagedResource(byte[]) (because it is broken)
• AssemblyBuilder.DefineUnmanagedResource(byte[]) (because it is broken)
• MethodBuilder.CreateMethodBody()
• Everything that doesn't make sense in a ReflectionOnly context.

Concepts that are not implemented:

• Most metadata tokens returned by Emit objects are not properly typed (and can't be used for anything, other than comparing them against other metadata tokens).
• When defining debugging symbols, a single method can only point to a single source document.
• All type/member lookup operations are case sensitive.
• Implementing a custom System.Reflection.Binder is not supported.
• Modules with unsorted metadata tables are not supported.
• When Type.GetMethods() (or __GetDeclaredMethods) is called on Array types it throws a NotImplementedException, instead of returning the special array accessor methods.
• Managed function pointer types are not supported. Like System.Reflection, they are returned as System.IntPtr instead.

Linker Prototype

To see if I did miss any important CLR features, I wrote a prototype assembly linker. It is pretty capable, but should not be confused for something that is usable for anything other than exploring. I've used it with C++/CLI (compiled with /clr:pure) to test the more esoteric CLR features. The source for the linker prototype is in the zip linked to below.

The IKVM.Reflection source code is available in cvs. If you just want the binary, the LinkerPrototype.zip contains it.

The BBC currently have a proposal before OfCom entitled Content Management on the HD FreeView platform. This proposes that OfCom allow them to compress the service information data and only provide the necessary Huffman table to those who license it from the BBC.

Not only does this not prevent anyone from using the service without the table — the video and audio are not encrypted, and the table could be worked out by brute force — but the BBC pretty much acknowledge this in their proposal. It’s all part of taking a chance that this will then allow them to enforce the inclusion of encryption technology on receivers, which will then cause more media companies to buy into HD on FreeView. That itself may or may not happen, but in the process they will make FOSS implementations illegal as they won’t be able to publish source code which includes the Huffman table (despite the idea in 5.28 of the OfCom consultion that FOSS will be fine because FreeSat receivers running Linux exist!)

I strongly suggest that you write a response to the consultation (you have until April the 2nd) so that OfCom understand the negative effect that allowing this will have on the usage of FreeView HD.

About two weeks ago I pushed the remaining changes (6894206, 6893081, 6829187, 6893268) for C2 invokedynamic support on x86 which were integrated into HotSpot 17 b07.  Meanwhile HS17b07 has been integrated into JDK 7 b80 and the latter has been released.

With two changesets John Rose pushed lately (6891770, 6914665), which have been integrated into JDK 7 b79, you can run now invokedynamic enabled programs with decent performance.  Additionally we tuned inlining heuristics a little (6912063) to be able to use inline-related switches in a product VM and to not count generated MethodHandle adapters.

And here is how it works (gwt is a simple testcase for guardWithTest):

$ java -server -XX:+UnlockExperimentalVMOptions -XX:+EnableMethodHandles -XX:+EnableInvokeDynamic gwt
370655678

You can also run JRuby benchmarks, like bench_fractal.rb which prints a nice fractal (we need to tune inlining heuristics a little to get good performance):

$ bin/jruby --server -J-Djruby.compile.invokedynamic=true -J-XX:+UnlockExperimentalVMOptions -J-XX:+EnableMethodHandles -J-XX:+EnableInvokeDynamic -J-XX:InlineSmallCode=2500 -J-XX:MaxInlineSize=50 bench/bench_fractal.rb
Rendering
<snip>

(To see the fractal you have to try it yourself :-)

The next thing will be C1 support.  I'm currently working on it and it's almost finished.

Did an interview for FOSDEM about SystemTap. It discusses a wide range of topics. About when I got involved with Free Software, working for Red Hat, how FOSDEM helped the libre Java community, getting Fedora more observable by adding static markers into programs, the history of observation tools (tracers, profilers, debuggers) on GNU/Linux, comparisons to other tools like DTrace, GUI frontends, Eclipse integration, the future of SystemTap and of course why you should come to FOSDEM.

Today I‘ve seen something very boring.

It‘s supposed to be some sort of artist, but the fact is that he doesn‘t add anything to the visual arts. I think he is a broken copy of Andy Warhol. Ah, the name of this guy, right. He's called Douglas Gordon...

I‘ve created an artistic simple applet in his honour. The applet draws a square every 900000 milliseconds. If you have patience enough to wait 84600000 you‘ll also see one letter appearing on the screen. Then a new one every other 3600000 milliseconds :)

You need to leave the browser window open for a lot of time to see this, and it‘s not worth it :)

• libffi 3.0.9 left the building on Dec 31.


• I continue to toy around with my lisp-based web app framework for Java servlet containers, blow. I'm hoping to get it running on Google App Engine soon.


• moxie development continues, although I haven't blogged about it in a really long time. What's exciting is that I'm very close to a major milestone in the development of my libre core.

It‘s said that one image is best than thousands of words. Good luck everybody, and thanks guys, for all the hard work, for Java, Solaris, for all the great things. For “The Network is the Computer”. I owe you most of my professional existence. And a beer.

• David Pollak on Scala Basics - Byte-code Fancypants


• Charles Nutter on JRuby, Duby, and Surinx: Building a Better Ruby


• Neal Gafter on Statically Dynamic Typing


• Cliff Click on A Crash Course in Modern Hardware

Yesterday, I released 0.42 and as seemingly always happens a bug was reported right after that. So with that we're now on the road to the 0.42 Update 1 release. This is release candidate 0.

Changes:

• Updated version to 0.42.0.4.
• Added fix to mangle all artificial type names if they clash with Java type names in the same assembly.
• Fix for http://gcc.gnu.org/bugzilla/show_bug.cgi?id=41696.
• Fixed exception sorter to be correct when invoked with two references to the same object.

Binaries available here: ikvmbin-0.42.0.4.zip

Sources: ikvm-0.42.0.4.zip, openjdk6-b16-stripped.zip

It's exactly one year ago that I started to work for Sun.  Nice!  I wasn't sure that I will make the anniversary facing the merger with Oracle.  Anyway, I learned a lot in this first year and I'm really looking forward to the next years at Sun (or whatever the company will be called then).