After some discussion in IRC (Thanks rootard!) I was able to find a workaround. The problem appears to be that the pre/post scripts in the package call bash, which chokes on the newly installed libncurses library. Copying libncurses.so.5 to /tmp/nlulib and then installing the package works. I don't know if it is considered an acceptable solution to copy libncurses into nexenta-lu or not.
After some discussion in IRC (Thanks rootard!) I was able to find a workaround. The problem appears to be that the pre/post scripts in the package call bash, which chokes on the newly installed libncurses library. Copying libncurses.so.5 to /tmp/nlulib and then installing the package works. I don't know if it is considered an acceptable solution to copy libncurses into nexenta-lu or not.