The is plenty of information on the web about configuring VNC server on Solaris, most of it misleading and of little use, especially in the context of Solaris 11 express. For the record I'm running 151a updated to SRU4 although I suspect that the SRU is of little relevance.

vi /etc/gdm/custom.conf
[xdmcp]
Enable=true

This is the only entry you need here,trust me.

svccfg -s xvnc-inetd
setprop inetd_start/exec = astring: "/usr/bin/Xvnc -geometry 1920x1200 -inetd -query localhost -once passwordFile=/home/alan/.vnc/passwd
setprop inetd/wait = boolean: true
end

So what have we done here, change the default geometry for VNC windows because I like it a little larger, set a password file to authenticate against when I make the connection, and lastly make my VNC session persistent so when I disconnect I can just be right back to my session. A little side not the password file is populate by running the vncpasswd command.

Next we need to restart, refresh or enable the stuff we have just tampered with

svcadm refresh xvnc-inetd

or

svcadm enable xvnc-inetd

if you haven't turn it on already

also restart gdm

svcadm restart gdm

Now the two biggest issue I had one, the built in screen sharing app on the Mac only seems to like other Mac's is probably nobbled by Apple and I recall from someplace else it's outdated. So go get a VNC client that works for you on your desktop platform, I used Chcicken of VNC seemed to the job.

Second problem Gnome. I would connect and the connection would crash, not sure why but running gnome-cleanup fixed the problem