MSX demo: Utopia

Van viejo_archivero

Paragon (1395)

29-04-2007, 10:24

Another IMPRESSIVE demo! you guys really know how to stretch MSX1 limits! congratulations for this awesome release

Van Ivan

Ascended (9341)

29-04-2007, 11:01

Another impressive demo from dvik and Vincent van Dam!

Btw, dvik sent us a note saying that some screenshots included in utopia_v1.0.zip are not from the final release.

Van hap

Paragon (2042)

29-04-2007, 11:09

Nice =)
Does it run as it should on the current blueMSX?

Van KNM

Master (172)

29-04-2007, 12:21

WooooowwwWWW!!!!!INCREDIBLE DEMO!Dvik & Joyrex RULEZ!!

Van ARTRAG

Enlighted (6932)

29-04-2007, 15:40

Great demo!! greetings!
Strangely bluemsx does not show the last effect with the 3 girls, where open works fine.
On the contrary openmsx has some sync problems when displaying the 105 color images (that works much better on bluemsx)

Van Maggoo

Paragon (1217)

29-04-2007, 16:05

Great demo and great effects ! MSX1 still can surprise me
Congratulation !

Van dvik

Prophet (2200)

29-04-2007, 18:40

The demo does indeed not run 100% on neither blueMSX or openMSX. The last part requires pixel accuracy in screen 0 and last release did not have it. since then I've added support for it and it will work fine in next version.
The images does not show that well in openMSX. It looks like some minor timing issue but I haven't investigated it that much. Also interlaced images or images that change from frame to frame usually show better in blueMSX than in openMSX. The reason is that blueMSX has a frequency conversion between the MSX frequency and the PC frequency. The improvment is minor but noticable in for example the interlaced images.

Van dvik

Prophet (2200)

29-04-2007, 19:19

@Ivan: The screenshots are from the release but they shouldn't be part of the actual download if you want to host the demo. I just added them so you didn't have to do your own screenshots.

Van hap

Paragon (2042)

29-04-2007, 19:26

When can users expect to see this if they don't have the necessary MSX hardware? <-- polite way of asking when the next version of blueMSX is due

Also interlaced images or images that change from frame to frame usually show better in blueMSX than in openMSX.That's an external issue. Setting the PC screen refresh rate at that of the emulated MSX (or a multiple of it), will fix it as I've just witnessed on my PC with openMSX.

Van dvik

Prophet (2200)

29-04-2007, 19:34

polite way of asking when the next version of blueMSX is due

We are in the final stages preparing the release and it will most likely come in a week or two (unless something unexpected shows up).

Apart from the two 105 color images, openMSX shows the demo quite well. There is also a minor glitch in the end part where the top and bottom border is one pixel off sometimes.

Van [WYZ]

Champion (451)

29-04-2007, 19:59

What a good demo!. I enjoy a lot with this stuffs, VDP is up to limits and some PSG FX are great! (Fade out with envelopes...O_o)

it's a pitty, I couldn't watch it at Barcelona.

Congratulations Dvik & Joyrex.

Van Ivan

Ascended (9341)

29-04-2007, 20:46

@Ivan: The screenshots are from the release but they shouldn't be part of the actual download if you want to host the demo. I just added them so you didn't have to do your own screenshots.

I don't remember seeing the picture of 'utopia_multicolor.png' in the demo... I will run it again.

Van dvik

Prophet (2200)

29-04-2007, 23:19

I don't remember seeing the picture of 'utopia_multicolor.png' in the demo... I will run it again.

Its only sown if you run it in an MSX1 machine (or rather one with a MSX1 VDP)

Van Latok

msx guru (3927)

29-04-2007, 23:23

I tried running the ROM-file on my turboR with loadrom 1.8. Didn't work

Van dvik

Prophet (2200)

29-04-2007, 23:25

The demo doesn't work on TR or some MSX2+ machines, mainly because the TR is too slow. If you run the disk version I think you'll get a warning. Did it not run at all or did it stop after a while. Some parts runs ok on TR and some don't. I suppose I could make a TR version with only the parts that runs ok on TR.

Van Latok

msx guru (3927)

29-04-2007, 23:34

I just used PUTDISK and loaded the DSK-file onto a diskette. Booting that diskette immediately gave a 'Direct Statement in File'

Van dvik

Prophet (2200)

29-04-2007, 23:37

I can look at it but since the demo isn't intended to run on TR anyways I don't think its a big issue. You need to buy a real MSX to run the demo

Van Manuel

Ascended (19302)

29-04-2007, 23:37

Heh, the 105 colour pictures are not shown on an openMSX MSX2, but they were visible on my real MSX2 (with wobbly effect we talked about). Weird?

Van Latok

msx guru (3927)

29-04-2007, 23:39

It also gives a direct statement in file on my philips nms8280

Van dvik

Prophet (2200)

29-04-2007, 23:41

@manuel: That is actually a bit weird. I suppose the demo thinks its an MSX1 then. The reason why we wanted to disable that part is that its not running properly on MSX2.

@latok: Then its probably something wrong with the actual disk. We've tried on quite a few different MSXes and not seen this problem.

@latok (again): The disk is single sided. That may be the problem.

Van Latok

msx guru (3927)

29-04-2007, 23:43

But why is everybody else able to watch the demo then? Do they all use emulators?

Van wolf_

Ambassador_ (10093)

29-04-2007, 23:46

yes

Van wolf_

Ambassador_ (10093)

29-04-2007, 23:47

scared?

Van Manuel

Ascended (19302)

29-04-2007, 23:48

I actually used the MegaFlashROM SCC (And openMSX as well, of course.)

Van Latok

msx guru (3927)

29-04-2007, 23:50

Alright, now I tried booting the DSK-file with FDE.COM on my turboR and that works. At least....I get this warning you told me about dvik. That the demo won't run properly on my turboR. But that indicates the DSK-file isn't corrupt or something.

Maybe you're right the problem is the fact it's a single sided DSK-image. Probably PUTDISK.COM (I don't have another utility like that) can't handle single sided diskimages?

I also can't put the DSK-file on a diskette on a PC, I don't have a PC with a diskdrive anymore ^_^

Guess I'll just have to run the demo on an emulator then

Van dvik

Prophet (2200)

29-04-2007, 23:53

Maybe someone can help you get a double sided disk. And if MRC hosts the demo they usually do an lzh packing as well with the individual files on the disk. But until then I guess you're stuck with the emulator

Van Latok

msx guru (3927)

30-04-2007, 00:46

Alright, extracted all files from the DSK-image with Diskmanager 0.11. Then copied the files onto a CompactFlash Card and from there back onto a diskette. That worked

The demo is great to watch! And of course....I especially like the sample-part. Crystal clear sounds with nice visual effects! Great stuff!!!

Van dvik

Prophet (2200)

30-04-2007, 07:36

@Ivan: I suppose its a good idea to make a 720kB dsk image for the download database to avoid the issue Latok had.

Van spl

Paragon (1470)

30-04-2007, 12:44

I saw this demo at last 31 BCN RU and I was very impressed! I like these kind of demos! Also... if they do this on MSX 1.... what they will do on a MSX 2? o_O

Van wolf_

Ambassador_ (10093)

30-04-2007, 13:41

As long as things stay tiles (e.g. screen 2 or 4) then: not much more, other than palette and more sprites. Well, scrolling perhaps, but that's an aged -thus boring -demo-technique.

Van Manuel

Ascended (19302)

30-04-2007, 20:40

Daniel and Vincent: you two need name for your cooperation. What about DaVinDa? (Daniel Vik & Vincent van Dam. Or ViDaDaVi, ViDaVi, DaViDa, DaViViDa, whatever. Or something that sounds a lot cooler and more original than this crap

Van wolf_

Ambassador_ (10093)

30-04-2007, 21:15

Davilex ^_^

or just "D&V" .. like T&E

Van Vincent van Dam

Hero (513)

30-04-2007, 22:04

Davirex

I kinda like the 'dvik & joyrex' logo

Van poke-1,170

Paragon (1783)

30-04-2007, 22:09

Impressive stuff indeed ! I suppose it would look even better on a real tv (instead of on en emulator), with the flickering extra colour effects.
The stuff with the eyes and the way it's displayed looks very charming, is it made with characters or something ?
nice music as well, including the filtered tone in the loading parts

Van ARTRAG

Enlighted (6932)

01-05-2007, 09:09

D&V:
Is it possible to have a short text that explains the key point of all the parts of the demo?
It would be very interesting for many of us and would allow to maintain and preserve
your knowhow in the time.

Van dvik

Prophet (2200)

02-05-2007, 01:10

Here is a draft of the spoler: www.bluemsx.com/demos/utopia.txt

If you miss some info, let us know and we'll try to add it.

Van Huey

Prophet (2694)

02-05-2007, 09:01

Grrrrrrreat!

Van ARTRAG

Enlighted (6932)

02-05-2007, 12:24

Great!!! Thanks!! The Utopia spoiler could get as much accesses as the demo

Van ARTRAG

Enlighted (6932)

02-05-2007, 12:31

Big Eyes
========

The graphics in this part is basically a simple 2-D transformation effect doing
offset, rotation and scaling. The part uses screen 2 characters with dithered
patterns to simulate different color tones. The source image is a 256x128 pixel
image with 8 bit color depth.

The transormation matrix is


  | x' |     | x0 |   | dx  -dy | | x |
  |    |  =  |    | + |         | |   |
  | y' |     | y0 |   | dy   dx | | y |

To make the math easy on the Z80, fix point arithmetics is used. 16 bit
registers are used in the calculation and the high 8 bits are the integer part.

Each source pixel (x, y) represents one character on the MSX screen. The
character shown depends on the pixel in the source image closest to (x', y').

Each frame in the animation is defined by four values. x0, y0 is the top left
corner and dx, dy which are the transformation parameters. So each frame, the
engine reads the transformation data from a table and renders the output image.

Do you use vector quantization for defining the characters you display?
Do you define tables to associate pixel (x, y) to each character ? Which metric do you use ?
I see you use the same character set for all the scales:
Do you use different tables to associate pixel (x, y) to each character depending on the scale ?

Van Prodatron

Paragon (1839)

02-05-2007, 19:38

Really impressive, what an MSX1 can do! Great work!

Van dvik

Prophet (2200)

02-05-2007, 19:55

Do you define tables to associate pixel (x, y) to each character ?

I use a 256x128 bitmap with 8bit pixels, so yes its a table.

Each pixel represent an MSX character. The character is made up by two colors and a pattern to get as close
as possible to the color of the source bitmap.

Do you use different tables to associate pixel (x, y) to each character depending on the scale ?
No its only one table and its quite big, 32kB.

Van ARTRAG

Enlighted (6932)

02-05-2007, 23:43

I see, each point in the 256x128 map is a char.

The character is made up by two colors and a pattern to get as close
as possible to the color of the source bitmap

how do you define the characters? do you use some specific algorithm ?

Van dvik

Prophet (2200)

03-05-2007, 00:45

how do you define the characters? do you use some specific algorithm ?

Its a quite simple algoritm. From a given RGB value, it picks the two closest colors and calculates the percentage of each color. Then it generates the pattern by starting at the top left pixel with the bg color, and to set the pixels in the pattern it does sortof linear interpolation of a step signal that is 0 for time 0-x and 1 for x-1 where x is the percentage of the background color.

Van ARTRAG

Enlighted (6932)

03-05-2007, 23:22

Why do you blend only 2 colors?
in theory you could blend up to 16 colors in the same tile...
this could lead to better results without almost any effort

Van dvik

Prophet (2200)

04-05-2007, 00:24

Thats true. You'll get quite a few colors with the current method as well. You can combine 15 colors in 105 ways. Then you have 256 ways of mixing each pair of colors. So you'll get 105*256 different colors which is quite a lot, 26880 to be exact. And as a bonus you get interesting patterns (at least some times).

Van poke-1,170

Paragon (1783)

04-05-2007, 00:59

would you be able to convert videos in such a way too ? I must say the patterns add a nice touch to it.

Van dvik

Prophet (2200)

04-05-2007, 01:32

Its definately possible. The question is how often you need to update the color table though. The part in this demo uses a 256 color palette (the number of possible characters) and if you would do a video you probably need to update the palette every now and then since a 256 color palette may not be sufficient for an entire video.

An option would be do do the part in screen 1 and create the most compatible palette based on the color combinations you can get in screen 1. The benefit is that you could update the palette every frame without much cost.

I think that wolf_'s patterns he use in Sphere may work better for video though, mainly because I think they give higher resolution (It looks like sphere resolution is 64x48 or 64x24, correct me if I'm wrong).

Van ARTRAG

Enlighted (6932)

04-05-2007, 13:47

About reproducing videos (or even the rotation of pre-defined objects - it can be considered a video as well)
many well known methods exist in order to find the best definition of the tile set for a given video sequence.

Assume you have a video sequence of 100 frames 256x192, 8 bit per pixel.
Assume we use the s.c. hybrid mode, i.e. screen 2 with only one tileset (or the same tile set replicated 3 times, it is the same for now)

Split the frames in 8x8 blocks: you get 76800 vectors of 64 bytes (i.e. 8x8 = 64 )
This is usually called "test set".

You need to find the best 256 tile set to represent the video.

This is the set of 256 64 bytes vectors that, accordingly some criteria, best approximate the 76800 test set.
This is usually called "code set".

The additional complexity is that the code set is chosen among the tiles that can be represented by the VDP (“valid” tiles).

Without this last constrain, the easiest way to define the code set is to use an algorithm like
K-means http://en.wikipedia.org/wiki/K-means_algorithm

The steps are:
1) start with a random code set, (256 random vectors of 64 bytes)
2) for each code vector, find what are the test vectors that should be represented by that code vector: call it cluster (you get 256 clusters in our case)
3) define the centroid of each cluster you have found above (it is the mean of the cluster)
4) replace the code set with the centroids of the clusters (you get 256 new code vectors)
5) repeat from 2) till convergence (i.e. when centroids does not move any more)

Adding the constrain on the fact that not all the vectors can be centroids (i.e. not all the tiles can be displayed by the VDP due to color clash), the K-means could be restated like this:

1) start with a random code set, among those that can be displayed by the VDP
2) for each code vector, find what are the test vectors that should be represented by that code vector: call it cluster (you get 256 clusters in our case);
3) define the centroid of each cluster you have found above (mean of the cluster);
4) convert to screen 2 the centroids, i.e. find the best valid tile that approximate each centroid (any existing conversion strategy can be used here)
5) replace the code set with the converted centroids of the clusters (you get 256 new code vectors)
6) repeat from 2) till convergence (i.e. when centroids does not move any more)

Actually MSX1 could do even better:

Split vertically the video in 3 separate videos accordingly the 3 tile sets available on the MSX1 and do the process above for each part of it. You get 3 different tile sets each one fitted on the part of video that should be displayed in its area.

IMHO all this is doable and is the best that can be done on msx1

Naturally quality improves when the code vectors are fitted on sequences of fewer frames. This implies that instead of changing the tileset each 100 frames you need to change it each 50 frames or less. As always it is a trade off between video quality and z80 resources.

Van wolf_

Ambassador_ (10093)

04-05-2007, 14:37

64x48 for Sphere and Wings. However, the diff with for example screen 3 is that our chars are not just setup like:
00001111
00001111
00001111
00001111
22223333
22223333
22223333
22223333

but more like

00001111
00110011
00001111
22112211
11331133
22332233
22223333
22332233

or sumfin', dunno exactly. Anyway, point is that it'll look a little less blocky that way, more like mpeg blocks.

Van ARTRAG

Enlighted (6932)

04-05-2007, 16:04

A small update about the feasibility of my proposal.

I did some experiments in matlab rigth now,

Standard K-means on a test set of 7680 vectors (10 frames of 256x192 at 8 bit per pixel) - random noise at the moment - needs about 51 minutes to extract the 256 optimal tiles (on a AMD 3400+).

I know, having 10 frames per sec, with my 7680 vectors I optimized the tileset of 1 sec of video only, never the less everithing seems doable...

I'm sure taht the k-means I proposed is more complex than the standard, but you need to take into account that C can be much more efficent than matlab (as in matlab all the data are represented as double, while here we deal with bytes and ints)

DVIK what do you think ?

Van ARTRAG

Enlighted (6932)

04-05-2007, 20:24

There are only 4,73304E+35 possible values for a 8x8 screen 2 tile.

Someone proposing exhaustive search?

Van dvik

Prophet (2200)

04-05-2007, 23:22

I've used vector optimizations in the past, mainly to do compression of images. Never really though of applying it on MSX though.

I cheated and had Photoshop do the color optimization. Just convert the 24bit color image to 256 indexed colors in PS and you're almost there. Just need to create the optimal partern and color match for the 256 indexed colors. Would this method be significantly less good than doing an MSX optimization from the 24 bit color source image?

Van ARTRAG

Enlighted (6932)

05-05-2007, 01:18

We do not speak of the same thing. You associate a pint in the image to a char on the screen.

I speak of a general method to reproduce videos.
In my proposal the source video has frames of 256x192 points (and 8 bit per pixel) and the proposed vector quantization does a true compression, as at each block of 8x8 bytes is associate only one byte (the number of the tile in screen 2)

The algorithm I proposed is able to define the codebook finding the best 256 tiles of screen2 in order to reproduce the video.
This is tile set, or the 3 tile sets, to be used in screen 2.

Once you have the codebook (i.e. the tile sets), you store only the char number instead of the 8x8 blocks.

This saves a lot of room in rom.
The final video has a compression rate of 0.1250 bits per pixel - very poor for color images - I know, but still something.

In theory the process is able to reproduce any video with resolution 256x192.
In practice I guess that reproducing videos with lower resolution - but expanded at 256x192 - should lead to good results in terms of codebook generation.
(I mean that if you start from a video that has few bit per pixel, the compressed version at 0.125 bit per pixel will be very close to the original)

1 min of 256x192 video at 10fps could fit in about 450Kbytes of ROM (tiles apart)

Van ARTRAG

Enlighted (6932)

05-05-2007, 15:43

I'll try some quick and dirty development in matlab...
ready for VDPENC ?

Van poke-1,170

Paragon (1783)

06-05-2007, 22:44

hmmm, what if you stick to indexed colour, and have a fixed palette ?
when you use monochrome colours 16 is often enough.
it's graphically an appealing effect I must say, done with the patterns
used in the eye rotation part.
so it that all calculated realtime ? Would be an interesting video coded
sort of thing hehe

Van poke-1,170

Paragon (1783)

06-05-2007, 22:45

now that I think of it, didn't bandwagon do some video in monochrome (or 2 colours even)
video in some of their demos ?

Van NYYRIKKI

Enlighted (6033)

06-05-2007, 23:17

Dvik & ARTRAG, check out this idea that I wrote with you in mind:
http://www.msx.org/forumtopicl7311.html
:D

Van dvik

Prophet (2200)

06-05-2007, 23:41

bandwagon did indeed a short video in one of their demos. Its a 64x48 pixel video made in screen 1 or screen 2. The colors are black and blue and there are four different intensities per pixel, so two bits per pixel. Similar patterns to the big scroller in the utopia demo. The tunnel effect also has two bits per pixel color depth but instead of using different patterns, its using different colors.

Van ARTRAG

Enlighted (6932)

07-05-2007, 00:07

Some preliminary results at
http://ragozini.googlepages.com/vdpenc

The VDP now is only in cascade to kmeans, but WIP to bring it into the loop!
stay tuned...

Van dvik

Prophet (2200)

08-05-2007, 01:43

@Ivan, wolf_ or snout: I know you may be busy with other things but I just wanted to ask if you still want to host the demo?

Van wolf_

Ambassador_ (10093)

08-05-2007, 02:12

From those 3, only snout can access the MRC-FTP, and he's far from motivated recently. Bear with him/us.

Van dvik

Prophet (2200)

08-05-2007, 17:58

I saw you mentioned something similar in another thread. I hope he is ok and its nothing serious. I'll keep the temporary location as long as needed, no problem.

Van dvik

Prophet (2200)

13-05-2007, 01:18

Emu users: Now you can download blueMSX2.7 and it runs the entire demo pretty well. Use a European MSX1 machine for best result (or rather to show all parts and effects).

Van wolf_

Ambassador_ (10093)

13-05-2007, 01:37

I hope he is ok and its nothing serious.

Euhm, well, serious enough to have been out of the msx-running for months, and prolly for a while to come..

Van Yukio

Paragon (1540)

23-10-2007, 23:37

Used BlueMSX version 2.4 for this.
Later will download a newer version of BlueMSX.
The demonstration is very nice ... Sounding astonishing!
Very cool text scrolls.

Van dvik

Prophet (2200)

24-10-2007, 21:25

use bluemsx 2.7 for best result and run the demo in a european MSX1 machine (e.g. the default MSX1 machine).

Van Yukio

Paragon (1540)

24-10-2007, 23:40

I completed the download of BlueMSX version 2.7 generic.
The connection and services over here are troubleshot and the server does not support GetRight. Because of it I (at this time) can't get the SS2 version 2.7. Am I missing something for not use the SS2 extensions ?

Van dvik

Prophet (2200)

26-10-2007, 09:02

earlier versions of bluemsx had different builds for non-SSE, SSE1 and SSE2 but 2.7 is the same build regardless of what you try to download. There are some hand made SSE optimizations in the build but these are automatically detected.

Share

Related content

Reacties (68)