Yet another tool to convert MSX Basic written in modern code standards + Syntax Highlight for Sublime Text 3

By Grauw

Ascended (10821)

22-05-2019, 23:44

Neat stuff! And I like how you slam a nice tool like that together in under a week’s time.

p.s. Difficulty understanding Git really has nothing to do with age or you as a person .

By farique

Resident (40)

23-05-2019, 16:28

Thank you, Grauw

By Louthrax

Prophet (2497)

23-05-2019, 21:38

Impressive achievement for a first Python project !

Maybe a next step could be to generate tokenized BASIC ? Just suggesting... there are already existing tools, but IIRC they were not command-line oriented and hard to integrate in a "make-like" build-chain (I'd definitively be interested by that )

By Pencioner

Scribe (1612)

23-05-2019, 21:57

@farique great to have such tools developed

@Louthrax for tokenized, is there maybe option for running openmsx (headless? without video output?) and control it via script or so - make it enter a command to load an ascii basic program and save back tokenized? it is a trick but it would definitely do the job and take less effort than making a tokenizer in python (which also could be not completely error prone in the end, because it is quite complicated and there are no guarantee that it will work exactly like a native one). i'm not 100% sure but i have a gut feeling that openmsx has enough of features for this task

By turbor

Hero (529)

24-05-2019, 10:17

@pencioner Using openmsx headless is possible and would work, the main drawback with this proposed solution is that there would be no feedback in case of errors. For instance lines to long, program to big to fit in memory etc etc
So you would need to check these conditions first and then assume that the emulated MSX will always be possible to parse the input without any error...

By NYYRIKKI

Enlighted (6094)

24-05-2019, 14:58

Few tips, if someone wants to make a new tokenizer:

In openMSX there is also "listing"-script in the scripts folder that pretty well explains how the tokenized program is stored. If the line is too long (>254 visible characters IIRC) the encoded program still works, but you can't edit it anymore in MSX full screen editor or the invisible extra information is lost from end of the line. It should be noted also that the memory pointers in encoded file don't really matter as MSX will calculate them again after loading anyway.

By farique

Resident (40)

24-05-2019, 17:39

@Louthrax, I have to admit I gave one minute of consideration to this possibility but it is a whole new ball game and, as @Pencioner pointed, it is also a pandora box, there is hope on the bottom but madness to get there

For the sake of the new guy here (me), apart from being smaller, faster to load (and more compatible?) what are the benefits of a tokenized approach during production?

The way I worked during the tests was to have a two column Sublime (source and converted on each side), openMSX with throttle turned on (to help on the loading speed), dir as disk inserted and F1 set to 'CLS:RUN"convert.bas"+enter'.
I worked on the source, saved, converted, and pressed F1 on openMSX. If I wanted to test or debug something quickly, straight on the emulator, I used the converted window on sublime to find the offending line. The back and forth was very fast.

By NYYRIKKI

Enlighted (6094)

24-05-2019, 18:52

farique wrote:

For the sake of the new guy here (me), apart from being smaller, faster to load (and more compatible?) what are the benefits of a tokenized approach during production?

No, there are no compatibility benefits. The smaller size & faster to load is pretty much all there is... If this would be done correctly then I could imagine that this kind of program could help if someone is planning to put BASIC program inside ROM-file as MSX internal BASIC editor is not really good for this kind of conversion... but I would say this would be very theoretical use case... especially in 2019.

By ricbit

Champion (438)

24-05-2019, 20:54

turbor wrote:

@pencioner Using openmsx headless is possible and would work, the main drawback with this proposed solution is that there would be no feedback in case of errors. For instance lines to long, program to big to fit in memory etc etc
So you would need to check these conditions first and then assume that the emulated MSX will always be possible to parse the input without any error...

You can place a breakpoint in basic's error messages and catch the exception using tcl scripts. This way you can get the error reporting you need. I have done that in the code coverage tool for my msx json parser. In my case I placed the breakpoint at the FF07 hook ("Ok" handler), if basic prints anything but Ok then it's considered an error to me.

By ricbit

Champion (438)

24-05-2019, 20:54

turbor wrote:

@pencioner Using openmsx headless is possible and would work, the main drawback with this proposed solution is that there would be no feedback in case of errors. For instance lines to long, program to big to fit in memory etc etc
So you would need to check these conditions first and then assume that the emulated MSX will always be possible to parse the input without any error...

You can place a breakpoint in basic's error messages and catch the exception using tcl scripts. This way you can get the error reporting you need. I have done that in the code coverage tool for my msx json parser. In my case I placed the breakpoint at the FF07 hook ("Ok" handler), if basic prints anything but Ok then it's considered an error to me.