Yep, TomH is correct: MSX-HID is a protocol
Sorry to say this out loud, but I don't consider MSX-HID as a protocol. I think MSX-HID is a scam.
It is good effort to try to explain the world as it is, but it works more like a religion than technical definition. Why I say it is a scam is that it makes it look like it would be a technical standard or something, but when you read it more carefully, you realize it is just big pile of empty words. There are even things like "List of known signatures" but not a single word about where they come from... I think it is intentional. I bet if people would see the process used they would realize this is all just a fantasy and those IDs are not read, but generated pretty much out of thin air by using all kinds of tricks. It is not hard to come up with a bed time story like this by making rules out of physical limitations, taking nice parts of existing real standards and then spicing the soup with some imagination. If the result is not ok, you just tell that the device is not compatible with the "standard", easy!
NYYRIKKI,
I have great admiration for your technical skills, but your social skills are appalling.
I recommended this to you once, and I'll recommend again: you should read
this book ASAP: https://www.amazon.com/Nonviolent-Communication-Language-Lif...
Just because one begins with "I'm sorry", it doesn't mean that it's Ok for them to rape someones wife afterwards. If you don't like the MSX-HID, you're entitled to your opinion. What's not your right is to humiliate and defame me in front of everyone by calling me a scammer. At this side of the globe this is a very grave offence, and I doubt very much that this is socially acceptable where you live too. There's a line between being direct and socially inept, and you have just kicked your way over it.
The only reason why I'm replying to this clear act of trolling is because of your past technical contributions. If it was someone else, I would just have told him to f* up and shove his destructive criticism and insinuations up his ass. But I won't be exactly sweet on this reply, since you like to be harsh on others, I believe you must be able to handle the same kind of tone.
Let me tell you something obvious you seem to have missed: The MSX is commercially dead for more than 30 years. Reverse-engineering, programming, creating content for this platform is a complete waste of time when it comes to any commercial value or personal profit. There are plenty of better opportunities elsewhere. This means that when someone does it, it's just for personal hobby. The whole idea of a scam or hidden agenda in this context is so paranoid that makes anti-vaxxers seem pretty rational (at least there's a lot of money involved). Do you really think that the MSX-HID will be used to fool someone to obtain personal profit? Yeah, maybe it will look great on my resume and pave the way for a high paying salary. Probably some big university will call me to do presentations about this subject, or maybe I'll get rich by selling MSX-HID devices. Hah.
I repeat over and over that I come here just like I go to a bar: for the good friends, good talk, to exchange ideas, tips and share projects. But the negative attitude that's polluting these forums is getting on my nerves. It's hard to find a motivation to go back to a bar where there's a loud trigger-happy minority that don't lose a chance to expose their destructive opinions about anything they don't agree, and don't measure efforts to public humiliate people. Just the idea of praising others seems to make their bowls twist, otherwise what would explain that quite good news posts get less than a dozen of positive feedback messages.
It's really getting hard find motivation to use my scarce free time for any MSX project, when I know that when released, some troll will come out just because he doesn't like or consider that sacred soil was touched, be it a BIOS, sound fix, a turbo, a driver, a game patch, some pixel art collection, etc. I'm even getting out of options on things to work on, because more and more areas are becoming a minefield. The dogmas are well established, and anything different from the establishment seems to be target for being tossed on fire.
Sharing projects and reverse-engineering documentation info here should be just like preparing a dinner to some friends. And at least at this side of the globe, even if the dinner is horrible people will try to find a way to make constructive comments so the person who cooked it can improve. Destructive comments about the food? That's beyond being impolite.
Going past the attitude problem and back to the MSX-HID subject, I'll use your own arguments here: "As general feeling I think you are just very hard trying to misunderstand the whole point." Just like an anti-vaxxer, you seem to ignore all the evidence and only focus on the exceptions. Then, instead of calling me to a friendly talk to clarify your doubts, you preferred just to make a scene in public.
There are even things like "List of known signatures" but not a single word about where they come from..
It seems that the cognitive BIAS caused you to ignore this whole paragraph: "Since this standard was created in the 80s, the majority of the signatures were created with very simple circuitry by a clever selection of how the device bits were connected to the joystick port, sometimes forcing some bits to a specific value under certain conditions. This also means that the older devices took the simpler signatures, and the newer devices had to use the more complex bit patterns."
You already fiddled with the joystick port devices yourself. And you know that, for example, the bit3 of the two trackball coordinates is always sent inverted. Why do you think that it was designed this way? Because they were stupid, or wanted to make the life of the programmers harder? And you've seen that the GTPAD routine dumps the signature of the device just after it has read the mouse/trackball offsets. If it is 00h, it will decode the offsets as a mouse, and if it's 88h, it will decode the coordinates as a trackball. Are you intentionally ignoring this fact?
The same goes for the light-gun. They intentionally opted to invert the light-sensor return value so it could be detected when idle, and they chose a bit that wouldn't conflict with the older devices. Otherwise it was easier to just make the signal active-low just like any other device that connects to the joystick port, and that would have made it easier to use the ON STRIG GOSUB statement on BASIC.
The guy who developed the NinjaTap intentionally sets the pin-7 when the joystick port be the inverse of the pin-8 so this device can be detected with the very same algorithm.
Sega also followed exactly the same algorithm for the megadrive joypad, and as you can see on its schematics, the LEFT/RIGHT were intentionally wired to return zero when the port TH pin is set to zero (equivalent to the MSX pin-8 being set to 1). This was then a brand new console that had absolutely no other reason to return such signature other than to allow that controller to be detected. They could simply have just left those two bits unused to allow two more buttons to be easily added later in the future.
I really cannot see why do you find so hard to understand how the signatures are generated. All that the device has to do is to react to the pin-8 state on a predictable way when it's on idle position. If your device already has a flag that reacts to the pin-8, you can use that too as part of the signature. As I said in the article, things were constructed under the 80's philosophy of simplicity/cheapness. You don't need any advanced hardware to return a signature, just a keen engineer.
In the article, I listed all signatures I could find, to follow the same idea of the MSX I/O ports overview page: to have an exhaustive list to help both software and hardware developers. If it isn't clear, i'll repeat: I'm trying to help the developers. I really don't understand the destructive attitude against that.
You seem to be upset by the statement that some devices aren't compatible with the standard. That statement means just that, without any "religious meaning": such devices weren't designed to be detected, period. This stops people from repeatedly asking why their <insert exotic joystick/device here> isn't detected by JoyTest.
The standard MSX joystick itself can't be detected, since it was based on the Atari's that is older than the standard. It was just deliberately wired in a way that it wouldn't produce codes that would conflict with the signatures.
If all that explanation wasn't enough to demonstrate how it works and convince whoever skeptics are out there, I really don't give a sh*t about it. It will just prove to me that I should stop wasting my time with this platform.
OTOH, if anyone would like to design a device that can be detected using the same algorithm that works for plenty of other devices, they have the article as a guidance.
Sd_snatcher, do not focus on comments that are clumsy. As you say, everyone does with the time he has. And sometimes we do not take the time to see if our message can be misinterpreted. I do not think NYYRIKKI came to humiliate you. He just reacted too quickly and heatedly on what does not seem to him a good specification to his eyes.
The wiki is moving forward, it has clarified a lot about this standard. It is a very useful help that saves time for those looking for it. Before, I had not imagined how rich it was.
I did not find any trace about the MSX-HID in the documentations and it's almost unused in software but it's undeniable this protocol is usable. This can improve our programs. Thanks for it.
sd_snatcher: I do agree it was a bit of a troll post, but relax a bit. This is just one strong opinion, no need to feel quite that bad. If it helps at all, generally I think I agree what you said about me.
This means that when someone does it, it's just for personal hobby. The whole idea of a scam or hidden agenda in this context is so paranoid that makes anti-vaxxers seem pretty rational
This might be a bit of a language problem. It seems the word "scam" is now what makes your blood boil. What I mean by that has nothing to do with commercial value, trying to get personal benefit or anything like that. In my dictionary "scam" is just "something that is not true" (while I feel you know it).
I just mean that the theory you have come up with is not rational in my eyes at all and it leads people towards wrong ideas about how the system was designed. It glorifies the random design decisions like there would have been some big idea behind while I have no reason to believe it ever existed. I could have replaced this word with "false claim", "misleading", "prank" or something like that, but I don't know if it would have made your day any better as I still think MSX-HID belongs strongly to "conspiracy theory"-category.
I'm well aware you have done good job ie. with Joytest that is a very useful tool, but probably your work on that has fuzzed your judgement on this issue. You probably don't like me saying that either, but please consider this option for a while. Just think, can it be possible that you have looked the noise long enough to start finding patterns from there although no one placed them in?
Sorry that I made you feel that bad (again). In reality I would rather let issues fight than people, but I realize that in this case these complaints drop pretty directly to your desk...
You already fiddled with the joystick port devices yourself. And you know that, for example, the bit3 of the two trackball coordinates is always sent inverted. Why do you think that it was designed this way? Because they were stupid, or wanted to make the life of the programmers harder?
You know it is the highest bit, that is usually called a "sign". This is just matter of representing numbers signed/unsigned. We can see this used also ie. in V9938 adjust register. -> I can't see any "hidden agenda".
And you've seen that the GTPAD routine dumps the signature of the device just after it has read the mouse/trackball offsets.
Indeed GTPAD is quite clever routine, but I don't think there is hidden agenda here either. The thing is that mouse and trackball have very different protocols and it needs to do all these tricks in order to get coordinates from both of the devices at a same time without making them go out of sync or losing data. When you send additional "not expected" clocks to trackball, it does not send you any "signature", it simply returns no movement since the counters have been just reset by previous clocks. Yes, the guy who wrote the routine figured this out, but "signature" or "standard"? Come on...
The same goes for the light-gun. They intentionally opted to invert the light-sensor return value so it could be detected when idle, and they chose a bit that wouldn't conflict with the older devices. Otherwise it was easier to just make the signal active-low just like any other device that connects to the joystick port, and that would have made it easier to use the ON STRIG GOSUB statement on BASIC.
There are many different light transistors and light resistors that can be connected in various ways... and actually there are both ways used on MSX depending of what light gun you select. If now one model has this bit set by default right way around to make such a "signature" it is not that major innovation... If you take a random schematics from a book you have pretty much 50/50 possibility to make it "MSX-HID compatible".
The guy who developed the NinjaTap intentionally sets the pin-7 when the joystick port be the inverse of the pin-8 so this device can be detected with the very same algorithm.
Sorry, this device is unknown to me.
Sega also followed exactly the same algorithm for the megadrive joypad, and as you can see on its schematics, the LEFT/RIGHT were intentionally wired to return zero when the port TH pin is set to zero (equivalent to the MSX pin-8 being set to 1). This was then a brand new console that had absolutely no other reason to return such signature other than to allow that controller to be detected.
So, Sega must have been really interested about MSX as they have made MSX-HID compatible device even before no one had mentioned MSX-HID in public! This must be signature for us or a message from gods! Too bad they were so drunk they did not hit the pin8 you mention, so we can't really use it... and it even uses the signal inverted, so that if we could use it, it would be harder to use as is with MSX-BIOS... but of cource they wanted it to be MSX-HID compatible... or then they just accidentally sold 74HC157 chips to Sega as well. (Ok, I'll stop this ranting right now.)
I still think this whole conspiracy theory especially involving Sega & others is just insane. I think in reality they wanted to make a controller that is Atari compatible, but so that Atari controllers can't be used on Sega. End of story. These are just typical marketing tricks.
If you would say that you have figured out a pattern to detect lot of different kind of controllers, then I would be more than fine with this and I would show my thumbs up... especially if you share your knowledge... but now instead you have created this Wiki-page with mystical ID-numbers, talk about signatures and technical usage specs like (and I quote) "sometimes forcing some bits to a specific value under certain conditions." without any examples. Although it may give someone some idea, I don't think this represents you really being helpful.
Guys, I'm am aware that the article is far from perfect. It's just what was possible to put together with very limited resources.
Yes, it could have been written this or that way, and maybe I should have setup some crowd funding campaign in order to have some university-quality historic research. But there's one small detail that gets in the way: RealLife (TM).
OTOH, I have the impression that some very important details of the article are getting just a quick skim by who's reading it. I'll highlight:
- De-facto standard: This means that it's an informal standard. There's no written documentation.
There are plenty of such examples in the industry, and maybe the most well known is the 3.5mm audio connector. There's no centralised authority that publishes the specs for this connector. Sometimes de-facto standards become de-jure standards (meaning a standard with official specs). - Aggregated evolutions: This is normally part of a de-facto standard evolution. Unrelated parties make their own extensions to the standard, and it end up being adopted/cloned by the other parties. Some of the extensions end up being incompatible with each other. Taking again the 3.5mm audio connector as an example, first it was extended to TRS for stereo. And recently it was extended to TRRS for mic support/remote control. Some of the TRRS extensions are incompatible.
So, Sega must have been really interested about MSX as they have made MSX-HID compatible device even before no one had mentioned MSX-HID in public! This must be signature for us or a message from gods!
Hey, sarcasm seems to be clouding your judgment...
IMHO, it seems that Sega just cloned some ideas that they found useful, and kept improving it on their own. Sorry, but no big conspiracy theory to be seen here. You're just being paranoid about a very irrelevant detail.
Anyway, I should also have mentioned the convergent evolution effect. Given the same resources, constraints and requirements, there's a great chance that convergence evolution happens. This is why so many unrelated people designed similar planes back in the pioneering days.
Anyway again, I'm under the impression that too much attention is being paid to the package, and not enough attention to the contents.
Why? As I said, the article is far from perfect, but I tried to follow some good practices guidelines, and structured it the following way:
1) Name
The goddam thing needed a name, otherwise it's not even possible to create an article. Even the 3.5mm audio connector never had an official name, then some day someone coined the term name TRS as an acronym for "tip, ring, sleeve" because they needed to call it by some name. (BTW, in Brazil, this connector was named P2, and maybe not even God knows why).
2) Introduction
Paragraph-1: Gives an instant idea of what the article will cover.
Paragraph-2: Explains what it does, some brief explanation about what a de-facto standard is. I thought it wouldn't be necessary to dwell deeply inside the de-facto explanation because it would derail the article from its subject.
Also gives some overview about where it was used and how flexible it is.
Paragraph-3: Gives some insight about where where the standard was also used. It's more of a curiosity/trivia information told in relaxed tone, just to break the ice. I really thought that people would find interesting to know that it was used elsewhere, for whatever reason. I also did not want to go into a deep analysis about the reasons this happened because it would derail the article from its subject. It's just some fun fact, and not some precise historical research. Something to talk in the bar with your friends. It was meant just as a treat for the social side that I expected everyone here had.
3) Detection
Well, after the introduction has broken the ice with some (I believed so) fun information, now it was time to get to the dry technical part.
Paragraph-1: This is a technical overview. Explains that the detection takes some steps and that it was created with very simple hardware circuitry/tricks. They used the absolute minimal amount of hardware that was just enough to make it work.
Paragraph-2: Explains the algorithm per se. It's so simple that at least to me it didn't require more than that: Just read three 6-bit values from the port, and change the state of the pin-8 after between reads.
(Hummm. Reading it now again, I see that I forgot to mention that the pin-8 must be left LOW after the last read. This also means that the algorithm requires the pin-8 state to be flipped just two times).
Note: My original idea was to publish the sources of JoyTest and it's accompanying library soon after I published this article, so people could check the code and clarify their doubts themselves. But guess what? RealLife (TM) decided against this. I've an unreleased version of the tools and the code that I couldn't find the time to package, document and publish yet.
3) Premises
I thought it was important to describe some requisites/premises for things to work smoothly. Some of them are also rules of the thumb for simply using many of the devices that can be connected to the joystick port, regardless of MSX-HID.
Others, besides being pretty obvious, just had to be stated, like "do not create any signatures that will conflict with the standard MSX joystick operation".
4) Communication protocols
I wanted to make clear that there are no restrictions on the protocols used. Once the device is detected, it can communicate using whatever protocol it needs. I thought about recommending a good practice to prefer to use some of the existing protocols, but I feared that it could be controversial, so I left that part out.
5) Signatures of devices
As I mentioned before, this was meant as an exhaustive list of all devices I could find. New IDs can be inserted by whoever creates new hardware, or reverse engineer some existing device that I didn't try.
6) Incompatible devices that don't comply with the standard
It's a gentle/subtle way of saying: "Sorry, but the devices listed here just can't be detected this way. Please don't waste our time asking."
But I also felt that some explanation about why they can't be detected would be kind, so it's there on the last paragraph, alongside the explanation that they can still be supported with a manual menu selection.
---
Come on. This is getting tiresome. This whole justification here costed me way more time than it should. If it will be necessary to defend every article I publish like if I was publishing some doctorate thesis, it will just be more rational for me to rather use that damn time to write a real thesis on something that might get a god position on some university.
I don't think it causes any loss to anyone, nor hurts nobody reputation. It doesn't kill any daisies or polar bears. For Chrissake, it was just meant to help whoever needed to detect a damn device that will be connected to the joystick port! I just can't get why on Earth it is causing such heated reaction and so much opposition. Can anyone here explain me why? Because it must be something culturally related, that I'm just not aware that could possibly offend someone.
If you don't like the MSX-HID and don't find it useful/helpful, please spare me for this torment and just IGNORE it. You can just add some manual selection on your software and live without it. Plenty of ZX-Spectrum have some way to select the input device to be used and nobody ever died because of that.
If you find MSX-HID useful but don't like the way I've written the article, feel free to contribute to improve it. Your time will be better invested there than to keep needling me here about how I should have written the f*cking article.
The guy who developed the NinjaTap intentionally sets the pin-7 when the joystick port be the inverse of the pin-8 so this device can be detected with the very same algorithm.
Sorry, this device is unknown to me.
The PCCM NinjaTap is an adapter made by a circle of Japanese users that allows to use up to 4 joysticks by port.
http://www.gigamix.jp/entry/mlr/spec.html
I know some games that supports this adapter: Magical Labyrinth Remix, Atachatte 25% and a racing game like F-Zero.
I've already explained the article, but there are some specific claims that need clarifying.
When you send additional "not expected" clocks to trackball, it does not send you any "signature", it simply returns no movement since the counters have been just reset by previous clocks. Yes, the guy who wrote the routine figured this out, but "signature" or "standard"? Come on...
Aren't you being anal (*1) with the term "signature"? You know that the mouse will return 0,0,0 for the detection steps (as zero coordinates) when it is in idle state, or just after it has been reset by a full read. If you don't want to call this a "signature", call it a "fingerprint" then. I have just chosen the name that I felt that would allow the reader to understand the concept more easily. If you really prefer fingerprint over signature, please feel free to edit the article and replace all the places where it is mentioned.
If you would say that you have figured out a pattern to detect lot of different kind of controllers, then I would be more than fine with this and I would show my thumbs up... especially if you share your knowledge...
The wiki article was just the way I thought it was the best to share the knowledge... And, sorry, but I cannot take authorship for the idea. Besides the BIOS, other MSX programs used the same method way before me, at least to some degree, to detect some device(s) connected to the joystick port. It would be unethical from my part to claim that I was the author.
Nearly all software from HAL Laboratory I've seen, use this method to detect the Trackball. I've also seen some software from Sony doing it too, either to detect a mouse or a trackball. Sega also describes the same method of detection in its Saturn documentation. Before the Saturn, Sega of Japan used it to detect many devices on the Mega drive, but AFAIK there was no formal documentation published.
I'm sure there might be others.
If someone would be given credit for the original idea, I think it's the guys from HAL. The others seem to have expanded/improved on the same concept later. Unless we find some older software that implements it too.
but now instead you have created this Wiki-page with mystical ID-numbers, talk about signatures and technical usage specs like (and I quote) "sometimes forcing some bits to a specific value under certain conditions." without any examples. Although it may give someone some idea, I don't think this represents you really being helpful.
Well, I just thought that a list full of signatures (fingerprints, if you prefer) of how the devices respond when the pin-8 is flipped would be more than enough examples of how the bits react on that specific detection condition. If that's not enough, then I'm out of ideas of how to make everyone happy with the description, sorry.
*1: yes, that's a real English expression. Bizarre, but it is.
I tried to follow some good practices guidelines, and structured it
... and actually you have done a fine job with that. This structure is much better than on the Wiki pages that I've done for example. No any complaints about that.
Aren't you being anal (*1) with the term "signature"? You know that the mouse will return 0,0,0 for the detection steps (as zero coordinates) when it is in idle state, or just after it has been reset by a full read. If you don't want to call this a "signature", call it a "fingerprint" then. I have just chosen the name that I felt that would allow the reader to understand the concept more easily. If you really prefer fingerprint over signature, please feel free to edit the article and replace all the places where it is mentioned.
Actually now I can see that this is exactly where I did go wrong... In my head ID, "signature", part- or serial-number is something that manufacturer intentionally puts to the device in order for it to be identified later and that is why my head went to the wrong track and I did read rest of the text as instructions of how to read these ID-numbers that manufacturers have placed there back. Now if I switch these to fingerprints, I can actually jump out of this mode where my brains immediately after each few seconds kick in with internal "bullshit warning"-messages. :)
Yes, fingerprint is something that someone may leave behind unintentionally and keeping that in mind your text starts to make a whole lot of more sense although this should be somehow made more clear to the reader so that people know that this is not that kind of "exact science", but rather good tips that are results from physical properties & practical tests. I need to think about how to do this before I start fixing the page.
Well, I just thought that a list full of signatures (fingerprints, if you prefer) of how the devices respond when the pin-8 is flipped would be more than enough examples of how the bits react on that specific detection condition.
I now also understood that these "mystic numbers" are not so mystic after all... I think I was too deep in my mind on thinking things like reading serial communication from graphics tablets and such... If I got it correctly these are just list of results after sending 1.5 "clocks" and reading PSG register 14 (AND #3F) @ each rising and falling edge... Not really magical ID-bytes that are read back with various different undocumented ways...
Yes, this thing now makes lot more sense. Maybe I was too tired or something when I started to read this first time... You know... When you get it wrong once, it is very hard to turn your brains to read it correctly after that... but I must say your explanation helped a lot. Thanks (and sorry again)
FYI: I rewrote the MSX-HID page descriptions... I think I managed to keep all of the actual information, but I removed all the confusing stuff.
Indeed, 'fingerprint' makes way more sense than 'signature', as the latter connotes a 'de jure' standard.
Words are a bitch!
Speaking of the devil, I have another example - I hope less controversial - of an innacurate term on that wiki page: The word PWM is being used to categorize the devices that acts like the standard MSX paddle, but while the monostable inside the paddle _do_ respond to a pulse with a variable time, it _does_not_ modulate a cyclic wave. Another term, more exact, would fit better, like "Time Encoded" or "Variable Pulse Width" devices.