The ineffability of the RISC OS Data Transfer and Clipboard Protocols

Look and feel

The Data Transfer Protocol (DTP) is arguably the most important set of messages in RISC OS. Indeed, as it controls and enables loading, saving, transfer of data from program to program (and ultimately got roped into the Clipboard Protocol against its better judgement too) in many ways it is RISC OS, to most of its users.

It is important. It must be well understood and correctly implemented.

Things started OK in RISC OS 2 (with a big fat proviso). It was just about possible to implement it correctly given only the official documentation and a bit of luck. The documentation was bad, but not bad enough to break anything. Most implementations were wrong to some degree, just not noticeably.

But over time changes have been made. Some of those changes have not been well considered, and as of 2001 not only have all existing implementations been retrospectively broken, but the unintended consequences have gone as far as breaking Wimp_SendMessage itself.

I’m sorry Dave, I can’t do that

As we know, the DTP is mostly a four-way-handshake:

1. DataSave asks for permission to send

2. DataSaveAck gives that permission

3. DataLoad informs it has been sent

4. DataLoadAck confirms receipt

Programs can start at the third step, DataLoad, if the data is already on disc. Programs like the Filer can make use of a fifth message, DataOpen, to “launch” a file when double-clicked. (My Filer has a sixth – DataCheck – which allows the most appropriate app to load the file)

The DTP is elegant and well engineered, and would have been simple to document… but for a problem with the hardware.

The problem with the hardware is that at the launch of RISC OS 2 in October 1988, that hardware was most likely an A305 – half a meg of RAM, one floppy disc… and the !Scrap directory on the System disc which probably isn’t in the floppy drive anyway. (I remember it well)

Between steps 2 and 3 there was therefore a long grinding pause if you were lucky… and “Please insert disc ‘System'” otherwise. A work-around was required.

That work-around was badly designed and had knock-on consequences that are still felt today: Memory Data Transfer (MDT). A nasty, hacky add-on to the DTP. These days I/O is so incredibly rapid and hard discs (or their modern equivalent) are so ubiquitous that I recommend not bothering with MDT at all, it was always optional. But the damage was already done.

We will come to the MDT soon, but first we must look more closely at the DTP messages. The documentation in the PRMs is pretty clear. Unfortunately it is also Not Even Wrong, as we will see. What is not wrong is the most important sentence in the PRMs:

“A reply can always use the previously received message’s block just by altering a couple of fields”

We must define those terms, because this is crucially important to making sense of the DTP messages.

Reply has a very particular meaning for Wimp messages. It does not mean “a message you chose to send because of a message you received” – that is merely a response. A reply references a particular message using its your_ref field*.

Always means always. It does not mean sometimes. DataSaveAck is always made from a DataSave. DataLoad always becomes DataLoadAck, and so on.

Previously used to mean the message you have just received. In RISC OS 2 that was the only message you could be replying to. Unfortunately this changed as we shall see. “Previous” means the message whose my_ref reference you have placed in your_ref.

Altering hardly needs explaining but it is useful to stress: Those fields that are not explicitly altered will be whatever was received in the “previous” message. This is extremely important.

And Always really does mean always.

Every message given to Wimp_SendMessage is allocated a non-zero reference number, even if that message doesn’t get delivered to anyone. Every message has a reference. I refer to this field as this_ref – the reference of this message.

A reply also quotes the reference of the “previous” message to which it is replying. Originally this could only have been the chronologically previous message, but as we shall see this is no longer the case. I refer to that as prev_ref – the reference of the “previous” message. I hope this will be clearer. this_ref = my_ref, prev_ref = your_ref.

Now we must turn to the Memory Data Transfer (MDT) hack.

Daisy, daisy

The intention was reasonable enough: If the floppy is slow and inconvenient, why not transfer the data entirely in memory? If you drag something to the Filer then you intend it to go to disc, of course. But if you drag something from your application to another task, why touch the disc at all?

So there are two possibilities, and the sender can’t tell the difference – it is up to the recipient (the Filer knows it is not an editor and vice versa). An alternative to step 2 is introduced: When an editor receives a DataSave, it can choose to reply with a RamFetch message instead of a DataSaveAck.

RamFetch is an invitation for the sender to write data directly into the recipient’s memory. As tasks share the same address range and are nominally protected from each other, a method for achieving that write is provided by the Wimp: Wimp_TransferBlock. This is a horrific security blunder, which not only allows the sender to write anywhere in the recipient’s memory (not just that buffer), it can also be used in reverse to read any part of the recipient’s memory.

In fact every task can read and write every other program. This is why you must never run a web browser in RISC OS. If you do your internet banking on RISC OS you may as well paint your login and password on the front of your house. Literally every other program in RISC OS can read every keypress, every password, absolutely everything. Are you mad?

However we’re not concerned with security today, but with the messages themselves. Because unlike the DTP messages, the MDT messages are stupid. Really very stupid indeed. Idiotic.

The recipient sends a RamFetch message, which contains only a buffer address and the length – an invitation to write the data (but not outside that buffer! please no more than that length!). The sender then writes the data and replies with RamTransmit, which includes the same buffer address (which the recipient already knows) and how much data was written (which cannot be larger than the offered length), but not the size of the buffer (which the recipient must remember), nor whether the transfer has actually finished!

Inexplicably there’s no way for the sender to inform the recipient how much data will be sent. The sender can fill the buffer, but it can’t say “and there’s another megabyte still to go”. There’s literally no clue at all. So the recipient has to allocate more memory, send another RamFetch, and wait for the reply. If that buffer is filled, the recipient still has no idea how much data is left. It must simply repeat the whole process until, finally!, a RamTransmit arrives that does not include the whole buffer.

In fact, that RamTransmit may say that zero bytes were transferred in this final cycle. Regardless, the recipient finally knows how much data was received, and can modify its buffer length accordingly. It replies with DataLoadAck (step 4) just as originally intended.

Hint, hint...

The interesting question is this: How big should the recipient make the initial buffer? You might think that’s straightforward. The PRMs certainly seem to imply so. Let’s see the DTP message structure, as described in the PRMs:

• +20 destination window handle

• +24 destination icon handle

• +28 destination X coordinate

• +32 destination Y coordinate

• +36 estimated size of data in bytes

• +40 file type of data

• +44 proposed leafname of data, zero-terminated

“Estimated” is an interesting word, isn’t it? Estimated. Not “exact”. How “estimated” is estimated? If the file is five bytes but is “estimated” to be seven is that OK? What if it’s “estimated” to be 7KB, is that “wrong” in some sense? What if it’s 7MB?! 700MB! How near does “estimated” have to be? Is there any way at all for “estimated” to be wrong?

If not, then any value at all is an estimation! Is there any other interface in RISC OS that uses an “estimated size”? What if you know the exact size, should you lie about it? Is the word “estimated” actually defined?

There is a documentation disaster here, and to understand it we must start at Step 2 – DataSaveAck, for it has specific but apparently incomplete instructions:

“if the [recipient of the DataSave] is not the Filer, then it should set the word at +36 to -1. […] The Filer, on the other hand, will not put -1 in this word”

The -1 magic value is referred to either as “not secure” or “unsafe”, but the rest of the wording is highly ambiguous: If the Filer does not put -1 in that word, what value does it put there? The answer is “nothing” – it does not change it.

Recall the most important sentence – whatever is in +36 in the DataSaveAck will become +36 in the DataLoad, which will be +36 in the DataLoadAck. That is the purpose of this instruction in DataSaveAck: When the four-way-handshake has completed, and the sender of some modified data has received the final DataLoadAck confirming that the transfer has succeeded, it finally gets to update its modified status and adopt the filename in the message as the filename of the file… but only if the transfer is “safe”.

So the sender looks at +36 in the DataLoadAck and, if it is not -1, knows that the data is permanently stored. But is there any situation in which the sender knows in advance that regardless of how successful the transfer is it will not be changing its modified status or adopting the new filename?

Well yes, there are two cases:

1. Saving a selection

2. Exporting (destructively) as a non-native format

When saving a selection, even if successfully transferred to disc, you do not want to mark the document as “saved”. Equally, if you export your complex word processor document as Plain Text you do not want it marked “unmodified” – the formatting, pictures and layout have not been saved, just the text.

If you know that you’re not saving the whole document when you send the DataSave, wouldn’t it be nice if you could document that easily? Of course, as the most important sentence says, if you were to put -1 in the unsafe flag at the start, it would definitely still be there at the end: Your DataLoadAck code will look at +36, see it is -1 meaning “unsafe”, and therefore not change the modified status or adopt the filename. “But… estimated size?” you say. Well, is -1 not just a very bad “estimation” of the actual size? How estimated is too estimated?

In fact, this was the original intention of this field, and we have evidence of that in every version of RISC OS ever released:

Too big or not too big, that is the question

There are three DTP messages that initiate a transfer of some kind:

• DataSave

• DataLoad

• DataOpen

The Filer emits two of those: DataLoad and DataOpen. It sends DataLoad if you drag a file to a program, and it uses DataOpen if an object is double-clicked (or dragged to the empty iconbar). Both of those will end up as a DataLoadAck by the usual process… and we know what +36 is in DataLoadAck.

The crucial question to ask is, “Does the Filer know how big files are?”.

This is not a silly question – perhaps the Filer only knows how big things are when the Filer window is in “Full info” mode? Or perhaps only if the File Info window is open? Well, no. In fact the Filer always knows how big files are. All versions of the Filer know how big the files are, even as far back as the RISC OS 2 Filer 1.00 in 1988.

The Filer always knows how big files are.

So… if the field at +36 is “estimated size”, obviously the Filer puts the file size in that field.

Doesn’t it?

Nope

No version of the Filer has ever put a file size in that field.

Not in RISC OS 2. Not RO3. Nor 3.5, 4, 6 or even 5. Not for DataLoad. Not for DataOpen. Never.

The Filer NEVER puts a file size in a Data Transfer Protocol message.

The reason for this is very simple: The field at +36 is not a file size, and never was. It is the unsafe flag. It has another function to which I will return at the end as some sweet sugar to take away the taste of the nasty medicine. But we’re not even half-way yet. It’s going to get a lot nastier.

We know that this field is not a length in DataSaveAck, because it is the unsafe flag as per the specific instructions. We know that it is not a length in DataLoad or DataOpen, because the Filer does not put the file size in there (in any version of RISC OS ever released). As those messages immediately become, thanks to the most important sentence, a DataLoadAck, then we know it’s not a size there either.

So what on earth made some poor documenter write “estimated size”? To understand that, we have to return to the horrible Memory Data Transfer hack.

If I were going to go there, I wouldn’t start here

As we mentioned what must seem like a week ago, the MDT is very badly designed. It doesn’t know how far through it is. It requires the recipient to speculatively allocate a buffer and find out afterwards whether it was too big by a known amount or possibly too small by an unknown amount.

So where do you start? There’s a big difference between transferring 5 bytes and 5MB. So this is where some bright spark looked at the unsafe flag: -1 means unsafe, ok, but anything that isn’t -1 means something else, right? Safe. So maybe it could be used as a hint for how big a buffer to allocate to start the Memory Data Transfer?

Now there are problems right away. Firstly, the MDT is so badly designed that even if you know exactly how much data to expect, allocating that size buffer will merely result in you not knowing whether you’ve finished – the transfer hasn’t completed until a buffer was not completely filled. The data keeps coming until it stops… there is no absolute length.

And even if it is helpful to know whether 5 bytes or 5MB are likely to come, nevertheless that can only be a hint, because the protocol continues until a buffer is not filled. And however big your first buffer was is not a good guide for how big your next buffer should be:

If you are expecting to get 5MB of data, and that buffer is filled, either the transfer has finished but thanks to the stupid message design there’s no way of knowing yet, or there is more to come… but whether that’s 1 byte or another 15MB is impossible to tell. So blindly allocating another 5MB is probably not a good idea.

So the field at +36 in a DataSave and ONLY a DataSave is not the “size of the file”, nor is it an “estimate” of the size of the file. It is merely a hint as to what size buffer to allocate if you’re going to reply with RamFetch.

It thereafter plays no other part in the transfer. If another buffer must be allocated (and it probably will because of the stupidness) using the same size is likely to be overkill. The value may not be sensible anyway – it might be the unsafe value of -1 because the program is saving a selection, or the “estimate” may be insane. And if you don’t use RamFetch then you are to ignore it anyway (and overwrite it with -1 if you aren’t the Filer).

There’s no point refusing to start the RamFetch if the estimate is larger than available memory because it’s only an “estimate” anyway, whatever that means (it might say 5MB when the data is actually 5 bytes). And as app slots are allocated in whole pages, you might as well round up the buffer allocation to a multiple of the page size anyway… perhaps the sender already did that? Who knows. It’s not defined.

Not even wrong

You can sort of see how “estimated size” came about. You can imagine the arm-waving description of the hackery of the ill-designed Memory Data Transfer into the elegant Data Transfer Protocol. It’s difficult to communicate the subtleties of “a hint for an initial buffer allocation when using the RamFetch but not an indicator of the actual size and in fact the optimal value is a larger value than reality in order to avoid further buffer allocations, but this can only be used when not used as an unsafe flag, so recipients are free to ignore, override or use any heuristic they see fit”.

“Estimated size” in DataSave is Not Even Wrong, but let’s be clear:

DataSaveAck, DataLoad, DataLoadAck and DataOpen do NOT contain a file size. And DataSave only contains a hint for a buffer allocation if using RamFetch… maybe. Unless you completely ignore it. And it can’t be negative as we’ll see.

My advice is this: Never put a file size there. Either put zero (when saving) or -1 (when exporting). No RamFetch user should ever have taken this value as gospel. The word “estimated” is simply not defined. Never was. If you feel like helping RamFetchers, choose a sensible size; make it slightly too big; make sure it’s not negative; or just use zero, that’s fine too.

And you must never, never treat any value in +36 of DataSave as being even vaguely related to the actual size of the data. It is not, and never was, an accurate representation. It might be in one case, it definitely isn’t in another. And because “estimated” was never defined, there’s no way of knowing how wrong it is. It could be seven orders of magnitude wrong, and still not be “wrong”.

And you might think we’ve got to the end, but we have not. Because this is where things really do start to go seriously wrong.

Unsafe becomes safe... becomes very unsafe

Everything I’ve described so far relates to RISC OS 2. In 1991 there was a significant change, and its repercussions were not adequately understood. A new Data Transfer Protocol message was added… and it’s amazing.

DataSaved was added in RISC OS 3. Its purpose was straightforward – what if you transferred your modified document to another program and then that program saved the document… doesn’t that mean your data is now “safe”?

RISC OS was intended to feature many little programs that did little things, all working together. So the concept of dragging data from one program to another was a core concept. But this scenario wasn’t originally envisioned. When you only have half a meg of RAM, not many programs can be run at the same time even if they are small. But by 1991 some machines had lots more memory. So DataSaved was born: A method for an editor which had previously reported itself as “unsafe” (via the unsafe flag in the DataSaveAck) to retrospectively announce that in fact it had saved the file, so the sender could modify its status and adopt the new filename.

This is an amazing thing to see in action… but you won’t have seen it in action because, apart from me, no one implemented it. How could they? The PRMs get this message completely Not Even Wrong. They don’t even adequately describe why you’d want to implement it, never mind how.

This is a problem, because the how introduces some mind-bending new requirements on existing implementations… and also managed to break the entire concept of Wimp_SendMessage at the same time.

Give me your references... no, not that one

Firstly we must recall the most important sentence, because DataSaved seriously taxes the definitions. First, under what circumstances would you send a DataSaved? Well, obviously, when the data that was transferred to you (unsafely, remember) has become “safe”. That can happen in three ways:

1. You transferred it elsewhere, and they sent you a DataSaved. This is the DTP equivalent of Panspermia

2. You dragged it somewhere and you’ve just received a DataLoadAck saying it is “safe”

3. You typed a full pathname and pressed Return, which saves the file directly without using the DTP

In the last case there wasn’t a “previous message”, so the most important sentence doesn’t apply – you have to build the appropriate message from scratch. We’ll come back to this after we’ve examined the other two because the PRMs are no help at all – we will eventually understand why.

In the first case it’s easy – you just received a DataSaved, so you can pass it on to the sender of your data… but that doesn’t help us understand what a DataSaved looks like.

The second case is the helpful one – you’ve just received a DataLoadAck which says the destination was “safe”, and as usual it contains the filetype and the full pathname the data was saved as. The most important sentence is clear that if you now need to send a DataSaved, it will be done by “altering a couple of fields”. But which ones?

Well, the message type, obviously. And, because it is sent as a reply, the prev_ref… but that won’t be the this_ref of this message – this message is a DataLoadAck from the recipient of your data, you need the reference from the original sender of the data. So the two fields that change are the type, and the prev_ref. And what does the useless PRM say?

• R1+12 reference from DataSave message

• R1+16 13 (the DataSaved message number)

Well, yes, those are the only two fields we have to alter to turn a DataLoadAck into a DataSaved… but the PRMs make a really good job of making it look like those two are the only fields in that message. They’re not! We know they’re not because of the most important sentence, but even so, that’s a fantastically unhelpful bit of documentation.

DataSaved is one of the Data Transfer Protocol messages, so of course it has the same structure as all the others. This means it contains the filename the data was ultimately saved as.

And that means that when an editor receives a DataSaved confirming that the previously unsafe data has now become safe, not only do we expect the “modified” status to change… we expect the filename of that document to change too. Just as if the original editor had saved to the final destination.

It’s quite magical when this happens. No other OS works like this. Sadly the only machine on the planet that does behave like this is mine. Perhaps that may change now.

And now we know what to build in the third case above (when the user has typed a full pathname and pressed Return) – a full DTP message, just like the others.

So that’s all good news, yes?

Nope. No. Because there are a couple of extraordinary decisions enshrined in DataSaved. Firstly there’s the reference used to identify the original data – I quoted it above, the reference from the DataSave. The DataSave? What?!

DataSave, as we recall from last year when you started reading this article, is a request for permission – it can be refused. Even if it elicits a DataSaveAck, the sender may fail to send DataLoad. Even if there is a DataLoad, there may be no DataLoadAck. We already know that it is the DataLoadAck that confirms that the transfer has occurred successfully. When data is transferred to us then it’s our DataLoadAck that confirmed it… and we’d definitely have that reference, as would the sender.

But DataSaved wants the DataSave reference. It wants to reply to the DataSave… exactly like a DataSaveAck does. It’s at this point we have to look even more closely at how messages are handled by applications.

Here’s your receipt for your husband, and here’s my receipt for your receipt

It was so simple in RO2: You send a DataSave, and remember its this_ref. There can only be one Data Transfer occurring – you can only drag one icon – so you only need to remember one this_ref. When (if) you receive a DataSaveAck, you compare its prev_ref to the this_ref you stored – if they don’t match then this isn’t a reply to your DataSave and you should ignore it.

Aside: There’s loads of other things you should check too. For example, the length of the message. There’s no point checking the filetype at +40 if the message was only 32 bytes long – you’re checking whatever was lying about in your memory. “But nothing sends the wrong length messages” you’re probably thinking. Oh yes they do. They often do. Acorn authored programs do. Paint does, for example. Even the latest RO5 version. Just cos it’s “official” doesn’t mean it’s right. In fact the worst implementation of the DTP in the world – one that’s so bad it can cause uninvolved tasks to crash – is still distributed with RO5 today: Maestro. Anyway.

So you’ve got your DataSaveAck and it matched your stored ref, so you save the data and send DataLoad. Of course you set its prev_ref to the this_ref of the DataSaveAck, and you remember the this_ref of the DataLoad.

Then, finally, when you get DataLoadAck, and its prev_ref matches your stored this_ref and you’ve dealt with the safe/unsafe thing… you don’t need that this_ref any more. In any case, you only had the one. You probably called it savingref or something. Ah yes, RISC OS 2 was nice and easy.

Unfortunately RO3’s DataSaved rides a coach and horses through that. It requires the this_ref from the initial DataSave. Not the final DataLoadAck that confirmed the transfer worked. No, the DataSave that invited it. Which you therefore have to keep until DataLoadAck has been sent and you’ve got the data in memory – and you then have to store that DataSave this_ref (and the sender’s task handle) with the data until one of three things happen:

1. You save the data and send the DataSaved

2. You discard the file

3. You change the data in some way

The third one is easy to forget, but if you change the data, then even if you save it, the data you’ve saved is not the data that was transferred to you, so you mustn’t send a DataSaved. And if you’re not going to send a DataSaved, you don’t need to remember that reference.

So that’s a bit of faff, keeping a DataSave reference knocking about until the DataLoadAck has been sent… but that’s not the nasty bit. The nasty bit is at the other end – what does the sender have to keep?

How many?

The sender has to remember the this_ref of the DataSave it sent to initiate the transfer to you. Now it might get no reply, or it might abort the DataLoad, or it might not get the DataLoadAck, but assuming all that happened, it must then store the this_ref of the initial DataSave so that it can match it with the prev_ref in the DataSaved if it arrives. But which DataSave?

You could reasonably drag the unsaved modified data to three different programs. That’s three different DataSaves, three DTPs, three recipients. Each of those recipients will remember the this_ref of their respective DataSaves, and any of those recipients might save the data and send a DataSaved.

So that means… the sender has to store the this_ref of every DataSave it sends. Not one reference, but all the references. However many of them there are.

You will note that there is no un-DataSaved: There’s no way for a recipient of an unsafe DataSave transfer to say “oh I’ve discarded that now, so you don’t need to keep that this_ref because I won’t be using it”. It can be discarded if it appears in a DataSaved, but otherwise it must be kept. They must all be kept… for how long?

How long?

By symmetry there are three reasons why the sender can discard a DataSave reference:

1. The data has been saved

2. The data has been discarded

3. The data is modified again

But if none of those happens, the sender must keep a list of all DataSave refs, just in case one of them comes back in a DataSaved.

None of this is described adequately in the PRMs, but the natural consequence of this hasn’t been described anywhere: This isn’t how references were designed to be used.

In RISC OS 2, you either replied to a message immediately, or you did not reply, and the reference of that message was no longer valid. But thanks to DataSaved an entirely new concept was introduced: the long-lasting reference. In RO2 a reference was only valid until the next poll. If you hadn’t used it by then, you were never going to use it. But now, both sender and multiple recipients must keep hold of message references until they don’t need them… and if you just get up and walk away from your computer, that could mean weeks.

This is a problem, because the repercussions were not considered. As I said during the Mesozoic Era when I started writing this article, every time you call Wimp_SendMessage the message gets a unique non-zero reference. In RO2 this was uncontroversial. But thanks to DataSaved, there are now a number of message references that continue to live on and may be used minutes, hours, or days from now.

Wimp_SendMessage doesn’t know this. It will happily wrap around, avoid zero, but then allocate the exact same message references again. Which, if any of these programs receive them, will result in data being marked “saved” that isn’t, filenames changing. Bad things!

What should have happened is to have introduced a new Wimp call to allow sender and recipients to register their interest in a message reference, so that the Wimp could avoid reusing it (something like Wimp_Extend,516,ref,±1, he writes, overspecifically). Once everyone had deregistered interest, it would be free for reuse. But no, this was not anticipated. Back in the RO3 days, having your Archimedes run all day without crashing was quite an achievement. The chances of a 32bit message reference actually wrapping round were effectively zero.

But these days computers run for days, weeks without being reset. I can keep 20 or more windows open all week. And if you have interactive help running, or a TaskWindow generating output, the number of messages being generated is huge – the Wimp is ticking off message refs multiple times per second. Wrap-around is to be expected.

So DataSaved is very badly documented, requires big changes to the way we’ve always implemented the DTP in order to operate correctly, and it broke Wimp_SendMessage without anyone noticing.

So we’re there right, that’s the end of the DTP badness?

Nope, because now we have to look at the Clipboard Protocol. Because this is where absolutely every program there is, completely broke.

Cut that right out

I forget now whether the Clipboard Protocol was designed by Iota, by Acorn in conjunction with them, or merely enthusiastically embraced by the authors of DataPower. Regardless, the Clipboard Protocol caused problems immediately, because it wasn’t what other people wanted.

The protocol was designed to work in a particular way. And unlike the DTP, this fully included the user experience. This is how you should experience the clipboard functionality it says, in effect. Unfortunately they (whoever ‘they’ were) got it wrong. This is how the Clipboard Protocol is designed to be used:

1. User presses Ctrl-C in a program, which causes it to claim the clipboard (this task is the clipboard owner)

2. User switches to a different program (the focussed task) and presses Ctrl-V

3. The focussed task requests the clipboard data from the clipboard owner

4. The data is delivered by DTP

This seems entirely reasonable. But there’s two honking great misdesigns at step 1. And authors were having none of it. The problem is with the claim of ownership. This is the ClaimEntity message. Unlike DataRequest, which includes a filetype list, ClaimEntity (as defined) does not. This means its contents say I have the clipboard and I’m not telling you what it is.

Not only that, but it specifically says that if the program had already claimed the clipboard when it copied something different, even a different filetype, it shouldn’t send another message, which means I have the clipboard and I’m not telling you I’ve changed it.

The belief of the protocol authors was that the user will not try to paste a Draw rectangle into an Edit text file. That every request from the user will be sensible, and actionable. Let’s expand the above into actual messages:

1. Clipboard owner broadcasts ClaimEntity

2. User switches focus, presses Ctrl-V, and editor sends DataRequest containing a list of up to 53 acceptable filetypes

3. The clipboard owner replies with DataSave – if it can supply one of the requested filetypes it will choose that filetype, otherwise it must use its “native” filetype

4. The editor receives the DataSave and if it does not accept the filetype, silently ignores it

From the user’s point of view, the editor has “Paste” on its menu, or a Paste icon on a toolbar which, when clicked, does nothing at all.

That is how the clipboard is designed to work – mysteriously, intermittently, inscrutably. Needless to say, many software authors thought this was a bit rubbish, so they implemented a work-around (and this is a common pattern):

1. Editor receives ClaimEntity

2. Editor immediately sends DataRequest

3. Editor will eventually receive a DataSave containing a filetype

4. It ignores the DataSave but, depending on the filetype, enables or disables “Paste” in its user interface, or even augments the text with the filetype – “Paste Sprite”.

This is a much better user experience. If the clipboard contents aren’t suitable, paste is not presented as an option. Some programs defer doing this until just before opening their menu, but for any program with a toolbar that isn’t an option – the DataRequest must be sent immediately. What was not anticipated is that this completely breaks all implementations of the DTP, and changes a fundamental tenet of the protocol.

It doesn’t work like that

The above transaction seems to make sense as a few bullet points, in isolation. But the tasks concerned aren’t in isolation, they are running alongside other tasks. Any of those other tasks may do the same thing. This is where a crucial detail becomes terribly important: ClaimEntity is a Broadcast.

It’s easy to think of the DTP as a game of tennis, with “the ball” (a message) bouncing back and forth between the tasks. It’s a different ball each time, but thanks to those prev_refs it’s all the same volley. But this is merely an illusion. There isn’t one ball, there are potentially many balls… but only one of them is in the air at a time. The rest of the balls get queued up. There aren’t two players on court, there are many – and they are all playing different games, usually.

But a Broadcast is different. This is a ball that gets served at every player (yes, even the player that sent the ball, weirdly). And it stays in the air until everyone has received it or it has been destroyed. A message is destroyed by replying or acknowledging it. This means the above pattern does not work the way you picture it. Instead, this happens:

1. Editor receives ClaimEntity

2. Editor sends DataRequest but that message just gets queued up and is not delivered yet

3. Another editor gets the ClaimEntity

4. That editor sends a DataRequest and it gets queued up too

5. A third editor gets ClaimEntity…

This should already be rather alarming.

Finally the ClaimEntity broadcast has been delivered to all tasks and is discarded, so the Wimp has to decide what to do next. Oh, it’s got a pile of messages queued up. And they’re all for the same task: The clipboard owner. The clipboard owner may well be your task, that you wrote. The clipboard protocol uses the DTP, that you’ve implemented following the PRM documentation. This is old code. You know how it works. So now this disaster happens:

1. Clipboard owner gets a DataRequest, chooses a filetype and replies with a DataSave

2. Being a normal implementation of the DTP, it remembers the this_ref of the DataSave, perhaps calling it savingref – it will need it to check the DataSaveAck and, if it’s an editor too, it may implement DataSaved so will need all the DataSave refs for that

3. It calls Wimp_Poll

You probably expect the Wimp to deliver that DataSave don’t you? But the Wimp does not work like that. It really doesn’t like swapping tasks. Not when there’s something else for this task to do, and there is: The Wimp still has a pile of DataRequests to deliver, so the Wimp_Poll returns immediately to the clipboard owner with another DataRequest.

Your task does the exact same thing: replies with a DataSave and remembers the reference. In the process it entirely forgets the previous reference. After all, we all know that you can’t have multiple data transfers in progress simultaneously right? So it calls Wimp_Poll and guess what?!

Another DataRequest, another DataSave, another this_ref lost. It was a fundamental truism of RISC OS 2 that there could not be multiple simultaneous data transfers, so programs only keep the one reference to check against replies. The Clipboard Protocol inadvertently destroyed that assurance.

Why did nobody notice this breakage? Well, fortunately none of the DataRequesters actually mean it – they just want the filetype from the DataSave, which they otherwise ignore. The problem is there, it just isn’t apparent… yet.

This is why I recommend you add a filetype list to every ClaimEntity you send, and look for one if you handle the message. Wimp messages can be easily and compatibly extended – that’s why they have a length field. Backwards compatibility is assured because tasks ignore fields they don’t look for. So ClaimEntity can have a filetype list added with no ill effects. Editors that recognise that there’s a list (thanks to the length) can immediately check whether or not to enable Paste as an option, without any further messages. This avoids a lot of agro and delay. You should also emit a ClaimEntity when you change the clipboard even if you already own it, because a new task may have just been started that didn’t get the original claim.

Just when it couldn’t get worse

We’re only up to RO3 here. Then RO4 came along, and with it an innovation to work around another misdesign of the Clipboard Protocol: If you copy something onto the clipboard and then quit the clipboard owner, the clipboard disappears. This is just bad.

So RO4 added ClipboardHolder. Other third-party equivalents exist. The concept is simple: When something sends ClaimEntity, ClipboardHolder requests the contents of the clipboard and becomes the new owner. The originator can be removed and the clipboard contents persist. Much better.

There are significant problems with the RO4/6 implementation of ClipboardHolder, but the more fundamental problem is that it is doing the exact same thing as the editors above – it responds to a ClaimEntity with DataRequest.

That means that in addition to an unknown number of editors sending DataRequests but ignoring the resulting DataSave, with the addition of ClipboardHolder one of them actually means it. That means one of those DataSaves will produce a corresponding DataSaveAck returned to the owner. And unless it happens to be the most recent DataSave, the owner will have forgotten the this_ref and will ignore the message.

So sometimes ClipboardHolder works, and sometimes it doesn’t (and this applies to anything else using the same pattern).

That all of this has happened is not down to stupid people. It is caused by bad documentation. Vague wording. Incomplete structures. Undefined terms. Almost all RISC OS APIs are ill-defined – incomplete. This is why no one can say what the API of OS_WriteC is (ridiculous but true, but a story for another day).

And... relax

So what have we learnt?

• The PRMs are terrible

• The field at +36 is not a file size, ever

• RamTransfer is terrible

• DataSaved means you have to store one sender this_ref (and its task handle) and any number of DataSave refs with every file you edit

• DataSaved means refs can last forever and Wimp_SendMessage is broken

• The clipboard protocol means multiple data transfers can occur simultaneously

• ClaimEntity should have a filetype list and be sent every time the clipboard changes

• Paint is bad but Maestro is worserer

• nemo has a magic bonus surprise that has never been documented…

New and improved is actually very old

Picture the scenario: You have a fancy word processor, it’s really good. But true to the RISC OS principle of small-is-better it doesn’t have a wizard, or a set of blank templates from which to start, just a blank A4. But this is RISC OS! You could add that functionality with a separate program!

You can see how it would work – present some thumbnails in a Filer-like window. The user clicks on the “New newsletter” icon, and the wizard launches the appropriate file. The word processor loads it and hey presto, the template has been opened. You even know which message to use – DataOpen, just like the Filer. It has nearly written itself!

But there’s a flaw in this design.

The user starts typing their ten-thousand word diatribe on The Evils of the Data Transfer Protocol when the phone rings and Significant Other enquires where, precisely, the author is waiting outside the station to pick them up as promised… to choose a scenario entirely at random.

So the author (that other author, not this author you understand) jumps to attention, mashes F3, Return to save the file, and flees. Can you see what just happened? The user has just overwritten the blank template file with their work. Easily done.

You could lock all the template files. But it’s still rather horrible that the titlebar will bear the full pathname of the template file even though it should never be saved back there, and the save dialog will contain the same.

If only there were a way of launching a file in “template mode” so that the editor loads it, but thereafter treats it as a brand new file.

That would be good wouldn’t it? That would have been a great feature of the Data Transfer Protocol. If only they’d thought of that and put it in from the start…

Well…

They did.

It’s called template mode and the DTP includes exactly this functionality.

It is selected by putting a different magic value in that field which definitely isn’t the file size. If you set unsafe to -2 in a DataOpen it means “open this file as a new file” – discarding the full pathname. The editor should then retain only the leafname of the file.

You will note that this has never been documented. We are perhaps less surprised by that than we used to be. Has anything ever implemented it? Well yes, and I don’t just mean all my programs and the copy of Zap I patched.

Edit and SrcEdit both implement template mode (though they do so differently) and have done so since RISC OS 2.00!

Unfortunately it was so not-documented-at-all that the authors of Paint and Draw did not get the non-memo, so they did not implement template mode, which is a great shame. And this is the first you’ve heard of it, so you haven’t implemented it either.

But you will now, won’t you. There is a program here that will help you test your implementation.

In fact, Edit and SrcEdit are rather wonky in their implementation. They actually treat any negative number that isn’t -1 as being -2, but that’s just laziness – it ought to be tighter. After all, maybe we’ll think of other magic values one day.