ASH - Automatic Syntax highlight (Attempt 2)
[Please note: I have my Delphi Editor colors set-to the [Ocean] colour speed settings for testing purposes. This setting works well on the default RichEdit white background, and most TokenTypes are in different colors from each other]
Okay now to do some real work. Most of the function have been written thereabouts. As a basis for writing this ASH I'm going to use Project1.dpr which comes out of mpas2rtf.zip in the YourPasEdit zip file yrpasedit.zip. This is because it much smaller than YourPasEdit, and thus quicker to compile.
I suggest you put the contents of the mpas2rtf.zip into a separate directory. Also copy mwPas2Rtf.pas to testinput.pas using the Explorer shell - we'll be using this file as a sample pascal file for benchmarking.
Open Project1.dpr in Delphi, compile Project1, run it, and open the file testinput.pas by pressing [Button 1] and selecting it in the [OpenFile Dialog]. Do it a number of times, and record the time taken for each once the file is stabilised in the system cache. On my system it averages about 0.47 - 0.41 seconds once its in the cache (P133 - 16M - Win95b)
Preparing Project1's Unit1.pas
Now replace the contents of mpas2rtf.pas with that code in jhdpas2rtf.pas. Recompile. Now open up the testinput.pas sample file again by using [Button 1]. As you see - we get color - but it takes a "lot" longer: 1.20-1.25 seconds.
Try and speed it up if you like. You can start by commenting out the pascal-code that codes in the different Font and FontSizes in TPasConversion.SetRtf. Recompile and run again. This time it improves a bit to 1.10-1.15. Now try commenting out the code for different Colors. Wow - the speed decreases down to 0.49 - 0.44.
Hmm. This font and color stuff really packs a punch. We may need to look at this later in more detail if things end up too slow. For the moment we'll leave the code back in full working condition (so you'll need to go back and uncomment the code).
Now put the following base code into the [OnChange] event of the RichEdit1 in Unit1.pas of Project1. Most of this code is just based on what we have already covered elsewhere.
procedure TForm1.RichEdit1Change(Sender: TObject);
var
WasSelStart,WasRow,Row,BeginSelStart,EndSelStart: Integer;
MyRe: TRichEdit;
MyPBuff: array[0..255] of char;
begin
MyRe := TRichEdit(Sender);
WasSelStart := MyRE.SelStart;
WasRow := MyRE.Perform(EM_LINEFROMCHAR, MyRE.SelStart, 0);
BeginSelStart := MyRe.Perform(EM_LINEINDEX, Row, 0);
EndSelStart := BeginSelStart + Length(MyRE.Lines.Strings[Row]);
Row := WasRow;
end;
Were going to use the GetToken() function to do all the hard work. We'll need some extra variables to pass to the GetToken function, so add to the var section:
MyTokenStr:string;
MyTokenState:TTokenState;
MyRun:PChar;
MySelStart: Integer;
These are similar to the variables we used in the ConvertReadStream - in fact we want to do "exactly" the same thing, just one single line at a time. Add this code before the last end;
StrPCopy(MyPBuff,MyRE.Lines.Strings[Row]);
MYPBuff[Length(MyRE.Lines.Strings[Row])] := #0;
MySelStart := BeginSelStart;
MyRun := MyPBuff;
while(MyRun^ <> #0) do
begin
MyRun := PasCon.GetToken(MyRun,MyTokenState,MyTokenStr);
//
// ScanForRtf;
// SetRtf;
// WriteBuffer(Prefix + TokenStr + Postfix);
//
end;
end;
NB: As we will be using PasCon you'll have to move it from being a local variable of TForm1.Button1Click to be a global variable. This will mean you'll have to move all the initialising:
PasCon:=TPasConversion.Create;
PasCon.UseDelphiHighlighting(3);
to a TForm1.Show, and the PasCon.Free to TForm1.Close procedure. It will still work if you only move the variable definition - but not for long... :-)
I've left the code from the old ConvertReadStream in the example above to show what we "logically" still need to implement in the current context - that is manipulating the RichEdit Control directly. What we have now is the ability to cut up the current line in to different tokens, and know what type they are. We now have to add these tokens to current line with the right attributes (Fonts,Colors,Bold etc).
But wait. They are already on the line - well the text is anyway, but maybe not in the correct format (Color,Bold etc). So what actually could do is to select each token in its corresponding positon in the RichEdit control and just apply the appropriate attributes to them.
We did this back in the beginning remeber? When we set the >10 character lines to the color red. But how do we do this now? Lets look at what we have in the variables at hand when we hit "// SetRtf" the first time:
(these example uses Uni1.pas as the input file as its more interesting)
VARIABLES
01234567901234567890
Lines.Strings[R0]
unit Unit1;
MyPBuff
unit Unit1;
MyTokenState
tsIdentifier
MyTokenStr
unit
MyRun
Unit1;
So what we need to do is select the word 'unit' in the RichEdit control, and set its attributes. We do this by setting SelStart to the position of 'unit' in the RichEdit control, and SelLength to the length of the word 'unit'. And since 'unit' is at the beginning of the current line - thats position is BeginSelStart (which I conveninently have stored in MySelStart - you'll see why). Lets replace the "pseudo" comment code with the following:
MyRe.SelStart := MySelStart;
MyRe.SelLength := Length(MyTokenStr);
MyRe.SelAttributes.Assign(PasCon.FParseFont[MyTokenState]);
end;
But remember we are in a loop - when we go around again we'll have the next token in the line, and the variables will look like this:
VARIABLES
01234567901234567890
Lines.Strings[R0]
unit Unit1;
MyPBuff
unit Unit1;
MyTokenState
tsSpace
MyTokenStr
(space character)
MyRun
Unit1;
But (space character) isn't at BeginSelStart (#0) in the RichEdit control. Its further along (at position #4). Which just happens to be BeginSelStart + Length('unit'). We need to update MySelStart after we process the preceeding token, but before we go around the loop again:
MySelStart := MySelStart + Length(MyTokenStr);
end;
Okay - this is where we are standing at the moment:
procedure TForm1.RichEdit1Change(Sender: TObject);
var
WasSelStart,WasRow,Row,BeginSelStart,EndSelStart: Integer;
MyRe : TRichEdit;
MyPBuff: array[0..255] of char;
MyTokenStr:string;
MyTokenState:TTokenState;
MyRun:PChar;
MySelStart: Integer;
begin
MyRe := TRichEdit(Sender);
WasSelStart := MyRE.SelStart;
WasRow := MyRE.Perform(EM_LINEFROMCHAR, MyRE.SelStart, 0);
Row := WasRow;
BeginSelStart := MyRe.Perform(EM_LINEINDEX, Row, 0);
EndSelStart := BeginSelStart + Length(MyRE.Lines.Strings[Row]);
StrPCopy(MyPBuff,MyRE.Lines.Strings[Row]);
MyPBuff[Length(MyRE.Lines.Strings[Row])] := #0;
MySelStart := BeginSelStart;
MyRun := MyPBuff;
while(MyRun^ <> #0) do
begin
MyRun := PasCon.GetToken(MyRun,MyTokenState,MyTokenStr);
MyRe.SelStart := MySelStart;
MyRe.SelLength := Length(MyTokenStr);
MyRe.SelAttributes.Assign(PasCon.FParseFont[MyTokenState]);
MySelStart := MySelStart + Length(MyTokenStr);
end;
MyRE.SelStart := WasSelStart;
MyRE.SelLength := 0;
end;
Now: put the Debugging code on, do [Build All] and then [Run], and set a breakpoint on the first line of this Event. Open up the testinput.pas. When the debugger stops in the OnChange event, Press <F9> to continue on, and press <F9> again, and again - Do you see? We keep going back into the Event again and again (and again). What’s happening?
Somehow in our event we are triggering off another [OnChange] event. This call to the [OnChange] event code is stored in the message queue. When the event were currently in is finished, a new one is just waiting on the Event queue, which executes and creates more events... a re-entrant loop.
This behaviour is not surprising - after all we are actually changing the control in the process of our code, so no wonder another [OnChange] event is being triggered.
The way to fix such things is to ensure our actions do not trigger of the Event. We can do this by "temporarily" storing the RichEdits.OnChange property (which contains a reference to call our procedure TForm1.RichEdit1Change) in our own internal variable, and then setting the OnChange property to nil.
We then do all the processing we want to do - if it happens to trigger an [OnChange] event - there is nothing to call as OnChange is nil, and so the Event doesn't go onto the Message queue. When we're finished however we must return the OnChange property to it original value, otherwise the reprocessing want happen next time around.
If we look at the Delphi Helpfile we see that the OnChange property is of a certain type, the same type we have to make our SaveOnChangeIn variable:
var
SaveOnChangeIn: TNotifyEvent;
~~~~~rest of code
begin
MyRe := TRichEdit(Sender);
SaveOnChangeIn := MyRe.OnChange;
MyRe.OnChange := nil;
~~~~~rest of code
MyRe.OnChange := SaveOnChangeIn;
end;
Try it out!!!
Compile and Run
Open up Unit1.pas in the "editor" we have written
Click in the RichEdit in the center of the first lines, in the middle of "unit".
Press the [space bar]
Press the [backspace key]
Arrow to the end of the line
Press [Enter]
Press [BackSpace]
[BackSpace] away the entire line
Re-type the entire line
Result: "functionally" the Control should look that same as it did before we clicked in it. The line "unit Unit1;" should highlighted properly as per your Delphi 3.0 Editor (save the background colour). However its slow and flickers a great deal. Try opening up a new line and just type a long phrase - e.g "if (RichEdit = Santa) then GetPresents('box of choclate'); " and you'll agree with me that:
GOOD - It is highlighting properly
BAD - There is flickering
BAD - The longer the line gets, the longer it takes to do the re-highlighting
BAD - you get the "someone is chasing me effect"
The flickering is due to a number of components. We'll have to deal with each seperately.
The most obvious is the "selecting" of each Token. Visually the control is just repeating what we were able to do manually - when a piece of text is selected it becomes highlighted by the black stripe. We need to stop this from happening. Back to the helpfile(s) again. Have a search around, and come back after a snack break with some ideas... I'm hungry :-)
Death to the black stripe
Marks: 5/10
Most of you would have found the HideSelection property of the RichEdit control. When it is set to TRUE and the RichEdit looses the focus (the user clicks onto another control) the selection bar (the black stripe) is hidden. In fact if you try it out by selecting some text in the RichEdit1 then clicking in the Edit1 control at the top of the "editor" you'll see the selection disappears! [Tab] back into the RichEdit control and it reappears. Lets do this programmatically:
begin
Edit1.SetFocus;
~~~~~
MyRe.SetFocus;
end;
Take my word for it, but if you look closely, the black strip is gone. Pity we got stuck with a new one in the Edit1 control :-(
If your programmed in Delphi you may know a little trick:
Delphi Rule #4: you can't SetFocus on a disabled Control.
The converse however is also true:
Delphi Rule #4b: a disabled Control is not "Focused"
So try instead we can just Disable (then Enable) the RichEdit control like this:
begin
MyRe := TRichEdit(Sender);
MyRe.Enabled := False;
~~~~~~
MyRe.Enabled := True;
end;
Oops. I should have known. After all I said it: a disabled Control is not "Focused" - barely ten lines ago! When the RichEdit is enabled again, we also have to SetFocus back to it. Shees.. :-)
begin
MyRe := TRichEdit(Sender);
MyRe.Enabled := False;
~~~~~~
MyRe.Enabled := True;
MyRe.SetFocus;
end;
Try it again. This time things are working better, and we're leaving poor old Edit1 Control alone. Thats good practice, as it may have had an [OnFocus] event that does wierder things than what we're trying to do. Maybe not now, but it could in the future!
Marks: 10/10
On the other kind, some of you may have found instead the EM_HIDESELECTION message in the Win32.HLP. If you had delved in, you would have found something very interesting. The Delphi HideSelection property only implements half the capabilities of this message. You can also, by calling it direct, tell it to Temporarily hide the black stripe even when the control has the focus. So instead you could use the following lines of code:
begin
MyRe := TRichEdit(Sender);
MyRe.Perform(EM_HIDESELECTION,1,0);
~~~~~~
MyRe.Perform(EM_HIDESELECTION,0,0);
end
Yummy. Nice clean coding:
Death to the FLICKER
The next major problem is this bloody flicker. You should pop back into Delphi for a second, and types some lines in its editor, to see if it flickers at all. It does. But only when it is changing colors when it recognizes a change has occured. Otherwise it doesn't bother. Now look at what’s happening in our "editor". Do you see?
The problem is that we are not "conserving" what we are doing. If something is still the same TokenType it doesn't need to be re-highlighted because it already correct on the screen. We need to check if the TokenType of each token has changed since last time we repainted this line, and only then repaint to.
In fact we don't need to do even that - we can just check whether the SelAttributes (which represents the current selection's attributes) is any different from what we want to change it to i.e. FParseFont[MyTokenType]. This way if even the TokenType had changed, but the new and old TokenType shared the same display attributes, we would still conserve our drawing.
Actually the problems is that the RichEdit isn't doing the conserving. In the old text based system I used to use, if you printed something to the screen, and it was the same as something already on the screen, in the same position, then the program would not rewrite it to the screen. It would "conserve" the amount of writing it did, as in the old days 1200 baud screens were SLOW, and printing the same characters was a waste of time.
Huh - and people said we have come so dar with windows. Sloppy, Sloppy, Sloppy I say! :-)
So lets replace:
MyRe.SelAttributes.Assign(PasCon.FParseFont[MyTokenState]);
with:
If MyRe.SelAttributes.Name <> PasCon.FParseFont[MyTokenState].Name then
MyRe.SelAttributes.Name := PasCon.FParseFont[MyTokenState].Name;
If MyRe.SelAttributes.Color <> PasCon.FParseFont[MyTokenState].Color then
MyRe.SelAttributes.Color := PasCon.FParseFont[MyTokenState].Color;
if MyRe.SelAttributes.Style <> PasCon.FParseFont[MyTokenState].Style then
MyRe.SelAttributes.Style := PasCon.FParseFont[MyTokenState].Style;
And off you go and try it out... (PS. Yes the last bit of code is bad programming...)
