Todo:

• Add more advanced highlighter stuff (explain the logic of the generated highlighter source).

Introduction

This document tries to give a explanation, on how to create your own custom highlighter for the TSynEdit component. This document starts with a step-by-step explanation on how to create a simple highlighter, but this will be expanded to cover advanced highlighting techniques.

Preparation

Creation of a new highlighter is best started with the creation of a grammar file. This grammar file (.msg file), is then used by the program SynGen to generate a basic skeleton of the highlighter source. For most highlighters this generated source is nothing more then a point to begin with, but for simple keyword highlighters, the result is directly usable.

The layout of the grammar file, is rather straight forward (you can insert empty lines if you like. Text between { and } is considered as comment, and thus is ignored):

• The first line contains the name of the highlighter class to be created (e.g. TSynSampleSyn).

• The second line contains the prefix of the enumeration type to be created for the various types of tokens. The current series of SynEdit highlighters use the prefix ‘tk’ for this.

• The third line contains either the keyword ‘Sensitive’ to indicate that the keywords of the highlighter are to be used case sensitive, or it will contain the keyword ‘IdentStart’, followed by a list of characters that can be used to make up valid identifiers.

• After this the keyword ‘KEYS’ is in the file, after which all the keywords to be highlighted are mentioned. Each keyword starts on a new line. This section is ended by the terminator string ‘|<>|’.

• After this the file contains the list of token types the highlighter will recognize. By default the token types Identifier and Key should always be there. Each token type starts on a new line, and the section is again terminated by a line containing ‘|<>|’.

• After this the keyword ‘CHARS’ is in the file, after which all the special character handling is coded in Pascal. This is just a kind of a case statement, which will be clarified in the examples listed later in this document. This section is also terminated by a line containing ‘|<>|’.

Based on this .msg file, we can run the SynGen program, which will generate the basic highlighter source for us. After that we can take the generated source file, and fine tune it for our special needs.

Example: creating a basic highlighter

In our first example we will attempt to create a new highlighter, which will highlight just one simple keyword ‘Hello’. This highlighter is not case sensitive, and will be based on a SynGen grammar file. This grammar file should look like this:

Once we have created this file (as sample.msg), we can startup SynGen. On startup, SynGen will prompt us to select the grammar file we have just created. After that, we are presented with a four-page window, which allows us to do some customizations:

• On the first page (Highlighter) we will fill in some general information about the highlighter. This contains the name of the author, a short description, and the version of the highlighter. This page also contains two checkboxes which give the user the option to have the SynEdit standard comment header (with the GPL/MPL information), and the option to have a GetKeywords function which returns all keywords being highlighted.

• On the second page (Language) we can fill in the default filter (for this example we select ‘All files’), which can be used for the FileOpen dialogbox in Delphi. We can also fill in the name of the language being highlighted (for this example we call it ‘Sample’).

• On the third page (Attributes), we can assign different constants to the Identifier and Reserved word tokens. We can keep the default values now. On this page we can also set which type of token should be used for non-keyword identifiers. In this example we will leave it to the default value (Identifier).

• On the fourth page (Private fields), we can add our own private field declarations to the highlighter class. We don’t need this now, so we leave this page empty.

After having filled in all fields in SynGen, we can press the ‘Start’ button. After that SynGen will generate a Delphi unit, called sample.pas, which contains the implementation of the highlighter. You can use this highlighter in SynEdit, and it will only highlight the keyword ‘Hello’ in bold.

If we now want to add more keywords to this highlighter, we simply add them in the grammar file, as new lines in the KEYS section. After regenerating the Pascal unit using SynGen, we will have the additional keywords being highlighted in SynEdit.

Adding support for comments and strings

After creating a simple highlighter like this, we want to add support for comments to the highlighter. In our sample language, comments are started with a ‘{‘ (brace open) character, and closed with a ‘}’ character (Pascal style comments).

To achieve this we must make two modifications to our grammar file:

• Add a new token kind ‘Comment’

• Add a new section “ENCLOSEDBY” which specifies the delimiters of the comment section

Based on this modification, our grammar file will now look like (with two keywords: ‘Hello’ and ‘World’):

The ENCLOSEDBY section, can contain 1 or multiple lines to specifies token types which are recognized by a starting and ending sequence of characters. The syntax of each line is:

<Token name> , <Procedure name> , <starting sequence> , <ending sequence> [ , MultiLine]

The “Token name” should be an already defined token name in the higher part of the grammar file.

The “Procesdure name” should be unique for each line. This name is used to generate procedure names for each of the lines in the source code of the highlighter.

• The “Starting sequence” specifies the string that denotes the start of the token kind.

• The “Ending sequence” specifies the string that denotes the end of the token kind.

• The last identifier “MultiLine”, is optional. If it is specified then the token kind can continue on the next line, if no ending sequence was found. If it is not specified, the token kind will end on the end of the line.

After you have generated the source code via the SynGen utility, you will see that you have a highlighter supporting two keywords (hello and world), and support for comments starting with { and ending with }.

Based on this logic we can now simply add support for /* .. */ C-style comments, by adding this line to the ENCLOSEDBY section:

We can also add support for strings delimited by “ and “. For this we add a new token kind “String”, and then this line to the ENCLOSEDBY section (note that strings are not allowed to cross multiple lines):

So now our grammar file looks like this:

Once you have generated the source code for this highlighter, and use it in your application, you can test it to see if it suits your needs. Once you have the generated source code from SynGen, you can further enhance it by modifying this code in Delphi.

Highlighting different kind of keywords

In our example till now, the highlighter will always recognize all keywords in the same way. All keywords are considered to be equal, and this will be highlighted with the same highlighter attribute. In practice you will often want to create a highlighter that will highlight e.g. data types in a different color then the other keywords. So now we will expand our grammar file, to add a new keyword ‘SynEdit’, which will be highlighted as a new token type ‘Test’. Our grammar file will become to look like this:

As you’ll notice, that there is an extension to the KEYS section, which allows you to specify the token type to use for the keywords in that section. If you don’t specify a token type, it will consider the token type to be ‘Keys’. You can create as many KEYS section as you, which in the grammar file, and even multiple sections with the same (or no) token type are allowed. The SynGen program will merge them automatically.

After you have run this grammar file through the SynGen utility you can test your highlighter, and you’ll notice that the SynEdit keyword is highlighted using a different attribute than the Hello and World keywords.

Setting the default highlighter attribute settings

One annoying thing so far about the generated highlighter is, that we should always modify the source code of the highlighter, if we want to provide default attribute styles for the different token types. Suppose we want to provide the user with the default settings of red strings, Italic and blue comments, and bold keywords, we now must modify our highlighter source code. This is not very nice, since this means that we might loose changes, once we want to regenerate the source code from the grammar file (e.g. because we added new keywords).

So what we will do now, is that we’ll modify our TOKENTYPES section, so that it includes the default styles for the different attributes. The section will then look like this:

As you’ll notice, each token type can have 0 to 3 additional parameters, which specify the foreground, background and font style of the highlighter attribute. The different parameters are divided by a pipeline character (‘|’), and the order in which the parameters are mentioned is irrelevant. You are also free to leave out any of the parameters, and then the highlighter attribute will keep it’s default value for that parameter.

The value at the right sign of the equal operator, is considered to be valid Delphi code. The SynGen utility will assign this value directly to the associated attribute, and thus will not take any attempts to interpret it. This means that any of the following assignments are valid:

Adding sample source to the highlighter

Many highlighters in the SynEdit package, have implemented the GetSampleSource function. This function should return a code snippet, which demonstrates all the features of the highlighter. You can add sample source to your highlighter, by adding a SAMPLESOURCE section to your grammar file. The source you will enclose between the keyword ‘SAMPLESOURCE’ and the terminator |><|, will be taken literally by your new highlighter as the result of the GetSampleSource function. For our sample, our grammar file will be extended with this section (at the end of the file):

Contributing your highlighter to the SynEdit package

Once you have finalized your highlighter, and want to contribute it to the SynEdit package, it should meet the following requirements:

• The name of the highlighter class should be formatted as ‘TSyn’ + LanguageName + ‘Syn’. So in our example it becomes ‘TSynSampleSyn’.

• The name of the highlighter unit should be formatted as ‘SynHighlighter’ + LanguageName. So in our example it becomes ‘SynHighlighterSample’.

• If the highlighter contains definitions of resourcestrings or consts starting in the implementation section (SynGen will generate here ‘SYNS_Lang*’, ‘SYNS_Filter*’ and optionally ‘SYNS_ATTR*’), these should be moved to the SynEditStr.pas unit.

• An icon should be added for the highlighter in the SynEditReg.dcr file.

• The highlighter should be registered on the “SynEdit highlighters” component page in the SynEditReg.pas unit.

Sample sources

The sample sources from this tutorial are available in the SynEdit demos folder, as ‘HighlighterDemo’. This demo contains the grammar (.msg) file + a demo program showing the generated highlighter in use.