Update syntax readme and docs

runtime/README.md

@@ -1,5 +1,5 @@
 # Runtime files for Micro
 
 This directory will be embedded in the Go binary for portability, but it may just as well be put in `~/.config/micro`. If you would like to make your own colorschemes
-and syntax files, you can put in in `~/.config/micro/colorschemes` and `~/.config/micro/syntax` respectively.
+and syntax files, you can put them in `~/.config/micro/colorschemes` and `~/.config/micro/syntax` respectively.
 

runtime/help/colors.md

@@ -117,42 +117,95 @@ Colorschemes can be placed in the `~/.config/micro/colorschemes` directory to be
 
 The syntax files specify how to highlight certain languages.
 
-The first statement in a syntax file will probably the syntax statement. This tells micro
+Syntax files are specified in the yaml format.
-what language the syntax file is for and how to detect a file in that language.
 
-Essentially, it's just
+#### Filetype defintion
+
+You must start the syntax file by declaring the filetype:
 
 ```
-syntax "Name of language" "\.extension$"
+filetype: go
 ```
 
-For the extension, micro will just compare that regex to the filename and if it matches then it
+#### Detect definition
-will use the syntax rules defined in the remainder of the file.
 
-There is also a possibility to use a header statement which is a regex that micro will compare
+Then you can provide information about how to detect the filetype:
-with the first line of the file. This is almost only used for shebangs at the top of shell scripts
-which don't have any extension (see sh.micro for an example).
-
----
-
-The rest of a syntax file is very simple and is essentially a list of regexes specifying how to highlight
-different expressions.
-
-It is recommended that when creating a syntax file you use the colorscheme groups (see above) to
-highlight different expressions. You may also hard code colors, but that may not look good depending
-on what terminal colorscheme the user has installed.
-
-Here is an example to highlight comments (expressions starting with `//`):
 
 ```
-color comment "//.*"
+detect:
+    filename: "\\.go$"
 ```
 
-This will highlight the regex `//.*` in the color that the user's colorscheme has linked to the comment
+Micro will match this regex against a given filename to detect the filetype. You may also
-group.
+provide an optional `header` regex that will check the first line of the file. For example for yaml:
-
-Note that this regex only matches the current line. Here is an example for multiline comments (`/* comment */`):
 
 ```
-color comment start="/\*" end="\*/"
+detect:
+    filename: "\\.ya?ml$"
+    header: "%YAML"
+```
+
+#### Syntax rules
+
+Next you must provide the syntax highlighting rules. There are two types of rules: patterns and regions.
+A pattern is matched on a single line and usually a single word as well. A region highlights between two
+patterns over multiple lines and may have rules of its own inside the region.
+
+Here are some example patterns in Go:
+
+```
+rules:
+    - special: "\\b(break|case|continue|default|go|goto|range|return)\\b"
+    - statement: "\\b(else|for|if|switch)\\b"
+    - preproc: "\\b(package|import|const|var|type|struct|func|go|defer|iota)\\b"
+```
+
+The order of patterns does matter as patterns lower in the file will overwrite the ones defined above them.
+
+And here are some example regions for Go:
+
+```
+- constant.string:
+    start: "\""
+    end: "(?<!\\\\)\""
+    rules:
+        - constant.specialChar: "%."
+        - constant.specialChar: "\\\\[abfnrtv'\\\"\\\\]"
+        - constant.specialChar: "\\\\([0-7]{3}|x[A-Fa-f0-9]{2}|u[A-Fa-f0-9]{4}|U[A-Fa-f0-9]{8})"
+
+- comment:
+    start: "//"
+    end: "$"
+    rules:
+        - todo: "(TODO|XXX|FIXME):?"
+
+- comment:
+    start: "/\\*"
+    end: "\\*/"
+    rules:
+        - todo: "(TODO|XXX|FIXME):?"
+```
+
+Notice how the regions may contain rules inside of them.
+
+Also the regexes for region start and end may contain more complex regexes with lookahead and lookbehind,
+but this is not supported for pattern regexes.
+
+#### Includes
+
+You may also include rules from other syntax files as embedded languages. For example, the following is possible
+for html:
+
+```
+- default:
+    start: "<script.*?>"
+    end: "</script.*?>"
+    rules:
+        - include: "javascript"
+
+- default:
+    start: "<style.*?>"
+    end: "</style.*?>"
+    rules:
+        - include: "css"
 ```

runtime/syntax/README.md

@@ -1,5 +1,27 @@
 # Syntax Files
 
-Here are highlight's syntax files. If you would like to make a new syntax file, you should first check it
+Here are micro's syntax files.
-with the `syntax_checker.go` program. Just place it in this directory and run the program (`go run syntax_checker.go`)
+
-and it will let you know if there are issues with any of the files in the directory.
+Each yaml file specifies how to detect the filetype based on file extension or headers (first line of the file).
+Then there are patterns and regions linked to highlight groups which tell micro how to highlight that filetype.
+
+Making your own syntax files is very simple. I recommend you check the file after you are finished with the
+[`syntax_checker.go`](./syntax_checker.go) program (located in this directory). Just place your yaml syntax
+file in the current directory and run `go run syntax_checker.go` and it will check every file. If there are no
+errors it will print `No issues!`.
+
+You can read more about how to write syntax files (and colorschemes) in the [colors](../help/colors.md) documentation.
+
+# Legacy '.micro' filetype
+
+Micro used to use the `.micro` filetype for syntax files which is no longer supported. If you have `.micro`
+syntax files that you would like to convert to the new filetype, you can use the [`syntax_converter.go`](./syntax_converter.go) program (also located in this directory):
+
+```
+$ go run syntax_converter.go c.micro > c.yaml
+```
+
+Most the the syntax files here have been converted using that tool.
+
+Note that the tool isn't perfect and though it is unlikely, you may run into some small issues that you will have to fix manually
+(about 4 files from this directory had issues after being converted).