diff --git a/README.md b/README.md index a5359c1..843db46 100644 --- a/README.md +++ b/README.md @@ -13,6 +13,7 @@ We have some guides and general documentation in the [docs](docs/) directory. Am - [CLI usage](docs/running.md) - [Tests](docs/tests.md) - [HTTP Router](docs/router.md) +- [Template engine](docs/template.md) - [List of security fixes](docs/sec-fixes/) ## Dependencies diff --git a/docs/template-examples.md b/docs/template-examples.md new file mode 100644 index 0000000..4b2e3d7 --- /dev/null +++ b/docs/template-examples.md @@ -0,0 +1,193 @@ +# HTTP.sh: template usage examples + +## Basic example + +Create a new .shs file with the following contents: + +``` +#!/usr/bin/env bash +declare -A str +str[title]="Hello, world!" +str[test]="meow" + +render str "${cfg[namespace]}/templates/main.htm" +``` + +`render` is the core of the templating engine; it takes an assoc array, iterates over it, applies +additional magic and outputs the response directly to stdout. It is likely the final thing you want +to run in your script. + +The script above has referenced an HTML file; For this example, we put it under +`app/templates/main.htm`, but you're free to use any directory structure for this. + +``` + + + + + {{.title}} + + + {{.test}} + + +``` + +![netscape 3.06 gold screenshot, showing our awesome page](https://f.sakamoto.pl/IwILCemig.png) + +## Boolean if statements + +Following is an example script which simulates a coin toss: + +``` +#!/usr/bin/env bash +declare -A str +str[title]="Coin flip!" + +if (( RANDOM%2 == 0 )); then + str[?random]=_ +fi + +render str "${cfg[namespace]}/templates/main.htm" +``` + +And the corresponding template: + +``` + + + + + {{.title}} + + + {{start ?random}} + It's heads! + {{else ?random}} + It's tails! + {{end ?random}} + + +``` + +![another netscape screenshot. the page is titled Coin flip! and it shows that it rolled heads](https://f.sakamoto.pl/IwIQT0d6w.png) + +50% of the time the variable will be set, 50% it won't. Hence, it will display either heads or tails :) + +Of note: if you hate repeating yourself, this template can be done inline: + +``` +It's {{start ?random}}heads{{else ?random}}tails{{end ?random}}! +``` + +The effect is exactly the same. This is quite useful for adding CSS classes. + +## Loop example + +This API is pending a rewrite due to how convoluted it is. + +``` +#!/usr/bin/env bash +declare -A str +str[title]="foreach example" + +nested_declare list # "array of arrays" +declare -A elem # temporary element +for i in {1..32}; do + elem[item]="$i" # assign $i to the temporary element + nested_add list elem # add elem to list; this creates a copy you can't modify +done +# once we have a full list of elements, assign it to the array passed to render +str[_list]=list + +render str "${cfg[namespace]}/templates/main.htm" +``` + +And the template... + +``` + + + + + {{.title}} + + + {{start _list}} + {{.item}}
+ {{end _list}} + + +``` + +The result repeats the whole "subtemplate" between list start and end: + +![list so long that the numbers go off-screen!](https://f.sakamoto.pl/IwI0slukA.png) + +This is very useful for rendering data in tables: + +``` + + + + + {{start _list}} + + + + + {{end _list}} +
number
{{.item}}whatever...
+``` + +![our example, now rendered as a table](https://f.sakamoto.pl/IwIf39cYw.png) + +### integration with notORM + +notORM's `data_iter` function works great with nested_add; Body of a callback function can be +treated as equal to a for loop: + +``` +declare -A elem +nested_declare list +x() { + elem[ns]="${data[2]}" + elem[domain]="${data[1]}" + nested_add list elem +} +data_iter storage/zones.dat "$username" x + +str[title]="SERVFAIL :: zone list" +str[_list]=list +``` + +## date pretty-printing + +``` + + + + + {{.title}} + + + Current time is {{+time}} + + +``` + +``` +#!/usr/bin/env bash +declare -A str +str[title]="time pretty-print" +str[+time]="$EPOCHSECONDS" + +render str "${cfg[namespace]}/templates/main.htm" +``` + +![netscape displays the current date and time](https://f.sakamoto.pl/IwIvf3Axw.png) + +If you get quirky with the ``, you can even make it +auto update! (don't) + +![oh gosh it refreshes now. whyyyyyyyyyyyyyyyy](https://f.sakamoto.pl/simplescreenrecorder-2025-04-05_22.57.36.png) diff --git a/docs/template.md b/docs/template.md new file mode 100644 index 0000000..19c1a19 --- /dev/null +++ b/docs/template.md @@ -0,0 +1,157 @@ +# HTTP.sh: template engine + +We have a basic template engine! It's somewhat limited in capabilities compared to engines you might +have previously used, but we're working on making it better :3 + +Note: the `templates` subdirectory in the HTTPsh repo is entirely unrelated to the template engine, +and it will be removed in a future release. Please ignore it. + +For practical examples, see the [template examples](template-examples.md) page. + +## Tag schema + +- Tags always start with `{{` and end with `}}`. +- Tags can't include whitespace, outside of special iter/boolean tags defined below +- Tag identifiers can contain letters, numbers, dashes and underscores (`[a-zA-Z0-9_-]`). + Other characters may work but are NOT RECOMMENDED. +- Tag identifiers are always prefixed by the tag type. This is also reflected in the code, outside + of simple replaces which MUST skip the dot in the array assignment. +- Identifiers are represented by `` later in this document. + +## API + +`render [recurse]` + +The first param points to an associative array containing the replacement data. Second one points +to a file containing the template itself. Third is optional, and controls whether `render` will +recurse or not. This is mostly used internally, you likely won't ever need to set it. + +## Simple replace + +| | | +| --- | --- | +| In the template | `{{.}}` | +| In the code | `array[]=""` | +| Notes | For your convenience, code representation skips the dot. | + +**Important**: to simplify your life (and protect your application), simple replaces ALWAYS use +html_encode behind the scenes. This means that you're safe to assign any value to them without +prior sanitization. + +## Raw replace + +| | | +| --- | --- | +| In the template | `{{@}}` | +| In the code | `array[@]=""` | + +Same as a simple replace, but doesn't do html_encode. Useful if you want to guarantee unmangled +output (for filling out hidden form values, etc.) + +## Template includes + +| | | +| --- | --- | +| In the template | `{{#}}` | +| In the code | n/a | + +Template includes are special, in that you don't have to define them in the array. +They get processed first to "glue together" one singular template. + +Currently, the path starts at the root of HTTPsh's directory. We don't support expanding variables +inside the include tag, so for now you'll need to hardcode `{{#app/templates/...}}`. This will +likely get changed in a future release, starting the path in your namespace. + +**Warning**: No recursion is supported within included templates; This means that you can't have +an "include chain". Furthermore, some interactions between included templates and loops/ifs are +a bit wonky; This will get ironed out at some point (sorry!) + +## Boolean if statements + +| | | +| --- | --- | +| In the template | `{{start ?}} ... {{end ?}}` | +| In the template (alt.) | `{{start ?}} ... {{else ?}} ... {{end ?}}` | +| In the code | `array[?]=_` | +| Notes | Can be used both inline and not. See [examples page](template-examples.md) for more details. | + +**Important**: Currently, you can't have two checks for the same variable. If needed, set a second +variable in the code and check for that. Fix TBD. + +This is a *boolean operator*. The only supported mode of operation is checking whether +a variable is set or not. + +## Loops + +| | | +| --- | --- | +| In the template | `{{start _}} ... {{end _}}` | +| In the code | `array[_name]=""` | + +Each loop extracts the area between start/end markers, and executes another `render` internally. +You have to provide it with an "array of arrays", essentially an intermediate holding references. +This is usually done through `nested_declare ` and `nested_add `. + +Essentially, this boils down to: + +``` +nested_declare list # "array of arrays" +declare -A elem # temporary element +for i in {1..32}; do + elem[item]="$i" # assign $i to the temporary element + nested_add list elem # add elem to list; this creates a copy you can't modify +done +# once we have a full list of elements, assign it to the array passed to render +str[_list]=list +``` + +A more detailed usage description is available on the [template examples](template-examples.md) page. + +### Leaky temporary array + +You should excercise caution when handling the temporary arrays; Calling `unset elem` on the end +of each loop may be a good idea if you can't guarantee that all of your elements will always have +values. Otherwise, values from previous iterations may leak to the current one, potentially causing confusion. + +## Loop indexes + +| | | +| --- | --- | +| In the template | `{{-index}}` | +| In the code | n/a | +| Notes | Doesn't resolve at all outside loops. Counter starts at 0 and gets incremented with every element. | + +## Date pretty-printing + +| | | +| --- | --- | +| In the template | `{{+}}` | +| In the code | `array[+]=""` | + +This saves you from a few messy calls to `date`. Input is a UNIX timestamp. + +The date format can be overriden by changing a config variable. Default is +`cfg[template_date_format]='%Y-%m-%d %H:%M:%S'`. + +## URI slices + +| | | +| --- | --- | +| In the template | `{{-uri-}}` | +| In the code | n/a | +| Notes | Level must be a number. URI is always terminated with a slash, even if your last object is a file. | + +Takes the current URI path, and slices it using `/` as a delimeter, going from the left. + +Given an URL `http://localhost:1337/hello/world/asdf`... + +- `{{-uri-0}}` -> `/` +- `{{-uri-1}}` -> `/hello/` +- `{{-uri-2}}` -> `/hello/world/` +- `{{-uri-3}}` -> `/hello/world/asdf/` +- `{{-uri-4}}` -> none (higher values are always empty) + +This is very useful when creating menus; Instead of relying on hardcoded values, if the page is always +on *the same URI level*, one can create links such as `(...)`, which will always +resolve to the same file; This eliminates a whole class of bugs where trailing slashes would break some +poorly-written relative URLs. diff --git a/src/server.sh b/src/server.sh index 756a612..6be454e 100755 --- a/src/server.sh +++ b/src/server.sh @@ -252,11 +252,13 @@ if [[ "${r[post]}" == true ]] && [[ "${r[status]}" == 200 || "${r[status]}" == if [[ "${r[content_length]}" ]]; then read -r -N "${r[content_length]}" data else - data= - while read -r line; do - data+="$line" - done - unset line + if read -t0; then + data= + while read -r line; do + data+="$line" + done + unset line + fi fi if [[ "${r[payload_type]}" == "urlencoded" ]]; then diff --git a/src/template.sh b/src/template.sh index 89acac1..42270b2 100755 --- a/src/template.sh +++ b/src/template.sh @@ -13,13 +13,43 @@ function render() { local template="$(tr -d "${_tpl_ctrl}" < "$2" | sed -E 's/\\/\\\\/g')" fi local buf= + local garbage= local -n ref=$1 + # process file includes; + # recursion is currently unsupported here, i feel like it may break things? + if [[ "$template" == *'{{#'* && "$3" != true ]]; then + local subtemplate= + while read key; do + # below check prevents the loop loading itself as a template. + # this is possibly not enough to prevent all recursions, but + # i see it as a last-ditch measure. so it'll do here. + if [[ "$file" == "$2" ]]; then + subtemplate+="s${_tpl_ctrl}\{\{\#$key\}\}${_tpl_ctrl}I cowardly refuse to endlessly recurse\!${_tpl_ctrl}g;" + elif [[ -f "$key" ]]; then + local input="$(tr -d "${_tpl_ctrl}${_tpl_newline}" < "$key" | sed 's/\&/�UwU�/g')" + garbage+="$input"$'\n' + input="$(tr $'\n' "${_tpl_newline}" <<< "$input")" # for another hack + subtemplate+="s${_tpl_ctrl}\{\{\#$key\}\}${_tpl_ctrl}${input}${_tpl_ctrl};" + _template_find_special_uri "$(cat "$key")" + fi + done <<< "$(grep -Poh '{{#\K(.*?)(?=}})' <<< "$template")" + + buf+="${subtemplate}" + fi + local key IFS=$'\n' for key in ${!ref[@]}; do if [[ "$key" == "_"* ]]; then # iter mode - local subtemplate="$(grep "{{start $key}}" -A99999 <<< "$template" | grep "{{end $key}}" -B99999 | tr '\n' "${_tpl_newline}")" + # THE MOST EVIL OF ALL HACKS: + # we're scraping a subtemplate from our main template. + # HOWEVER: this fails on included templates, because they're not real. + # this means that iterators can't work on included templates + # + # workaround? collect all includes, concatenate them all together and just. + # use that pile of garbage here along with the real template. it works! + local subtemplate="$(grep "{{start $key}}" -A99999 <<< "$template"$'\n'"$garbage" | grep "{{end $key}}" -B99999 | tr '\n' "${_tpl_newline}")" local -n asdf=${ref["$key"]} local j local value='' @@ -31,7 +61,7 @@ function render() { (( _index++ )) done - buf+="s${_tpl_ctrl}\{\{start $key\}\}.*\{\{end $key\}\}${_tpl_ctrl}\{\{$key\}\}${_tpl_ctrl};s${_tpl_ctrl}\{\{$key\}\}${_tpl_ctrl}$(tr -d "${_tpl_ctrl}" <<< "$value" | sed -E "s${_tpl_ctrl}"'\{\{start '"$key"'\}\}'"${_tpl_ctrl}${_tpl_ctrl};s${_tpl_ctrl}"'\{\{end '"$key"'\}\}'"${_tpl_ctrl}${_tpl_ctrl}")${_tpl_ctrl};" + buf+="s${_tpl_ctrl}\{\{start $key\}\}.*\{\{end $key\}\}${_tpl_ctrl}\{\{$key\}\}${_tpl_ctrl};s${_tpl_ctrl}\{\{$key\}\}${_tpl_ctrl}$(tr -d "${_tpl_ctrl}" <<< "$value" | sed "s${_tpl_ctrl}{{start $key}}${_tpl_ctrl}${_tpl_ctrl};s${_tpl_ctrl}{{end $key}}${_tpl_ctrl}${_tpl_ctrl}")${_tpl_ctrl};" unset "$subtemplate" elif [[ "$key" == "@"* && "${ref["$key"]}" != '' ]]; then local value="$(tr -d "${_tpl_ctrl}${_tpl_newline}" <<< "${ref["$key"]}" | sed -E 's/\&/�UwU�/g')" @@ -51,12 +81,7 @@ function render() { elif [[ "$key" == '?'* ]]; then local _key="\\?${key/?/}" - # TODO: check if this is needed? - # the code below makes sure to resolve the conditional blocks - # *before* anything else. I can't think of *why* this is needed - # right now, but I definitely had a reason in this. Question is, what reason. - - buf+="s${_tpl_ctrl}"'\{\{start '"$_key"'\}\}((.*)\{\{else '"$_key"'\}\}.*\{\{end '"$_key"'\}\}|(.*)\{\{end '"$_key"'\}\})'"${_tpl_ctrl}"'\2\3'"${_tpl_ctrl};${buf}" # MAYBE_SLOW + buf+="s${_tpl_ctrl}"'\{\{start '"$_key"'\}\}((.*)\{\{else '"$_key"'\}\}.*\{\{end '"$_key"'\}\}|(.*)\{\{end '"$_key"'\}\})'"${_tpl_ctrl}"'\2\3'"${_tpl_ctrl};" elif [[ "${ref["$key"]}" != "" ]]; then if [[ "$3" != true ]]; then @@ -71,32 +96,12 @@ function render() { done unset IFS - # process file includes; - # achtung: even though this is *after* the main loop, it actually executes sed reaplces *before* it; - # recursion is currently unsupported here, i feel like it may break things? - if [[ "$template" == *'{{#'* && "$3" != true ]]; then - local subtemplate= - while read key; do - # below check prevents the loop loading itself as a template. - # this is possibly not enough to prevent all recursions, but - # i see it as a last-ditch measure. so it'll do here. - if [[ "$file" == "$2" ]]; then - subtemplate+="s${_tpl_ctrl}\{\{\#$key\}\}${_tpl_ctrl}I cowardly refuse to endlessly recurse\!${_tpl_ctrl}g;" - elif [[ -f "$key" ]]; then - subtemplate+="s${_tpl_ctrl}\{\{\#$key\}\}${_tpl_ctrl}$(tr -d "${_tpl_ctrl}${_tpl_newline}" < "$key" | tr $'\n' "${_tpl_newline}" | sed 's/\&/�UwU�/g')${_tpl_ctrl};" - _template_find_special_uri "$(cat "$key")" - fi - done <<< "$(grep -Poh '{{#.*?}}' <<< "$template" | sed 's/{{#//;s/}}$//')" - - buf="${subtemplate}$buf" - fi - _template_find_special_uri "$template" buf+="$(_template_gen_special_uri)" if [[ "$3" != true ]]; then # are we recursing? - tr '\n' ${_tpl_newline} <<< "$template" | sed -E -f <( - tr '\n' "${_tpl_newline}" <<< "$buf" | sed -E $'s/\02;\01/\02;/g;s/\02g;\01/\02g;/g' # i'm sorry what is this sed replace?? + tr '\n' "${_tpl_newline}" <<< "$template" | sed -E -f <( + tr '\n' "${_tpl_newline}" <<< "$buf" | sed $'s/\02;\01/\02;/g;s/\02g;\01/\02g;/g' # i'm sorry what is this sed replace?? echo -n 's/\{\{start \?([a-zA-Z0-9_-]*[^}])\}\}(.*\{\{else \?\1\}\}(.*)\{\{end \?\1\}\}|.*\{\{end \?\1\}\})/\3/g' ) | tr "${_tpl_newline}" '\n' | sed -E 's/�UwU�/\&/g' else diff --git a/tests/01-http-basic.sh b/tests/01-http-basic.sh index e769f43..e89b6c5 100644 --- a/tests/01-http-basic.sh +++ b/tests/01-http-basic.sh @@ -158,7 +158,7 @@ server_req_header_dup() { server_req_header_invalid() { tst() { # we have to trick curl into sending an invalid header for us - curl -s "localhost:1337/meow.shs" -H $'a:\nasdf asdf asdf asdf' -H "meow: asdf" + curl -s "localhost:1337/meow.shs" -H $'meow:\nasdf asdf asdf asdf' -H "a: aaaa" } match_not="asdf"