HSC and scripting

If you thought this page was going to break with HSC's tradition and start the "Wizardry" series with ScrollerScript¹ stuff, you are mistaken. This article is about integration of scripting languages like Perl or Python with HSC.

Introduction

Looking at some of HSC's "competitors", I found a particulary nice specimen called WML for "Website META Language". No, I don't want to convince you to use this instead of HSC ;-) It just has one feature that I found quite useful (among loads of others that I don't need, which is why I'll stick to HSC): a built-in Perl interpreter.

Now why would you want to have Perl support in an HTML preprocessor?
First of all, because Perl is cool². Second, because you can actually do useful things to your HTML code with it!

Perl in WML

For instance, let's try the script WML's author uses to demonstrate this feature:

40|  <table border=2> <:
41|  $end = 9;
42|  print "<tr><th></th>";
43|  for ($i=1; $i <= $end; $i++) {
44|      print "<"."th bgcolor=\"#9090f0\">$i</th>";
45|  }
46|  print "</tr>";
47|  print"</tr>\n";
48|  for ($i=1; $i <= $end; $i++) {
49|      print "<tr><"."th bgcolor=\"#9090f0\">$i</th>";
50|      for ($j=1; $j<= $end; $j++) {
51|          print "<td>", $i*$j, "</td>";
52|      }
53|      print"</tr>\n";
54|  }
55|  :> </table>

(taken verbatim from the WML support page, apart from some surrounding lines that demonstrate WML's i18n gimmicks)
Apart from creating slightly wrong HTML code that WML doesn't seem to catch, the above code writes a nice HTML multiplication table. As you can see³, the <TABLE> tags are included in the source verbatim, while the entire rest of the table is generated by the script, which is enclosed in a pair of special tags: <: ... :>. First, the colored header cells are printed in a loop, then the first row is closed by a </TR> tag (twice, which is the mistake here). The rest of the script consists of two nested loops that calculate the table's contents. They simply generate HTML code by printing it.

This is arguably not a very useful example, but at least it demonstrates what can be done. A few lines of code have made extending the table to 20x20 (or 200x200, if your browser can digest that) cells a matter of changing a single number instead of doing loads of calculations and typing.⁴

Getting it into HSC

As you will have noticed from HSC's documentation, an encapsulation of code like the above doesn't fit with HSC's syntax, at least it cannot be implemented as a macro (it does have <| ... |> and similar constructs built in). The closest we get would be a container macro—which fits better into familiar HTML paradigms anyway. Something like this:

<$macro PERL /CLOSE>
...
</$macro>

Inside such a "container macro" we have two ways of accessing the macro's contents, i.e. the stuff between <PERL> and </PERL>: the special tag <$content>, which inserts the contents at its position (possibly expanding any other macros that might be contained therein), and the special attribute HSC.Content which can be passed to other macros as a parameter and used in expressions.

What we need to do is pass the entire contents of the macro to the perl interpreter and insert in their place the output of the script. Fortunately, HSC has two special tags to help us do exactly this: <$export> and <$exec>. The former used to be undocumented in versions <=0.917, but from my experience it works exactly as it's supposed to, so we'll use it here.

<$export> takes at least a filename and a string to be written to this file, so we can write out a macro's contents using:

<$export FILE="perlscript" DATA=(HSC.Content)>⁵.

After writing out the script, all we have to do is run it at read the output back, which is what exec does if we call it like this:

<$exec COMMAND="perlscript" INCLUDE>

Et voilá—that's it for HSC's Perl integration! The final macro looks like this:

<$macro PERL /CLOSE>
  <$export FILE="perlscript" DATA=(HSC.Content)>
  <$exec COMMAND="perl perlscript" INCLUDE>
</$macro>

This version has the disadvantage that it always leaves a file called "perlscript" around after compilation. Using <$exec> you can try to remedy that—unless you want to spoil the fun and look at the macro file on the download page!

A more general scripting macro

Don't like Perl? Well, you're out of luck so far. Maybe you have figured that you could substitute your favorite interpreter for "perl" in the last version, and yes, you could do that, and it would work. But what about extending the previous macro to accept the interpreter as a parameter, so we could just call any script? Other macros like <PERL> could be derived from it as simple shortcuts. Maybe we could even add an extra parameter for additional arguments that makes deriving other macros easier? Then the general macro, let's call it <INTERPRET-SCRIPT>, could also do the cleaning up, i.e. delete the script after use. What about this:

<$macro INTERPRET-SCRIPT /CLOSE INTERPRETER:string/R PARAMS:string>
  <$define cmd:string=(INTERPRETER + ' ')>
  <$define tempfile:string/CONST="script">
  <$if COND=(set PARAMS)>
    <$let cmd=(cmd + PARAMS + ' ')>
  </$if>
  <$export FILE=(tempfile) DATA=(HSC.Content)>
  <$exec COMMAND=(cmd + tempfile) INCLUDE>
</$macro>

Getting more complex, eh?
This macro uses expressions, local variables and conditionals already. To be exact, it's one variable to hold the command, and one constant for the script file. It makes sense to define a constant for clarity, not because it's faster. Both the variable and the constant get a default value as they are defined: the command is the content of the INTERPRETER parameter with a blank appended⁶, and the tempfile is just a filename—if you want to change it later (e.g. to be the current source name with a suffix, or to put it in T: or /tmp), you only have to change it in one place. As PARAMS may or may not be set (note it's not a /REQUIRED parameter!), we have to check whether it is, and if it is, append its contents plus a blank to the previous cmd. The rest of the macro is basically the same as the last version, it just uses some expressions where <PERL> had literals.

There is one important difference though: <INTERPRET-SCRIPT> is no longer a container macro! Instead, it gets the script as a string parameter. Couldn't it just be a container macro as well, so you could use it easily without wrapping it into one? No, it can't! That's because, as HSC's documentation says:

You should be aware of the fact that hsc, when scanning for the end-tag for the macro, does not process other macros or <$include>-tags, but only looks at the text. Therefore, the end-tag has to show up within the same input file as the start-tag.

The underlying principle is that every construction is expanded at the lowest possible level, i.e. if you had two nested containers like:

<$macro PERL /CLOSE>
  <INTERPRET-SCRIPT INTERPRETER="perl">
    <$content>
  </INTERPRET-SCRIPT>
</$macro>

<$macro INTERPRET-SCRIPT /CLOSE INTERPRETER:string/R>
  ...
  <$export FILE="foo" DATA=(HSC.Content)>
  ...
</$macro>

The "inner" macro, INTERPRET-SCRIPT, would be passed the string "<$content>" instead of the actual contents of <PERL>. It could then expand the contents using <$content>, but this expansion is not done as long as the parameter is only passed on to other tags like <$export>.

Anyway, writing macros for specific languages around our generalized scripting macro is trivial now:

<$macro PERL /CLOSE>
  <INTERPRET-SCRIPT INTERPRETER="perl" SCRIPT=(HSC.Content)>
</$macro>

<$macro PYTHON /CLOSE>
  <INTERPRET-SCRIPT INTERPRETER="python" PARAMS="-O" SCRIPT=(HSC.Content)>
</$macro>

Now we can try an equally trivial example with both of them—this time, it will only be a single-row table with multiples of PI, so the code doesn't get too long :-) Below are both scripts in source form on top, and the resulting HTML output below it⁷:

<PERL>
print "<TABLE BORDER=\"1\"><TR>\n";
$pi = 3.1415;
print "<TD STYLE=\"background-color:#ffffff\">".$_*$pi."</TD>"
  foreach(1..4);
</PERL>
</TR></TABLE>

<PYTHON>
print "<TABLE BORDER=\"1\"><TR>";
pi = 3.1415
for x in range(1,5): 
  print "<TD STYLE=\"background-color:#ffffff\">"
  print x * pi, "</TD>"
</PYTHON>
</TR></TABLE>

The seemingly strange creation of the <TABLE> tags—one printed by the program, the other included in the HTML code outside the scripting part is intentional. It is by no means necessary to produce them like that, it's just to demonstrate that it doesn't matter the least where they come from. Stuff printed out by the script is inserted as the content of the <whateverlanguage> tag, and that's it.

Another very nice aspect of this concept is that you don't have to restrict yourself to pure HTML output in your script. You can just as well print HSC code! If you make sure that the macros used are available when the scripting macro is called, you can write simple scripts that do powerful things. You don't have to waste your time printing loads of tags just because you are generating HTML algorithmically, if a handful of HSC macros can do the same.
Hint: to ensure all necessary macros are available, you can easily print something like "<$include FILE="mymacros.hsc">"!

Dynamic scripts

The scripts developed so far are all fine and dandy if you have some external data source to integrate in your HTML documents, but they fail to do one important thing: take data from HSC and work with it. Say you wanted our multiplication table as a macro that can vary the table's size according to a numeric parameter. For the sake of "information hiding" it would be nice to have this, instead of letting potential users of this macro fiddle with the source or include a completely new script for every differently-sized table they need. For the same reasons as for the nested container macros, the following doesn't work:

<$macro MULTTALBE SIZE:num=9>
  <TABLE BORDER=2>
  <PERL>
    $end = <(SIZE)>;
    ...
  </PERL>
</$macro>

That is, Perl will happily run it, put "(SIZE)" in $end—and then fail trying to do maths with that string value. But of course there's a solution: dynamic scripts. If you take a look at the <INTERPRET-SCRIPT> macro again, you'll see that it gets the script to run as a string parameter, and strings can be used in expressions! Thus we could directly use <INTERPRET-SCRIPT> and build the required script dynamically as a string, instead of using HSC.Content. Like this:

<$macro MULTTALBE SIZE:num=9>
  <TABLE BORDER=2>
  <INTERPRET-SCRIPT INTERPRETER="perl" SCRIPT=(
     '$end = ' + SIZE + ';' +
     '
  print "<tr><th></th>";
  ...
  ')>
</$macro>

Admittedly, this is not as nice as the previous version, but if you initialize a few variables from the HSC macro parameters at the start of the script and write the rest like above, you can still keep it quite readable. To avoid lots of warnings when compiling code like this you will want to switch off message 33: "linefeed found inside string". It's rarely helpful anyway, and ignoring it keeps you from having to quote and concatenate each line of your script individually. Especially with Perl you should however watch your quotes, as at times you will have to split lines and quote them separately if both single and double quotes are needed.

To avoid having to specify the interpreter and parameters every time, you can of course wrap another thin layer around <INTERPRET-SCRIPT>:

<$macro DYN-PERL SCRIPT:string/R>
  <INTERPRET-SCRIPT INTERPRETER="perl" SCRIPT=(SCRIPT)>
</$macro>

Following is the complete new multiplication table macro, implemented using the above wrapper, along with two calls and the resulting tables. The part of the script separated from the rest by blank lines is exactly the same as in the container-macros version. The difference is that it's now part of a string starting at the end of the preceding line and ending right before the closing parenthesis that closes the expression for SCRIPT:

<$macro MULTTABLE SIZE:num=10>
<table border="2">
<DYN-PERL SCRIPT=(
    '$end=' + SIZE + ';' + '

print "<tr><th></th>";
for ($i=1; $i <= $end; $i++) {
    print "<"."th style=\"background-color:#9090f0\">$i</th>";
}
print "</tr>\n";
for ($i=1; $i <= $end; $i++) {
    print "<tr><"."th style=\"background-color:#9090f0\">$i</th>";
    for ($j=1; $j<= $end; $j++) {
        print "<td>", $i*$j, "</td>";
    }
    print"</tr>\n";
}

')>
</table>
</$macro>

<MULTTABLE SIZE="3">

<MULTTABLE SIZE="5">

Using shell parameters

Dynamically built scripts may become a little unwieldy when they get longer, because due to their being HSC strings you have to watch your use of quotes and eventually split the program into several differently-quoted strings that you concatenate in one or more expressions. Perl in particular is a language that assigns different quote characters different semantics, and often you can't but use both single and double quotes in an expression.

To avoid the confusing source layout with mixed scripting language and HSC syntax but retain the possibility of passing information from your HSC code to the script, you can use INTERPRET-SCRIPT's PARAMS attribute. This passes additional shell parameters to the interpreter when HSC calls it, and usually these can be accessed easily from within your script. As an example, take the following alternative way of writing the multiplication table script:

<$macro MULTTABLE SIZE:num=10>
<table border="2">
<PERL PARAMS=SIZE>
$end = shift;
print "<tr><th></th>";
for ($i=1; $i <= $end; $i++) {
    print "<"."th style=\"background-color:#9090f0\">$i</th>";
}
print "</tr>\n";
for ($i=1; $i <= $end; $i++) {
    print "<tr><"."th style=\"background-color:#9090f0\">$i</th>";
    for ($j=1; $j<= $end; $j++) {
        print "<td>", $i*$j, "</td>";
    }
    print"</tr>\n";
}
</PERL>
</table>
</$macro>

Last change: 21-Feb-2006, 06:43