mailing-list for TeXmacs Users

[Enrique Perez-Terron <address@hidden>]

I have tried again now and then to get a working Axiom and TeXmacs
combination.  From time to time I read in mailing lists about tricks
that work for others, but they never worked for me.

I found tm_axiom.c in TeXmacs-1.0.7.2-src.tar.gz, but it was so hard to
read that I had to partially translate it into more well-known idioms
and more analizable programming patterns, just to get an idea about what
it does.  I have not read the corresponding code in texmacs that
communcates with it, so I do not know what "language" tm_axiom is
supposed to speak.

In the end I found the source of my problems, in Axiom: it issues two
copies of the initial prompt. I run Fedora11.  Tm_axiom stops reading
after the first prompt, issues a command, reads to the next prompt, etc.
(If it were to read past the last prompt it would hang in the read,
presumably for ever.  The program does not invoke non-blocking IO
modes.)  So, when I issue "2 + 2", I get the answer to my previous
command, if 2+2 is the first, there are the ")set so-and-so-mode" that
tm_axiom issues during session initialization.  But I did not have the
patience to discover this pattern, I just noticed it did not work.
Perhaps I also added a semicolon to some commands, not being an Axiom
expert yet. (Semicolon suppresses output.)  Software usability problems
obey the law 1+1 = infinite.

I have also tried on the laptop, that runs Ubuntu (and before that,
Debian), but I don't remember the details of how it failed.  Everything
takes its time and I have other things... I have also tried to run Axiom
under TeXmacs on a friend's Vista computer, no luck.

I have tried a couple of times to recompile Axiom; it crashes in the
subcompile of gcl (gnu common lisp) in a variety of ways, depending on
the tarball/cvs gold, silver, source rpm, and the phase of the moon. The
last separate download of gcl from cvs failed due to new compiler
features in gcc-4.4. Now that I have found the compiler incantation that
gets me through the compile of gcl (add -fno-strict-aliasing in TCFLAGS
in configure), perhaps I will manage to compile the rest of Axiom in a
few weeks or months, as time and opportunities allow. Then I might read
enough code to find the code responsible for the repeated prompt - in
only a few years.

Having found the root cause, it seems easier for the moment to adapt
tm_axiom to read the prompt twice before sending anything to Axiom. But
having invested so much in reading this little file, I could not resist
turning my partial translations into a working program, and here it is;
it works for me. It is longer than the original and probably no better
except for using select() to determine if it should read the prompt
again. 

(If a new Axiom comes along with just one prompt, my program will wait
one second before timing out. This happens only in the beginning, during
initialization. It is of course easy to disable this particular code,
and the select() call allows much shorter timeouts, 0.1 seconds probably
works.)

On the other hand, I do believe that its code structure is better for
future maintainability, much better.  It is not just my personal
preferences with regard to indentation.  It is about making the
subroutines perform more general operations that are easier to describe,
and therefore easier to remember when you see the call site.  When
subroutines become more general, the "strategic" decisions shift up to
the callers, and are communicated down to the subroutines using
parameters when necessary.  I have strived to have the logic of, e.g.,
when to enter and exit "math mode", all in the same function, rather
than having global mode variables that change all over the place.

Also, by using parameters to pass modes to subroutines, it becomes much
easier to follow the trails of the information, determining where a
setting could possibly come from, etc.

So, if anyone want to try it out and tell me about bugs, I will gladly
keep maintaining it for the next few months.  But I need to travel a lot
this fall and next year, and will not always be available.  The code is
heavily commented, and I am convinced that you will find it a piece of
cake to fix problems yourselves.  (Ouch, that was a mouthful of
boasting, but what can I do? :)

Once I began to understand what the program was doing it could not
resist changing some of the logic, so the behavior will differ in corner
cases.  I doubt there was strong reasons for everything in the old
program, but you will likely be bitten by something.  

(The language seems fragile anyway. For example, a prompt is a string
"-> " at the start of the line. Is there any guarantee that no such
string will occur in other output? I have seen " -> " in error messages.
Could long domain names force a line break just before the hyphen?  I
have thought of adding "select()" to the main loop as well, to catch
continued output after an apparent prompt.  A logical move would be to
use select() for all IO, to guarantee there is no hanging in reads from
sources that are themselves waiting for us. This is standard procedure
for programs that do four-way communication, like telnet, ssh, etc.
With that model, the program would always read any pipe that had data
pending and never sleep unless all input pipes are empty.  But all this
takes time and, as I said, tm_axiom now works for me as it is.)

Just to remind potential users: if you (or your package manager)
install(s) Axiom in, say, /usr/libexec/axiom, there is probably a
directory  /usr/libexec/axiom/mnt/_something_ that contains the bulk of
the associated files, and you need to set the environment variable AXIOM
to point to this _something_ directory. You also need to have $AXIOM/bin
in the PATH. Do not forget to "export" AXIOM. If you start texmacs from
desktop menus or from icons, you will have to set and export AXIOM and
PATH in login scripts or boot scripts.  Some distributions of Axiom may
supply a trampoline script in /usr/bin/axiom that set the environment
before starting the real Axiom. That solution may take care of all this
old-fashioned hassle with environment variables.

I have seen versions of tm_axiom on the web that start axiom as
"AXIOMsys".  My tm_axiom starts "axiom" wherever that is found in the
path.  Adjust as required for your version.

Compile tm_axiom.c with "gcc -o tm_axiom tm_axiom.c", and copy tm_axiom
to /usr/libexec/TeXmacs/bin, or wherever tm_axiom resides in your
TeXmacs installation. "locate tm_axiom" is your friend.

Some mailing lists don't like attachments, but I have seen a couple of
attachments in this list. Bear with me if that is wrong.

I do not include a patch, as the rewrite is so nearly total. Those who
need can generate a patch with "diff -u tm_axiom.c new-tm_axiom.c".

Oh, and Windows users - forget it. Unless you like to fiddle.

-Enrique
/******************************************************************************
* MODULE     : tm_axiom.c
* DESCRIPTION: Glue between TeXmacs and Axiom
* COPYRIGHT  : (C) 1999  Andrey Grozin
*******************************************************************************
* This software falls under the GNU general public license and comes WITHOUT
* ANY WARRANTY WHATSOEVER. See the file $TEXMACS_PATH/LICENSE for more details.
* If you don't have this file, write to the Free Software Foundation, Inc.,
* 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
******************************************************************************/

// Heavly modified by Enrique Perez-Terron 2009

// Task: filter data in two directions, stdin/stdout talks to TeXmacs,
//  axin/axout talks to Axiom. We are not playing with "select()" like telnet,
//  so we must take care to not read from Axiom when Axiom is waiting for
//  input. That is, we must detect when the input from Axiom contains a prompt.
//
// Otherwise we also do some transformations.

// Comment out to disable logging
#define LOGFILE "/tmp/tm_axiom.log"

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>

/* According to POSIX.1-2001 we need this for select() */
#include <sys/select.h>

/* According to earlier standards */
#include <sys/time.h>
#include <sys/types.h>
#include <unistd.h>

// Codes for type of data seen from Axiom. LONG means buffer full.
#define NORMAL 0 // Complete data are being returned, not of more specific type
#define LONG   1 // Expect more data. Neither prompt nor newline seen. (Buffer full.)
#define END    2 // EOF, no data being returned.
#define PROMPT 3 // Propmt seen
#define MATH   4 // Normal data with Math string recognized
#define TYPE   5 // Normal data with Type string recognized
#define DASHES 6 // Normal data with dashes property recognized

#ifdef LOGFILE
char *Codename[] = {
  "NORMAL: ",
  "LONG  : ",
  "END   : ",
  "PROMPT: ",
  "MATH  : ",
  "TYPE  : ",
  "DASHES: "
};
#endif

// Put data in "buf" while processing. "len" holds strlen(buf).
// Mathmode data from Axiom is copied to "mathbuf", for some conversions.
#define LEN 256
#define MATHBUFSIZE 40960
char buf[LEN];
char mathbuf[MATHBUFSIZE];
int len;

// Some strings we need to recognize in the input
char Prompt[] = "-> ";
char Math[]   = "$$\n";
char Type[]   = "Type: ";

// Axin, like stdin, is where this program reads. *out is for writing.
FILE *Axin, *Axout;
#ifdef LOGFILE
FILE *Log;
#endif

// tex_to_latex() - convert "\\root{blah} \\of {" to "\\sqrt[blah]{"
// Auxiliary: return pointer to first char after balanced {} pairs.
char *balanced(char *ptr)
{
  int level = 0, c;
  while (c = *ptr++) {
    if (c == '{') { level++; continue; }
    if (c == '}') { level--; if (!level) return ptr; }
  }
  return NULL;
}

// convert; return number of chars buf is shortened.
int tex_to_latex(char *buf)
{
  char *A = buf, *B = buf, *C, *D;
  int shortened = 0;

  // Remove all newlines from buffer. Memmove, not strcpy, because of overlap.
  while (A = strchr(A, '\n')) {
    memmove(A, A+1, strlen(A+1)+1);
    shortened ++;
  }

  // Convert all "\\root {blah} \\of {" to "\\sqrt[blah]{"
  while ((A = strstr(B, "\\root {"))
	 && (C = balanced(A+6)) // A+6 includes the '{'
	 && (strncmp(--C, "} \\of {", 7) == 0)) { // --C includes the '}'
    B = A + 7; // just after "\\root {", this is the earliest point where 
               // another "\root {" could be, and where the next turn in the loop will seach
    D = C + 6; // Includes the '{' after " \\to ".
    strncpy(A, "\\sqrt[", 6); A += 6;
    memmove(A, B, C-B);       A += C-B;
    *A++ = ']';
    memmove(A, D, strlen(D) + 1);
    shortened += 6;
  }

  return shortened;
}

// Get a chunk of data from TeXmacs. 
// We assume TeXmacs only sends '\n'-terminated data, one line at a time.
// Return the terminating condition, LONG == buffer full, END == end of file.
int fromTeXmacs(void)
{
  static eof_seen = 0;
  char *p = buf;
  char *e = buf + sizeof buf;
  int code = LONG;

  if (eof_seen)
    code = END;
  else {
    while (p < e) {
      int c = getchar();
      if (c == EOF) {
	// Report actual data seen before reporting END.
	code = (p==buf ? END : NORMAL); 
	eof_seen = 1; 
	break;
      }
      *p++ = c;
      if (c == '\n') {code = NORMAL; break;}
    }
  }

  *p = '\0';
  len = p - buf;

#ifdef LOGFILE
  fputs("From TeXmacs: ", Log);
  fputs(buf, Log);
  // Always terminate the log output with a newline.
  if (code == END)
    fputs("--end of TeXmacs data--\n", Log);
  else if (len == 0 || buf[len-1] != '\n')
    fputs("\\--unfinished--\\\n", Log);
  fflush(Log);
#endif

  return code;
}

// Read a chunk of data from Axiom. Return code for ending condition.
// If a prompt is seen, it is copied to the buffer, just like other input.
// Only \r is suppressed.
// Return kind of terminating condition, or result of tests ordered through flags.

// It might seem like an error in this program that if we are looking for a string
// like the prompt or the Math, if the string appears stradling a buffer limit, we do not
// detect it.  However, we only expect these strings in the initial part of any line.
int fromAxiom(int flags, int prompt_limit)
{
  static int eof_seen = 0;
  char *p = buf;
  char *e = buf + sizeof buf;
  int prompt_chars_seen = 0;
  int code = LONG;
  if (eof_seen)
    code = END;
  else {
    while (p < e) {
      int c = getc(Axin);
      if (c == EOF) {
	// Report actual data seen before reporting END.
	code = (p==buf ? END : NORMAL); 
	eof_seen = 1; 
	break;
      }
      if (c == '\r') continue;

      *p++ = c;

      if (c == '\n') {code = NORMAL; break;}

      if (flags & (1<<PROMPT)) {
	if (c == Prompt[prompt_chars_seen]) {
	  prompt_chars_seen++;
	  if (prompt_chars_seen == sizeof(Prompt) - 1)
	    { code = PROMPT; break; }
	  continue;
	}
	prompt_chars_seen = 0;
	if (p >= buf + prompt_limit)
	  flags &= ~(1<<PROMPT);
      }
    }
  }

  *p = '\0';
  len = p - buf;

  // Detect particular types of lines
  if (code == NORMAL) {
    if (flags & (1 << MATH))
      if (strcmp(buf, Math) == 0) code = MATH;

    if (flags & (1 << DASHES))
      if (len == 78 && strspn(buf, "-") == 77) code = DASHES;

    if (flags & (1 << TYPE))
      if (len == 78) {
	p = buf; while(*p == ' ') p++;
	if (strncmp(p, Type, sizeof(Type)-1) == 0)
	  code = TYPE;
      }
  }

#ifdef LOGFILE
  fputs(Codename[code], Log);
  fputs(buf, Log);
  // Always supply a newline to the log
  if (code == END)
    fputs("--end of Axiom data\n", Log);
  else if (code == PROMPT)
    fputs("\n", Log);
  else if (len <= 0 || buf[len-1] != '\n')
    fputs("\\--unfinished--\\\n", Log);
  fflush(Log);
#endif

  return code;
}

#ifdef LOGFILE
void log_line(char *prefix, char *msg)
{
  fputs(prefix, Log);
  fputs(msg, Log);
  if (msg[strlen(msg) - 1] != '\n')
    fputs("\\--no-newline\n", Log);
  fflush(Log);
}
#endif

typedef enum {Noflush = 0, Flush} Flush_t;
void toAxiom(Flush_t flush, char *msg)
{
  fputs(msg, Axout);
  if (flush) fflush(Axout);
#ifdef LOGFILE
  log_line("To AXIOM: ", msg);
#endif
}

void toTeXmacs(Flush_t flush, char *msg)
{
  fputs(msg, stdout);
  if (flush) fflush(stdout);
#ifdef LOGFILE
  log_line("To TeXmacs: ", msg);
#endif
}
  
int send_and_get_prompt(char *msg)
{
  int code, silent = 1;
  // Send
  toAxiom(Flush, msg);

  // Get response.
  do {
    code = fromAxiom(1 << PROMPT, 0);
    // Expect just a prompt, but if anything goes wrong...
    if (code != PROMPT) {
      if (silent) {
	silent = 0;
	int msglen = strcspn(msg, "\n");
	printf("\2latex:\\red Command \"%.*s\" produced response: \n", msglen, msg);
#ifdef LOGFILE
	fprintf(Log, "RETURNED: \2latex:\\red Command \"%.*s\" produced response: \n", msglen, msg);
#endif
      }
      toTeXmacs(Noflush, buf);
    }
  } while (code != END  && code != PROMPT);

  if (! silent)
    toTeXmacs(Flush, "\\black\5\5");

  return code;
}

// Check if Haystack contains Needle, or if a final substring of Haystack is an initial 
// substring of Needle. The point is that haystack is only the initial bufferfull of
// a longer haystack, and we want to detect needle if it straddles the buffer boudary.
char *detect_possible(char *needle, char * haystack, char* end_of_haystack)
{
  char *p = strstr(haystack, needle);
  if (p) return p;

  char *e = end_of_haystack;
  for (p = e - strlen(needle) + 1; p < e; p++)
    if (strncmp(needle, p, e - p) == 0)
      return p;

  return NULL;
}

void session(void)
{
  int code;
#ifdef LOGFILE
  Log = fopen(LOGFILE, "w");
  if (! Log) {
    fprintf(stderr, "Could not open log file \"" LOGFILE "\"\n");
    fprintf(stderr, "Logging to stderr instead.\n");
    Log = stderr;
  }
#endif

  // Get the banner. Suppress output after seeing second line of dashes
  int dashlines = 0;
  code = fromAxiom((1<<PROMPT)|(1 << DASHES), 10);
  while (code != END && code != PROMPT) {
    if (code != PROMPT && dashlines < 2)
      toTeXmacs(Noflush, buf);
    if (code == DASHES) dashlines++;
    code = fromAxiom((1<<PROMPT)|(1 << DASHES), 10);
  }
  fflush(stdout);
  if (code == END) goto finish;

  // Currently Axiom outputs the initial prompt twice. We have to read it to
  // get commands and responses in synch. I will bite the bullet, and use select().
  fd_set readfds, writefds, exceptfds;
  struct timeval timeout;

  FD_ZERO(&readfds);
  FD_ZERO(&writefds);
  FD_ZERO(&exceptfds);

  FD_SET(fileno(Axin), &readfds);
  timeout.tv_sec = 1;		// timeout one second.
  timeout.tv_usec = 0;
  
  int readable = select(fileno(Axin) + 1, &readfds, &writefds, &exceptfds, &timeout);

  if (readable) {
    do { code = fromAxiom(1<<PROMPT, 10); } while (code != END && code != PROMPT);
    if (code == END) goto finish;
  }

  // Send initial commands
  if (send_and_get_prompt(")set messages prompt plain\n") == END) goto finish;
  if (send_and_get_prompt(")set messages autoload off\n") == END) goto finish;
  if (send_and_get_prompt(")set quit unprotected\n")      == END) goto finish;
  if (send_and_get_prompt(")set output tex on\n")         == END) goto finish;
  if (send_and_get_prompt(")set output algebra off\n")    == END) goto finish;

  // Main prompt-read-write loop
  do {
    // prompt TeXmacs
    toTeXmacs(Flush, "\2channel:prompt\5\2latex:\\red$\\rightarrow$\\ \5\5");
    toTeXmacs(Noflush, "\2verbatim:");

    // Get the user input, send to axiom. Read until newline.
    do {
      code = fromTeXmacs();
      toAxiom(Noflush, buf);
    } while (code == LONG);

    if (len > 0 && buf[len-1] != '\n')
      toAxiom(Noflush, "\n");
    fflush(Axout);

    // Get Axiom's response
    int flags = (1 << PROMPT) | (1 << MATH) | (1 << TYPE);
    int mathmode = 0;
    char *mathptr = mathbuf;
    while ((code = fromAxiom(flags, 0)) != END && code != PROMPT) {
      if (code == MATH) {
	// MATH lines come in pairs, wrapping math mode lines.
	mathmode = ! mathmode;
	if (mathmode) {
	  // Don't look for TYPE lines until mathmode is off again.
	  flags &= ~(1 << TYPE);
	  // Alert TeXmacs that now comes mathmode data
	  toTeXmacs(Noflush, "\2latex:$\\displaystyle\n");
	} else {
	  // When turning off math mode, send contents of mathbuf to TeXmacs.
	  flags |= 1 << TYPE;
	  *mathptr = '\0';
	  tex_to_latex(mathbuf);
	  toTeXmacs(Noflush, mathbuf);
	  mathptr = mathbuf;
	  // Alert TeXmacs that math mode is ending
	  toTeXmacs(Noflush, "$\5\n");
	}
	// Don't send MATH lines to TeXmacs
	continue;
      }

      if (! mathmode) {
	if (code == TYPE) {
	  char *p = strstr(buf, Type) + strlen(Type);
	  toTeXmacs(Noflush, "\2latex:\\axiomtype{");
	  toTeXmacs(Noflush, p);
	  toTeXmacs(Noflush, "}\5");
	} else
	  toTeXmacs(Noflush, buf);
	continue;
      }

      // During mathmode, collect all output in mathbuf, with current position mathptr.
      if (mathptr + len + 1 >= mathbuf + sizeof mathbuf) {
	// Data does not fit in mathbuf. Ship some of the data to TeXmacs, to make room.
	// Avoid splitting data in the middle of structres that need conversion 
	// by tex_to_latex() - best effort.
	*mathptr = '\0';
	mathptr -= tex_to_latex(mathbuf); // idempotent - fortunately
	char *p = detect_possible("\\root {", mathbuf, mathptr);
	if (p && (mathptr - p) + len < sizeof mathbuf) {
	  // Ship portion before "\\root {"
	  *p = '\0';
	  toTeXmacs(Noflush, mathbuf);
	  *p = '\\';
	  int plen = strlen(p);
	  memmove(mathbuf, p, plen);
	  mathptr = mathbuf + plen;
	} else {
	  // Ship all of it and hope the best
	  toTeXmacs(Noflush, mathbuf);
	  mathptr = mathbuf;
	}
      }
      memcpy(mathptr, buf, len);
      mathptr += len;
    }
    
    fflush(stdout);

  } while (code != END);

 finish:
  toTeXmacs(Flush, "\2latex:\\red The end\\black\5\5");
}

void fatal(char *mess)
{
  fprintf(stdout,"\2latex:\\red Cannot %s\\black\5\5",mess);
#ifdef LOGFILE
  fprintf(Log,"To TeXmacs: \2latex:\\red Cannot %s\\black\5\5\\--no-newline\n",mess);
#endif
  exit(1);
}

int main()
{ int p1[2],p2[2];
  if (pipe(p1)<0) fatal("create pipe");
  if (pipe(p2)<0) fatal("create pipe");
  switch (fork())
  { case -1: fatal("fork");
    case 0: /* Axiom */
      dup2(p1[1],1); close(p1[1]); close(p1[0]);
      dup2(p2[0],0); close(p2[0]); close(p2[1]);
      execlp("axiom","axiom","-noclef",0);
      fatal("exec axiom");
    default: /* parent */
      close(p1[1]); close(p2[0]);
      Axout=fdopen(p2[1],"w"); Axin=fdopen(p1[0],"r");
      session();
  }
  return 0;
}