Backtalk File Formats

Version 1.4.9

© 2002-2003: Jan Wolter, Steve Weiss

Table of Contents

 0.Tag Files
 1.List of Conferences
 1.1.Yapp-style Conflist File
 1.2.Picospan-style Conflist File
 2.Menu of Conferences
 2.1.Backtalk confmenu File
 2.2.Grex-style public.txt File
 2.3.M-Net-style public.txt File
 2.4.Other public.txt Files
 3.Conference Configuration
 4.Conference Access Control List
 5.Conference Summary
 6.Item File
 6.1.Tradition Picospan/Yapp Item Files
 6.2.HTML Responses and Alternate Texts
 6.4.Item IDs
 7.Index File
 8.Participation File
 8.1.Traditional Picospan Participation Files
 8.2.Backtalk Participation Files
 8.3.Backtalk Participation Extension Files
 9.Authentication/Identification Databases
 9.1.Unix Logins
 9.2.Backtalk Logins
10.Dump Files
10.1.User Dump Files


This document describes some of the file formats used in Backtalk.

Backtalk is designed to be able to share its conference database with Picospan or Yapp. Picospan is a conferencing program designed by Marcus Watts and distributed by NETI. It was the original software used by M-Net and The Well. It is still used today on Grex, but is no longer available due to the collapse of NETI. Yapp is a clone of Picospan written by David Thaler. Yapp 2.3 source is available as shareware. Yapp 3.0 is an extended version including a web interface. Both use file formats very similar to, but not completely compatible with Picospan.

Backtalk can be built to coexist with Picospan, Yapp, or neither. The file formats it uses will be slightly different in each case.

0. Tag Files

Backtalk uses several different "tag files". These are files that give string or integer values for various variables ("tags"). Tag files can contain fairly arbitrary information. They are not normally editted manually, but the can be. Different Backtalk interfaces may store different information in them. Tag files used by Backtalk include the following:

Filename Location Description
.backtalk User's home directory User's personal settings
id User's home directory User's login name, uid number, and gid number. Used only in configurations built with --ident=tagfile.
settings Conference directory Settings for a conference.
Backtalk-style participation files are modified tag files.

0.1. Sample

Here's a sample settings file:
   TER>\n&nbsp;\n<BR>\n<FONT SIZE=+1>Welcome to the</FONT><BR>\n<FONT SIZE=+3>TEST<
   /FONT><BR>\n<FONT SIZE=+1>Conferencce</FONT><P>\n\(Fairwitness:  Marcus Watts\)<
Note that the "login" field is all one line, broken across four lines here for readability.

0.2. Description

Each line starts with a tag or variable name, terminated by an equal sign. After the equal sign is either a number, if it is an integer tag, or an string enclosed in parentheses. Inside the string, any backslash or parenthesis characters have a preceeding backslash and any newlines characters are converted to the two character sequence "\n".

In some tagfiles some tag names will be prefixed with a '*' character. This indicates that the field can only be editted by a conference administrator.

1. List of Conferences

The conflist file resides in the top level "bbs" directory. It is used to map typed names of conferences to the actual directory names. It allows you to have multiple names for the same conference, or to abbreviate the names of conferences. These features are mainly useful when people are typing in conference names, which isn't that common in most Backtalk interfaces - you more often select the conference from a list, and having alternate names for it isn't that useful there. However, it's a nice feature to have for command-line interfaces like Picospan and Fronttalk.

Yapp and Picospan use slightly different formats here. Backtalk uses the Picospan format, unless it was configured for Yapp compatibility at compile time.

1.1. Yapp-style Conflist File

1.1.1. Sample


1.1.2. Description

The first line is always ``!<hl01>''. This is a magic number identifying the file type and version.

The second line is the directory name of the default conference. This is rarely used in Backtalk's web interfaces, but it is in command-line interfaces.

The remaining lines each consist of a conference name, a colon, and a conference directory. Conference names usually have an underscore inserted in them somewhere. When a user types the name of a conference, he may abbreviate it by leaving off letters after the underscore. Thus, in the sample above, you could join version 2 of the general conference by typing ``gen,'' ``gene,'' ``gener,'' ``genera,'' ``general,'' or ``general2.'' To join version 1, you'd have to type ``general1'' however, because there is no underscore in the name so no abbreviations are allowed. Note that there can be more than one line for the same conference. Thus, in the sample above, we can also go to version 1 of the general conference by using the name ``oldgen.''

Lines may be commented out by starting them with a # sign.

1.2. Picospan-style Conflist File

Newer versions of Picospan support some compression of the conflist file. The old syntax still works though.

1.2.1. Sample

The following Picospan conflist file is equivalent to the Yapp conflist in section 1.1.1:

1.2.2. Description

Here the % sign is replaced with ``/usr/local/backtalk/bbs/'' while the file is being read in. Actually, it is replaced with whatever default location was defined for conference directories during the installation process. Otherwise the format is the same. Of course, if any conference directories are stored in a different place, then their full path names must be given, as before.

2. Menu of Conferences

When you request a list of all conferences from Backtalk, Yapp or Picospan, you do not get a listing of the conflist file. Instead, there is a separate conference menu file. Conferences can be listed in the conflist, but not the conference menu, in which case it will still be possible to join them normally, if you happen to know they exist and if the interface you are using offers a way to type in conference names. This makes "hidden" conferences possible.

When Picospan or command-line Yapp are given a command to list all conferences, they simply print the public.txt file. This is a file with no fixed format that lives in the conference directory.

Backtalk needs a more formally formatted list of conferences that it can read, parse apart, and display in different formats in different interfaces. So Backtalk defines a new file called "confmenu" which lives in the conference directory.

In places where Backtalk co-exists with Picospan or command-line Yapp, Backtalk can be configured to use the traditional "public.txt" file instead, but it requires that it be entered in a consistant format so that it can be parsed. However, what that format should be can be configured. You can write your own regular expressions to parse a fairly broad range of file formats. Two sets of regular expressions are pre-written, one for a Grex-style public.txt file, one for an M-Net-style public.txt file. Those will be described here.

Web versions of Yapp seem to have a different conference menu database. I haven't figured out what exactly this is.

2.1. Backtalk confmenu File

2.1.1. Sample

:Terrestrial Conferences:Conferences primarily relating to the planet earth.
general:General:Our conference for general discussions
oldgeneral:Old General:Previous version of the general conference
:Interplanetary Conferences:Sponsored by your friends from NASA
mars:Martian:Talk about Mars here
venus:Venusian:Talk about Venus here
:Test Conferences:
test:Test:A place to experiment

2.1.2. Description

The conference menu contains two kinds of lines. The lines starting with colons are "header" lines. They describe related groups of conferences. The other lines describe individual conferences. The conferences will normally be listed in the order that they appear in the file, with each header applying to the conferences listed after it.

You don't need to have any headers in the conference menu.

The first field in each line is the internal name of the conference. (This is blank for header lines). This must match a line in the conflist file. This is normally all lower case with no spaces.

The second field is the name of the conference to be displayed in the conference list. This can be mixed case and can have spaces, but should generally be something quite short. For headers it is the text of the header.

The third field is a longer description of the conference or group of conferences.

Lines starting with # signs are comments and are ignored.

2.2. Grex-style public.txt File

2.2.1. Sample


Terrestrial Conferences:

 general    - Our conference for general discussions
 oldgeneral - Previous version of the general conference

Interplanetary Conferences:

 mars       - Talk about Mars here
 venus      - Talk about Venus here

Test Conferences:

 test       - A place to experiment

    (last modified 12/23/02)

2.2.2. Description

Like a confmenu file, this contains header lines and conference lines. It also contains various other lines. Anything that isn't a conference or a header is ignored (like the "OUR CONFERENCES" title and the "last modified" date).

Header lines start in column 0 (no leading spaces) and have a colon in them. The header text is from the start of the line through the colon. You cannot give descriptions.

Conference lines start with exactly one space. After the space is the internal conference name, followed by white spaces and dashes. After this is the description of the conference. The description must all fit on one line. The title of the conference will simply be a capitalized version of the conference name.

2.3. M-Net-style public.txt File

2.3.1. Sample

 You may JOIN any of the following conferences:
  ********Terrestrial Conferences*********

general......Our conference for general discussions
oldgeneral...Previous version of the general conference

  *******Interplanetary Conferences********

mars.........Talk about Mars here
venus........Talk about Venus here

  ***********Test Conferences*************

test.........A place to experiment

2.3.2. Description

Same basic idea, slightly different format. Headers start with some white space. The text of the header is surrounded by one or more stars on each side. Conferences start with no white space, and the internal conference name is separated from the description by dots or white space. Any other lines are ignored.

2.4. Other public.txt Files

Other systems which have used Yapp or Picospan for a long time have probably developed their own favorite formats for the public.txt file. Very likely you can teach Backtalk to use it, though you may have to clean the file up a bit.

All you have to do is provide two regular expressions - one that matches header lines, and one that matches conference lines. They should contain parenthesis that pick out the interesting substrings of the lines - the header text for headers, and the conference name and description for conferences.

This means that (1) headers and conferences must be described in one line, and (2) the conference name must appear before the conference description in conference lines. If this is true for your public.txt file, then Backtalk can probably use it.

3. Conference Configuration File

Every conference directory must contain a conference configuration file named config. This stores some basic conference configuration information.

3.1. Sample


3.2. Description

Magic Number
The first line is always '!<pc02>'. It identifies the file type.
Participation File Name
The second line is the name of the participation file used for the conference. Each user that joins the conference will have a participation file by this name created in his home directory. The name should be unique among all conferences that have ever existed in the history of the system.
Time Limit
The third line is always zero. Backtalk and Yapp always ignore it. I don't know if non-zero values mean anything to Picospan. Some references refer to it as a time limit.
Fairwitness List
The fourth line is a comma-separated list of the login names of the users who are fair witnesses in this conference.
Conference Mode
The fifth line, if given, is the conference mode. If not given, it defaults to zero. The meanings of the mode values are as follows:
0 all users may read and post [default] public
4 only users in the ulist or glist may read and post ulist
5 only users who know the password may read or post password
6 only users who are in the ulist or glist and know the password may read or post paranoid
8 all users may read and post (but item files not directly readable) protected
20* only users in the ulist or glsit may post / all may read ulist readonly
21* only users who know the password may post / all may read password readonly
22* only users who are in the ulist or glist and know the password may post / all may read paranoid readonly
* Note that read-only conferences are not recognized by Picospan, though they are recognized by Yapp. If you are using Backtalk with Picospan, modes 20, 21, and 22 cannot be used.
Later versions of Yapp allow sequences of keywords to be used instead of numeric modes. Backtalk also recognizes these. They are given at right in the table above. Picospan does not recognize these keywords.

If Backtalk was compiled to be Yapp-compatible mode, then the modes given in the config file will be ignored if an acl file exists. In non-Yapp compatible installations, the acl file will only be checked if the mode in the config file is 'acl' or a negative number.

E-Mail Address
Later versions of Yapp use the sixth line for mailing addresses when conferences are linked via email. If Backtalk is compiled for Yapp compatibility, it will ignore this line, and put the conference title in the seventh line. Otherwise, the conference title is in the sixth line.
Conference Title
The conference title, if given, is usually mixed case and possibly has some spaces in it. Something that will fit nicely into the "The X Conference". Normally it would be the same as whatever conference title is given in the confmenu file, but it doesn't have to be. If the sixth line is omitted, a capitalized version of whatever name the user used to join the conference is used as the conference title.

4. Conference Access Control List

Conference access control lists files were introduced in late versions of Yapp. They allow more fine-grained control over which users have which kinds of access to the conference. The file is called "acl" and it resides in the conference directory. Backtalk supports "acl" files, but somewhat half-heartedly.

Normally Backtalk will only pay attention to "acl" files only if the conference mode specified in the conference "config" file is negative or "acl". However, if compiled in Yapp-compatible mode, Backtalk will always check for the "acl" file and will ignore any conference modes set in the "config" file if it is found.

4.1. Sample

    r +all
    w +registered -f:observer
    c +f:ulist
    a +fwlist

4.2. Description

Lines of the acl file may begin with one of the following symbols:
r    Who may join and read the conference.
wWho may respond to existing items in the conference.
cWho may create new items in the conference.
aWho may edit the acl file.
The lines may appear in any order. Blank lines are accepted. Currently, the 'a' lines are ignored by Backtalk, since it doesn't actually include any tools to edit acl files.

After these characters will appear a sequence of keywords describing the tests that a user must pass to have that access. All tests on the line must be passed by a user to have the given access. If the keyword is prefixed with a '+' or has no prefix, then the user must pass the test. If the keyword is prefixed with a '-' then the user must fail the test.

Recognized tests are:

all All users.
registeredUsers who have logged in.
originlist Users from sites listed in the 'originlist' file in the conference directory. Ignored by Backtalk.
password Users who can give the conference password in the 'secret' file in the conference directory.
f:filename Users whose logins are included in the named file in the conference directory.
u=userlist Users whose logins appear in the comma-separated list. Not Yapp compatible.
g:filename Users who are in at least one of group names listed in the named file in the conference directory.
g=grouplist Users who are in at least one of comma-separated group names listed. Not Yapp compatible.
sysop System administrators. Equivalent to 'g=cfadm'
# Comment.
The '+all' keyword is actually a no-op, but the '-all' keyword is useful for banning anyone from doing something. Backtalk supports '+all' for item entry or responding only if 'anonposter' is defined in the file. Otherwise you must always be 'registered' to perform these actions.

All the keywords except 'all' and 'secret' imply 'registered'. You have to be logged in to have a chance of passing them.

In Backtalk (though probably not in Yapp) 'fwlist' implies 'sysop', since system administrators are considered to be fairwitnesses in all conferences. If you wanted to permit something to fairwitnesses, but not sysops, you'd have to say '+fwlist -sysop'.

The 'f:' keyword is most commonly used as '+f:ulist' to say that a user must be in the ulist file, or as '-f:observer' to say that a user may not be in the observer file. Similarly, '+g:glist' says the user must be a group in the 'glist' file. However these may be used with any filename. Filenames starting with '/' are treated as full paths. Otherwise they are sought in the current conference's directory.

5. Conference Summary File

The oldest versions of Picospan did not have conference summary files. The information contained in them is all redundant. If the conference summary file is removed, Backtalk, Yapp or Picospan will reconstruct it. The purpose of the summary file is to provide a rapidly readable index of all the items in a conference. It is used to improve performance.

Because of this, it is one of the few binary files used by Backtalk. It must be rapidly accessible, and it does not have to be easily read or editted by humans. If a file is corrupted, you don't edit it, you delete it. The Backtalk distribution includes two programs, sumcheck and dumpsum that can be mildly useful in reading and repairing sum files, in the unlikely event that you should wish to do so. See man pages in the doc directory for more information.

The sum file contains a 24-byte header record, followed by a sequence of 16-byte item records. The sum file entry for item number N is always stored at offset 24 + 16(N-1).

5.1. Header Record

The fields of the 24-byte header record are as follows:
8 bytescharMagic Number Always "!\n". Identifies file type and version.
4 bytesintChecksum A checksum on the name of the participation file given in the config file.
4 bytesintUnknown Picospan always sets this to 0x00001537. I don't know what it means.
8 byteslongSequence The value 0x12345678. Used for byte order checking.
Given a participation file name, we compute Checksum by doing
    set Checksum = 0;
    for each character Char in the name
       Checksum = (Checksum * 4) XOR Char
Backtalk puts this checksum into sum files it creates, but does not check them when it reads an existing sum file.

Sum files are written using the native byte order of the computer that generated them. So if you have conferences on a shared file system, it is possible that different sum files may be created with different byte orders. We check the Sequence field to determine the byte order of the participation file. If it is not the native byte order, then the program must bytes when reading or writing.

5.2. Item Record

The fields of the 16-byte item records are as follows:
4 bytesunsigned intFlags Item flag bits. A boolean OR of the following:
   0x0002     Item is retired.
0x0030Item is active.
0x0040Item is frozen.
0x0080Item is synchronous.
0x0100Item is linked.
4 bytesunsigned intResponse Count The number of responses on the item, including the item text.
4 bytesunsigned intResponse Date Date/time of last response to the item.
4 bytesunsigned intItem Date Date/time item was entered.
In Picospan, ordinary items may have flag values 0x0030 or 0x003C. I don't know what makes the difference. In Backtalk and Yapp ordinary items flagged 0x0030. Additional bits are set for retired, frozen, synchronous or linked items. Entries for deleted items have the flags set to zero.

The times are standard Unix time integers (seconds since Jan 1, 1970). The response date should always match the 'mtime' value for the item file - the time at which the file was last modified.

The system is less picky about the item date, and it is set to slightly different values in different circumstances. When a new item is entered, it is set to the value the file's mtime had after the item was created. If an item is linked in from another conference, it is set to the time that the link was made. If the sum file ever has to be reconstructed, then it is set to match the time stamp on response zero (the item text). Picospan and Backtalk are not entirely consistant in the way they set this value, but I don't think it matters.

6. Item File

Item files store the actual text of a Picospan item and all of the responses to it. Item files reside in the appropriate conference directory. The name of the file is the item number, with an underscore prepended.

For the most part, Backtalk uses the traditional item file format used by Picospan and Yapp. The main extension is in how HTML and plain-text versions of responses are stored in installations that must be Picospan or Yapp compatible. This is done slightly different for Picospan and Yapp, and will be discussed at the end of this section.

6.1. Tradition Picospan/Yapp Item Files

6.1.1. Sample

The following is an item with two responses:
    ,HOur First Test Item
    ,AJan Wolter
    This is an item entered to test Backtalk.
    This is the item text for that item.
    It is a very good item.
    ,AJoseph Cantata
    This is the first response to the very dull item
    that was entered by Jan Wolter.  This response too
    is very dull.
    ,AJan Wolter
    How very dull!  It is very good that this item is
    so very dull.

6.1.2. Description

All item files begin with the magic text string "!<ps03>" which identifies it as an item file and gives the version number.

The second line is normally the item title, identified by a ",H" prefix.

The rest of the item file consists of a sequence of responses. Note that, the item text is simply the first response.

Each response consists of a ",R line that contains various flags for the response, a ",U" line that contains the login name and UID number of the author, a ",A" line that contains the author's full name, a ",D" line that contains the date of the response in hexadecimal, and then the text of the response, enclosed by ",T" and ",E" lines.

Any lines in the text that start with a comma are stored with two initial commas in a Picospan item file, or three initial commas in a Yapp item file. Backtalk follows Picospan in this, unless configured for Yapp compatibility.

Response flags are as follows:

,R0000    Regular response
,R0001Hidden response
,R0003Scribbled response

For some reason, it is common for the ",E" tag to be missing. Programs reading item files should tolerate this.

Yapp supports three additional directives. The "parent" link, ",P" usually appears between the ",D" and the ",T" lines:

    ,AKate Middleton
    This is a response to response number 2.
This causes the response to be displayed with a "<response to #2>" message beside it. There are also ",N" and ",M" lines that can appear immediately after the ",T" line, but before the message text. This are used when importing Usenet news messages and give the original article number and message ID respectively. Backtalk understands the ",P" line but not the other two.

For editted responses, Backtalk stores an edit date in addition to the post date. This is a second hexadecimal string on the ",D" line.

   ,D3b977d6a 3b978ac1

The item modes are mostly stored in the item files permission bits. Linked items are detected by checking if the item's link count is greater than one. Frozen items have their owner-write permissions turned off. Retired items have their owner-execute permissions turned on. I forget how synchronous items are flagged - group execute, I think.

6.2. HTML Responses and Alternate Texts

Backtalk allows responses to be either plain text or HTML. HTML Responses are flagged by setting another bit in the response flags, like ",R0010" for an HTML response that has neither been hidden nor erased.

When Backtalk is configured for compatibility with Yapp or Picospan, then a plain text version of any HTML response is posted in addition to the HTML response. The plain text response is placed so that Yapp or Picospan will find that, while the HTML text is stashed someplace slighly unusual where Yapp or Picospan will ignore it.

For Picospan, this works like this:

    ,AJan Wolter
    I'm feeling very <STRONG>strong</STRONG>
    and <EM>emphatic</EM> today.
    I'm feeling very strong
    and emphatic today.
Here the HTML response is inserted immediately before the ",T" line.

For Yapp, we additionally prefix each line of the HTML version of the response with ",:" characters:

    ,AJan Wolter
    ,:I'm feeling very <STRONG>strong</STRONG>
    ,:and <EM>emphatic</EM> today.
    I'm feeling very strong
    and emphatic today.

6.3. Attachments

Backtalk allows attachments to be associated with responses. Every attachment is identified by a unique string called an "attachment handle", which can be used to find the content of the attachment in the attachment directory, or to lookup up information about the attachment in the attachment index. Attachment handles tend to be somewhat human-readable strings, including the item number and response number, but this is only to make them look pretty in URLs. Programs should expect to be able to parse them apart for any really meaningful information. (If the item has been linked for example, the item number in the attachment handles is likely to be wrong.)

Responses that have attachments include attachment lines, which are simply a ",X" prefix followed by an attachment handle. These appear after the ",A" line and before the ",D" line. More than one my appear. For example:

    ,AJan Wolter
    Here are some picture of my uncle doug.

6.4. Item IDs

In some configurations, backtalk assigns a unique integer item ID value to each item. These may be stored on a ",I" line, which is typically the second line of the file:
Item ID's are normally 24 character strings, but shorter ones will work fine and longer ones will be truncated. It is fairly important that every item on a system have a different item id.

7. Index File

Index files are a Backtalk extension. They are not supported by Picospan or Yapp. An index file is maintained for each item file. It is a binary file containing the offsets into the item file where each response begins and is used to rapidly find the Nth response to an item. If an index file is missing, incomplete, or inaccurate, Backtalk will usually detect the fact and repair the file automatically.

Index files reside in a subdirectory of the conference directory called "indexdir". The index file's name is the item number prefixed with an "@" character.

When Backtalk links an item, the index files will also be linked. If a link is created by Picospan or Yapp, no link is created and the items will likely end up with separate index files.

The format of an index file is simple. It is a sequence of 4-byte integers. The first is the offset of response 0 (the item text), the next is the offset of response 1, and so on. The offsets always point to the start of the ",R" field of the response. So to find response 412 in an item, we'd seek to offset 412*4 in the index file, read the long integer there, and then seek that offset in the item file.

8. Participation File

Participation files record information about a user's participation in a particular conferences. Traditionally this includes a record of which responses have been read, which items have been forgotten, and the user's alias in the conference. Backtalk will expand on this a bit.

8.1. Traditional Picospan Participation Files

Picospan participation files reside in the user's home directory, or, if it exists, a directory name .cfdir under the user's home directory. They are owned by the user. There is one file for each conference which the user has joined. The names of the files for the individual conferences are specified in the conference configuration files.

In most Unix-login configurations, Backtalk changes the group of the files to one it can read/write. In Backtalk-login and Yapp 3.0 compatible Unix-login configurations, the files are owned by the cfadm account.

8.1.1. Sample

    Jan Wolter
    1 76 3D1C5899
    2 8 3D1C5A80
    3 3 3D1C5A8A
    4 20 3D1C5B69
    5 18 3D1C5B80
    6 58 3D1C5BA3
    7 52 3D1C5CD8
    8 -9 3D1CEF3F
    9 -30 3D1CEF6A

8.1.2. Description

The first line identifies the file type. It is always !<pr03> in Picospan and Yapp.

The second line gives the user's alias in this conference.

The remaining lines describe items that the user has read. Each contains three columns.

The number of the last response read may be prefixed with a minus sign. This means that the item has been forgotten. Picospan recognizes "-0" as a distinct value from "0" - the former is an item that has never been read, but has been forgotten. The later is an un-read, un-forgotten item. Yapp does not understand "-0" and does not allow unread items to be forgotten.

The time of last reading is represented as a standard unix time value (the number of seconds since the begining of 1970), written in hexidecimal notation.

The time of last reading is used to distinguish between new and unseen items. An item has unseen responses if the number of responses in the item exceeds the last response read. An item has new responses if it is unseen AND the date of the last response in the item exceeds the date on which we last read responses.

8.2. Backtalk Participation Files

Backtalk likes to store some additional per-user/per-conference and per-user/per-item information that is not included in traditional Picospan Participation Files. So a more flexibly formatted file is used for Backtalk systems that do not need to be Picospan compatible.

These files are named the same as a Picospan file would have been, and are stored in the same locations, except that Backtalk accounts don't usually have .cfdir directories.

8.2.1. Sample

    alias=(Jan Wolter)

8.2.2. Description

This is basically a series of tagfiles, separated by item numbers alone on a line. The tags defined before the first number apply to the entire conference. The tags after each number apply to only that item. Some tags have meaning to the Backtalk interpretor, others are only used by the interfaces. Ones that are specific to a particular interpretor usually have names like 'pistachio.background'.

The tag syntax is actually slightly different than regular tag files, for no good reason. A tag without a value (like "fav") is equivalent to setting it to integer 1 (that is, "fav=1"). If a string contains newlines, actual newlines are put in instead of "\n" values. Oh well.

In the conference-wide section, the 'alias' tag defines the user's name in the conferences.

In the per-item sections, the 'r' field is a selector describing all unread responses in the item. There is an implied '-$' at the end of each, so 'r=(9)' means we have not read responses 9-$. Currently these are always single digits, because, like Picospan, if we have read response 109, then we assume that we have read all that went before. But to support tree-style items, we need to remember exactly the set of responses which are still unread, so these will become more general selectors. If this is a single number (as it usually is), it can be written out as a number instead of a string, without the parentheses.

The d field is the date the item was last read. This is exactly like the date field in the traditional picospan participation file, except that it is written in decimal instead of hexadecimal.

Forgotten items are marked by a 'forgot' tag.

Extensions to the standard Picospan format include the 'fav' tag which is included if the item is one of the user's favorites, and the title flag which gives the user's personal title for the item.

8.3. Backtalk Participation Extension Files

When Yapp or Picospan compatibility is needed, we must maintain two files for each item the user reads. One is the standard Picospan participation file. The other is an extension file. We can't try to jam our extra information into the Picospan file, because Picospan and Yapp destructively rewrite the file every time you enter and leave the conference. Even if we could hide some extra information in there in such a way that they wouldn't choke on it, they probably wouldn't write it back out for us.

So we maintain our extra information in an extra file, called the participation extension file. It's name is the same as the participation file, but with an 'x' appended to the name.

8.3.1. Sample


8.3.2. Description

This files format is just like a Backtalk participation file, except that it starts with a !<px01> line and it includes only fields that have information not included in the Backtalk file.

9. Authentication/Identification Databases

Backtalk users are identified by a unique login name. Other information about the user is stored in three conceptually separate databases, the authentication database, the identification database and the .backtalk tag file. The first two typically have rigid formats and can contain only a few different kinds of data. The .backtalk tag file is more free-format and can contain any kind of key/value pairs. The following user data resides in these three databases:
encrypted passwordauthentication
account validation status  authentication
uid numberidentification
gid numberidentification
full nameidentification
home directoryidentification
privacy flagidentification or .backtalk
email addressidentifcation or .backtalk
email privacy flagidentifcation or .backtalk
email validation statusidentifcation or .backtalk
anything else.backtalk
The identity and authentication databases are sometimes the same database.

Account validation status takes three possible values: validated, unvalidated, and invalidated. Unvalidated accounts exist only if Backtalk is configured to require new user accounts to be manually validated by an administrator before they can log in. Invalidated accounts are formerly active accounts that have been "turned off" by an administrator for whatever reason. It is sometimes stored as a numeric value (0=validated, 1=unvalidated, 2=invalidated) and sometimes as a prefix code on the encrypted password.

The privacy flag is 1 if it is OK to display user information to other users. Passwords, of course, are never displayed. If it is 0, this information is to be kept secret.

If email privacy flag controls the privacy of the user's email. If 0, only the user and administrators can see it. If 1, it is still secret, but users can use Backtalk's mailer to send mail to the address. If 2, any logged in user may see it, but others cannot. If 3, any user may see it, but it is displayed as a series of gifs and not made clickable when anonymous users try to see it. If 4, it may be seen by anyone.

Email validation codes are used to validate email addresses. If Backtalk is configured to validate email, then whenever a new account is created, or an old account changes it's email address, a validation code is assigned. A copy of this value is mailed to the new address. Users must click on a link in the email, sending back the code, to confirm their email addresses. So, if the email address has been validated (or if Backtalk is not configured for email validation) then the email validation string is an empty string. If the email address is not validated, it is the validation code.

9.1. Unix Logins

If Backtalk is configured with Unix logins, then the unix user database is used as the authentication and identification database. Passwords, uids, gids, fullnames, and home directories are taken from that database. All other data is stored in the .backtalk file.

9.2. Backtalk Logins

If Backtalk's own accounts are used, then authentication and identification databases can be chosen from many alternatives. The authentications databases are mostly ones designed to be readable by various Apache authentication modules, which actually do the authentication if we are using basic authentication. Identity information can be put in a similar parallel database, the same database, or in a tag file named id. The various possible databases are described below.

9.2.1. Flat Text Authentication Files

The Text authentication file is used if you configure Backtalk with the --auth=text option, but not the --ident=authfile option. It resides in the passwd subdirectory of the Backtalk data directory and is named bt_passwd. Unless suexec or cgiwrap is being used, it will normally be owned by the user Apache runs as instead of by the Backtalk user, and be permitted only to that user.

It is just a standard Apache-style htpasswd file, as used by plain old mod_auth. Each line contains a login and an encrypted password separated by a colon. Like this:

The encryption used for the password is standard Unix crypt(). If an account has been invalidated, then the encrypted password is prefixed with a '*' character. If it has not yet been validated, the it is prefixed with a '?' character. The presence of such a character prevents logins to that account until it is removed.

9.2.2. Flat Text Identification Files

The text identification file is used if you configure Backtalk with the --ident=text. It usually supplements a flat text authentication file, but could in theory be used with any form of authentication database. It usually resides in the Backtalk data directory with the name etc/bt_user.

The file contains one line for each user, formatted like so:

For example:
  jan:*:7:1:Jan Wolter:/usr/local/backtalk/user/j/a/

9.2.3. Flat Text Authentication/Identification Files

If Backtalk is configure with --auth=text and --ident=authfile then a single flat text file is used for both authentication and identification. The lines of this file look just like those in section 9.2.2, but include the encrypted password instead of the star in the second field:
  jan:wMFXw97JJrmr2:7:1:Jan Wolter:/usr/local/backtalk/user/j/a/jan
This file would be stored in passwd/bt_passwd and would be normally owned by the user Apache runs as, and would be readable only to that user. Passwords are flagged with status as described in section 9.2.1.

Apache's mod_auth can do authentication out of such a file - it will ignore any information after the password. Having a single file doesn't actually gain you much though. It still needs to be read twice - once by Apache while authentication, the by Backtalk to get the rest of the user data.

9.2.4. Hashed Authentication Files

Backtalk can be built with any of several hash libraries, including the old dbm library, ndbm, Gnu's gdbm or Sleepy Cat's Berkeley db libraries of various vintages. Authentication files built with ndbm or Gnu or Sleepy Cat's emulation of ndbm can be used with Apache's mod_auth_dbm. Backtalk can be configured to use such a file with the --auth=dbm option. Authentication files hashed with a version of Berkeley db can be used by Apache's mod_auth_db. Backtalk can be configured to use such a file with the --auth=db option.

The file will reside in the passwd subdirectory, and it's name will start with bt_passwd. The name may have various different suffixes depending on the hash library used. It will normally be readable only by the Apache account.

For these files the key is the user's login name (not including the terminating nul). The values are the encrpyted passwords (including the terminating nul), possibly with a '*' or '?' prefix as described in section 9.2.1.

9.2.5. Hashed Identification Files

Identification information can be stored in second hash file if Backtalk is configured with the --ident=hash option. It is usually used to supplement a hashed authentication file. It is stored in the etc subdirectory and the name starts with bt_user but may have various suffixes depending on the hash library used.

The key is user's login name (not including the terminating nul). The content will be a string formated as below, terminated by both a newline and a nul:

For example:
  *:7:1:Jan Wolter:/usr/local/backtalk/user/j/a/

9.2.6. Hashed Authentication/Identification Files

If Backtalk is built with --auth=dbm or --auth=db and with --ident=authfile, then a single hash file maintained. It's name, location and permissions are just like those of a hashed authentication file. The data is formatted like a hashed identification file, except that the star at the start of the string is replaced by the user's encrypted password, possibly prefixed by a '*' or '?'.

9.2.7. id Identification Tag File

Identfication tag files are used if Backtalk is built with --ident=tagfile. This is a common option with either flat or hashed authentication files. With this option, the user's directory path is not stored at all. It is contructed from the login name. There is a separate identification tagfile for each user. It resides in the user's directory and is named 'id'. It uses the standard tagfile format described in section 0. A typical example looks like this:
  fullname={Jan Wolter}
Note that the uid and gid tags are flagged with a star to indicate that they can only be editted by the conference administrator. The full name tag is not so flagged, because it is OK for the user to change it. If the email address was not validated, there would be a tag like:
The fullname, email, valcode, privacy and eprivacy field are all optional, and will not be included if they are not defined.

9.2.8. Yapp-Style passwd Identification Tag File

For compatibility with web versions of Yapp, Backtalk can be built with the --ident=yapp option. The user's uid and gid numbers are stored in a tagfile named 'id' and similar in format to the one in the previous section, except that it does not contain the fullname or email tags. For example:
A file named 'passwd' also resides in the user's home directory, containing one line:
For example:
The passwd file is sharable witth Yapp. The id file is unique to Backtalk.

9.2.9. SQL Authentication and Identification Databases

Backtalk can also keep authentication and identification information in SQL database tables. The table formats are defined by the config files in the sql directory of the backtalk configuration. They are slightly different for different SQL servers, and can be editted to be different on different sites.

Maybe someday I'll document them here, but in the mean time, I suggest inspecting the appropriate SQL config file.

10. Dump Files

The Backtalk distribution includes some tools to dump and restore user databases. Some day similar tools may be created to dump and restore the conference database. The dump files produced are supposed to look pretty much the same no matter how Backtalk is configured. They can be used in a variety of ways, including:
  1. When upgrading to a version of Backtalk whose user database formats are not compatible with previous version, you can dump the user database with the old version of Backtalk, and then restore them with the new version.

  2. When converting the Backtalk user database from one configuration to another, say from a flat file database to an SQL database, you can make a dump with the old version and restore it with the new version.

  3. Dumps can be used to make backups of all the users, or just of selected users that are about to be deleted.

  4. If bugs are encountered relating to the user database, a bug report might include a dump of the user who was having the problem. That user could then be restored on a test site to try to duplicate the problem.

Dump files are a mixture of Backtalk tag file syntax and XML syntax. The basic framework is vaguely XML-like, but most of the data is tag-file-like.

10.1. User Dump Files

A user dump contains data describing collections of Backtalk users. It might include all users on the system, or just a few.

10.1.1. Sample

A typical dump file looks like this:
    <userdata date="1105117836" version="1.3.17">
    <user login="cfadm">
    usrname=(Conference Administrator)
    <participation conf="general">
    alias=(Conference Administrator)
    <item number="1">
    lastdate=(Tue, 23-Nov-2004 06:20:29 GMT)
    <user login="jan">
    usrname=(Jan Wolter)
    <participation conf="general">
    alias=(Jan Wolter)
    <item number="1">
    lastdate=(Wed, 24-Nov-2004 02:47:48 GMT)

10.1.2. Description

The file resembles XML, but isn't. The first difference is that each line contains either an XML-ish element, starting with a "<" character, or tagfile style data lines, never a mixture. Second, in an XML file any '<', or '&' characters in a character data section would be converted to '&lt;' or '&amp;'. They aren't in dump files. Probably dump files should adhere more closely to standard XML so that standard XML parsing code code be used to read them, but this would require extra complexity in Backtalk, and I don't really believe that any programmer other than me is ever going to write code to read these things.

So the top level element is '<userdata>'. This has two optional attributes:

The date the file was generated. All dates in the dump file are Unix-style time integers (that is seconds since 00:00:00 UTC, January 1, 1970)
The Backtalk version that produced this dump.
The <userdata> element can contain zero or more <user> elements.

Each <user> element describes one user. It has one mandatory attribute:

The login ID of the user.
The <user> element can contain several different kinds of subelements: zero or one <data> element, zero or more <file> elements, and zero or more <participation> elements.

The <data> element has no attributes and contains tag file formatted data about the user. The user tag data may contain the following standard tags:

usrnameUser's full name
usridUser's uid number
grpnameUser's primary group name
grpsComma-separated list of user's secondary group names
encpasswdUser's encrpyted password
usrstat0=validate, 1=unvalidate, 2=invalidated
dirpathPath of home directory
emailUser's email address
evalidValidation string if email hasn't been validated. Empty if it has.
eprivacyUser's email privacy setting, from 0 (most private) to 4 (least private)
privacy0 if other users may see info, 1 if not
There may be (and probably are) lots of other tags that come out of the user's .backtalk file.
contact page Tue Dec 14 10:44:09 EST 2004