Backtalk File Formats

Table of Contents

Introduction

Backtalk is designed to be able to share its conference database with Picospan or Yapp. Picospan is a conferencing program designed by Marcus Watts and distributed by NETI. It was the original software used by M-Net and The Well. It is still used today on Grex, but is no longer available due to the collapse of NETI. Yapp is a clone of Picospan written by David Thaler. Yapp 2.3 source is available as shareware. Yapp 3.0 is an extended version including a web interface. Both use file formats very similar to, but not completely compatible with Picospan.

Backtalk can be built to coexist with Picospan, Yapp, or neither. The file formats it uses will be slightly different in each case.

0. Tag Files

Filename	Location	Description
.backtalk	User's home directory	User's personal settings
id	User's home directory	User's login name, uid number, and gid number. Used only in configurations built with --ident=tagfile.
settings	Conference directory	Settings for a conference.

0.1. Sample

   allowhtml=1
   allowgraphics=1
   login=(<CENTER>\n<TABLE BORDER=4 CELLSPACING=4 CELLPADDING=8>\n<TR><TD ALIGN=CEN
   TER>\n&nbsp;\n<BR>\n<FONT SIZE=+1>Welcome to the</FONT><BR>\n<FONT SIZE=+3>TEST<
   /FONT><BR>\n<FONT SIZE=+1>Conferencce</FONT><P>\n\(Fairwitness:  Marcus Watts\)<
   BR>\n&nbsp;\n</TD></TR></TABLE>\n</CENTER>)
   new_isel=(1,3,5,10,11,$)
   new_since=7
   bgcolor=(00FFFF)
   regbutton=(bgn)
   bgimage=()
   highcolor=(FF1493)

0.2. Description

In some tagfiles some tag names will be prefixed with a '*' character. This indicates that the field can only be editted by a conference administrator.

1. List of Conferences

Yapp and Picospan use slightly different formats here. Backtalk uses the Picospan format, unless it was configured for Yapp compatibility at compile time.

1.1. Yapp-style Conflist File

    !<hl01>
    /usr/local/backtalk/bbs/general2
    gen_eral2:/usr/local/backtalk/bbs/general2
    general1:/usr/local/backtalk/bbs/general1
    oldgen_eral:/usr/local/backtalk/bbs/general1
    tes_t1:/usr/local/backtalk/bbs/test1
    mar_tian1:/usr/local/backtalk/bbs/martian1
    mars:/usr/local/backtalk/bbs/martian1
    bar_soom:/usr/local/backtalk/bbs/martian1

1.2. Picospan-style Conflist File

    !<hl01>
    %general2
    gen_eral2:%general2
    general1:%general1
    oldgen_eral:%general1
    tes_t1:%test1
    mar_tian1:%martian1
    mars:%martian1
    bar_soom:%martian1

2. Menu of Conferences

When Picospan or command-line Yapp are given a command to list all conferences, they simply print the public.txt file. This is a file with no fixed format that lives in the conference directory.

Backtalk needs a more formally formatted list of conferences that it can read, parse apart, and display in different formats in different interfaces. So Backtalk defines a new file called "confmenu" which lives in the conference directory.

In places where Backtalk co-exists with Picospan or command-line Yapp, Backtalk can be configured to use the traditional "public.txt" file instead, but it requires that it be entered in a consistant format so that it can be parsed. However, what that format should be can be configured. You can write your own regular expressions to parse a fairly broad range of file formats. Two sets of regular expressions are pre-written, one for a Grex-style public.txt file, one for an M-Net-style public.txt file. Those will be described here.

Web versions of Yapp seem to have a different conference menu database. I haven't figured out what exactly this is.

2.1. Backtalk confmenu File

2.1.1. Sample

:Terrestrial Conferences:Conferences primarily relating to the planet earth.
general:General:Our conference for general discussions
oldgeneral:Old General:Previous version of the general conference
:Interplanetary Conferences:Sponsored by your friends from NASA
mars:Martian:Talk about Mars here
venus:Venusian:Talk about Venus here
:Test Conferences:
test:Test:A place to experiment

2.1.2. Description

You don't need to have any headers in the conference menu.

The first field in each line is the internal name of the conference. (This is blank for header lines). This must match a line in the conflist file. This is normally all lower case with no spaces.

The second field is the name of the conference to be displayed in the conference list. This can be mixed case and can have spaces, but should generally be something quite short. For headers it is the text of the header.

The third field is a longer description of the conference or group of conferences.

Lines starting with # signs are comments and are ignored.

2.2. Grex-style public.txt File

2.2.1. Sample

    @@@@@@@@@@@@@@@@@@@
    @ OUR CONFERENCES @
    @@@@@@@@@@@@@@@@@@@

Terrestrial Conferences:

 general    - Our conference for general discussions
 oldgeneral - Previous version of the general conference

Interplanetary Conferences:

 mars       - Talk about Mars here
 venus      - Talk about Venus here

Test Conferences:

 test       - A place to experiment

    (last modified 12/23/02)

2.2.2. Description

Header lines start in column 0 (no leading spaces) and have a colon in them. The header text is from the start of the line through the colon. You cannot give descriptions.

Conference lines start with exactly one space. After the space is the internal conference name, followed by white spaces and dashes. After this is the description of the conference. The description must all fit on one line. The title of the conference will simply be a capitalized version of the conference name.

2.3. M-Net-style public.txt File

2.3.1. Sample

 You may JOIN any of the following conferences:
 
  ********Terrestrial Conferences*********

general......Our conference for general discussions
oldgeneral...Previous version of the general conference

  *******Interplanetary Conferences********

mars.........Talk about Mars here
venus........Talk about Venus here

  ***********Test Conferences*************

test.........A place to experiment

2.3.2. Description

2.4. Other public.txt Files

All you have to do is provide two regular expressions - one that matches header lines, and one that matches conference lines. They should contain parenthesis that pick out the interesting substrings of the lines - the header text for headers, and the conference name and description for conferences.

This means that (1) headers and conferences must be described in one line, and (2) the conference name must appear before the conference description in conference lines. If this is true for your public.txt file, then Backtalk can probably use it.

3. Conference Configuration File

3.1. Sample

    !<pc02>
    .general.cf
    0
    bob,roger
    0
    General

3.2. Description

Magic Number

Participation File Name

Time Limit

Fairwitness List

Conference Mode

0	all users may read and post [default]	public
4	only users in the ulist or glist may read and post	ulist
5	only users who know the password may read or post	password
6	only users who are in the ulist or glist and know the password may read or post	paranoid
8	all users may read and post (but item files not directly readable)	protected
20*	only users in the ulist or glsit may post / all may read	ulist readonly
21*	only users who know the password may post / all may read	password readonly
22*	only users who are in the ulist or glist and know the password may post / all may read	paranoid readonly

* Note that read-only conferences are not recognized by Picospan, though they are recognized by Yapp. If you are using Backtalk with Picospan, modes 20, 21, and 22 cannot be used.

If Backtalk was compiled to be Yapp-compatible mode, then the modes given in the config file will be ignored if an acl file exists. In non-Yapp compatible installations, the acl file will only be checked if the mode in the config file is 'acl' or a negative number.

E-Mail Address

Conference Title

4. Conference Access Control List

    r +all
    w +registered -f:observer
    c +f:ulist
    a +fwlist

5. Conference Summary File

Because of this, it is one of the few binary files used by Backtalk. It must be rapidly accessible, and it does not have to be easily read or editted by humans. If a file is corrupted, you don't edit it, you delete it. The Backtalk distribution includes two programs, sumcheck and dumpsum that can be mildly useful in reading and repairing sum files, in the unlikely event that you should wish to do so. See man pages in the doc directory for more information.

The sum file contains a 24-byte header record, followed by a sequence of 16-byte item records. The sum file entry for item number N is always stored at offset 24 + 16(N-1).

5.1. Header Record

8 bytes	char	Magic Number	Always "!\n". Identifies file type and version.
4 bytes	int	Checksum	A checksum on the name of the participation file given in the config file.
4 bytes	int	Unknown	Picospan always sets this to 0x00001537. I don't know what it means.
8 bytes	long	Sequence	The value 0x12345678. Used for byte order checking.

    set Checksum = 0;
    for each character Char in the name
       Checksum = (Checksum * 4) XOR Char
    end

Sum files are written using the native byte order of the computer that generated them. So if you have conferences on a shared file system, it is possible that different sum files may be created with different byte orders. We check the Sequence field to determine the byte order of the participation file. If it is not the native byte order, then the program must bytes when reading or writing.

5.2. Item Record

4 bytes	unsigned int	Flags	Item flag bits. A boolean OR of the following:     0x0002     Item is retired. 0x000C Unknown. 0x0030 Item is active. 0x0040 Item is frozen. 0x0080 Item is synchronous. 0x0100 Item is linked.
4 bytes	unsigned int	Response Count	The number of responses on the item, including the item text.
4 bytes	unsigned int	Response Date	Date/time of last response to the item.
4 bytes	unsigned int	Item Date	Date/time item was entered.

The times are standard Unix time integers (seconds since Jan 1, 1970). The response date should always match the 'mtime' value for the item file - the time at which the file was last modified.

The system is less picky about the item date, and it is set to slightly different values in different circumstances. When a new item is entered, it is set to the value the file's mtime had after the item was created. If an item is linked in from another conference, it is set to the time that the link was made. If the sum file ever has to be reconstructed, then it is set to match the time stamp on response zero (the item text). Picospan and Backtalk are not entirely consistant in the way they set this value, but I don't think it matters.

6. Item File

For the most part, Backtalk uses the traditional item file format used by Picospan and Yapp. The main extension is in how HTML and plain-text versions of responses are stored in installations that must be Picospan or Yapp compatible. This is done slightly different for Picospan and Yapp, and will be discussed at the end of this section.

6.1. Tradition Picospan/Yapp Item Files

6.1.1. Sample

    !<ps02>
    ,HOur First Test Item
    ,R0000
    ,U232,janc
    ,AJan Wolter
    ,D3d7a9a89
    ,T
    This is an item entered to test Backtalk.
    This is the item text for that item.
    It is a very good item.
    ,E
    ,R0000
    ,U123,joe
    ,AJoseph Cantata
    ,D3dbdfad5
    ,T
    This is the first response to the very dull item
    that was entered by Jan Wolter.  This response too
    is very dull.
    ,E
    ,R0000
    ,U232,janc
    ,AJan Wolter
    ,D3dbdfd2f
    ,T
    How very dull!  It is very good that this item is
    so very dull.
    ,E

6.1.2. Description

The second line is normally the item title, identified by a ",H" prefix.

The rest of the item file consists of a sequence of responses. Note that, the item text is simply the first response.

Each response consists of a ",R line that contains various flags for the response, a ",U" line that contains the login name and UID number of the author, a ",A" line that contains the author's full name, a ",D" line that contains the date of the response in hexadecimal, and then the text of the response, enclosed by ",T" and ",E" lines.

Any lines in the text that start with a comma are stored with two initial commas in a Picospan item file, or three initial commas in a Yapp item file. Backtalk follows Picospan in this, unless configured for Yapp compatibility.

Response flags are as follows:

,R0000	Regular response
,R0001	Hidden response
,R0003	Scribbled response

For some reason, it is common for the ",E" tag to be missing. Programs reading item files should tolerate this.

Yapp supports three additional directives. The "parent" link, ",P" usually appears between the ",D" and the ",T" lines:

    ,R0000
    ,U771,kate
    ,AKate Middleton
    ,D3b977d6a
    ,P2
    ,T
    This is a response to response number 2.
    ,E

For editted responses, Backtalk stores an edit date in addition to the post date. This is a second hexadecimal string on the ",D" line.

   ,D3b977d6a 3b978ac1

The item modes are mostly stored in the item files permission bits. Linked items are detected by checking if the item's link count is greater than one. Frozen items have their owner-write permissions turned off. Retired items have their owner-execute permissions turned on. I forget how synchronous items are flagged - group execute, I think.

6.2. HTML Responses and Alternate Texts

When Backtalk is configured for compatibility with Yapp or Picospan, then a plain text version of any HTML response is posted in addition to the HTML response. The plain text response is placed so that Yapp or Picospan will find that, while the HTML text is stashed someplace slighly unusual where Yapp or Picospan will ignore it.

For Picospan, this works like this:

    ,R0010
    ,U232,janc
    ,AJan Wolter
    ,D3b977d6a
    I'm feeling very <STRONG>strong</STRONG>
    and <EM>emphatic</EM> today.
    ,T
    I'm feeling very strong
    and emphatic today.
    ,E

For Yapp, we additionally prefix each line of the HTML version of the response with ",:" characters:

    ,R0010
    ,U232,janc
    ,AJan Wolter
    ,D3b977d6a
    ,:I'm feeling very <STRONG>strong</STRONG>
    ,:and <EM>emphatic</EM> today.
    ,T
    I'm feeling very strong
    and emphatic today.
    ,E

6.3. Attachments

Responses that have attachments include attachment lines, which are simply a ",X" prefix followed by an attachment handle. These appear after the ",A" line and before the ",D" line. More than one my appear. For example:

    ,R0000
    ,U232,janc
    ,AJan Wolter
    ,Xtest-1-8/doug1.gif
    ,Xtest-1-8/doug2.jpg
    ,D3b977d6a
    ,T
    Here are some picture of my uncle doug.
    ,E

6.4. Item IDs

    ,Ixw432lsk3sjdals3

7. Index File

Index files reside in a subdirectory of the conference directory called "indexdir". The index file's name is the item number prefixed with an "@" character.

When Backtalk links an item, the index files will also be linked. If a link is created by Picospan or Yapp, no link is created and the items will likely end up with separate index files.

The format of an index file is simple. It is a sequence of 4-byte integers. The first is the offset of response 0 (the item text), the next is the offset of response 1, and so on. The offsets always point to the start of the ",R" field of the response. So to find response 412 in an item, we'd seek to offset 412*4 in the index file, read the long integer there, and then seek that offset in the item file.

8. Participation File

8.1. Traditional Picospan Participation Files

In most Unix-login configurations, Backtalk changes the group of the files to one it can read/write. In Backtalk-login and Yapp 3.0 compatible Unix-login configurations, the files are owned by the cfadm account.

8.1.1. Sample

    !<pr03>
    Jan Wolter
    1 76 3D1C5899
    2 8 3D1C5A80
    3 3 3D1C5A8A
    4 20 3D1C5B69
    5 18 3D1C5B80
    6 58 3D1C5BA3
    7 52 3D1C5CD8
    8 -9 3D1CEF3F
    9 -30 3D1CEF6A

8.1.2. Description

The second line gives the user's alias in this conference.

The remaining lines describe items that the user has read. Each contains three columns.

• The item number.
• The number of first unread response. (Response 0 is the item text.)
• The date/time when the item was last read.

The number of the last response read may be prefixed with a minus sign. This means that the item has been forgotten. Picospan recognizes "-0" as a distinct value from "0" - the former is an item that has never been read, but has been forgotten. The later is an un-read, un-forgotten item. Yapp does not understand "-0" and does not allow unread items to be forgotten.

The time of last reading is represented as a standard unix time value (the number of seconds since the begining of 1970), written in hexidecimal notation.

The time of last reading is used to distinguish between new and unseen items. An item has unseen responses if the number of responses in the item exceeds the last response read. An item has new responses if it is unseen AND the date of the last response in the item exceeds the date on which we last read responses.

8.2. Backtalk Participation Files

These files are named the same as a Picospan file would have been, and are stored in the same locations, except that Backtalk accounts don't usually have .cfdir directories.

8.2.1. Sample

    !<pr10>
    alias=(Jan Wolter)
    pistacho.background=(C0FFE0)
    1
    r=(76)
    d=1025267865
    2
    r=(8)
    d=1025267352
    3
    r=(3)
    d=1025268362
    4
    r=(20)
    d=1025268585
    fav
    5
    r=(4-9,11,18)
    d=1025268608
    6
    r=(58)
    d=1025268643
    7
    r=(52)
    d=1025268952
    fav
    8
    r=(9)
    d=1025306431
    forgot
    9
    r=(30)
    d=1025306474
    forgot

8.2.2. Description

The tag syntax is actually slightly different than regular tag files, for no good reason. A tag without a value (like "fav") is equivalent to setting it to integer 1 (that is, "fav=1"). If a string contains newlines, actual newlines are put in instead of "\n" values. Oh well.

In the conference-wide section, the 'alias' tag defines the user's name in the conferences.

In the per-item sections, the 'r' field is a selector describing all unread responses in the item. There is an implied '-$' at the end of each, so 'r=(9)' means we have not read responses 9-$. Currently these are always single digits, because, like Picospan, if we have read response 109, then we assume that we have read all that went before. But to support tree-style items, we need to remember exactly the set of responses which are still unread, so these will become more general selectors. If this is a single number (as it usually is), it can be written out as a number instead of a string, without the parentheses.

The d field is the date the item was last read. This is exactly like the date field in the traditional picospan participation file, except that it is written in decimal instead of hexadecimal.

Forgotten items are marked by a 'forgot' tag.

Extensions to the standard Picospan format include the 'fav' tag which is included if the item is one of the user's favorites, and the title flag which gives the user's personal title for the item.

8.3. Backtalk Participation Extension Files

So we maintain our extra information in an extra file, called the participation extension file. It's name is the same as the participation file, but with an 'x' appended to the name.

8.3.1. Sample

    !<px01>
    pistacho.background=(C0FFE0)
    4
    fav
    5
    r=(4-9,11,18)
    7
    fav

8.3.2. Description

9. Authentication/Identification Databases

encrypted password	authentication
account validation status	authentication
uid number	identification
gid number	identification
full name	identification
home directory	identification
privacy flag	identification or .backtalk
email address	identifcation or .backtalk
email privacy flag	identifcation or .backtalk
email validation status	identifcation or .backtalk
anything else	.backtalk

Account validation status takes three possible values: validated, unvalidated, and invalidated. Unvalidated accounts exist only if Backtalk is configured to require new user accounts to be manually validated by an administrator before they can log in. Invalidated accounts are formerly active accounts that have been "turned off" by an administrator for whatever reason. It is sometimes stored as a numeric value (0=validated, 1=unvalidated, 2=invalidated) and sometimes as a prefix code on the encrypted password.

The privacy flag is 1 if it is OK to display user information to other users. Passwords, of course, are never displayed. If it is 0, this information is to be kept secret.

If email privacy flag controls the privacy of the user's email. If 0, only the user and administrators can see it. If 1, it is still secret, but users can use Backtalk's mailer to send mail to the address. If 2, any logged in user may see it, but others cannot. If 3, any user may see it, but it is displayed as a series of gifs and not made clickable when anonymous users try to see it. If 4, it may be seen by anyone.

Email validation codes are used to validate email addresses. If Backtalk is configured to validate email, then whenever a new account is created, or an old account changes it's email address, a validation code is assigned. A copy of this value is mailed to the new address. Users must click on a link in the email, sending back the code, to confirm their email addresses. So, if the email address has been validated (or if Backtalk is not configured for email validation) then the email validation string is an empty string. If the email address is not validated, it is the validation code.

9.1. Unix Logins

9.2. Backtalk Logins

9.2.1. Flat Text Authentication Files

It is just a standard Apache-style htpasswd file, as used by plain old mod_auth. Each line contains a login and an encrypted password separated by a colon. Like this:

   cfadm:7jspL5OIt9v1Y
   jan:wMFXw97JJrmr2

9.2.2. Flat Text Identification Files

The file contains one line for each user, formatted like so:

  login:*:uid:gid:fullname:directory:email:valcode:privacy:eprivacy

  jan:*:7:1:Jan Wolter:/usr/local/backtalk/user/j/a/jan:jan@unixpapa.com::0:2

9.2.3. Flat Text Authentication/Identification Files

  jan:wMFXw97JJrmr2:7:1:Jan Wolter:/usr/local/backtalk/user/j/a/jan

Apache's mod_auth can do authentication out of such a file - it will ignore any information after the password. Having a single file doesn't actually gain you much though. It still needs to be read twice - once by Apache while authentication, the by Backtalk to get the rest of the user data.

9.2.4. Hashed Authentication Files

The file will reside in the passwd subdirectory, and it's name will start with bt_passwd. The name may have various different suffixes depending on the hash library used. It will normally be readable only by the Apache account.

For these files the key is the user's login name (not including the terminating nul). The values are the encrpyted passwords (including the terminating nul), possibly with a '*' or '?' prefix as described in section 9.2.1.

9.2.5. Hashed Identification Files

The key is user's login name (not including the terminating nul). The content will be a string formated as below, terminated by both a newline and a nul:

  *:uid:gid:fullname:directory:email:valcode:privacy:eprivacy

For example:
  *:7:1:Jan Wolter:/usr/local/backtalk/user/j/a/jan:jan@unixpapa.com::0:2

  *:uid:gid:fullname:directory

9.2.6. Hashed Authentication/Identification Files

9.2.7. id Identification Tag File

  *uid=7
  *gid=1
  fullname={Jan Wolter}
  email=(jan@unixpapa.com)
  privacy=0
  eprivacy=2

  valcode=(Jd8lLKSs9LKs23KDLKS0323lLSDKFJ)

9.2.8. Yapp-Style passwd Identification Tag File

  *uid=7
  *gid=1
  valcode=()
  privacy=0
  eprivacy=2

  login:fullname:email

  jan:Jan Wolter:jan@unixpapa.com

9.2.9. SQL Authentication and Identification Databases

Maybe someday I'll document them here, but in the mean time, I suggest inspecting the appropriate SQL config file.

10. Dump Files

1. When upgrading to a version of Backtalk whose user database formats are not compatible with previous version, you can dump the user database with the old version of Backtalk, and then restore them with the new version.

2. When converting the Backtalk user database from one configuration to another, say from a flat file database to an SQL database, you can make a dump with the old version and restore it with the new version.

3. Dumps can be used to make backups of all the users, or just of selected users that are about to be deleted.

4. If bugs are encountered relating to the user database, a bug report might include a dump of the user who was having the problem. That user could then be restored on a test site to try to duplicate the problem.

Dump files are a mixture of Backtalk tag file syntax and XML syntax. The basic framework is vaguely XML-like, but most of the data is tag-file-like.

10.1. User Dump Files

10.1.1. Sample

    <userdata date="1105117836" version="1.3.17">
    <user login="cfadm">
    <data>
    usrname=(Conference Administrator)
    usrid=1
    grpid=0
    dirpath=(/home/jan/btd/tt1.3/user/cfadm)
    regdate=1100014718
    </data>
    <participation conf="general">
    <data>
    alias=(Conference Administrator)
    </data>
    <item number="1">
    lastresp=5
    lastdate=(Tue, 23-Nov-2004 06:20:29 GMT)
    favorite=1
    </item>
    </participation>
    </user>
    <user login="jan">
    <data>
    usrname=(Jan Wolter)
    usrid=2
    grpid=1
    dirpath=(/home/jan/btd/tt1.3/user/jan)
    regdate=1100019967
    privacy=(0)
    email=(jan@unixpapa.com)
    sex=(M)
    frames=0
    rpage=30
    rtype=(text/plain)
    confsummary=0
    dfltilist=(new)
    rfav=2
    ifav=1
    </data>
    <participation conf="general">
    <data>
    alias=(Jan Wolter)
    </data>
    <item number="1">
    lastresp=16
    lastdate=(Wed, 24-Nov-2004 02:47:48 GMT)
    </item>
    </participation>
    </user>
    </userdata>

10.1.2. Description

So the top level element is '<userdata>'. This has two optional attributes:

date

The date the file was generated. All dates in the dump file are Unix-style time integers (that is seconds since 00:00:00 UTC, January 1, 1970)

version

The Backtalk version that produced this dump.

Each <user> element describes one user. It has one mandatory attribute:

login

The login ID of the user.

The <data> element has no attributes and contains tag file formatted data about the user. The user tag data may contain the following standard tags:

usrname	User's full name
usrid	User's uid number
grpname	User's primary group name
grps	Comma-separated list of user's secondary group names
encpasswd	User's encrpyted password
usrstat	0=validate, 1=unvalidate, 2=invalidated
dirpath	Path of home directory
email	User's email address
evalid	Validation string if email hasn't been validated. Empty if it has.
eprivacy	User's email privacy setting, from 0 (most private) to 4 (least private)
privacy	0 if other users may see info, 1 if not