KISSFILT 1.3 - A Tool for Filtering the Contents of KISS Log Files rev. February 17, 1992 rev. March 8, 2002 (contact info only) Copyright 1991,1992 Paul Williamson, KB5MU You may use this program for non-commercial purposes, and you may distribute complete copies freely. KISSFILT allows you to remove unwanted packets from your KISS log files. For example, if you copy a pass of AMSAT-OSCAR 16 using the broadcast protocol receiver program PB, you end up with a KISS log file containing every frame received during the pass (other than the broadcast data frames themselves). This file will contain telemetry from the satellite, data frames sent from the satellite to other users, acknowledgements sent by the satellite to other users, and various other stuff. Suppose you want to archive telemetry from AO-16. If you store the KISS log file PB created, you will be storing lots of useless stuff. Unless you use KISSFILT first. KISSFILT provides several types of filtering: 1. Filter by source and destination callsigns 2. Filter out all non-information frames 3. Filter by Protocol ID (pid) 4. Filter by Pacsat Broadcast Protocol file number The callsign filter is by far the most complex and powerful of these features. Most of this document will be devoted to an explanation of how to use the callsign filter. KISSFILT is invoked from the DOS command line. You provide the input and output file names on the command line, as follows: C:> kissfilt filename.kss filename.out If that's all you provide, KISSFILT will drop into an interactive mode. It will scan through the input file, frame by frame, interpreting the KISS format. For each frame, it extracts the source and destination callsigns. For each new pair of callsigns, KISSFILT asks you whether you want to Keep or Discard frames from that source to that destination. You respond by hitting the K key for keep or the D key for discard. You may also hit the Q key for quit, which will end the program without processing that frame or any later frame. If you say Keep, KISSFILT copies the frame to the output file. If you say Discard, it just goes on to the next frame without copying the current frame to the output file. When you tell KISSFILT what to do with a frame, it remembers your answer and automatically does the same thing with all other frames from that source to that destination. As you might imagine, answering all those prompts can get quite tedious, especially if you know exactly what frames you want to keep and which you want to discard before you start. In that case, you can put your decisions in a "filter file". You specify the filter file on the DOS command line after the output file: C:> kissfilt filename.kss filename.out filtfile.ext The filter file can contain various commands. Each command must be on a separate line. Blank lines are OK, and you can put comments in the file by starting the line with a semicolon (";"). The following commands are accepted: Command Format Meaning ============== ======= keep call1 call2 Keep frames from call1 to call2 discard call1 call2 Discard frames from call1 to call2 ask call1 call2 Ask about frames from call1 to call2 else keep Keep any frames not matched else discard Discard any frames not matched else ask Ask about any frames not matched info-only Discard any non-information frames pid pid1 pid2 ... Accept frames with PID = pid1 or pid2 or ... pbpfile file_number Accept PBP frames for file # file_number In the Keep, Discard, and Ask commands, the callsign fields may simply contain callsigns, or they may contain what's called a "regular expression" specifying a pattern for what callsigns are to be matched. Regular expressions are like wildcards, only more powerful. For this application, they're probably more powerful than necessary, but I had a regular expression matching routine (written by Henry Spencer) handy, so I threw it in. Henry Spencer's description of regular expressions in included as RE.DOC in the archive. Unfortunately, it's written for computer scientists; if you're not familiar with formal languages, you may have trouble understanding his description. I'll just describe a simple subset of the language, and that'll probably be all you need. Rule 1: If you don't want to mess with fancy pattern matching, you can just ignore it. Just put callsigns in the callsign fields. Examples: LUSAT-1 matches LUSAT-1 (and nothing else) KB5MU matches KB5MU (and nothing else) KB5MU-0 matches nothing, since KB5MU-0 is KB5MU Rule 2: A dot ('.') matches any single character. It works like a question mark in DOS. Examples: KB.MU matches KB5MU, KB1MU, KBxMU, etc. ... matches ABC, XYZ, 4U1, etc. Rule 3: Anything followed by an asterisk ('*') matches any number of that thing, including zero. Examples: KB*MU matches KMU, KBMU, KBBMU, KBBBMU, etc. K.*MU matches KMU, KxMU, KxyMU, KxyzMU, etc. That last examples is particularly important. The sequence ".*" means "zero or more of any character". That description matches any string. It's like the way "*" works in DOS. Let's look at more examples of that technique: .* matches absolutely anything K.* matches anything that starts with K KB5MU-.* matches KB5MU-1, KB5MU-15, etc. .*5.* matches anything with a 5 in it. .*-.* matches anything with a nonzero SSID That's all you need to know about regular expressions. If you want to experiment with the more powerful expression matching capabilities of KISSFILT, consult RE.DOC. Let's take a look at an example filter file: keep PACSAT-1 TLM keep PACSAT-1 TIME ask PACSAT-1 .* else discard This is a typical filter file you might use to reduce the size of a telemetry archive from AO-16. The first line, "keep PACSAT-1 TLM" says that KISSFILT is to keep any frames addressed from PACSAT-1 to TLM. These are the telemetry frames you want. The second line, "keep PACSAT-1 TIME" says that you also want to keep the time frames. The third line, "ask PACSAT-1 .*" matches any frame addressed from PACSAT-1 to any address. It tells KISSFILT that you want it to ask you about any frames from PACSAT-1. Now frames from PACSAT-1 to TLM or TIME are already kept, according to lines 1 and 2. The earlier line rules, so line 3 actually matches any frame addressed from PACSAT-1 to any address *other than* the ones already specified. The last line, "else discard", tells KISSFILT that you aren't interested in any frames that don't match any of the other lines. In this case, any packet with a source address other than PACSAT-1 will be discarded. For another example, suppose you don't know what callsigns are in the file. You could run a program to translate the KISS file into an ASCII file (my program KISS2ASC, for instance), then go through the file with a text editor, and make up a filter file containing the instructions. Or, you could let KISSFILT do the work by using the following filter file: else ask That one-line filter file says that you want KISSFILT to ask you about every source+destination callsign pair it finds in the file. Even simpler, just don't use a filter file at all, since this is KISSFILT's default behavior. For yet another example, suppose you want to see just the information packets typed by users. You don't want to see the packets sent by the satellite, and you don't care about the acknowledgement and other overhead packets sent by the users either. This filter file does just that: discard PACSAT.* .* else keep info-only That last line, "info-only", is the key to eliminating the acknowledgement and other link overhead packets. With info-only specified, only frames with type I (information, used by connected stations), UI (unnumbered information, used for broadcasts, CQs, and roundtables), and FRMR (frame reject, which contains reject-reason information) are copied into the output file. All other frames are discarded. (You have to use a filter file to use info-only, pid, or pbpfile.) The protocol ID filter feature is almost as easy to use as info-only. If you are only interested in keeping frames with certain protocol ID's, just list them in a pid command in a filter file. You can have several pid commands. Any frame in the file with a PID not mentioned in any pid command in the filter file will be automatically discarded. Frames with a PID mentioned in one of the pid commands follow the usual rules. For example, if you want to keep frames with protocol ID 0xFF and discard all others, use the following filter file: pid ff else keep Here the "else keep" action overrides the default "else ask" action, and tells KISSFILT to automatically keep all frames (after filtering by PID). If you want to keep all frames BUT those with a certain PID, you'll have to use brute force. List all other PIDs on pid command lines. By the way, don't ask me why you'd want to filter on protocol ID. It was a user requested feature. On a similar note, KISSFILT now allows you to filter for Pacsat Broadcast Protocol frames from a particular file. You may name only one file number, in hexadecimal, on a pbpfile command line. To pass the PBP file filter, a frame must be a UI (unnumbered information) frame with PID 0xBB and a destination address of QST-1, per the Pacsat Broadcast Protocol specification. Frames that pass the PBP file filter are still subject to callsign filtering and/or default actions. For example, if you want to keep frames from PBP file 0x262b, use the following filter file: pbpfile 262b else keep Here the "else keep" action overrides the default "else ask" action, and tells KISSFILT to automatically keep all frames (after filtering by PBP file number). To help you figure out what file numbers to put on that pbpfile command line, I've included another utility program called PBPFILES. Run this program with the KISS logfile name on the command line, and it will tell you which PBP files have frames in that file. For example, PBPFILES 02171234.16R =================================================================== This release of KISSFILT probably has bugs. If you find any bugs, please let me know. If you have suggestions for improvements, please let me know. If you find KISSFILT useful, it wouldn't hurt to let me know that, too. Source code is available on request. KISSFILT is written in C, for the Microsoft version 6.00a compiler. There are only a couple of compiler specific things in the code, so it should be quite easy to compile KISSFILT on other compilers or for other (command-line oriented) machines. The regular expression matching routines were written by Henry Spencer, and are Copyright (c) 1986 by University of Toronto. He has granted to anyone permission to use the routines, and for that I thank him. The rest of KISSFILT is either derived from my earlier program KISS2ASC, or hacked up in a couple of late nights. Likewise PBPFILES. Thanks to Kazu Sakamoto for suggesting new features for KISSFILT 1.3. I can be contacted by email via: kb5mu@amsat.org 73 -Paul Williamson, KB5MU