Adding or changing data on the Image server

It's straightforward, really. And it's not hard at all.

Normally, the image server just mirrors the data contained on an author's local Macintosh. FrameBuilder has a feature called "Generate Transfer List" which produces a list of all the files that need to be transferred, and you can just turn file sharing & FTPd on for the Mac containing the data, log into the Unix box where the dataset is located, and use the transfer list with ftp to get the files. Easy as pie if the directories are already there, and there is a script that handles even this for a neophyte user. It's called "browser-update", and can be found in any of the data areas on the image server. Directions for using it are found in the file "F509 protocols" on the F509 Mac, a hardcopy of which is in the blue notebook in that room.

Why not just use Fetch? Or Tar?

Both of those would work, except they change spaces to underscores in batch transfers -- a really nasty thing for them to do, but hey, they're free. If you could tell the authors never to use a space character, Fetch would probably be the best way. (Fetch 3.0 is promising in that it doesn't do this replacement, but it freezes occasionally so I'm leery of using it for production jobs).

What if I only want to change a few files?

Fetch is the way to do it, but YOU MUST NOT MAKE ANY MISTAKES (of course). Don't let it change the filenames on you, and make sure you use "raw data" for the transfer mode on both frame files and image files; "text" or "MacBinary" will NOT work. (However, you do need to use MacBinary to transfer QuickTime movie files). And be quite sure that you test things after you're done, to make sure you did it all right.

Skills I use for this task:

- the use of Telnet and Fetch to do the periodic work
- shell script programming to alter & maintain the "browser-update" system
- knowledge of what kinds of files go where, without resorting to black magic

What if I only want to LOOK at the files that are there?

As of Jan. '96, you can log in on avalo, go to /usr/local/data2/browser, and look around. That's where all the data files are. In there, for instance, are an "Index" and a "Scans" folder, plus several specific folders for the Neuroanatomy, Thorax, and Knee datasets -- each has its own respective "Index" and "Scans" folder inside. All the image files in the Scans folders are normally PICT files, of course, in raw binary format; you can't look at them without downloading them to a Mac. (If you want to do so, save them with a file type of "PICT" and a creator code of either "ttxt" or "8BIM", and they'll open in TeachText/SimpleText or Photoshop).

To look at any of the frame files in the Index folders is easier:

   unix% cd /usr/local/data2/browser/Index
   unix% tr '\015' '\012' <"Open.Thorax" | more

The above command sequence gets you to the right location and then pipes one of the top-level frame files to the "more" utility so you can page through it; but since it's a Mac file with carriage returns on each line, it has to be translated first to map those to Unix linefeeds so it will look sensible. The "tr" utility does this.

Armed with the above information, plus knowledge of the frame file format, you can really dig through things and find out some interesting stuff. For example, you can concatenate all the frame files for a particular dataset together with the "cat" utility, do the CR-->LF mapping with "tr", extract just the structure names with "grep", sort them, and eliminate duplicates with "uniq"; this gives an alphabetical list of all the structure names used anywhere in the dataset, which is very useful in checking for misspellings and stray spaces. Here's the magic command (do it from a NeXT if possible, because an SGI has trouble with large groups of wildcarded files and will probably fail -- to avoid the error you'll have to do groups of files like with [a-k]*/* and [l-z]*/* and then join the results.)

unix% cd /usr/local/data2/browser/Neuroanatomy/Index
unix% cat */* */*/*|tr '\015' '\012'|fgrep STRUCTURE|sort|uniq >~/results

Finally, there's a similar sequence of operations that will result in a list of frame names and their individual structure names; Dr. Sundsten found this useful and asked me to include it somewhere so he could obtain such a list in the future. So here it is:

Go to your home directory on a Biostr Unix machine, preferably a NeXT:
    cd

Create a file with the following contents (call the file "d"):
echo "FRAME:" $1
cat "$1" |tr '\015' '\012'|fgrep STRUCTURE|sort|uniq
echo
echo

Make the file executable:
    chmod +x d

Go collect the names of all the files to process:
    cd /usr/local/data2/browser/Neuroanatomy/Index
    ls */* */*/* >~/neuro.dat

Modify the list, removing files you don't want (like those in the SUBJECTS
directory) and add the call to the "d" script on each line:
    vi ~/neuro.dat
      :g/^/s/^/\~\/d "/g
      :g/$/s/$/"/g
      :wq

The above editing session leaves you with a file that contains lines like:
	~/d "3D Object Composites/Accumbens Caudate Putamen"
	~/d "3D Object Composites/All Objects"
	~/d "3D Object Composites/Amygdala Caudate"

Execute the script with:
    sh ~/neuro.dat >~/neuro.out