On Sun, Apr 24, 2005 at 06:48:07PM -0400, B. Bogart wrote:

Hi IX!

Hmmm, I figure specs are a good way to descibe a possible solution that

well hardly a spec, i should have looked to see how concise it was first..

Where does your grep search pick up "hanning-window"? From the name of the array, from the comments? Indeed scanning for "keyword" would match

my point exactly..between the unlikeliness of retrofitting all the help files, and no idea where it will be...sometimes its a #X text, other times its the name of a graph, or a message - i guess in the indexer would give KEYWORD: annotations a higher relevance in search results...

I think right now the help menu is NOT dynamic, only the submenus (5.reference) are actually dyanamic. I should put a note in the wiki that the menu should be dynamically generated... (though this would be interesting when the -help files are centralized (in extra for example)

should the auto-generated doc menu be scanning extra/ as well as help/ then? extra/ seems to be mainly used for abstractions and binary libraries...

carmen

Hans-Christoph Steiner wrote:

I think the auto-generated menu should just recurse thru doc/ and mirror that structure in the menu structure. Its not a great solution, but better that what's there.

why ? in my opinion, the doc/ is pretty much fixed (after packaging).

additional libraries and externals are normally installed into extra/ (or wherever additional "path"s point to) since p>=0.37 help can be installed besides them and will still get found.

these are the real problematic (and interesting) objects to auto-index. (i mean, for the fixed objects in doc/, we can build an offline (this is: packaging-time) index.

mfg.a.dsr IOhannes

This does bring up the issue in the deployment plan for such "help" menu features. I figure we should have consensus on a plan for what to do and then figure out what steps in what order to develop it slowly.

As a quick and easy point in the right direction I think HC's idea is fine in the short term. The problem with the idea is that I would argue the doc/ structure leaves a lot to be desired as an interface to documentation for an end user.

If we could reuse the code used to decent into the doc/ tree then it would help, if there is not other use for this in the short term...

Oh there is some tcl code for decending into a directory tree in LML (no longer maintained -

www.ekran.org/ben/research/lml/lml-v0.1-TEST8-linux.tgz

May be useful for something.

Ok, onto other emails in this thread, ug!

I think I'll have to sum up the comments and respond in one big email.

B>

Hans-Christoph Steiner wrote:

On Apr 25, 2005, at 8:00 AM, IOhannes m zmoelnig wrote:

...

Hans-Christoph Steiner wrote:

...

I think the auto-generated menu should just recurse thru doc/ and mirror that structure in the menu structure. Its not a great solution, but better that what's there.

why ? in my opinion, the doc/ is pretty much fixed (after packaging).

additional libraries and externals are normally installed into extra/ (or wherever additional "path"s point to) since p>=0.37 help can be installed besides them and will still get found.

these are the real problematic (and interesting) objects to auto-index. (i mean, for the fixed objects in doc/, we can build an offline (this is: packaging-time) index.

mfg.a.dsr IOhannes

Because its easy, its almost done, and its a temporary hack.

.hc

---

---

"[W]e have invented the technology to eliminate scarcity, but we are deliberately throwing it away to benefit those who profit from scarcity." -John Gilmore

---

---

PD-dev mailing list PD-dev@iem.at http://lists.puredata.info/listinfo/pd-dev

Krzysztof Czaja wrote:

hi Ben,

B. Bogart wrote: ...

...

• spec for the searching of help files.
• organization of pd "help" pull-down menu.
• functional catagories for PD externals/objects.

great you are pushing this. Some random thoughts and questions...

Have you considered other ways of browsing -- choosing something that better scales? Not sure if hierarchical pull-down menu is the right way of browsing thousands of items.

How would the help files structure relate to eventual structuring of class names into a browsable ``object list'' (btw. such list should have its own "open help file" popup)? And how would these relate to eventual structuring of classes themselves into packages, Guenter's namespaces, or anything that would finally be chosen?

yes, i still think that full-object-browsing will lead to information-overdose if we don't limit ourselves to some structuring mechanism, which in turn i think will be problematic (given the different characters of pd-developers)

The structure seems closed. I mean, even though a keyword comment may contain any word that its author finds suitable, only the predefined words would categorize the file for browsing -- is this the case? and if so, why?

i don't see, why the browser system should not be extensible to include the "randomly chosen" keywords an author finds suitable.

probably we should ping bryan, as he might be the one with most knowledge on auto-clustering words.

mfg,.as.dr IOhannes

Its good to dream, and we've come up with a lot of interesting ideas. But I think at this point we need some basic implementation before we can really go into details. A simple hack that parses a pre-defined set of keywords from a specially tagged comment in a help patch would be relatively easy to implement. Then we can test it out, and take it from there.

So... the question remains, who actually wants to do some implementation? I am sure I'll try my hand at this at some point, but I have no idea when.

.hc

On Apr 25, 2005, at 9:15 AM, Bryan Jurish wrote:

morning folks,

Just to add my -b¤ 0.02, I think Ben's idea of an integrated help-file-A structuring and searching mechanism would be quite useful -- I'm currently using a perl script and grep to index my own abstractions along similar lines (topmost comment in the file together with some rather arbitrary string matching conventions which probably only apply for me), but a more comprehensive catalog, especially for builtin objects and externals, would be very helpful indeed.

On 25 April 2005 at 14:05:18, IOhannes m zmoelnig wrote:

...

yes, i still think that full-object-browsing will lead to information-overdose if we don't limit ourselves to some structuring mechanism, which in turn i think will be problematic (given the different characters of pd-developers)

Agreed. Still, I think the option of adding one's own (arbitrary) keywords / structuring conventions is important too -- pre-defined hard categorization schemes have a disturbing tendency to go all goopy at the edges when they're not dictatorially enforced, which I don't think anyone here really wants to do or to be done; maybe the solution is just as simple as differentiating between "browsing" and "searching"?

...

probably we should ping bryan, as he might be the one with most knowledge on auto-clustering words.

Consider me ping'd [icmp_seq=1 ttl=64 time~=3600000 ms] ;-)

As it turns out, I happen currently to be working on an unsupervised word clustering system, although not directly along semantic lines. Still, there are methods for generating full-blown hierarchies and/or "flat" clusters based only on, say, word co-occurrences. There's even a nifty technique to find the most salient dimesions (highest variance) in a feature data space. Problems include (as you might expect): unsupervised clustering doesn't necessarily get you meaningful groupings; and perhaps more importantly from a user standpoint, it doesn't get you meaningful cluster labels (there are ways to work around this too, but they don't solve the basic problem); lastly, auto-clustering is computationally very expensive -- I might try running documentation comments through a co-occurence clustering algorithm, but I don't see any of these techniques becoming really useful at runtime -- at most we could use them to help induce an initial breakdown for existing objects...

marmosets, Bryan

---

PD-dev mailing list PD-dev@iem.at http://lists.puredata.info/listinfo/pd-dev

________________________________________________________________________ ____

Man has survived hitherto because he was too ignorant to know how to realize his wishes. Now that he can realize them, he must either change them, or perish. -William Carlos Williams

Hans-Christoph Steiner wrote: ...

A simple set of keywords would be easiest to implement to start with. Then we can test the basic idea out, and see whether its worthwhile to invest a lot of time coding an dynamic, flexible keyword system.

dynamic is easier... in tcl it could be

while {[gets $ch line] >= 0} { if {[regexp $skipbadentries $line] == 0} { ;# for safety set helpfile [lindex $line 0] foreach c [lrange $line 1 end] { lappend categories($c) $helpfile } } }

which might be fed by

find . -name "*-help.pd" -exec awk \ '/KEYWORDS/ {print FILENAME substr($0,index($0,"KEYWORDS")+9)}' {} ;

Krzysztof

Hans-Christoph Steiner wrote: ...

...

dynamic is easier... in tcl it could be

...

Looks good, except find and awk are not standard on Windows. If we are

neither is "package require fileutils", I am afraid, but copy/pasted code seems to be supported at least...

doing to make a dynamic system, it should be dynamic at run time, not just build time. It should run whenever Pd starts so that user-added patches are also included.

And I don't think we want to start making Pd dependent on find and awk.

sure, that was only for readability... The implementation is quite simple either way. It is the design that is more important: predefined vs dynamic set of categories, storing meta-data in .pd files (imo wrong) or separately, etc.

Krzysztof

Hey Krzysztof,

Thanks for contributing this.

Maybe I'm just too tired but I'm having trouble imagining the structure of the index file. something like:

counter-help.pd control sound

Could put the counter-help in both the control and sound catagories?

Any developer added catagories would be automagically added to the list?

I was thinking that a short description of each entry (help-file) would be very handy to show when searching. Since the name of the object/extra may not be meaningful. So lets say we have DESCRIPTION KEYWORDS CATAGORY in the template. The patch parsing script could: (in sketchy pseudo tcl)

set path /extra/ # all libs and all *-help.pd files should be here. cd $path set help_files [glob *-help.pd] # list -help files. set help_index 0 foreach help_file $help_files { set FILENAME($help_index) $help_file set id [open $help_file r] set help_contents [read $id] close $id

foreach help_line [split $help_contents "\n"] { # loop over lines foreach help_feild [list DESCRIPTION KEYWORDS CATAGORY] if ($help_line contains $help_feild) { # Extract from $help_feild to ";" put in variable value

# Create a different array for each feild set $help_feild($help_index) $value } } }

incr help_index

# generate index file (is this needed if we rebuild on PD start?) }

# The searching could look something like:

# the forloop could be changed for the array searching functions for {set i 0} {$i < [array size KEYWORDS]} {incr i} { if {[string match -nocase $KEYWORDS($i) $keyword_search]} { # create a nice output list of matches $w.listbox insert end "$FILENAME($i)\t$DESCRIPTION($i)" } }

# Generating the help menu could contain something like:

# generate list of unique catagories foreach catagory $catagories { # generate list of help files for this catagory in $w.menu.$catagory $w.menu add cascade -label $catagory -menu $w.menu.$catagory }

Notes: -help files COULD be also PDDP all_about_* patches. Any patches in the correct folder(s) with the right feilds would be searchable.

The code above depends on each -help file containing all three Feilds.

Anyhow its just a sketch of how things could work. I've put this on the wiki: https://puredata.org/Members/bbogart/SimpleImplimentation

I agree with HC that we should only depend on tcl/tk for the searching system.

B.

Hans-Christoph Steiner wrote:

On Apr 27, 2005, at 4:46 AM, Krzysztof Czaja wrote:

...

Hans-Christoph Steiner wrote: ...

...

A simple set of keywords would be easiest to implement to start with. Then we can test the basic idea out, and see whether its worthwhile to invest a lot of time coding an dynamic, flexible keyword system.

dynamic is easier... in tcl it could be

while {[gets $ch line] >= 0} { if {[regexp $skipbadentries $line] == 0} { ;# for safety set helpfile [lindex $line 0] foreach c [lrange $line 1 end] { lappend categories($c) $helpfile } } }

which might be fed by

find . -name "*-help.pd" -exec awk \ '/KEYWORDS/ {print FILENAME substr($0,index($0,"KEYWORDS")+9)}' {} ;

Krzysztof

Looks good, except find and awk are not standard on Windows. If we are doing to make a dynamic system, it should be dynamic at run time, not just build time. It should run whenever Pd starts so that user-added patches are also included.

And I don't think we want to start making Pd dependent on find and awk.

.hc

---

---

"Information wants to be free." -Stewart Brand

Ok,

Here is a sum up of comments made on list, I'm only pulling out the key points and my comments.

ix@replic.net wrote:

my point exactly..between the unlikeliness of retrofitting all the help files, and no idea where it will be...sometimes its a #X text, other times its the name of a graph, or a message - i guess in the indexer would give KEYWORD: annotations a higher relevance in search results...

Consistancy between help files is sorely missed in PD. If we have a standard for help-patches then there is a template that is easily used for new abstractions. Indeed it is quite a chore to convert the old help files, but having a functional browsing and searching system should be reason enough for developers to convert thier own help patches.

Krzysztof Czaja wrote:

Have you considered other ways of browsing -- choosing something that better scales? Not sure if hierarchical pull-down menu is the right way of browsing thousands of items.

This connects to what HC says below about the catagories themselves. Is three catagories with 600 items each better than 9 subfolders with only 200 each? I think it is a scale of granularity, how much browseable resolution vs how much bulk listing. I agree that pull-down menus with 1000s of items is not a great solution (at least jumping with "p" works in pulldowns!!!) Maybe each subfolder does not bring up a list of objects but a new window containing a list of objects, with a little description of each and be able to link from this list to the help file.

Krzysztof Czaja wrote:

How would the help files structure relate to eventual structuring of class names into a browsable ``object list'' (btw. such list should have its own "open help file" popup)? And how would these relate to eventual structuring of classes themselves into packages, Guenter's namespaces, or anything that would finally be chosen?

Damn good question. I think what I'm imaging with the help menu is a kind of browseable object list. This need not be created on object instanciation ala max/impd but perhaps is just one in the same with the help menu. Or the object browser could be a separate floating window, that would deal with the 1000 items more easily. Each item would have a popup for "help" or "put". As for class structure this is a big issue, as I recall all the discussion around name-spaces was a solution to the name-conflict problem. These name-spaces are based on project name, not function. If the classes were organized by function then they would no longer solve the namespace issue, that is counter and cyclone counter are the same in terms of function. Mathieu, what do you think about how the help-menu, object browser and object namespaces relate?

Krzysztof Czaja wrote:

The structure seems closed. I mean, even though a keyword comment may contain any word that its author finds suitable, only the predefined words would categorize the file for browsing -- is this the case? and if so, why?

The central point here is that the structure of the browsing is user-centric and sorted by function. For this to be successful certain names/catagories would need to be standardized. If a new object does not fit into the existing catagories then A new catagory could be created for those items "other" ? I think the basis of the catagories need to be fixed, with the option of the developer adding more. I do not think leaving the community of developers to entirly self-structure will create a viable organization of objects.

Bryan Jurish wrote:

Agreed. Still, I think the option of adding one's own (arbitrary) keywords / structuring conventions is important too -- pre-defined hard categorization schemes have a disturbing tendency to go all goopy at the edges when they're not dictatorially enforced, which I don't think anyone here really wants to do or to be done; maybe the solution is just as simple as differentiating between "browsing" and "searching"?

Having a starting point functional organization that gets extended when needed seems to be like a fine solution. What other options are there? I *am* thinking of browsing and searching as being different. Searching is to find an object to solve a particular problem. Browsing is poking around to find out what is possible. Browsing being most important to novice users. I think fixed (general) catagories will be more meaningful than an automated clustering system.

Hans-Christoph Steiner wrote:

But I think at this point we need some basic implementation before we can really go into details. A simple hack that parses a pre-defined set of keywords from a specially tagged comment in a help patch would be relatively easy to implement. Then we can test it out, and take it from there. So... the question remains, who actually wants to do some implementation?

I've added a "deployment" page on the wiki: https://puredata.org/Members/bbogart/Deployment

Of course I imaging I'll be making a major contribution, since I'm pushing this approach.

Hans-Christoph Steiner wrote:

• The workshops should absolutely tie into the help system. That is

one of the key ideas of the open source workshops for me: that all of the materials would be built into Pd and integrated into the Pd help system.

I do think this makes a lot of sense. Miller, how do you feel about this? How do you approach the problem of teaching PD?

Hans-Christoph Steiner wrote:

• I think that if we have a search function, then the browse categories

should be as broad as possible, like just "control", "audio", "video" sounds pretty good to me.

By "broad" do you mean flat, as all objects fit into these tree catagories with no sub-folders? One thing I did not consider is objects that fit into more than one catagory. pix_sig2pix~ is an example right in the middle of audio and video. Maybe the help-patch format could allow one external to be in multiple catagories.

Ok thats it for me. Please feel free to post comments on the wiki so we end up with less information in email boxes.

Thanks for all the comments. B.