hello, could anyone recommend a library for, or method of, manipulating lists of strings in textfile format? i need to be able to grep, sort, file etc. up until now i have been using [coll] but its not powerful enough for my needs. recently i have been experimenting with [shell] but it doesn't respond fast enough - if i dump a load of information triggered by an [until] then shell gives me the "old process still running message" i could regulate the flow if info into the [shell] but before i start in that direction i was wondering is there a library or method out there that would do it more efficiently. thanks, rob c
Hallo, robcanning hat gesagt: // robcanning wrote:
could anyone recommend a library for, or method of, manipulating lists of strings in textfile format? i need to be able to grep, sort, file etc. up until now i have been using [coll] but its not powerful enough for my needs.
What about an objectclass written in Python or Lua?
Frank Barknecht _ ______footils.org__
Frank Barknecht wrote:
Hallo, robcanning hat gesagt: // robcanning wrote:
could anyone recommend a library for, or method of, manipulating lists of strings in textfile format? i need to be able to grep, sort, file etc. up until now i have been using [coll] but its not powerful enough for my needs.
What about an objectclass written in Python or Lua?
Ciao
well my programming skills don't really extend far pass writing the odd bash script but yes, writing an objectclass would be ideal (you mean a pd objectclass?) any pointers on where to start with this using python?
i have been using [shell] to echo "$1">>"write this text to a file" but i am finding it very slow (between 20-30ms) to write one line - i am quite happy working with this method but am finding it way too slow - an tricks on how to speed up the shell? - seems strange that is so slow.
thanks
rob c
robcanning wrote:
Frank Barknecht wrote:
Hallo, robcanning hat gesagt: // robcanning wrote:
i have been using [shell] to echo "$1">>"write this text to a file" but i am finding it very slow (between 20-30ms) to write one line - i am quite happy working with this method but am finding it way too slow - an tricks on how to speed up the shell? - seems strange that is so slow.
shell is fast, but it takes long to start it up. whenever you send a command, a new shell environment is created.
for alternative solutions with running "outside" programs, i once proposed a "server" that communicates with Pd via pdsend/receive on this list.
for appending lines to a file, you can use zexy's [fwriteln].
this doesn't solve your problem of grep/sed/awk though.
gfma.dr IOhannes
i have been using [shell] to echo "$1">>"write this text to a file" but i am finding it very slow (between 20-30ms) to write one line - i am quite happy working with this method but am finding it way too slow - an tricks on how to speed up the shell? - seems strange that is so slow.
shell is fast, but it takes long to start it up. whenever you send a command, a new shell environment is created.
for alternative solutions with running "outside" programs, i once proposed a "server" that communicates with Pd via pdsend/receive on this list.
for appending lines to a file, you can use zexy's [fwriteln].
this doesn't solve your problem of grep/sed/awk though.
gfma.dr IOhannes
this is useful - i can write the files with fwriteln - then use the shell to grep them - the speed was needed at the writing end more than the grepping end - so this has solved my problem for now though i do need to come up with a more elegant solution in the long run thanks rob c
Hallo, robcanning hat gesagt: // robcanning wrote:
well my programming skills don't really extend far pass writing the odd bash script but yes, writing an objectclass would be ideal (you mean a pd objectclass?) any pointers on where to start with this using python?
Unless you already have pyext installed, I think installing the Lua loader is a bit easier. You need to check it out from the subversion repo at goto10, install the lua5.1-dev packages for your distro, then on Linux a simple "make" should compile it. Load it with "-lib lua".
Learning Lua or Python is useful in other contexts as well, and actually it's not that hard to turn that knowledge into a Pd object class. You may want to start at http://lua-users.org/wiki/LuaTutorial
IMO writing shell scripts is harder than writing Lua/Python scripts.
The general structure for both Python and Lua Pd classes is very simple. I attached a quickie demonstrating this for Lua: "ltxt" is a kind of simplified "coll" or "textfile", where you can store data under float and symbol keys and look it up again. Starting with that it's easy to extend this with methods to search for keys, sort the data storage array etc, delete data where keys match a certain pattern etc. The example is more meant to illustrate the general structure of a Lua Pd class than as a real object to use.
Frank Barknecht _ ______footils.org__
Frank Barknecht wrote:
Hallo, robcanning hat gesagt: // robcanning wrote:
well my programming skills don't really extend far pass writing the odd bash script but yes, writing an objectclass would be ideal (you mean a pd objectclass?) any pointers on where to start with this using python?
Unless you already have pyext installed, I think installing the Lua loader is a bit easier. You need to check it out from the subversion repo at goto10, install the lua5.1-dev packages for your distro, then on Linux a simple "make" should compile it. Load it with "-lib lua".
Learning Lua or Python is useful in other contexts as well, and actually it's not that hard to turn that knowledge into a Pd object class. You may want to start at http://lua-users.org/wiki/LuaTutorial
IMO writing shell scripts is harder than writing Lua/Python scripts.
The general structure for both Python and Lua Pd classes is very simple. I attached a quickie demonstrating this for Lua: "ltxt" is a kind of simplified "coll" or "textfile", where you can store data under float and symbol keys and look it up again. Starting with that it's easy to extend this with methods to search for keys, sort the data storage array etc, delete data where keys match a certain pattern etc. The example is more meant to illustrate the general structure of a Lua Pd class than as a real object to use.
Ciao
frank this is great! - thanks so much - i'll have a proper look at it and let you know how i get on with it. i know i really need to learn some programming outside pd - i just never know where to start - so many choices! - maybe i will just get stuck into this lua/python stuff thanks rob c
hi, ok - i have started learning to use lua with pdlua and have a question with regards to getting stuff out of the my pdlua object
in lua this works: for i, v in ipairs(self.mydata) do if string.match(v, "score = 1", 1, true) then print(v) end end
so i thought if i change print(v) with pd.post(v) i would get an output in the pd terminal but no...
also how would i get the output of the above to come out of outlet self:outlet(v) i would have guessed but obviously i have the wrong approach.
thanks
robc
-- what to do on "scores" into 1st inlet: function M:in_1_scores() for i, v in ipairs(self.mydata) do if string.match(v, "score = 1", 1, true) then print(v) end end
i'll clarify a bit...
i add a list to a table with:
-- "insert [list]" into 1st inlet: function M:in_1_insert(list) table.insert(self.mydata, list) end
then i search the table for the pattern "score = 1" and ask it to print the lines containing the matched pattern
-- what to do on "scores" into 1st inlet: function M:in_1_scores() for i, v in ipairs(self.mydata) do if string.match(v, "score = 1", 1, true) then self:outlet(1, "list", v) end end end
the problem is that the string.match is expecting a string and getting a table and i'm not sure how to fix this - i thought the ipairs was iterating the table into a string but its not.
here is the error from pd: [string "rob2"]:19: bad argument #1 to 'match' (string expected, got table)
lua and lua files attached
any guidance welcome,
thanks
rob c
#N canvas 123 334 512 437 10; #X msg 52 64 print; #X msg 53 86 scores; #X obj 17 131 rob2; #X msg 17 43 insert s d d d 2 2 2 2 w score = 1; #X msg 54 106 reset; #X obj 17 161 prepend set; #X msg 17 183; #X obj 17 11 loadbang; #X connect 0 0 2 0; #X connect 1 0 2 0; #X connect 2 0 5 0; #X connect 3 0 2 0; #X connect 4 0 2 0; #X connect 5 0 6 0; #X connect 7 0 3 0;
also, the code works in lua but not in the pdlua object somewhere i am getting mixed up between floats lists strings atoms tables etc. but i cant see where
here is the working lua code
tbl = {} table.insert(tbl, '1, s22df, 1, score = 3') table.insert(tbl, '1, s22df, 1, score = 2') table.insert(tbl, '1, s22df, 1, score = 1') for i, v in ipairs(tbl) do if string.match(v, "score = 1", 1, true)
then print(v) end end 1, s22df, 1, score = 1
thanks
rob
i'll clarify a bit... i add a list to a table with:
-- "insert [list]" into 1st inlet: function M:in_1_insert(list) table.insert(self.mydata, list) end
then i search the table for the pattern "score = 1" and ask it to print the lines containing the matched pattern
-- what to do on "scores" into 1st inlet: function M:in_1_scores() for i, v in ipairs(self.mydata) do if string.match(v, "score = 1", 1, true) then self:outlet(1, "list", v) end end end
the problem is that the string.match is expecting a string and getting a table and i'm not sure how to fix this - i thought the ipairs was iterating the table into a string but its not.
here is the error from pd: [string "rob2"]:19: bad argument #1 to 'match' (string expected, got table)
lua and lua files attached
any guidance welcome,
thanks
rob c
local NAME_OF_CLASS = "rob2" local M = pd.Class:new():register(NAME_OF_CLASS)
function M:initialize(name, args) self.outlets = 2 self.inlets = 1 self.mydata = {} return true end
-- "insert [list]" into 1st inlet: function M:in_1_insert(list) table.insert(self.mydata, list) end
-- what to do on "scores" into 1st inlet: function M:in_1_scores() for i, v in ipairs(self.mydata) do if string.match(v, "score = 1", 1, true) then self:outlet(1, "foo", v) end end end
-- what to do on "print" into 1st inlet: function M:in_1_print() for k,v in pairs(self.mydata) do pd.post(k .. ": " .. table.concat(self.mydata[k], " ")) end end
-- what to do on "reset" into 1st inlet: function M:in_1_reset() self.mydata = {} end
#N canvas 123 334 512 437 10; #X msg 52 64 print; #X msg 53 86 scores; #X obj 17 131 rob2; #X msg 17 43 insert s d d d 2 2 2 2 w score = 1; #X msg 54 106 reset; #X obj 17 161 prepend set; #X msg 17 183; #X obj 17 11 loadbang; #X connect 0 0 2 0; #X connect 1 0 2 0; #X connect 2 0 5 0; #X connect 3 0 2 0; #X connect 4 0 2 0; #X connect 5 0 6 0; #X connect 7 0 3 0
Hallo, robcanning hat gesagt: // robcanning wrote:
i add a list to a table with:
-- "insert [list]" into 1st inlet: function M:in_1_insert(list) table.insert(self.mydata, list) end
then i search the table for the pattern "score = 1" and ask it to print the lines containing the matched pattern
-- what to do on "scores" into 1st inlet: function M:in_1_scores() for i, v in ipairs(self.mydata) do if string.match(v, "score = 1", 1, true) then self:outlet(1, "list", v) end end end
the problem is that the string.match is expecting a string and getting a table and i'm not sure how to fix this - i thought the ipairs was iterating the table into a string but its not.
here is the error from pd: [string "rob2"]:19: bad argument #1 to 'match' (string expected, got table)
Okay, that's an issue with the general kinds of types in Lua, which can be a bit confusing at first. Basically in Lua/Pd you will have to deal with these three types: numbers, strings and tables. (There are more types e.g. functions.)
Now the first two are easy: numbers are 1, 2, 3.141, ... and strings are "this" and "that, too". Tables however are a bit trickier: For one they can be like lists in Pd:
counts = {1, 2, 3, "four", "and five"} print(counts[4]) -- prints: "four" print(type(counts[4])) -- it's a string
These kinds of tables which are indexed with numbers are also called "arrays" in Lua parlance. In Pd they are called lists.
Arrays also can contain other arrays and tables:
counts = {1, 2, 3, "four", "and five", {"x", "y", "z"} }
But the outlet functions in pdlua generally only work with flat arrays! So you first need to convert your data to a flat array using a pairs/ipairs iterator for example.
In the "in_*_something(args)" methods, args generally is an array as well (except for the bang, float and symbol methods).
The self.mydata table in rob2.lua is not flat, so it needs to be made flat like that.
self.mydata also is a key/value table (containing flat arrays indexed by string or numeric keys).
Tables in Lua can be key/value containers, which is something, Pd doesn't have directly:
veggies = {apples = 4, potatoes = 10} print(veggies.apples) -- prints: 4
And to make things worse, tables can contain tables:
stuff = {counts, veggies} print(stuff[2].potatoes) -- prints: 10
or even stuff like functions:
stuff.func = function() -- does nothing end
for k,v in pairs(stuff) do print (k,v) end
So, you see in that part here:
for i, v in ipairs(self.mydata) do if string.match(v, "score = 1", 1, true) then self:outlet(1, "list", v) end end
You have one "v" which is expected to be a string in string.match(), and another which is expected to be a table/array in self:outlet()!
This won't work, and the error message of Lua tells you, that "v" (the values of self.mydata) is a table, so it won't work in string.match()
I hope this clears it up a bit.
Now for the fix: You can convert a (flat) list/array to a string with table.concat(somelist, sep) where sep is the separator. If you use a space for sep, you can use
string.match(table.concat(v, " "), "score = 1")
or a bit better:
string.find(table.concat(v, " "), "score = 1", 1, true)
Better because string.find doesn't try to build and return a string, which you wouldn't be interested in anyhow.
Frank Barknecht _ ______footils.org__
Frank Barknecht wrote:
Hallo, robcanning hat gesagt: // robcanning wrote:
i add a list to a table with:
-- "insert [list]" into 1st inlet: function M:in_1_insert(list) table.insert(self.mydata, list) end
then i search the table for the pattern "score = 1" and ask it to print the lines containing the matched pattern
-- what to do on "scores" into 1st inlet: function M:in_1_scores() for i, v in ipairs(self.mydata) do if string.match(v, "score = 1", 1, true) then self:outlet(1, "list", v) end end end
the problem is that the string.match is expecting a string and getting a table and i'm not sure how to fix this - i thought the ipairs was iterating the table into a string but its not.
here is the error from pd: [string "rob2"]:19: bad argument #1 to 'match' (string expected, got table)
Okay, that's an issue with the general kinds of types in Lua, which can be a bit confusing at first. Basically in Lua/Pd you will have to deal with these three types: numbers, strings and tables. (There are more types e.g. functions.)
Now the first two are easy: numbers are 1, 2, 3.141, ... and strings are "this" and "that, too". Tables however are a bit trickier: For one they can be like lists in Pd:
counts = {1, 2, 3, "four", "and five"} print(counts[4]) -- prints: "four" print(type(counts[4])) -- it's a string
These kinds of tables which are indexed with numbers are also called "arrays" in Lua parlance. In Pd they are called lists.
Arrays also can contain other arrays and tables:
counts = {1, 2, 3, "four", "and five", {"x", "y", "z"} }
But the outlet functions in pdlua generally only work with flat arrays! So you first need to convert your data to a flat array using a pairs/ipairs iterator for example.
In the "in_*_something(args)" methods, args generally is an array as well (except for the bang, float and symbol methods).
The self.mydata table in rob2.lua is not flat, so it needs to be made flat like that.
self.mydata also is a key/value table (containing flat arrays indexed by string or numeric keys).
Tables in Lua can be key/value containers, which is something, Pd doesn't have directly:
veggies = {apples = 4, potatoes = 10} print(veggies.apples) -- prints: 4
And to make things worse, tables can contain tables:
stuff = {counts, veggies} print(stuff[2].potatoes) -- prints: 10
or even stuff like functions:
stuff.func = function() -- does nothing end
for k,v in pairs(stuff) do print (k,v) end
So, you see in that part here:
for i, v in ipairs(self.mydata) do if string.match(v, "score = 1", 1, true) then self:outlet(1, "list", v) end end
You have one "v" which is expected to be a string in string.match(), and another which is expected to be a table/array in self:outlet()!
This won't work, and the error message of Lua tells you, that "v" (the values of self.mydata) is a table, so it won't work in string.match()
I hope this clears it up a bit.
Now for the fix: You can convert a (flat) list/array to a string with table.concat(somelist, sep) where sep is the separator. If you use a space for sep, you can use
string.match(table.concat(v, " "), "score = 1")
or a bit better:
string.find(table.concat(v, " "), "score = 1", 1, true)
Better because string.find doesn't try to build and return a string, which you wouldn't be interested in anyhow.
Ciao
ok great - i got the hang of that now i think
now what i cant figure out is how to pass arguments to the code from a pd message
ok, so this code finds all the scores with a value greater than one and sends them to outlet one when given the message [greaterthan(
function M:in_1_greaterthan() for i, v in ipairs(self.mydata) do if tonumber(string.match(table.concat(v, " "), "score=(%d+)")) > 1 then self:outlet(1, "", v) end end end
now i want to be able to send the message [greaterthan score 3(
so in the code where it says "score=(%d+)")) - i need "score" to be variable from second part of the message the ">1" needs to be a variable from the 3rd part of the message
i know its somewhere within the () below but i cant seem to get it right function M:in_1_greaterthan()
thanks,
rob c
robcanning wrote:
now i want to be able to send the message [greaterthan score 3(
so in the code where it says "score=(%d+)")) - i need "score" to be variable from second part of the message the ">1" needs to be a variable from the 3rd part of the message
The Lua operator .. is string concatenation, so you can do:
"foo" .. "bar"
to get:
"foobar"
i know its somewhere within the () below but i cant seem to get it right function M:in_1_greaterthan()
thanks,
rob c
Something like:
-- called when "greaterthan" message is received at first inlet -- atoms is a table of the atoms in the message following the selector function M:in_1_greaterthan(atoms) local name = atoms[1] -- could be: "score" local value = atoms[2] -- could be: 3 if name ~= nil and value ~= nil then -- could check types too.. -- do your stuff here end end
If you have a range of possible selectors, and don't want to have to write a method for all of them, you can use:
function M:in_1(selector, atoms) ... end
Then selector will be "greaterthan" (or whatever) and atoms will be as above.
More specific methods are called before less specific methods, read pd.lua to see how that works (it's in the dispatcher function(s)).
Hope this helps,
Hallo, Claude Heiland-Allen hat gesagt: // Claude Heiland-Allen wrote:
Something like:
-- called when "greaterthan" message is received at first inlet -- atoms is a table of the atoms in the message following the selector function M:in_1_greaterthan(atoms) local name = atoms[1] -- could be: "score" local value = atoms[2] -- could be: 3 if name ~= nil and value ~= nil then -- could check types too.. -- do your stuff here end end
If you have a range of possible selectors, and don't want to have to write a method for all of them, you can use:
function M:in_1(selector, atoms) ... end
Then selector will be "greaterthan" (or whatever) and atoms will be as above.
One thing I wanted to ask here: Is there a special reason that pdlua isn't using the (new in 5.1) vararg syntax with "..." for methods and maybe also in outlet(...) or similar places? Then it would be possible to just do self:outlet(outnum, sel, a, b, c) instead of having to pack it first like self:outlet(outnum, sel, {a,b,c}). And for things like bang-output, it could be even simpler: self:outlet(outnum, "bang") instead of: self:outlet(outnum, "bang", {})
Frank Barknecht _ ______footils.org__
Frank Barknecht wrote:
One thing I wanted to ask here: Is there a special reason that pdlua isn't using the (new in 5.1) vararg syntax with "..." for methods and maybe also in outlet(...) or similar places?
Yes. It makes it very difficult to do useful things with variable length inputs (such as iterating over them, storing them, etc).
See for example:
http://lua-users.org/wiki/VarargTheSecondClassCitizen
Hallo, Claude Heiland-Allen hat gesagt: // Claude Heiland-Allen wrote:
Frank Barknecht wrote:
One thing I wanted to ask here: Is there a special reason that pdlua isn't using the (new in 5.1) vararg syntax with "..." for methods and maybe also in outlet(...) or similar places?
Yes. It makes it very difficult to do useful things with variable length inputs (such as iterating over them, storing them, etc).
See for example:
Thanks for this pointer. As I only write Lua for some weeks now, I may miss some things, but it seems to me, many of these issues aren't really a problem in pdlua's inlet and outlet messages, as these AFAIK will practically never contain "nil"-values. And if they don't contain nils, then the varargs can be converted to a table with {...} (or unpack'd from a table) on demand without running into problems with holes in tables coming from nils. And often, especially for inlet-methods, a full table may not even be needed to construct.
Frank Barknecht _ ______footils.org__
Frank Barknecht wrote:
Hallo, Claude Heiland-Allen hat gesagt: // Claude Heiland-Allen wrote:
Frank Barknecht wrote:
One thing I wanted to ask here: Is there a special reason that pdlua isn't using the (new in 5.1) vararg syntax with "..." for methods and maybe also in outlet(...) or similar places?
Yes. It makes it very difficult to do useful things with variable length inputs (such as iterating over them, storing them, etc).
See for example:
Thanks for this pointer. As I only write Lua for some weeks now, I may miss some things, but it seems to me, many of these issues aren't really a problem in pdlua's inlet and outlet messages, as these AFAIK will practically never contain "nil"-values. And if they don't contain nils, then the varargs can be converted to a table with {...} (or unpack'd from a table) on demand without running into problems with holes in tables coming from nils. And often, especially for inlet-methods, a full table may not even be needed to construct.
Yes, I see your point - but I feel the syntactic sugar is outweighed by the other reason why I used tables instead of varargs:
Lua's default stack size is small (45 slots or so, iirc), and Pd messages can be (almost) arbitrarily large. A table uses 1 stack slot, varargs uses 1 slot per element.
Hallo, Claude Heiland-Allen hat gesagt: // Claude Heiland-Allen wrote:
Yes, I see your point - but I feel the syntactic sugar is outweighed by the other reason why I used tables instead of varargs:
Lua's default stack size is small (45 slots or so, iirc), and Pd messages can be (almost) arbitrarily large. A table uses 1 stack slot, varargs uses 1 slot per element.
Ah, that's indeed a very good reason!
Frank Barknecht _ ______footils.org__
Hallo, robcanning hat gesagt: // robcanning wrote:
ok, so this code finds all the scores with a value greater than one and sends them to outlet one when given the message [greaterthan(
function M:in_1_greaterthan() for i, v in ipairs(self.mydata) do if tonumber(string.match(table.concat(v, " "), "score=(%d+)")) > 1 then self:outlet(1, "", v) end end end
now i want to be able to send the message [greaterthan score 3(
so in the code where it says "score=(%d+)")) - i need "score" to be variable from second part of the message the ">1" needs to be a variable from the 3rd part of the message
i know its somewhere within the () below but i cant seem to get it right function M:in_1_greaterthan()
Okay, now function M:in_1_greaterthan() needs to accept additional data. This gets passed as an argument to the function. Arguments are wrapped in a table in pdlua. In the following I called that table "args", but you can call it anything. Then "args" is concatenated into a single search string.
function M:in_1_greaterthan(args) searchstring = table.concat(args, " ") for i, v in ipairs(self.mydata) do if tonumber(string.match(table.concat(v, " "), searchstring)) > 1 then self:outlet(1, "", v) end end end
Frank Barknecht _ ______footils.org__
Hallo, Frank Barknecht hat gesagt: // Frank Barknecht wrote:
function M:in_1_greaterthan(args) searchstring = table.concat(args, " ") for i, v in ipairs(self.mydata) do if tonumber(string.match(table.concat(v, " "), searchstring)) > 1 then self:outlet(1, "", v) end end end
Ah, this is in part bulls**t, better check out Claude's solution. I misread your goal a bit.
Frank Barknecht _ ______footils.org__