Since both the current Pd-SQL devs (Mike and Jamie :) have posted to
this thread, I want to bring up the interface again. What do you
guys think about the two inlet interface for SQL objects? I just
find the sql/sqlend thing very strange, and it could end up being
difficult to work with. I see the interface like this: the hot inlet
is for meta commands, like "open", "close", "export", "connect",
etc. Then the second, cold inlet is for building SQL queries.
Having a cold inlet devoted to building SQL queries would mean that
you would not need any special "sql/sqlend" style tags at all.

Instead, you'd just send messages to the cold inlet, those messages
would be appended to the existing SQL message buffer. Then when the
query is ready to be sent, send a "submit" or "run" or whatever to
the hot inlet, and it would run the buffer from the cold inlet and
clear it. There would also be a "clear" command to just clear the
buffer without sending it.

This would allow for the comma trick, etc. and I think it would fit
better into Pd's message processing. In addition, there could be a
single line message for the hot inlet, like this:

"submit insert into mine (name) values ('the_text')"

I'll help implement this if you want. I just have a strong aversion
to the "sqlend" stuff since I think it could end up being strange to
use within Pd. Plus I think these are very valuable objects to have,
so I think it's important to get the interface right. Basically, I
am thinking of something along the lines of the [textfile] interface,
as much as possible. Here's a sketch:

__hot inlet__ [open $1( [close( [export $1( - this could be used to export the database to a .sql file

[clear ( [submit ( [submit insert into mytable (name) values ('$1') (

__cold inlet__ [insert into mytable (name) values ('$1') ( [CREATE TABLE datatable(id INTEGER, duration FLOAT, type VARCHAR,
datetime DATETIME) ( [SELECT id, ABS((duration - 1500)/1500) AS error FROM datatable ORDER
BY error LIMIT 1(

Also, on a different note, I think it could be useful to have various
status and meta data queries output on the second inlet. Currently
in [psql], there is just a bang on complete. But there could be
things information about the database that aren't available via SQL.

Finally, there is the source code for the Max/MSP mysql external, so
I think it should be pretty easy to make a mysql version of this
interface as well. Then we'll have a nice trio of SQL options:
embedded, PGSQL, and MySQL.

http://www.publicbeta.cx/maxx/

.hc

On Dec 7, 2007, at 4:32 PM, Mike McGonagle wrote:

Andy, have you had a chance to check out the SQLite external yet?
You can get a copy of it here ( http://puredata.info/Members/ mjmogo ), the latest version is 0.13. While this is still under
developement, it is based on the work of Jamie (and others), as far
as how the SQL is handled. They are not exactly the same, but I
would like to make that one of the goals so that these things can
be interchangeable.

One thing that I have not done yet is implement any kind of Blob
data types. It appears that SQLite3 (not the external, but the
database) handles Blobs as binary strings. I still don't know quite
how this would work in PD though. I would like to get some feedback
from others about how to do this.

As SQLite3 stores its database in a file, rather than a server,
there is no need to set up anything for the database (other than
designing the tables). Also, if you need to, you can use the tools
from SQLite3 to work with the same database file.

I work on the Mac, so the Makefile is setup to build a Universal
Binary (thanks to Hans). If you are on another platform, it
shouldn't be difficult to compile these things, as SQLite3 is pure
ANSI C. When I compile it, there are a lot of warnings generated,
but everything works fine. I guess I should turn off all the
warnings for that part of the compile.

But, let me know if you try this. I am eager to see how it works
for others.

Thanks,

Mike McGonagle

On 12/7/07, andy.graybeal@casanueva.com <
andy.graybeal@casanueva.com> wrote: i installed postgresql, and got it running, created a user for myself, compiled [psql] from postlude (i'm using pd-extended from cvs).

i opened up the psql-help file and started with the first
instruction, and it created the database fine, i moved to the second command and get
this error: psql: Action failed. PQresultStatus is PGRES_FATAL_ERROR

i'm wondering where i can look for more information about what is
going wrong, and what i can do to fix it.

i think it would be wonderful if i could start using a database to
store values :)

-andy

---

PD-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/ listinfo/pd-list

-- Peace may sound simple—one beautiful word— but it requires
everything we have, every quality, every strength, every dream,
every high ideal. —Yehudi Menuhin (1916–1999), musician _______________________________________________ PD-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/ listinfo/pd-list

If you are not part of the solution, you are part of the problem.

On Dec 7, 2007, at 8:44 PM, Mike McGonagle wrote:

On 12/7/07, Hans-Christoph Steiner hans@eds.org wrote: This would allow for the comma trick, etc. and I think it would fit
better into Pd's message processing. In addition, there could be a
single line message for the hot inlet, like this:

Hans, I was thinking a little bit more about this, and I think I
see what you are getting at, I will have to try to mock this up at
home tonight. The only trouble that I can still see here is how to
indicate that an SQL statement is complete.

That is done by sending the [submit( message to the hot inlet. Or do
you mean having multiple SQL calls separated by semi-colons? If you
wanted to add semicolons, there would have to be a special message, I
think we could just reuse the "addsemi", "addcomma", "adddollar"
messages from message boxes.

As far as I know, the semi-colon at the end of the statement in SQL
triggers the execution of that statement, so I can't see an advantage
to having multiple, semi-colon terminated statements in a single
message box. Does it change how the SQL is executed if they are
submitted at the same time?

The other thing I think that would have to be is forcing all input
SQL to go into the second inlet. Allowing some things to go into
the hot inlet, while others are required to go into the second
outlet would make it difficult to use because you would have to
remember which type of statement can go where...

Any SQL statement would be allowed on the cold inlet. Because of the
limitations of Pd (no escape mechanism), and the fact that commas
already have a meaning in Pd messages, the hot inlet would not be
able to handle commas (unless someone comes up with something quite
clever).

One advantage of having the second, cold inlet is that it could be
devoted to SQL statements, so you don't need to start each message
with "sql" and there wouldn't be any words that the [sqlite] object
used, like "open", "close", etc. Having the comma handling on the
second/cold inlet only means that the first/hot inlet also doesn't
need the preceeding "sql" selector since SQL statements will only
start with a small set of selectors (CREATE, DROP, INSERT, SELECT, etc).

For clarity, I added another cold inlet example and a sketch of
building a big query. The big query example in the psql help points
out something that I think could get quite confusing, separating the
chunks with "sqlend, sql".



.hc

News is what people want to keep hidden and everything else is
publicity. - Bill Moyers

I'm not a big fan of [expr]'s syntax since it is custom syntax that
is not used anywhere else in Pd. Instead, you could achieve the same
result by using the interface I described, then embedded your SQL
statements with [sql] into a subpatch or an abstraction. This just
about any regular Pd user knows how to do.

.hc

On Dec 8, 2007, at 12:51 AM, Mathieu Bouchard wrote:

On Fri, 7 Dec 2007, Hans-Christoph Steiner wrote:

...

That is done by sending the [submit( message to the hot inlet. Or
do you mean having multiple SQL calls separated by semi-colons?
If you wanted to add semicolons, there would have to be a special
message, I think we could just reuse the "addsemi", "addcomma",
"adddollar" messages from message boxes.

The only SQL interface I'd really be happy to use (if I had to use
SQL) is something using placeholders, sort of like [expr], but
perhaps written in the usual SQL style:

[sql select * from candies where flavour=? and colour=?]

you wouldn't modify the statement any more than you modify an
[expr] at runtime. this allows the statement to be precompiled by
the server. you could still allow runtime change but perhaps it
would get messy with multiple inlets, so let's assume for now that
it is not supported.

in the above example, [sql] has two inlets, one for each question- mark. Sending a list in the hot inlet is also correct, and "set"
and "bang" do as usual. Each value will be quoted automatically,
with appropriate apostrophes and backslashes and stuff, so that the
user never has to do it. (my biggest problem with PHP is the lack
of this feature).

an alternate way to do it is to use the same syntax as [expr]. This
allows a repeated value to not have to be repeated in the input to
the object. However, this means that values are going to be typed,
which is a bad match with SQL, which doesn't type values (it only
types table cells themselves and casts everything automatically).

...

As far as I know, the semi-colon at the end of the statement in
SQL triggers the execution of that statement

With this interface, there would be no semicolon at the end (or it
would be optional and useless). The execution is done each time the
hot inlet is activated (bang, float, symbol or list, but not others
like set and whatever else if there's anything else).

...

so I can't see an advantage to having multiple, semi-colon
terminated statements in a single message box. Does it change how
the SQL is executed if they are submitted at the same time?

Multiple statements are compiled as one function to which you pass
all arguments you want at once. This could be useful if you want to
do both insert and select in one go. It may be a bit faster, or
maybe not, but it's convenient.

[expr] supports semicolons and uses them for multiple outputs. For
[sql], it might make only sense to have one outlet per query rather
than by statement: that is, only SELECT and EXPLAIN and such would
get an outlet, while INSERT and DELETE wouldn't, for example.

At this point, I can't think of a case where I'd like to modify a
SQL statement at runtime. Somehow it's possible to find a use, but
I think that for almost everything you don't.

_ _ __ ___ _____ ________ _____________ _____________________ ... | Mathieu Bouchard - tél:+1.514.383.3801, Montréal QC Canada

I spent 33 years and four months in active military service and
during that period I spent most of my time as a high class muscle man
for Big Business, for Wall Street and the bankers. - General
Smedley Butler

On Dec 9, 2007, at 9:41 PM, Mathieu Bouchard wrote:

On Sat, 8 Dec 2007, Hans-Christoph Steiner wrote:

...

I'm not a big fan of [expr]'s syntax since it is custom syntax
that is not used anywhere else in Pd.

A precedent has to start somewhere :)

I believe that the goal is to make an interface that is effortless
to use rather than try to be more dataflowish than the pope. The
goal is not to pass more messages and use more objects and
connections just to show off what's the concept of dataflow and how
deeply pd follows it. The goal is still to make patches work with
as little effort as possible.

I also believe that there are plenty of pd classes that have at
least one element of syntax that is not used anywhere else in pd.
It also depends on how you look at pd: are two occurrences in two
very related classes, counting as one occurrence, or as two? and
why would it be counted that way?

Too often "reducing effort" is equated with typing shortcuts and
things along that line. I think putting everything into the [psql]
object box is version of this. Things are a bigger concerns in the
push to reduce effort are:

• reducing bugs!
• reducing time spent learning new objects
• reducing time spent remembering how to use objects
• making flexible programming easier rather than basic programming

faster

.hc

...

Instead, you could achieve the same result by using the interface
I described, then embedded your SQL statements with [sql] into a
subpatch or an abstraction. This just about any regular Pd user
knows how to do.

But it's better to not have to do that.

_ _ __ ___ _____ ________ _____________ _____________________ ... | Mathieu Bouchard - tél:+1.514.383.3801, Montréal QC Canada

As we enjoy great advantages from inventions of others, we should be
glad of an opportunity to serve others by any invention of ours; and
this we should do freely and generously. - Benjamin Franklin

On Sat, 2007-12-08 at 00:51 -0500, Mathieu Bouchard wrote:

The only SQL interface I'd really be happy to use (if I had to use SQL) is something using placeholders, sort of like [expr], but perhaps written in the usual SQL style:

[sql select * from candies where flavour=? and colour=?]

Interestingly enough, that's pretty much how Mike started off with his SQLite object. Then I persuaded him that passing the queries as a list to the inlet would be more flexible. It also greatly reduces the number of objects required to send a query, if you have more than one query. I also agree with Hans about being consistent with other Pd objects esp. [textfile].

Jamie

On Sun, 2007-12-09 at 21:47 -0500, Mathieu Bouchard wrote:

On Sun, 9 Dec 2007, Jamie Bullock wrote:

...

Then I persuaded him that passing the queries as a list to the inlet would be more flexible. It also greatly reduces the number of objects required to send a query, if you have more than one query.

I don't understand the latter part. How does it work? I'm talking about putting any number of queries together in a single object and passing the arguments of those queries all together in a list. How can you reduce the number of objects more than that?

The way you are suggesting always requires at least 2 objects per query: an object to build the query and a message to send it. So if you have 5 different queries (I mean with different statements not just different data), then you would need at least 10 objects. This would be the case even if there was no variable data in the queries. Using the [psql] way of doing things, provided that the queries have no variable atoms, only 6 objects would be required, one for the database connection, and 5 containing the queries, which when passed to the connection object also trigger the sending.

Your way takes at least two objects instead of one and it does not provide any protection against SQL injection because it can't distinguish between a symbol passed as a SQL argument and a symbol representing part of the statement syntax itself.

True, this is a good argument for the [expr]-style SQL object. Although there may be other ways to provide some protection against injection like allowing the user to lock the number of statements in the query.

Jamie

On Mon, 10 Dec 2007, Mike McGonagle wrote:

Could someone please explain that IMPORTANCE of worrying about SQL injection? Just how would it effect users of PD?

We don't know what pd users will do with [psql]. Do you?

_ _ __ ___ _____ ________ _____________ _____________________ ... | Mathieu Bouchard - tél:+1.514.383.3801, Montréal QC Canada

On Mon, 10 Dec 2007, Mike McGonagle wrote:

I guess what I am getting at is that I don't see how we can prevent people from using this maliciously.

Using true placeholders or other form of automatic quoting.

If they are creating the SQL and putting the data into it, how can we stop them from being idiots?

If you have automatic quoting, you don't even have to think about who is an idiot and who isn't. I don't want to think about who's an idiot and who isn't. I don't want you to think about it.

Are you saying that we need to do data checking prior to the data being sent to the server?

If you quote your data properly then you don't need to check whether the data will garble the query's syntax or not. Therefore, no, I don't think what you need to do on the data is a "check"... though at the character level, you have to "check" in order to know which chars have to be replaced.

_ _ __ ___ _____ ________ _____________ _____________________ ... | Mathieu Bouchard - tél:+1.514.383.3801, Montréal QC Canada

On Mon, 2007-12-10 at 09:55 -0600, Mike McGonagle wrote:

Well, Jamie, at the same time, I think that Mathieu might be refering how the output is handled from the 'sql' external. That is the part that would make having just a single instance of a database object difficult at best to work with. From some of the early tests that I have done, I have pretty much always assumed that each instance would be outputing a different result set. If you only used one database object, you would have to figure out how to route all the result sets.

Fair enough, I think I was probably stretching the point about using the [psql]-style of database connectivity requiring less objects.

However, it's perfectly possible to have multiple [psql] instances connected to the same database, so the routing problem is a bit of a moot point.

Could someone please explain that IMPORTANCE of worrying about SQL injection? Just how would it effect users of PD?

It is certainly important in any situation where you would want to do variable substitution on queries using data sent to your Pd instance over a network. Some people might use Pd in this way via a web interface for example.

Jamie

On 12/10/07, Mathieu Bouchard matju@artengine.ca wrote:

But it requires multiple logins, perhaps many at once, which might be wasteful. For sharing database connections between several [psql] objects, what do you think would be the best ways to do it? This is supposing that several different database connections can still be used at once, as specified in the patches.

In one of the early prototypes I wrote for this, I set it up so that the connections were shared. Basically, when an object tried to open a connection, it would check if there was already one open, and if so, it would use that. Else it would create a new connection, which would then get stored in the global connection object.

One thing that I would be curious about would be any collisions in using a shared object, but I also think that should be something left to think about in building the PD patch.

Mike

_ _ __ ___ _____ ________ _____________ _____________________ ...

| Mathieu Bouchard - tél:+1.514.383.3801, Montréal QC Canada

On Mon, 2007-12-10 at 16:43 -0500, Mathieu Bouchard wrote:

On Mon, 10 Dec 2007, Mike McGonagle wrote:

...

In one of the early prototypes I wrote for this, I set it up so that the connections were shared. Basically, when an object tried to open a connection, it would check if there was already one open, and if so, it would use that. Else it would create a new connection, which would then get stored in the global connection object.

I think that the first argument of [psql] could be a receive-symbol, followed by the query itself. Then [psql] would not use the receive-symbol to send messages to, it would look at the receiver directly, checking that it's a [psql.connection] object, and then calling a function of that object directly, and get called back when a response is received. This is quite close to how [tabread] works except it inserts a [delay] before the result. It's how [tabread] would work if it were trying to access an array on another computer.

I like this idea. What do you think about using an implicit receive-symbol, '$0-psql.1001', '$0-psql.1002' etc? Would it be possible for psql to automatically discover the psql.conn's receive symbol by using pd_findbyclass(), and then reading a variable in the object struct?

Jamie

On Dec 10, 2007, at 3:30 PM, Mike McGonagle wrote:

On 12/10/07, Mathieu Bouchard matju@artengine.ca wrote: But it requires multiple logins, perhaps many at once, which might be wasteful. For sharing database connections between several [psql] objects, what do you think would be the best ways to do it? This is supposing that several different database connections can still be used at once, as specified in the patches.

In one of the early prototypes I wrote for this, I set it up so
that the connections were shared. Basically, when an object tried
to open a connection, it would check if there was already one open,
and if so, it would use that. Else it would create a new
connection, which would then get stored in the global connection
object.

One thing that I would be curious about would be any collisions in
using a shared object, but I also think that should be something
left to think about in building the PD patch.

It is useful to represent the pieces in Pd space, so you can
understand what's going on. That's one reason why I advocate having
the core object represent the connection to the database rather than
a query. Otherwise, it's starts to become more like Max/MSP's mega- objects (coll, zl, etc) that are really like mini-applications than
programming.

There still can be query objects, they would just be designed to feed
to the core database objects. These query objects would then be
usable if we maintain the same interface.

.hc

News is what people want to keep hidden and everything else is
publicity. - Bill Moyers

On Mon, 2007-12-10 at 15:00 -0500, Mathieu Bouchard wrote:

On Mon, 10 Dec 2007, Jamie Bullock wrote:

...

Fair enough, I think I was probably stretching the point about using the [psql]-style of database connectivity requiring less objects. However, it's perfectly possible to have multiple [psql] instances connected to the same database, so the routing problem is a bit of a moot point.

But it requires multiple logins, perhaps many at once, which might be wasteful.

Not especially.

For sharing database connections between several [psql] objects, what do you think would be the best ways to do it? This is supposing that several different database connections can still be used at once, as specified in the patches.

I think this is a less trivial problem than it might appear. Having a global pointer representing the database connection seems dangerous and a bit naive. Perhaps some kind of callback-based mechanism between the database connection 'server' object and the 'client' query objects would be the way to go.

Jamie

On Mon, 10 Dec 2007, Hans-Christoph Steiner wrote:

For a place where you are expecting a number, you can protect against a SQL injection attack by merely putting a [float] before the message box with the SQL in it. In other situations, I think that Perl has a pretty decent idea: a "SQL quote" function.

Perl has also a pretty decent idea, which is to allow placeholders, which automatically quotes so that you don't have to do it nor even think about it. I rarely ever wrote any Perl code that would access a SQL database in any other way than using placeholders. It's for safety but also not to have to think about strings, so that using SQL feels most like using an array.

I know that you know about Perl's (and most any other's) placeholders, but I really mean that one should almost never have to use [sqlquote] at all, and things are easier if one doesn't have to use it.

• the names ones could be supported as selectors to the hot inlet:

what about selectors that conflict with existing functionality of the object? e.g. if a column is called "symbol" or whatever... what about columns with the same name as methods that will be defined in future versions of [psql] ?

_ _ __ ___ _____ ________ _____________ _____________________ ... | Mathieu Bouchard - tél:+1.514.383.3801, Montréal QC Canada

On Mon, 10 Dec 2007, Jamie Bullock wrote:

The way you are suggesting always requires at least 2 objects per query: an object to build the query and a message to send it.

I assumed that there would always be one variable part per query, so that there's always a float or symbol or list coming from somewhere to trigger the query. Some queries take no arguments but in my experience they are rather exceptional. So in the usual case it takes 1 object per query, and if the statement is completely non-variable, it takes a bang coming from somewhere, which could be counted as a "bang" messagebox or not, depending on how you count.

Using the [psql] way of doing things, provided that the queries have no variable atoms,

This almost *never* happens. As long as what you do in Pd is regular read-write activities involving tables that contain data that you only want to see a small part of at a time, you need variable atoms in almost any query:

select person from attendance where chatroom=? select chatroom from attendance where person=? select * from users where user=? select * from chatrooms where chatroom=?

apps more likely to have invariable queries are apps that started small, stayed small, and will never be expanded.

True, this is a good argument for the [expr]-style SQL object. Although there may be other ways to provide some protection against injection like allowing the user to lock the number of statements in the query.

checking the number of statements is not enough: what about a value like: (including quotes)

'or'1

put inside this statement:

delete from users where username='$1'

this query will delete the complete table.

_ _ __ ___ _____ ________ _____________ _____________________ ... | Mathieu Bouchard - tél:+1.514.383.3801, Montréal QC Canada

On 12/10/07, Jamie Bullock jamie@postlude.co.uk wrote:

PREPARE uses the '$' character as its placeholder identifier, and '$1'

Jamie, after thinking about this, I don't think we should worry about the specifics of the characters that we use, as long as we can agree on what gets entered into a PD message box, the specific instance of the external ('sqlite', 'psql', etc.) can handle the conversion to what the specific database uses.

From what I have seen on this thread, is it right that the '?' syntax does

NOT have an identifier following it? I think it would be very helpful if we use identifiers in the SQL if only to make the binding to the proper datatypes easier, as per my last email.

Mike

Peace may sound simple—one beautiful word— but it requires everything we have, every quality, every strength, every dream, every high ideal. —Yehudi Menuhin (1916–1999), musician

If you have applied the "add string support" patch you can use the blob atom, which is essentially a pointer to, and the length of, a binary string. The string can be anything at all.

Martin

Mike McGonagle wrote:

To: pd-list@iem.at Subject: Re: [PD] [psql] object hand-holding Date: Mon, 10 Dec 2007 13:29:16 -0600

On a different note, has anyone given any thought to handling Blobs? I know that in SQLite, they store the data as binary strings, but I am not quite certain how the external would handle the actual data going into and coming out of it. Mike

---

PD-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list

It probably would end up in vanilla if there was a demand for it, although Miller doesn't seem to like the idea, needs more convincing ;)

Martin

Mike McGonagle wrote:

To: "Martin Peach" martin.peach@sympatico.ca CC: pd-list@iem.at Subject: Re: [PD] [psql] object hand-holding Date: Mon, 10 Dec 2007 14:17:20 -0600

Is this something that is part of PD Vanilla? I would like to be able to have this external running there. Thanks,

Mike

On 12/10/07, Martin Peach martin.peach@sympatico.ca wrote:

...

If you have applied the "add string support" patch you can use the blob atom, which is essentially a pointer to, and the length of, a binary string. The string can be anything at all.

Martin

Mike McGonagle wrote:

...

To: pd-list@iem.at Subject: Re: [PD] [psql] object hand-holding Date: Mon, 10 Dec 2007 13:29:16 -0600

On a different note, has anyone given any thought to handling Blobs? I

know

...

that in SQLite, they store the data as binary strings, but I am not

quite

...

...

certain how the external would handle the actual data going into and

coming

...

out of it. Mike

...

---

PD-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list

-- Peace may sound simple—one beautiful word— but it requires everything we have, every quality, every strength, every dream, every high ideal. —Yehudi Menuhin (1916–1999), musician

On Mon, 10 Dec 2007, Hans-Christoph Steiner wrote:

What we really need is regression testing, I think that would make it easier to convince Miller to accept patches, and would be a great tool for finding bugs. Matju started that project, any word on that Matju?

No, I haven't done much progress. The problem with that thing is that no-one wants to do it, not even me. It's hard to consider it a priority as much as it should be.

Regression testing is only for past bugs, but you must mean unit tests. I would think that regression tests should be integrated with unit tests, and that acceptance tests should also be integrated, so this is why I say "automatic testing" in general. But I think that it's better to make testing part of one's process right now than to wait for anybody else's framework. The framework is secondary to actually having any kind of test. The framework is only there to factor out common things across separate tests.

I've been somewhat slowed down by the fact that Pd is missing some features like automatic error trapping. It really shows that Pd was not designed to be tested, and for that reason, it'll be hard to make a good test suite with Pd as it is now. This is why I merged PureUnity with DesireData.

I'd love to see that in action. :D

I'd also love to see that in action.

_ _ __ ___ _____ ________ _____________ _____________________ ... | Mathieu Bouchard - tél:+1.514.383.3801, Montréal QC Canada

IOhannes m zmölnig wrote:

at least i am using "kind-of" regression tests in zexy and iemlib.

this of course should read "iemmatrix" instead of "iemlib" sorry for the confusion.

mfg.asdr IOhannes

Hans-Christoph Steiner wrote:

On Dec 21, 2007, at 10:30 AM, IOhannes m zmölnig wrote:

...

Mathieu Bouchard wrote:

...

On Mon, 10 Dec 2007, Hans-Christoph Steiner wrote:

...

What we really need is regression testing, I think that would
make it easier to convince Miller to accept patches, and would be a great tool for finding bugs. Matju started that project, any word on that Matju?

No, I haven't done much progress. The problem with that thing is
that no-one wants to do it, not even me. It's hard to consider it
a priority as much as it should be.

at least i am using "kind-of" regression tests in zexy and iemlib. the "suite" is rather simplistic, but works well for my purposes.

see zexy/tests for how it works

Any interest in making this a general purpose testing framework? I

have you had a look at it?

would like to come up with a suite that is run every night
automatically, like the nightly builds. It sounds very similar to
what I was thinking.

If we just come up with a standard set of inlets, outlets, and a
standard output value,

well, i think i ahve documented this in the README. since i am using the same framework on several projects (not just the 2 mentioned), i would prefer to keep it rather fixed like it is. writing tests is not that much fun for me. right now there are more than 300 tests in iemmatrix...

then it will be easy to write a script to run
thru them all and report on them. Sounds like you might already have
that working to some degree.

have you had a look at it? i am using my suites for >2years now.

I would like to make a set that tests for the strict aliasing bugs so
I can safely use gcc optimizations. This means testing tilde objects.

while there are no explicit tests for ~-objects in any of my suites (i think), they are thought of.

I think it might make sense to start a section in CVS for this called
'tests', then we can start gathering tests there into subfolders
organized by library. Another option would be a standard folder in
each library called 'tests' like you have.

why would you keep them apart?

That's my two bits, but we really should be doing a lot more of this.

i agree.

and despite of saying how great my regression-test-suite is, in reality it is not. and i am using several different versions of it out there [1]

my advice:

• keep the tests as simple as possible, but not simpler

one should be able to write a test fairly easily; i think shortcuts are most likely not needed most of the times (e.g. iirc, matju introduced several shortcuts in his unity-test frameworks to test for mathematically important concepts (like assosiativity of operations); i think it is way simpler to just program 3 tests manually instead)

• as for results i have found it sufficient to have the following

return-states: SUCCESS, FAIL, WAIT i found it makes writing tests simpler if all of them implicitely fail if they do not respond with a return-state _immediately_ (this might make the FAIL-state unnecessary; but again it makes tests simpler if you can provide an explicit FAIL) if the test takes some time (e.g. for signal-objects or timing objects), i return the WAIT state which halts the testing-framework until the currently tested object returns a value.

i usually use "1" for success, "0" for fail, "-1" for wait, the rest is fail too.

• write tests to test the framework

• allow fail-tests which have to return FAIL-state in order to success

• handle crash-tests separately

• allow "manual" tests (not "scriptable" (as in "as fast as possible"))

• i found that tests-patches should not have arguments (especially:

default-arguments, like "$0"), as this sometimes makes testing of special features unneccessarily complicated.

• ...

fmga.sdr IOhannes

[1] the most current incarnation of my framework is currently used in http://ftm.cvs.sourceforge.net/ftm/ftm/externals/pd/regression/

Mathieu Bouchard wrote:

On Sat, 22 Dec 2007, IOhannes m zmoelnig wrote:

...

• keep the tests as simple as possible, but not simpler

one should be able to write a test fairly easily; i think shortcuts are most likely not needed most of the times (e.g. iirc, matju introduced several shortcuts in his unity-test frameworks to test for mathematically important concepts (like assosiativity of operations); i think it is way simpler to just program 3 tests manually instead)

I think that the tests for associativity are not going to be so useful, but they're still a good practice in some way. However, my goal for PureUnity was not just to make regression tests, but also invent a bunch of superclasses for all the Pd builtins (and some externals) in order to make a categorisation of objects that makes sense from the point of view of how you use the objects: e.g. [timer] and [realtime] would be in the same category not because they deal with time, but because the left bang starts and the right bang stops and outputs a float.

...

• allow fail-tests which have to return FAIL-state in order to success

How do you know that a method has failed properly?

because it returns the expected FAIL-state.

the problem is the same as "how do you know that a method has succeeded properly"

you need a-priori knowledge about the expected return state of the test. i usually do this by a discrimination based on the test-name. (prefixing "fail") e.g. "fail_blabla" will only success if it returns the state "FAIL" immediately or after a "WAIT".

fgamdsr. IOhannes

Hans-Christoph Steiner wrote:

In a binary system, anything that doesn't success would be a failure. I don't quite get the WAIT state. Do you have an example of where to use that?

how do you test objects that need time (e.g. test [delay 1000])

i found it much more easy to write tests if you can automatically fail tests (by not returning an explicit SUCESS) equally, i sometimes need to explicitely FAIL a test (instead of just _not_ returning SUCCESS)

if you discard these two requests, it would be sufficient to just return "bang" for success, everything else would fail. but how do you plan to test signal objects?

mgasdr. IOhannes

Mathieu Bouchard wrote:

On Sun, 23 Dec 2007, Hans-Christoph Steiner wrote:

...

...

e.g. "fail_blabla" will only success if it returns the state "FAIL" immediately or after a "WAIT".

In a binary system, anything that doesn't success would be a failure. I don't quite get the WAIT state. Do you have an example of where to use that?

In most tristate electronics, the third state is WAIT, but in Pd, you normally do that by not sending a message: if as binary you'd send a 0 or 1 while running a certain method, and want to introduce a WAIT state, you'd make it not output anything at first, introduce proper [delay], and only later send a 0 or 1 when it's ready. It's as simple as that. If the answer is not going to come, an explicit "wait" message isn't going to disambiguate nor solve that, so it might be a good idea to put some timeout protection *outside* of the tests themselves.

In digital electronics the third state is high-impedance, that is, it is not driven high or low. However, a device reading a Hi-Z output will itself always output either 0 or 1 and has no way of telling that it is connected to a Hi-Z output, so if you want Hi-Z to mean WAIT, you'll need another line that is driven to 1 or 0 to indicate that the result isn't in yet. It's an early result in computational theory that in general it is not possible to know if a process will terminate or not. Probably using a watchdog timer is the best way out: if the result isn't in after a reasonable delay, assume it's not going to happen, something like:

[test_suite( | | [b] | | | [delay] [object_under_test] | | | [fail( [pass( [stop(------->to [delay]

Martin

Mathieu Bouchard wrote:

On Mon, 24 Dec 2007, Martin Peach wrote:

...

Mathieu Bouchard wrote:

...

In most tristate electronics, the third state is WAIT,

In digital electronics the third state is high-impedance, that is, it is not driven high or low. However, a device reading a Hi-Z output will itself always output either 0 or 1 and has no way of telling that it is connected to a Hi-Z output,

No, I'm not talking about the high-impedance state, especially because it can't be read easily. The other kind of tristate is at a higher level than that and is implemented as "dual-rail" wires: 10 = false, 11 = true, 00 or 01 = wait. This is useful for making clockless devices.

OK, but that's not called tristate, and "dual-rail" usually means having two power supplies (e.g. plus and minus 5V instead of just 5V). What you are describing is like a two-wire serial setup where the data on the second wire is only considered valid when the first wire is high. It's not clockless, in fact the first wire is usually named "Clock".

Clockless devices are interesting because they can very gradually adapt to the "weather": if you remove the heat sink it will automatically slow down, and if you dip it in liquid nitrogen it will very much accelerate, without any clock, any thermometre, any power management device, etc..

The clock frequency is affected by temperature, but it's still a clock. I guess you're saying that a crystal oscillator is a clock and a RC oscillator is not, which is a new meaning to me. Even crystal oscillators are affected by temperature because the crystal gets bigger as it warms up, if you want a stable frequency you need to run it in temperature-controlled oven.

In clockful devices you usually don't have that many WAIT lines, so it does not make sense to count them as a 3rd state. Instead, a 8-bit bus would have 257 states (2^8 + 1) using 9 wires, and so on, but even then they are used as little as possible, and bunch of delays are hardcoded and missed deadlines make the CPU crash or do weird things (I've *seen* it in a Pd patch once!).

If you have 9 wires you have 512 possible states, so it's wasteful if 256 of them mean the same thing. That may be why the two-wire system is not much used. Cpus often have a line to indicate if the data on the bus is valid or not (as well as other lines that indicate what kind of data it is). And you can add wait states to memory access cycles so the ram has time to handle the data, which involves using a WAIT line to hold off the cpu until the time has elapsed.

...

Probably using a watchdog timer is the best way out: if the result isn't in after a reasonable delay, assume it's not going to happen, something like:

Most of the time, your [delay] is enough, and that's what I meant too. Sometimes the problem will also involve 100% CPU spent handling the same event, e.g. bang [until] without anything else, and so [delay] won't work at all. Sometimes it could be a [delay 0] within the test (or component being tested) doing an infinite loop at the same logical time instead. Because [delay] can only handle logical times, it won't be able to trap it. But I think that those two latter cases are relatively rare, and if ever they are not that rare, or that they are very inconvenient when they happen, then we might want to use a watchdog more like pd's watchdog process.

To be bulletproof you would need to schedule an external interrupt to kill the process in case of infinite loop. Maybe the pd watchdog could be adapted to do that? At the moment it doesn't seem to care if I banged an [until] because the connection to the gui is still functional.

Martin

Quoting Mathieu Bouchard matju@artengine.ca:

On Sun, 23 Dec 2007, Hans-Christoph Steiner wrote:

...

...

e.g. "fail_blabla" will only success if it returns the state "FAIL" immediately or after a "WAIT".

In a binary system, anything that doesn't success would be a
failure. I don't quite get the WAIT state. Do you have an example
of where to use that?

In most tristate electronics, the third state is WAIT, but in Pd, you normally do that by not sending a message: if as binary you'd send a 0 or 1 while running a certain method, and want to introduce a WAIT state, you'd make it not output anything at first, introduce proper [delay], and only later send a 0 or 1 when it's ready. It's as simple as that. If the answer is not going to come, an explicit "wait" message isn't going to disambiguate nor solve that, so it might be a good idea to put some timeout protection *outside* of the tests themselves.

again, this suggestion is based on practical experiences with tests. my first iteration of the framework did it exactly like you just proposed.

in practice i found it often simpler to write tests that only output a
result when they know that they have passed: you need to create far
less logic, which might minimize the chance to write buggy tests
(which i found is inevitable)

likewise, it is often simple to have a shortcut to tell the framework
that the test is known to have failed (most of these shortcuts could
be avoided by splitting the test into several sub-tests; in practice i
found that i prefer to write less tests)

apart from that, a mechanism to quit a test from outside after some
timeout might be a good idea.

fgmad.r IOhannes

This message was sent using IMP, the Internet Messaging Program.

Mathieu Bouchard wrote:

On Mon, 24 Dec 2007, zmoelnig@iem.at wrote:

...

in practice i found it often simpler to write tests that only output a result when they know that they have passed: you need to create far less logic, which might minimize the chance to write buggy tests (which i found is inevitable) likewise, it is often simple to have a shortcut to tell the framework that the test is known to have failed (most of these shortcuts could be avoided by splitting the test into several sub-tests; in practice i found that i prefer to write less tests) apart from that, a mechanism to quit a test from outside after some timeout might be a good idea.

But this is not really what we were talking about. We were thinking about the WAIT state. I sort of assumed that there were both PASS and FAIL results, and this is why I said tri-state, but we were talking about how to represent WAIT in Pd.

yes, this is what i was talking about (and was a bit unclear): a test can only have 2 results (PASS and FAIL), but it can return more states (WAIT,...) in order to communicate to the framework that it needs special handling (e.g. more time)

i shouldn't have confused these 2 things.

fmg.asdr IOhannes

On Sun, 23 Dec 2007, IOhannes m zmoelnig wrote:

...

...

• allow fail-tests which have to return FAIL-state in order to success

How do you know that a method has failed properly?

because it returns the expected FAIL-state.

Why do you need this instead of having a test PASS when the test checks that the error happened correctly?

the problem is the same as "how do you know that a method has succeeded properly"

I think that pretty much anyone doing automated tests so far agree that it's better to have just all tests be written in the positive way, so that you don't have to ever exchange the meanings of PASS and FAIL.

you need a-priori knowledge about the expected return state of the test. i usually do this by a discrimination based on the test-name. (prefixing "fail") e.g. "fail_blabla" will only success if it returns the state "FAIL" immediately or after a "WAIT".

I believe that this is definitely not a useful feature. I thought you had something for helping error handling, but the way you describe it now, it sounds like it doesn't actually help in any way.

_ _ __ ___ _____ ________ _____________ _____________________ ... | Mathieu Bouchard - tél:+1.514.383.3801, Montréal QC Canada

On Mon, 24 Dec 2007, zmoelnig@iem.at wrote:

Quoting Mathieu Bouchard matju@artengine.ca:

...

I think that pretty much anyone doing automated tests so far agree that it's better to have just all tests be written in the positive way, so that you don't have to ever exchange the meanings of PASS and FAIL.

i needed it to test the framework itself. i believe that this is important. (and i am doing automated tests and do not agree that just all tests ought to be written in a positive way) my proposals where not about a theoretic test-framework, but based in my practical experiences with an automated test-framework. i that i learned quite a lot about tests....

Oh, there are lot of very practical people doing very practical testing and never ever using fail-tests. Actually, you're the first one that ever told me about negative tests. Anyone else would test for an error like this:

def infinite_recursion infinite_recursion end

def test_infinite_recursion begin infinite_recursion rescue Exception => this_error return this_error == SystemStackError end return false end

So they would check that infinite_recursion causes an error, and then check whether it's the correct error, and then return true (PASS) if it does, and if it's the wrong error, return false (FAIL), and likewise if infinite_recursion ever returned. They don't ever return false to mean true and true to mean false.

I still don't know how you got there. Good for you that you found it from experience. I hope that it was a good experience, but if you keep it that much secret, there's no way I can relate to that experience.

_ _ __ ___ _____ ________ _____________ _____________________ ... | Mathieu Bouchard - tél:+1.514.383.3801, Montréal QC Canada

On Sun, 30 Dec 2007, IOhannes m zmoelnig wrote:

the implies a 2nd-order framework that evaluates the results of the 1st-order framework. i am not a cybernetic, so i don't know how to solve the problem of testing the 2nd-order framework.

Make the whole framework instantiable. Then you can instantiate the framework from within the framework, clearly distinguishing the testing framework that is testing, from the testing framework that is being tested. You need to make sure that all output of the nested framework can be trapped, so, no direct [print]s.

_ _ __ ___ _____ ________ _____________ _____________________ ... | Mathieu Bouchard - tél:+1.514.383.3801, Montréal QC Canada

On Dec 10, 2007, at 2:19 PM, Mike McGonagle wrote:

On 12/10/07, Jamie Bullock jamie@postlude.co.uk wrote: PREPARE uses the '$' character as its placeholder identifier, and '$1'

Jamie, after thinking about this, I don't think we should worry
about the specifics of the characters that we use, as long as we
can agree on what gets entered into a PD message box, the specific
instance of the external ('sqlite', 'psql', etc.) can handle the
conversion to what the specific database uses.

From what I have seen on this thread, is it right that the '?'
syntax does NOT have an identifier following it? I think it would
be very helpful if we use identifiers in the SQL if only to make
the binding to the proper datatypes easier, as per my last email.

It seems many things use the "?" alone as the unnamed placeholder,
but I only saw MySQL using ?name for named placeholders. It does
seem like a nicer syntax rather than using a different character.
Perhaps, we should follow pgsql and use "$" and "$name", since "$" is
already a marker for replacement vars in Pd. I suppose that could
get confusing in something like:

SELECT id, ABS(($duration - $1)/$2) AS error FROM datatable ORDER BY
error LIMIT 1

.hc

Mike

-- Peace may sound simple—one beautiful word— but it requires
everything we have, every quality, every strength, every dream,
every high ideal. —Yehudi Menuhin (1916–1999), musician _______________________________________________ PD-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/ listinfo/pd-list

If nature has made any one thing less susceptible than all others of
exclusive property, it is the action of the thinking power called an
idea, which an individual may exclusively possess as long as he keeps
it to himself; but the moment it is divulged, it forces itself into
the possession of everyone, and the receiver cannot dispossess
himself of it. - Thomas Jefferson

On Dec 11, 2007, at 3:59 AM, Jamie Bullock wrote:

On Mon, 2007-12-10 at 18:07 -0500, Hans-Christoph Steiner wrote:

...

...

It seems many things use the "?" alone as the unnamed placeholder,
but I only saw MySQL using ?name for named placeholders. It does seem like a nicer syntax rather than using a different character.
Perhaps, we should follow pgsql and use "$" and "$name", since "$" is
already a marker for replacement vars in Pd. I suppose that could get
confusing in something like:

SELECT id, ABS(($duration - $1)/$2) AS error FROM datatable ORDER BY error LIMIT 1

I vote that we use "?" as a placeholder, and only support unnamed placeholders. We can use this to convert to whatever the db-specific placeholder system is inside the external.

I am ok with using ?name for placeholders, but I think we should also
support the plain ? with the list input. I am fine if that happens
later tho. I'd like to get something working soon, we've done a lot
of talking :).

.hc

Using ReBirth is like trying to play an 808 with a long stick. - David Zicarelli

On Dec 12, 2007, at 11:52 AM, Jamie Bullock wrote:

On Tue, 2007-12-11 at 15:48 -0500, Hans-Christoph Steiner wrote:

...

On Dec 11, 2007, at 3:59 AM, Jamie Bullock wrote:

...

On Mon, 2007-12-10 at 18:07 -0500, Hans-Christoph Steiner wrote:

...

...

It seems many things use the "?" alone as the unnamed placeholder, but I only saw MySQL using ?name for named placeholders. It does seem like a nicer syntax rather than using a different character. Perhaps, we should follow pgsql and use "$" and "$name", since "$" is already a marker for replacement vars in Pd. I suppose that could get confusing in something like:

SELECT id, ABS(($duration - $1)/$2) AS error FROM datatable
ORDER BY error LIMIT 1

I vote that we use "?" as a placeholder, and only support unnamed placeholders. We can use this to convert to whatever the db-specific placeholder system is inside the external.

I am ok with using ?name for placeholders, but I think we should also support the plain ? with the list input. I am fine if that happens later tho. I'd like to get something working soon, we've done a lot of talking :).

I agree about getting on with it. However, I probably won't be able to commit any coding time to [psql] until after Christmas. This might be sensible though - I suggest we get the interface to one of the objects 'right' first and then copy this in the others. Mike seems to be
pretty active on the SQLite one, so I suggest we let that take the lead and follow that.

Sounds like a plan :)

.hc

Looking at things from a more basic level, you can come up with a
more direct solution... It may sound small in theory, but it in
practice, it can change entire economies. - Amy Smith

On Dec 10, 2007, at 2:55 PM, Mathieu Bouchard wrote:

On Mon, 10 Dec 2007, Jamie Bullock wrote:

...

On Mon, 2007-12-10 at 12:18 -0500, Mathieu Bouchard wrote:

...

This almost *never* happens.

I take your point, but I think you are exaggerating slightly.

Not in this context. We're only looking at the average query; I
don't even mean an actual query, but statistics about queries. If
only 10% of them don't have arguments, then unless there's a huge
difference in object count (which there is not), it doesn't make
much of a difference on the total object count. And that is only
supposing that you need something like an extra [bang] before your
sql query to prevent [psql] from getting non-bang messages.

...

I just looked at the code for a database driven web app I worked
on recently. It uses 30 queries, 3 of which have no variables. Out
of these three only one doesn't 'overlap' with other queries and
therefore can't be refactored. Maybe 1 in 30 is almost never?

Yes. to contrast this, 3% wouldn't be "almost never" if, for
example, the issue was whether to support something at all, or not.
In that case, if the cost of a workaround is 50x the wanted
feature, or if a workaround is impossible, it can weigh a lot in
the design priorities.

...

Good point! I think Hans' recent suggestion addresses the problem. It also occurs to me that for Postgres at least, we have the PREPARE statement, which addresses the optimisation and injection issues you have raised. Technically [psql] already supports PREPARE except that PREPARE uses the '$' character as its placeholder identifier, and
'$1' can't be passed around as a symbol in Pd. I think it might be interesting to use the '?' notation currently under discussion as an interface to PREPARE though.

The '?' notation is either MySQL-specific or PerlDBI-specific or
both. By PerlDBI I mean any database interface (in any language)
following's Perl DBI package closely enough. I guess that if you
use DBI-over-Postgres, then it replaces all ? by $ automatically.

The other somewhat common style that I saw in my searches was printf
patterns (%s, %f, etc). In Pd, [makefilename], [makesymbol],
[sprintf], and perhaps others use this syntax. The single ? notation
seems to be supported by at least these, if you want to call that
"specific": Qt, PerlDBI, Perl's DBD::Pg, RubyDBI, PHP PDO, Java
JDBC, MySQL, Oracle.

I think it is quite important to reuse existing syntax rather than
introducing new syntax. Minimal syntax is really one of Pd's biggest
strengths. Since these lines would be pure SQL, I think it would be
appropriate to use a common SQL syntax.

I just had a thought, SQL injection relies on being able to send semi- colons in text fields. You can't transmit a semicolon in a message
in Pd, and if you don't provide a means to explicitly send a semi- colon to the query (e.g. [addsemi( to the hot inlet), then no one
will ever be able to send a semi-colon to [sqlite]/[psql]. Pd would
always interpret the semi-colon before the object received it on its
cold inlet. AFAIK, that eliminates basically all of the really bad
SQL injection attacks.

.hc

$ alone can be used in pd as long as it is not followed by a digit,
but I wouldn't encourage that, if it's not used in a [expr] way,
because the use of '$' alone would prevent you from later
supporting an [expr] syntax in a way compatible with yourself.

Using $f1 or $s1 in [expr] style, or perhaps $e1 to mean "any
atom" (e stands for "element"), would be useful, though less
required than the placeholder feature itself.

_ _ __ ___ _____ ________ _____________ _____________________ ... | Mathieu Bouchard - tél:+1.514.383.3801, Montréal QC
Canada_______________________________________________ PD-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/ listinfo/pd-list

                                           http://at.or.at/hans/

On Dec 13, 2007 1:59 PM, Mathieu Bouchard matju@artengine.ca wrote:

On Mon, 10 Dec 2007, Hans-Christoph Steiner wrote:

...

The other somewhat common style that I saw in my searches was printf

patterns

...

(%s, %f, etc). In Pd, [makefilename], [makesymbol], [sprintf], and

perhaps

...

others use this syntax. The single ? notation seems to be supported by

at

...

least these, if you want to call that "specific": Qt, PerlDBI, Perl's DBD::Pg, RubyDBI, PHP PDO, Java JDBC, MySQL, Oracle.

Well, maybe I shouldn't have said "specific", but when I look at any PHP code that I find, it seems that they haven't discovered what's a placeholder yet, for example. So, it seems that it's not so universal.

As someone who has never really used Placeholders, the only sorts of things that I can see them being useful for are when you need to do a lot of inserts or deletes, or for other statements that will be executed repeatedly. From what I am gathering by these discussions is that the useage of placeholders allows the SQL statement to be "compiled" and then with each execution of the statement, the values of the placeholders are substituted.

This might be one reason you don't see them all that often in PHP, I would imagine that PHP doesn't really do a whole bunch of repetitive stuff.

...

I think it is quite important to reuse existing syntax rather than introducing new syntax. Minimal syntax is really one of Pd's biggest strengths. Since these lines would be pure SQL, I think it would be appropriate to use a common SQL syntax.

If you wanted to reuse existing Pd syntax, you could abstract out SQL syntax completely and make a database interface that fully feels like Pd. The Rails web framework has something like that.

I don't know about you guys, but my original goal on this was to basically allow a user to input SQL and it would return the result sets. I just wanted to keep it simple. But I can see a use for using Placeholders, especially when you have a lot of data to store (and it also kind of "vindicates" my original idea of putting the SQL directly in the creation args... [wink, wink, nudge...]).

This idea of doing this to make this more PD-like I think would be a waste of time, as SQL is pretty simple and a LOT of people already know it. Why create another "language"?

...

I just had a thought, SQL injection relies on being able to send

semi-colons

...

in text fields.

This is not true. I have already posted an example in this thread on how to delete a whole table using SQL injection without a semicolon.

At the same time, should our external be on the look out for these sorts of things? One of the original ideas was to not give the external any, if at all, knowledge of SQL. Meaning, it wouldn't "parse" the SQL, nor would it try to do any generation of SQL. It just expects that the user is HONEST (that is what these concerns over Injection are, right), and the SQL they entered is what they meant.

These things being said, I am not adverse to the new design model, and I hope to get something up over the weekend.

While we can try to protect against various things, those that want to be malicious will do so anyway.

Of course, we could eliminate these problems altogether, and just use an embedded database ONLY... (just kidding...)

Mike

On Thu, 13 Dec 2007, Mike McGonagle wrote:

As someone who has never really used Placeholders, the only sorts of things that I can see them being useful for are when you need to do a lot of inserts or deletes, or for other statements that will be executed repeatedly. From what I am gathering by these discussions is that the useage of placeholders allows the SQL statement to be "compiled" and then with each execution of the statement, the values of the placeholders are substituted.

You seem to think of placeholders as something extra that you necessarily add for improving the speed because inserting text within SQL seems perfectly normal. This is not the case. From another perspective, everything that can be abstracted out ought to be abstracted out, and that means that you don't want to touch SQL syntax with your data, you want to think of a SQL statement as a function that you call in your main programming language, you don't want to have to do type conversions and quoting because that's an aspect of SQL and not of your main programming language.

This might be one reason you don't see them all that often in PHP, I would imagine that PHP doesn't really do a whole bunch of repetitive stuff.

So far, all successful attacks on my server has been through PHP scripts that don't quote things properly. For example, embedding text directly into email headers without regard for email syntax.

This idea of doing this to make this more PD-like I think would be a waste of time, as SQL is pretty simple and a LOT of people already know it. Why create another "language"?

Because that other language would be more integrated with pd itself and integration is good, if it really makes things feel more familiar. You can witness the importance of this by looking at how much people avoid using other programming languages with pd (beyond the difficulties in compiling externals and distributing patches that use code like that).

The big downside of integration is that it usually does not cover the whole feature set of what is being wrapped, and especially not nonstandard extensions.

At the same time, should our external be on the look out for these sorts of things? One of the original ideas was to not give the external any, if at all, knowledge of SQL. Meaning, it wouldn't "parse" the SQL, nor would it try to do any generation of SQL. It just expects that the user is HONEST (that is what these concerns over Injection are, right), and the SQL they entered is what they meant.

You can't assume that the user is honest, because when you do, you also assume that the user does not make mistakes when including input from other sources that can't be assumed to be honest.

While we can try to protect against various things, those that want to be malicious will do so anyway.

This is not true. Every step is important in making it more difficult for abuse to happen.

_ _ __ ___ _____ ________ _____________ _____________________ ... | Mathieu Bouchard - tél:+1.514.383.3801, Montréal QC Canada

On Fri, 21 Dec 2007, Mike McGonagle wrote:

Well, this is one of those reasons why I am starting with using SQLite, I think it would be much easier than working with a networked database. And if someone is being 'dishonest', then they are only effecting themselves.

When we're talking about networking vs databases, it's not about the database connection being over TCP. We're talking about input that comes from non-trusted people and goes to a patch made by trusted people. It doesn't matter much whether the DB engine is just a library or also a daemon.

...

While we can try to protect against various things, those that want to

...

be malicious will do so anyway.

This is not true. Every step is important in making it more difficult for abuse to happen.

Well, if we are allowing these people to construct their own SQL and they are building their own stuff, just how can we stop them from being malicious? I mean, the only real way to stop this completely, is to not produce a network version of this, and only deal with standalone.

I'm not going to continue talking about this topic. I wish you good luck.

_ _ __ ___ _____ ________ _____________ _____________________ ... | Mathieu Bouchard - tél:+1.514.383.3801, Montréal QC Canada

Hi Hans,

I like this idea of a hot and cold inlet, and I can see where you are going with making the database access objects more pd-like. I also agree that the sql/sqlend delimiters are a rather inelegant workaround for Pd's lack of a string type. What you suggest shouldn't be too hard to implement either.

I have a few specific comments (see below).

On Fri, 2007-12-07 at 21:37 -0500, Hans-Christoph Steiner wrote:

That is done by sending the [submit( message to the hot inlet. Or do you mean having multiple SQL calls separated by semi-colons? If you wanted to add semicolons, there would have to be a special message, I think we could just reuse the "addsemi", "addcomma", "adddollar" messages from message boxes.

This feels like another workaround, but I suppose it's more idiomatic pd than 'sqlend'.

As far as I know, the semi-colon at the end of the statement in SQL triggers the execution of that statement, so I can't see an advantage to having multiple, semi-colon terminated statements in a single message box. Does it change how the SQL is executed if they are submitted at the same time?

No.

...

The other thing I think that would have to be is forcing all input SQL to go into the second inlet. Allowing some things to go into the hot inlet, while others are required to go into the second outlet would make it difficult to use because you would have to remember which type of statement can go where...

Any SQL statement would be allowed on the cold inlet. Because of the limitations of Pd (no escape mechanism), and the fact that commas already have a meaning in Pd messages, the hot inlet would not be able to handle commas (unless someone comes up with something quite clever).

I think we should either come up with something quite clever (using regex springs to mind), or not allow queries to be submitted to the hot inlet at all. Having a hot inlet that doesn't quite handle queries in the same way as the cold inlet is just confusing.

Another suggestion: perhaps it would be even more idiomatic to use "bang" instead of "submit" as the 'quert send' message?

Jamie

Ok, so I spent the weekend just trying to figure out if what Hans described is POSSIBLE at all... And NO, it is not possible in Plain Vanilla PD. You can't create a second inlet that will take arbitrary selectors. According to the 'pd-external-HOWTO.pdf' (by IOhannes), page 32; "It is not possible to addmethods for more than one select to a right inlet. Particularly it is not possible to add a universal method for arbitrary selectors to a right inlet."

And this is exactly what Hans is proposing. The 'cold' inlet would need to recognize arbitrary selectors. I don't know if it is possible to create a second inlet that takes a list, as there is no function call to add a list inlet.

The only way that I know to do that is to use Flext, and quite frankly, this is one of the things that I wanted to avoid in the first place. If we are forced to use Flext, then why not just get 'pool' running and be done with this mess...

Also, I don't think the idea of having other "hot" messages, like [use <database>( should be made into their own selectors, as ALL SQL should be processed in the same way, and no special cases should be created. I think it would be difficult for a user to know what SQL goes where, there may only be a few of them, but it still makes special cases out of those bits of SQL. The only selectors the external should recognize are ones that are for controlling the external itself.

So, if anyone has any ideas, I think I am just going to continue with what I have, you can use it if you want, but quite frankly, this is very frustrating when a suggestion like this comes from one of the lead developers, and it turns out not to be possible. I am not really thrilled about using Flext, as it is not part of the Extended build, and it is not part of Vanilla-PD (also, I have never been able to get anything to compile on my machine with Flext, not so say I put forth a huge effort, as I like things to be as simple as possible). I didn't want to force a user to have to download and install any additional packages (which include not just PD, but an extra database).

Or am I just freaking out over nothing.

Mike

On 12/9/07, Hans-Christoph Steiner hans@eds.org wrote:

The "addcomma", "addsemi", "adddollar" messages are a workaround, for sure, but yeah, I suppose a more pd-ish one. I think that if SQL gets submitted only on the right/cold inlet, then we would not need those messages. They might come in handy though, so they could be added later.

"bang" for submit sounds good too.

As for clever hacks, I can't think of any with regexs too. Plus from my experience, clever hacks can often lead to really strange and difficult bugs. IMHO, we'd better off with something simple that might be a bit harder to start on, but doesn't have odd conditions that could trigger bugs.

.hc

On Dec 9, 2007, at 9:26 AM, Jamie Bullock wrote:

...

Hi Hans,

I like this idea of a hot and cold inlet, and I can see where you are going with making the database access objects more pd-like. I also agree that the sql/sqlend delimiters are a rather inelegant workaround for Pd's lack of a string type. What you suggest shouldn't be too hard to implement either.

I have a few specific comments (see below).

On Fri, 2007-12-07 at 21:37 -0500, Hans-Christoph Steiner wrote:

...

That is done by sending the [submit( message to the hot inlet. Or do you mean having multiple SQL calls separated by semi-colons? If you wanted to add semicolons, there would have to be a special message, I think we could just reuse the "addsemi", "addcomma", "adddollar" messages from message boxes.

This feels like another workaround, but I suppose it's more idiomatic pd than 'sqlend'.

...

...

As far as I know, the semi-colon at the end of the statement in SQL triggers the execution of that statement, so I can't see an advantage to having multiple, semi-colon terminated statements in a single message box. Does it change how the SQL is executed if they are submitted at the same time?

No.

...

...

The other thing I think that would have to be is forcing all input SQL to go into the second inlet. Allowing some things to go into the hot inlet, while others are required to go into the second outlet would make it difficult to use because you would have to remember which type of statement can go where...

Any SQL statement would be allowed on the cold inlet. Because of the limitations of Pd (no escape mechanism), and the fact that commas already have a meaning in Pd messages, the hot inlet would not be able to handle commas (unless someone comes up with something quite clever).

I think we should either come up with something quite clever (using regex springs to mind), or not allow queries to be submitted to the hot inlet at all. Having a hot inlet that doesn't quite handle queries in the same way as the cold inlet is just confusing.

Another suggestion: perhaps it would be even more idiomatic to use "bang" instead of "submit" as the 'quert send' message?

Jamie

-- www.postlude.co.uk

---

---

As we enjoy great advantages from inventions of others, we should be glad of an opportunity to serve others by any invention of ours; and this we should do freely and generously. - Benjamin Franklin

Thomas, don't get me wrong, I am quite certain that you have written a very useful product. My only trouble is that I have never been able to get anything to compile that is using it. Over the weekend, I tried to compile Plain Vanilla PD, and that was a pain in the ass, too, and when I was done, it didn't work. One of the things that I have heard mention is that it would be nice to have flext as part of the Extended build. That would be a good thing, as we could take advantage of all the externals that use flext, without having to compile them ourselves. Also, it is not part of the Plain Vanilla PD, so that makes it suspect to forcing the user to integrate it themselves.

Could you point out where in the source for flext that allows the inclusion of lists to a 'cold' inlet, and that allows the use of arbitrary selectors?

Thanks,

Mike

On 12/9/07, Thomas Grill gr@grrrr.org wrote:

Hi Mike, it's unfortunate that you are not thrilled using flext. Since flext uses the PD api to get lists or other messages into secondary inlets there's nothing to stop you from doing the same in psql. The magic words are "proxy objects" which you might find in the mailing list archive (or the flext source code, if you prefer that). Needless to say flext saves you from coding these things yourself in a time consuming manner and makes other things really easy, once you got it running. You could also compile pool then.

gr~~~

Mike McGonagle schrieb:

...

Ok, so I spent the weekend just trying to figure out if what Hans described is POSSIBLE at all... And NO, it is not possible in Plain Vanilla PD. You can't create a second inlet that will take arbitrary selectors. According to the ' pd-external-HOWTO.pdf' (by IOhannes), page 32;

"It is not possible to addmethods for more than one select to a right inlet. Particularly it is not possible to add a universal method for arbitrary selectors to a right inlet."

And this is exactly what Hans is proposing. The 'cold' inlet would need to recognize arbitrary selectors. I don't know if it is possible to create a second inlet that takes a list, as there is no function call to add a list inlet.

The only way that I know to do that is to use Flext, and quite frankly, this is one of the things that I wanted to avoid in the first place. If we are forced to use Flext, then why not just get 'pool' running and be done with this mess...

Also, I don't think the idea of having other "hot" messages, like [use <database>( should be made into their own selectors, as ALL SQL should be processed in the same way, and no special cases should be created. I think it would be difficult for a user to know what SQL goes where, there may only be a few of them, but it still makes special cases out of those bits of SQL. The only selectors the external should recognize are ones that are for controlling the external itself.

So, if anyone has any ideas, I think I am just going to continue with what I have, you can use it if you want, but quite frankly, this is very frustrating when a suggestion like this comes from one of the lead developers, and it turns out not to be possible. I am not really thrilled about using Flext, as it is not part of the Extended build, and it is not part of Vanilla-PD (also, I have never been able to get anything to compile on my machine with Flext, not so say I put forth a huge effort, as I like things to be as simple as possible). I didn't want to force a user to have to download and install any additional packages (which include not just PD, but an extra database).

Or am I just freaking out over nothing.

Mike

On 12/9/07, *Hans-Christoph Steiner* <hans@eds.org mailto:hans@eds.org> wrote:

The "addcomma", "addsemi", "adddollar" messages are a workaround,

for

...

sure, but yeah, I suppose a more pd-ish one.  I think that if SQL
gets submitted only on the right/cold inlet, then we would not need
those messages.  They might come in handy though, so they could be
added later.

"bang" for submit sounds good too.

As for clever hacks, I can't think of any with regexs too.  Plus

from

...

my experience, clever hacks can often lead to really strange and
difficult bugs.  IMHO, we'd better off with something simple that
might be a bit harder to start on, but doesn't have odd conditions
that could trigger bugs.

.hc


On Dec 9, 2007, at 9:26 AM, Jamie Bullock wrote:

>
> Hi Hans,
>
> I like this idea of a hot and cold inlet, and I can see where
you are
> going with making the database access objects more pd-like. I also
> agree
> that the sql/sqlend delimiters are a rather inelegant workaround

for

...

> Pd's lack of a string type. What you suggest shouldn't be too
hard to
> implement either.
>
> I have a few specific comments (see below).
>
> On Fri, 2007-12-07 at 21:37 -0500, Hans-Christoph Steiner wrote:
>> That is done by sending the [submit( message to the hot
inlet.  Or do
>> you mean having multiple SQL calls separated by
semi-colons?  If you
>> wanted to add semicolons, there would have to be a special
message, I
>> think we could just reuse the "addsemi", "addcomma", "adddollar"
>> messages from message boxes.
>
> This feels like another workaround, but I suppose it's more
> idiomatic pd
> than 'sqlend'.



>> As far as I know, the semi-colon at the end of the statement in
SQL
>> triggers the execution of that statement, so I can't see an
advantage
>> to having multiple, semi-colon terminated statements in a single
>> message box.  Does it change how the SQL is executed if they are
>> submitted at the same time?
>
> No.
>
>>>
>>> The other thing I think that would have to be is forcing all

input

...

>>> SQL to go into the second inlet. Allowing some things to go
into the
>>> hot inlet, while others are required to go into the second

outlet

...

>>> would make it difficult to use because you would have to

remember

...

>>> which type of statement can go where...
>>
>>
>> Any SQL statement would be allowed on the cold inlet.  Because
of the
>> limitations of Pd (no escape mechanism), and the fact that commas
>> already have a meaning in Pd messages, the hot inlet would not be
>> able
>> to handle commas (unless someone comes up with something quite
>> clever).
>
> I think we should either come up with something quite clever

(using

...

> regex springs to mind), or not allow queries to be submitted to

the

...

> hot
> inlet at all. Having a hot inlet that doesn't quite handle
queries in
> the same way as the cold inlet is just confusing.
>
> Another suggestion: perhaps it would be even more idiomatic to use
> "bang" instead of "submit" as the 'quert send' message?
>
> Jamie
>
> --
> www.postlude.co.uk <http://www.postlude.co.uk>

---

...

----

As we enjoy great advantages from inventions of others, we should be
glad of an opportunity to serve others by any invention of ours; and
this we should do freely and generously.         - Benjamin Franklin

-- Peace may sound simple—one beautiful word— but it requires everything we have, every quality, every strength, every dream, every high ideal. —Yehudi Menuhin (1916–1999), musician

---

---

PD-list@iem.at mailing list UNSUBSCRIBE and account-management ->

http://lists.puredata.info/listinfo/pd-list

...

On 12/9/07, Thomas Grill gr@grrrr.org wrote:

Mike McGonagle schrieb: I know and it has been on my todo list for a long time. The point is that it hasn't made it to the top of this list since i don't need it. On the other hand it doesn't seem so important in general that any other person has cared to do it although i consider it a trivial task and i offered my support for any questions that might arise.

Well, this is not something that is within reach of many people. We need to understand not just PD's internals, but how flext uses them, and we also need to know how to integrate these things with the extended nightly builds. Not quite certain how many people know ALL of these things. Probably only a handful, and it would appear that none of them are interested in getting flext working.

Could you point out where in the source for flext that allows the

...

inclusion of lists to a 'cold' inlet, and that allows the use of arbitrary selectors?

I'm not sure if the flext code is easy to read because it handles Max and PD in the same file, but it is in flout.cpp (method flext_base::InitInlets ) and flproxy.cpp. There must be other externals using proxy inlets, probably zexy or other iem-based stuff.

Thanks for the pointers, I will check them out.

Does anyone know of any objects that are built without flext that use arbitrary selectors on a cold inlet? I would be more than happy to look at that code and see how this is done.

I also downloaded the latest version of flext, and will try again to see if I can get this working. That being said, if we do go the route of using flext, then we should decide now if we should continue developing an SQL interface, or just use 'pool'.

Mike

gr~~~

On 12/9/07, Mathieu Bouchard matju@artengine.ca wrote:

On Sun, 9 Dec 2007, Mike McGonagle wrote:

...

Well, this is not something that is within reach of many people. We need to understand not just PD's internals, but how flext uses them, and we also need to know how to integrate these things with the extended nightly builds. Not quite certain how many people know ALL of these things. Probably only a handful, and it would appear that none of them are interested in getting flext working.

GridFlow also uses proxy inlets. There are simpler examples of proxy inlets within pd itself. E.g. all [list] objects have it.

Thank you very much. I didn't think about 'list' objects, even though I have been using them more and more...

Great, now I have a place to look.

Basically you need to create another object that will be the "slave" of

your main object, and then create an inlet that will use that object. The second argument of inlet_new is for that: the inlet appears on the first argument's box, but messages will be sent to the second argument's object. In that case you pass 0,0 for the other arguments (which are usually symbols, but here are null pointers to mean "not applicable")

_ _ __ ___ _____ ________ _____________ _____________________ ... | Mathieu Bouchard - tél:+1.514.383.3801, Montréal QC Canada

Hans, I still have yet to learn about the WIKI stuff, but I posted a Inlet Proxy example on my page ( http://puredata.info/Members/mjmogo ). I don't know if you want to just put a link, or move the file to the Proxy page you created, either way works for me. Thanks,

Mike

On Dec 10, 2007 10:05 AM, Hans-Christoph Steiner hans@eds.org wrote:

On Dec 9, 2007, at 9:53 PM, Mathieu Bouchard wrote:

...

On Sun, 9 Dec 2007, Mike McGonagle wrote:

...

Well, this is not something that is within reach of many people. We need to understand not just PD's internals, but how flext uses them, and we also need to know how to integrate these things with the extended nightly builds. Not quite certain how many people know ALL of these things. Probably only a handful, and it would appear that none of them are interested in getting flext working.

GridFlow also uses proxy inlets. There are simpler examples of proxy inlets within pd itself. E.g. all [list] objects have it.

Basically you need to create another object that will be the "slave" of your main object, and then create an inlet that will use that object. The second argument of inlet_new is for that: the inlet appears on the first argument's box, but messages will be sent to the second argument's object. In that case you pass 0,0 for the other arguments (which are usually symbols, but here are null pointers to mean "not applicable")

I started a pdpedia page on the topic, please add anything useful:

http://wiki.puredata.info/en/proxy_objects

.hc

---

---

I spent 33 years and four months in active military service and during that period I spent most of my time as a high class muscle man for Big Business, for Wall Street and the bankers. - General Smedley Butler

On Mon, 10 Dec 2007, Hans-Christoph Steiner wrote:

I started a pdpedia page on the topic, please add anything useful: http://wiki.puredata.info/en/proxy_objects

You copied from my email. Emails are property of the sender unless otherwise specified. For publicly visible emails it doesn't seem like a big issue, but you can't claim a GNU FDL license on content that you don't own.

That said, I agree with putting any information from me from pd-list on the wiki unless I specify otherwise, but I want you to know that in theory you're supposed to ask and in practice it would be better to ask.

_ _ __ ___ _____ ________ _____________ _____________________ ... | Mathieu Bouchard - tél:+1.514.383.3801, Montréal QC Canada

Mike McGonagle schrieb:

On 12/9/07, *Thomas Grill* <gr@grrrr.org mailto:gr@grrrr.org> wrote:

Mike McGonagle schrieb:
I know and it has been on my todo list for a long time. The point is
that it hasn't made it to the top of this list since i don't need it.
On the other hand it doesn't seem so important in general that any
other
person has cared to do it although i consider it a trivial task and i
offered my support for any questions that might arise.

Well, this is not something that is within reach of many people. We need to understand not just PD's internals, but how flext uses them, and we also need to know how to integrate these things with the extended nightly builds. Not quite certain how many people know ALL of these things. Probably only a handful, and it would appear that none of them are interested in getting flext working.

It's not like that. It's certainly not a precondition to understand the inner workings of pd, flext or the operating system to compile an external. Integrating flext or anything else into the pd-extended build system, needs knowledge of the pd-extended build system and not much more. It's absolutely not necessary to understand flext to do this integration. I don't have sufficient knowledge of the pd-extended build system and i'm not going to spend time on dealing with it, but whoever is interested in the integration can dive into it, write the necessary make files (i could even provide skeletons for this) and stuff it into the build system in the usual free-style manner.

I also downloaded the latest version of flext, and will try again to see if I can get this working. That being said, if we do go the route of using flext, then we should decide now if we should continue developing an SQL interface, or just use 'pool'.

I would be very interested what exactly isn't working with flext for you. I just sshd into linux, checked out flext, pool and pd from cvs and compiled flext and pool without a single warning. I did exactly as described in the readme.txt files and it took about 2 minutes. I have heard "it doesn't work" many many times, but it's completely non-constructive to say that without saying why or what.

gr~~~

Thomas Grill wrote:

There must be other externals using proxy inlets, probably zexy or other iem-based stuff.

right: e.g. zexy's [mulitplex] (aka [mux]) uses proxy-inlets.

there are 2 reasons why it is not in the externals-howto: #1 it would have complicated the howto needlessly #2 when i wrote the howto, i had no idea about proxy inlets and/or how to use them.

having said that, i think it is usually a bad idea to all right-hand inlets accept arbitrary messages for _most_ objects (not for all). a message with a (nonstandard) "selector" selects a method of the object to be called and makes this explicit. e.g. it tells the object to do something (and eventually provides auxiliary data, to complete this task). if you are just passing around data to be collected by the object, there is already a certain type of message to do so: "list". it is possible to create a right-hand "list" inlet using the methods described in the externals-howto (iirc :-))

while i basically agree with matju, that a consistent use of the data-flow paradigm should not be made the holy grail (if we don't want to end up fighting green knights), i do think that we should bear data-flow (as opposed to control-flow) in mind and design objects accordingly.

mfg.adsf IOhannes

hi list !

...

...

more pd muzik >> drum n' drones

nvisible.taz >> features >>> segmentation fault

...

...

programming and muzik noish~+daax!

enjoy :P

:: ..: ::: ::: :: :. ... . :: :: . : : :. : . . : . ::....

/*segmentation fault*/* *is a research xperimental audio-software project, based on the fusion of drones and percutive broken rythms, programmed in Pd-patches. /*sf*/ emerges from the work-in-progress in the 'nerd-core' underground of barcelona, in openlabs like 'hangar' or 'csoa la fibra'. Spontanously we mixed the patches in two xperimental-jam-sessions first at HacklabAV@Nitcasm http://hangar.org/nitcasm/ (centre Santa monica), and after at Distorsion Festival (La Fibra) http://delcorp.org/distorsionfest/?page_id=18

patching resources >> granular synthesis, asymetric sequencer, phasevocoder, physycalmodelling applied to AudioFX, Fibonacci sequencer, and lots of noise.

The 3 tracks that you can download at: http://pitchvolley.com/nvisible.taz/segmentationfault.html

are recorded and composed in live, letting flow the freequencies and feed'bugs',..all trakks are made under UbuntuStudio Gnu/linux, and licensed in cc.

kontakkt >> isnotoursegmentationfault_at_gmail_dot_com http://pitchvolley.com/nvisible.taz/segmentationfault.html

.::: : : :: : :: :: : . . .. :::: : :: : : : :: : ... ..

PD-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list

Mathieu Bouchard wrote:

On Mon, 10 Dec 2007, IOhannes m zmoelnig wrote:

...

i do think that we should bear data-flow (as opposed to control-flow) in mind and design objects accordingly.

So do you want us to consider Gem as the bad example?

i don't claim that Gem is a good example. however, i also don't see how the data-flow vs control-flow is especially bad in Gem.

mfgasdr IOhannes

On Thu, Dec 13, 2007 at 03:12:18PM -0500, Mathieu Bouchard wrote:

On Tue, 11 Dec 2007, IOhannes m zmoelnig wrote:

...

i don't claim that Gem is a good example. however, i also don't see how the data-flow vs control-flow is especially bad in Gem.

As long as you pass a "gem" message around that is only a pointer to a shared state that all objects modify, it's all explicit control-flow all over the place. The contents of the "gem" messages doesn't matter at all, and the only thing that matters is the order in which the messages are sent. That's 100% controlflow and 0% dataflow.

It would be way cool if gem was truly dataflow, with the [cube] or another geometry source at the top of the stack and then geometry/colour/texture modifiers all the way down until a [render] object. Imagine doing audio style filtering on geometry streams.

One can dream I guess.

Best,

Chris.

PS This is not a criticism of Mark, IOhannes, Chris's work on Gem - it's a great library and I love using it! Thanks for all your hard work.

http://mccormick.cx

On Fri, 14 Dec 2007, Hans-Christoph Steiner wrote:

On Dec 13, 2007, at 8:31 PM, Chris McCormick wrote:

...

One can dream I guess.

I think that GEM is dataflow-ish, but it is representing OpenGL render chains rather than the physical shapes they generate...

There's no way to stretch any definition of dataflow to call GEM dataflow-ish. Execution order itself is not dataflow. If you just use bang (or equivalently just the "gem" message) then there's no dataflow aspect involved, there is only execution order that is important.

Chris means dreaming about making GEM more dataflowish, not just pretending that it is.

_ _ __ ___ _____ ________ _____________ _____________________ ... | Mathieu Bouchard - tél:+1.514.383.3801, Montréal QC Canada

On Fri, Dec 21, 2007 at 01:25:21PM -0500, Mathieu Bouchard wrote:

On Thu, 13 Dec 2007, Chris McCormick wrote:

...

It would be way cool if gem was truly dataflow, with the [cube] or another geometry source at the top of the stack and then geometry/colour/texture modifiers all the way down until a [render] object. Imagine doing audio style filtering on geometry streams. One can dream I guess.

Also imagine doing audio processing with the kind of flow that there is in gemchains. Urgh!

No thanks.

I had a crack at doing true dataflow gem processing - see attached Lua scripts and patches. You'll need Claude's pdlua and luagl installed - load 'geom.obj-test.pd'. [geom.obj] is a lua script to read in a .obj file and then spit out all of the faces as pd [list]s in the form [tri x1 y1 z1 x2 y2 z2 x3 y3 z3(. [render.plain] will take those lists of verticies and render them accordingly - you can replace it with [render.sketch] to see a different type of rendering. It would be possible to insert other things in that chain that manipulated the verticies to change the basic mesh.

Like my previous experiment a couple of years ago doing dataflow gem processing (which was in pure pd using [GEMglVertex3f] etc.), it's too slow to use on objects with lots of faces, but could be fun with small numbers of faces - I'm especially thinking here of doing NPR rendering like toon shading and sketch rendering, and also of mesh warping effects which I'm particularly fond of! Would love to hear if anyone has anything like that going.

Best,

Chris.

http://mccormick.cx

This is what the vertex_array objects basically do. The vertex_model object opens up a .obj and then passes the vertex, normal, texcoord and color data to other vertex_ objects for processing. The speed is pretty good - better than standard geos - but the better route is to use shaders. Shaders allow for more end user manipulation than the vertex_objects could ever offer.

On Dec 22, 2007 8:43 AM, Chris McCormick chris@mccormick.cx wrote:

On Fri, Dec 21, 2007 at 01:25:21PM -0500, Mathieu Bouchard wrote:

...

On Thu, 13 Dec 2007, Chris McCormick wrote:

...

It would be way cool if gem was truly dataflow, with the [cube] or another geometry source at the top of the stack and then geometry/colour/texture modifiers all the way down until a [render] object. Imagine doing audio style filtering on geometry streams. One can dream I guess.

Also imagine doing audio processing with the kind of flow that there is

in

...

gemchains. Urgh!

Forgot the attachment.

Best,

Chris.

---

http://mccormick.cx

---

PD-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list

On Sat, Dec 22, 2007 at 11:26:18AM -0600, chris clepper wrote:

This is what the vertex_array objects basically do. The vertex_model object opens up a .obj and then passes the vertex, normal, texcoord and color data to other vertex_ objects for processing. The speed is pretty good - better than standard geos - but the better route is to use shaders. Shaders allow for more end user manipulation than the vertex_objects could ever offer.

Alright, I'll have to take a closer look at shaders in Gem then. Thanks for the advice. Can anyone point me to a simple vertex manipulation shader I can run in Gem to get started?

Best,

Chris.

http://mccormick.cx

ah, i forgot to mention these tutorials which really helped me out. http://www.lighthouse3d.com/opengl/glsl/index.php?shaders

On Dec 22, 2007, at 8:14 PM, Chris McCormick wrote:

On Sat, Dec 22, 2007 at 11:26:18AM -0600, chris clepper wrote:

...

This is what the vertex_array objects basically do. The
vertex_model object opens up a .obj and then passes the vertex, normal, texcoord and
color data to other vertex_ objects for processing. The speed is pretty good

• better

than standard geos - but the better route is to use shaders.
Shaders allow for more end user manipulation than the vertex_objects could ever
offer.

Alright, I'll have to take a closer look at shaders in Gem then.
Thanks for the advice. Can anyone point me to a simple vertex manipulation shader I can run in Gem to get started?

Best,

Chris.

---

http://mccormick.cx

---

PD-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list

Excellent, it's good to see all this good stuff in Gem starting to
get some recognition! Example patches help so much with this, Jitter
is a good example of the power of examples.

.hc

On Dec 23, 2007, at 4:55 AM, cyrille henry wrote:

Thanks to popular demand, i just commited few shaders exemple in
gem exemple directory.

hope it will help Cyrille

Chris McCormick a écrit :

...

On Sat, Dec 22, 2007 at 11:26:18AM -0600, chris clepper wrote:

...

This is what the vertex_array objects basically do. The
vertex_model object opens up a .obj and then passes the vertex, normal, texcoord and
color data to other vertex_ objects for processing. The speed is pretty
good - better than standard geos - but the better route is to use shaders.
Shaders allow for more end user manipulation than the vertex_objects could ever
offer.

Alright, I'll have to take a closer look at shaders in Gem then.
Thanks for the advice. Can anyone point me to a simple vertex manipulation shader I can run in Gem to get started?

Best,

Chris.

---

http://mccormick.cx

---

PD-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/ listinfo/pd-list

---

GEM-dev mailing list GEM-dev@iem.at http://lists.puredata.info/listinfo/gem-dev

Terrorism is not an enemy. It cannot be defeated. It's a tactic.
It's about as sensible to say we declare war on night attacks and
expect we're going to win that war. We're not going to win the war
on terrorism. - retired U.S. Army general, William Odom

Frank Barknecht schrieb:

But as everyone knows that I don't really understand why a DB external has to be a C/C++ external, I'll better be quiet now. ;)

Right. Over and out

gr~~~

Hallo, Hans-Christoph Steiner hat gesagt: // Hans-Christoph Steiner wrote:

On Dec 10, 2007, at 6:14 AM, Thomas Grill wrote:

...

Frank Barknecht schrieb:

...

But as everyone knows that I don't really understand why a DB external
has to be a C/C++ external, I'll better be quiet now. ;)

Right. Over and out

Basically, only C and Pd externals have proven easy to manage.

lua is a C external (or rather a loader) and it's very easy to manage. Once the lua loader is built for your system (trivial on Linux and probably just as trivial on OS-X and MS-Windows, but I don't have access to these systems) a lua objectclasses is the equivalent to an abstraction, which especially means, that it's just as easily changed and edited by its users without every needing to recompile anything.

Ciao

Frank Barknecht _ ______footils.org__

Hallo, Claude Heiland-Allen hat gesagt: // Claude Heiland-Allen wrote:

Not quite as easy as abstractions - you still have to restart Pd.

Well, yes. But still the lua loader allows users to get the same "power" as developers have without the need to learn and setup all the stuff that's required to compile exernals. That's where the similarity between lua and abstractions is. And as Lua is *designed* as an embedded language, it can be shipped as a tiny binary without any dependencies easily. lua.pd_linux statically built with Lua included is 220 KB. Pd-extended is between 15 and 22 MB, zipped. What's easier to manage?

But who am I telling this? ;)

Ciao

Frank Barknecht _ ______footils.org__

Hallo, Hans-Christoph Steiner hat gesagt: // Hans-Christoph Steiner wrote:

From what I hear from Marius' struggles with pdlua, it's definitely
not trivial, and I would say, I probably wouldn't call it easy
either.

If you read up on that thread, most of it is related to problems building Pd. Also Marius wanted to build OpenGL for Lua, which adds dependencies, of course. But Lua itself doesn't have any dependencies besides ANSI C.

Building Lua on OS-X is just "make macosx", building pdlua then just is something like:

gcc $(CFLAGS) -bundle -undefined suppress -flat_namespace -o lua.pd_darwin lua.c -llua

or whatever the current fashion of compiling C externals on OS-X is. CFLAGS should include the -I and -L lines to your Lua directory of course. Add "-static" and you get a static build that you can copy around without even needing to "install" Lua itself.

I think, compiling an external doesn't get more simple than that.

Ciao

Frank Barknecht _ ______footils.org__

On Mon, 2007-12-10 at 10:39 +0100, Frank Barknecht wrote:

Hallo, Thomas Grill hat gesagt: // Thomas Grill wrote:

...

it's unfortunate that you are not thrilled using flext. Since flext uses the PD api to get lists or other messages into secondary inlets there's nothing to stop you from doing the same in psql. The magic words are "proxy objects" which you might find in the mailing list archive (or the flext source code, if you prefer that). Needless to say flext saves you from coding these things yourself in a time consuming manner and makes other things really easy, once you got it running. You could also compile pool then.

The Python and Lua objects also support arbitrary messages on their inlets. Both languages have DB modules for just about every DB, compilation of these externals only has to be done only once. But as everyone knows that I don't really understand why a DB external has to be a C/C++ external, I'll better be quiet now. ;)

Actually, please don't be quiet! When this was discussed some time ago, you mentioned Python as a solution, and my feeling was that if I write a piece and it uses some database connectivity, I don't really want to have to distribute a Python runtime, relevant Python SQL module, [py] and all the respective dependencies with it. I just (at most) want to include an external, and the databse library it uses. If I write a piece, I can't reasonably expect a performer to resolve all of this themselves - it needs to "just work". I also want to minimise as much as possible the number of discrete components in the system that could possibly go wrong.

However, Lua seems a slightly different story, ([lua] doesn't require flext, and Lua is very lean) - I'm starting to be convinced by it. It might be interesting to recreate [psql] using lua and luasql, and see how it measures up in terms of portability, ease of distribution and performance. I just downloaded [lua], and luasql, and It certainly didn't take long to get these things working on a basic level.

Jamie

On Dec 11, 2007, at 8:29 AM, Frank Barknecht wrote:

Hallo, Jamie Bullock hat gesagt: // Jamie Bullock wrote:

...

Actually, please don't be quiet! When this was discussed some time
ago, you mentioned Python as a solution, and my feeling was that if I
write a piece and it uses some database connectivity, I don't really want to have to distribute a Python runtime, relevant Python SQL module, [py] and all the respective dependencies with it. I just (at most) want to include an external, and the databse library it uses. If I write a piece, I can't reasonably expect a performer to resolve all of this themselves - it needs to "just work". I also want to minimise as
much as possible the number of discrete components in the system that could possibly go wrong.

That's quite sensible. As soon as a DB interface is involved, one already has entered dependency hell, so minimizig further dependencies may be a valid goal. However a dependency on flext is just a dependency at compile time, so it's not really an additional dependency at runtime. A dependency on Python is kind of a dependency at runtime, as you need a possibly large Python installation. A dependency on Lua is also a runtime dependency, however as Lua is small, including it with the binaries will make the runtime dependency not matter.

An immediate advantage of both Python and Lua is, that once the system is set up (easy with Lua, a bit tricky with pyext), the actual object code is just a textfile with a script. Especially as so far it's not even clear, how the SQL class in Pd should behave, this would allow rapid turnaround cycles in development. (For example I posted a working SQL object in Lua some weeks ago, and in Pyext some months ago, that took me about 20 minutes to write, including help-patch. In the case of Lua, I had never written SQL code in that language before. I wasn't so fast because I would be a genius programmer, but because the language is easier and no compilation is involved.)

For a general purpose DB interface I would generally still prefer Python because threading is possible, but if ease of installation is a higher goal, Lua IMO even beats C-externals.

Things are easy to compile only if your machine is setup for
compiling. Pd patches and binaries shipped as part of Pd are by far
the easiest to use. Getting distribution working in a non-trivial
task, so unless lua and/or python are built as part of Pd-extended, I
would not consider it easy.

Other than that, Python, PHP, Perl, Lua, etc all have native
interfaces to databases (some likely written in C), I think Pd should
also have a native interface for databases. Pd is written in C, we
have working C code, Pd externals written in C have a proven track
record of ease of use, so it makes sense to write this interface in C
as well.

That said, I think we have a solid interface sketched out, I think
it's time to write some code and try it out. I've started porting
the mysql external from Max to Pd. Let me know if you guys want
specific help with psql and sqlite. Ultimately, I think these all
should be part of a 'sql' library, and then we can also add the query
building objects too.

.hc

As we enjoy great advantages from inventions of others, we should be
glad of an opportunity to serve others by any invention of ours; and
this we should do freely and generously. - Benjamin Franklin

On Dec 11, 2007, at 5:27 PM, Mike McGonagle wrote:

On Dec 11, 2007 2:45 PM, Hans-Christoph Steiner hans@eds.org wrote: That said, I think we have a solid interface sketched out, I think it's time to write some code and try it out. I've started porting the mysql external from Max to Pd. Let me know if you guys want specific help with psql and sqlite. Ultimately, I think these all should be part of a 'sql' library, and then we can also add the query building objects too.

Hans, did you see my outline for the SQL stuff? While I think we
discussed a lot of things, I think that we should confirm that what
we discussed is what will be programmed. I have already done three
different versions of this, and I really would like to start using
it someday soon.

If you mean the SQL Object Model, I am looking now. If you mean
something else, please point me to it.

.hc

You can't steal a gift. Bird gave the world his music, and if you can
hear it, you can have it. - Dizzy Gillespie

I sketched up a quick patch to illustrate what I mean:



.hc

On Dec 7, 2007, at 4:32 PM, Mike McGonagle wrote:

Andy, have you had a chance to check out the SQLite external yet?
You can get a copy of it here ( http://puredata.info/Members/ mjmogo ), the latest version is 0.13. While this is still under
developement, it is based on the work of Jamie (and others), as far
as how the SQL is handled. They are not exactly the same, but I
would like to make that one of the goals so that these things can
be interchangeable.

One thing that I have not done yet is implement any kind of Blob
data types. It appears that SQLite3 (not the external, but the
database) handles Blobs as binary strings. I still don't know quite
how this would work in PD though. I would like to get some feedback
from others about how to do this.

As SQLite3 stores its database in a file, rather than a server,
there is no need to set up anything for the database (other than
designing the tables). Also, if you need to, you can use the tools
from SQLite3 to work with the same database file.

I work on the Mac, so the Makefile is setup to build a Universal
Binary (thanks to Hans). If you are on another platform, it
shouldn't be difficult to compile these things, as SQLite3 is pure
ANSI C. When I compile it, there are a lot of warnings generated,
but everything works fine. I guess I should turn off all the
warnings for that part of the compile.

But, let me know if you try this. I am eager to see how it works
for others.

Thanks,

Mike McGonagle

On 12/7/07, andy.graybeal@casanueva.com <
andy.graybeal@casanueva.com> wrote: i installed postgresql, and got it running, created a user for myself, compiled [psql] from postlude (i'm using pd-extended from cvs).

i opened up the psql-help file and started with the first
instruction, and it created the database fine, i moved to the second command and get
this error: psql: Action failed. PQresultStatus is PGRES_FATAL_ERROR

i'm wondering where i can look for more information about what is
going wrong, and what i can do to fix it.

i think it would be wonderful if i could start using a database to
store values :)

-andy

---

PD-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/ listinfo/pd-list

-- Peace may sound simple—one beautiful word— but it requires
everything we have, every quality, every strength, every dream,
every high ideal. —Yehudi Menuhin (1916–1999), musician _______________________________________________ PD-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/ listinfo/pd-list

Terrorism is not an enemy. It cannot be defeated. It's a tactic.
It's about as sensible to say we declare war on night attacks and
expect we're going to win that war. We're not going to win the war
on terrorism. - retired U.S. Army general, William Odom