From Buzztard

Gstreamer elements are usually processing just one stream or a fixed amount of streams. For music applications this is a bit unusual. Imagine an audio synthesizer. From the users point of view there will be one synthesizer instance. The user will select a sound-patch (e.g. a set of gobject properties) and then play several notes at once. Several notes mean different pitch, different volume, etc.

Of the ideas described below Idea 3 is currently the favourite. An implementation can be found in bugzilla.

Many people asked, why not using GstBin for it? Simple answer - GstBin manages GstElements, the proposed solution handles all kind of GObjects. For example in the poly audio case the Voice-Objects are just dummy GObjects to access voice-properties.

[edit] Example scenarios

[edit] Audio synthesizers

One requirement (for audio applications) is to differentiate between shared and per-voice parameters. An example: a simple synth has the following params:

• note
• volume
• attack
• decay
• oszillator

Note and volume are per-voice params, but attack, decay (volume envelope) and waveform are shared among voices. When the user plays several notes, he would like to select a sound (oszillator, attack and decay) and then trigger several note, volume pairs.

[edit] Mixer elements

Mixer elements mix contents from n incoming streams into 1 outgoing stream. It would be useful if one could provide properties per input stream. Some examples:

• A videomixer has a background image and composites the other streams on top of it. You might want to specify an offset of those images or maybe a colorkey for the alpha value.
• A simple audio mixer might also allow setting a volume on the incoming streams.

The properties would be local to each pad.

[edit] The problem

Currently GStreamer elements have only gobject properties. These properties are class wide - all instances of a class have them or don't have them. Therfore we can't dynamically add or remove property-sets as we add or removed voices. Furthermore gst-inspect would not like that too.

Just adding enough properties for e.g. 10 voices and name them like "note_%02d" would be quite limited. No one knows before how many voices one needs.

The current dparam approach, therefore added a parameter control system that works differently. The disadvantage is that elements need to implement it additionaly to their gobject properties. If they don't the element is not dynamically controllable.

[edit] Idea1: Instance Properties

Like

  g_object_class_install_property(GObjectClass *oclass,guint property_id,GParamSpec *pspec);
  g_object_interface_install_property(gpointer g_iface,GParamSpec *pspec);
  gtk_widget_class_install_style_property(GtkWidgetClass *klass,GParamSpec *pspec);

we will add

 gst_element_install_instance_property(GstElement *instance,guint property_id,GParamSpec *pspec);

that can be used in instance_init

  gst_sinesrc_init(GTypeInstance *instance, gpointer g_class);

Furthermore add methods similar to g_object_get|set ...

 GParamSpec** gst_element_list_instance_properties(GtkElement *instance,guint *n_properties);
 gst_element_instance_get|set_property();
 gst_element_instance_get|set();
 gst_element_instance_get|set_valist();

[edit] Compatibillity

An important point is, that it might be a good idea to have one voice as gobject properties anyway. This way the elemnt will work via gst-lauch etc. (but is monophon). The implementation of these gobject properties will directly access the data from voice 0:

 switch(property_id) {
   // shared properties
   ATTACK:
     self->priv->attack=g_value_get_xxxx(value);
     break;
   // voice property
   NOTE:
     self->priv->note[0]=g_value_get_xxxx(value);
     break;
  }

[edit] Using it for polyphonic elements

Elements using this for poly-audio will need to implement an interface. Voice property names have a number postfix "_%02d". The interface allows

• to find out which gobject-property controls the number of voices (return property name).
• get the param_specs of one set of voice_property basenames (return static array of property names).

[edit] GstController

It seems be possible for the GstController approach to change both gobject properties and instance properties (something like dparams, or style properties of gtk). Only gst_controller_sink_values() actually calls g_object_set(). So we could derive GstController so that this maintains both lists. This derived gst-controller would take care of using the instance property of voice 0, whenever one would use the gobject property of the same name.

[edit] Conclusion

[edit] Disadvantage

• multiple voices are not for free (need explicit implementation)
  • implementing the extra interface (two simple methods)
  • implementing the handling of instance properties (two methods)

[edit] Advantage

• clear design (is it?)
• application that are not aware of the poly stuff can use the elemnts as monophone elements
• elements can optimize calculations by share certain processing

[edit] Idea2: PolyBin

Another idea is to write a PolyBin elemnt. The element would provide polyphonic elements to applications. It will have two properties:

• element-name: the name of the gstelemnt that it wraps
• voices: the number of voice it provides

It further will provide a GST_POLYBIN_INTERFACE:

• GstElement *gst_poly_bin_get_element(GstPolyBin *polybin, guint voice);
• GstElement **gst_poly_bin_get_elements(GstPolyBin *polybin);
• GList *gst_poly_bin_class_list_shared_properties(GstPolyBin *polybin);
• GList *gst_poly_bin_class_list_voice_properties(GstPolyBin *polybin);

A requirement would be that we either introduce a flag like

 #define GST_PARAM_POLYPHONIC (G_PARAM_USER_SHIFT+n)

or use a naming scheme for polyphonic properties like

 "poly_pitch" (prefix "poly_")

or use an interface that the elements need to implement.

Technically the PolyBin would add one Tee and one Adder to itself. These elements would provide the ghost pads for the PolyBin. All instances of the element will go inbetween the Tee and the Adder.

When setting shared properties, the application needs to set the property in all active instances. Would it make sense to add something like:

 gst_poly_bin_set(GstPolyBin *polybin,gchar *name1, gpointer prop1, ...);

[edit] Compatibillity

Elements that are not aware of the concept would just give a bad UI (all properties are repeated for each voice).

[edit] Conclusion

[edit] Disadvantage

• elements can't share processing
• when wrapping plugins (like the buzz-machines that provide polyphony) one would need to instanciate several of them with one voice each

[edit] Advantage

• only little changes to elements (marking polyphonic properties)

[edit] Idea3: child objects

Give elements the possibility to have children. This could be integrated into GstObject (it already contains a hierarchy, although only for parents) or made an interface. The following examples assume an interface called GstParent (gstreamer/gst/interfaces/ or gstreamer/libs/gst/?).

[edit] the element's interface

An element implementing the GstParent interface would need to provide something like the following functions:

 GstObject * (* get_child_by_name) (GstParent *parent, const gchar *name);
 GstObject * (* get_child_by_index) (GstParent *parent, guint index);
 guint (* get_children_count) (GstParent *parent);

Furthermore add to this some signals like these:

 void (* child_added) (GstParent *parent, GstObject *child);
 void (* child_removed) (GstParent *parent, GstObject *child);

[edit] setting/getting properties

The API would provide an easy way to do something like this:

 gst_parent_get (polyphonic_element, "voice3::note", &old_note, NULL);
 gst_parent_set (polyphonic_element, "voice3::note", new_note, NULL);

This would simply split the provided string "voice3::note" into the child name "voice3" and the property "note", then call get_child_by_name and then g_object_set on that child.

If child objects are derived from GstObject then on could use the name field to map from property to child. In the above example "voice3" would be the name of the child object. An alternative way would be to use g_object_set_data(object,"GstParent::name","voice3") together with GObject.

[edit] use cases

[edit] polyphonic elements

The polyphonic element would just create a "private" GstVoiceObject with the required per-voice properties for every voice it wants and add those elements to itself based on its own "voice_count" property or similar. In the processing loop, the element would have no performance loss, since it could access all the values on its private GstVoiceObject directly.

To manage the number of voices elements would have a "voices" property that defines the number of children (and expose that through an additional interface like GstPolyVoice). Alternatively such an interface could provide API to add new and remove specific voices.

[edit] mixer elements

Mixer elements could just export their sinkpads as children and have the sinkpads have those properties.

Such elements would get children added via gst_element_get_request_pad. A per-channel volume element would get children added/removed on negotiation.

[edit] possible extensions

[edit] different objects

Elements could have different objects (for example voices and channels) with different properties. So you could set "voice0::note" and "channel0::volume", but not "channel0::note".

[edit] deep nesting

GstBin could export all the elements it contains as children. This would allow setting properties on pipelines directly, instead of first looking for the correct element. For example:

 gst_parent_set (pipeline, "audio_thread::volume::channel1::volume", 1.0, NULL);

which would get the element named "volume" in the "audio_thread" bin, get it's first channel and set the volume of that channel.

[edit] compatibillity

To allow application that are not aware of the interface use such elements as monophone element, one can register one set of properties as normat gobject properties. These would act as a proxy for the first voice.

[edit] tasks

get all children:

for(i=0;i<gst_parent_get_children_count(parent);i++) {
  child=gst_parent_get_child_by_index(parent,i);
}

[edit] conclusion

[edit] advantage

• elements can share processing
• powerful design
• simple interface for the user who can just use get/set

[edit] disadvantage

• not 100% backwards compatible with g_object_get/set
• might need lots of explicit boilerplate code for extra GstObject

[edit] Idea4: derive from GstBin

Plugins usually derive from GstElement. What if we derive from a GstBin subclass. That subclass can be named e.g. GstParalellBin or GstMultiChild.

We can overload the add_element and remove_element methods. We already have the signals in place. We just add the set and get methods.

Application would check if the type and not if it implmements an interface.

[edit] conclusion

[edit] advantage

• API is similar to bin

[edit] disadvantage

• children need to be GstElements