Thanks.  The XML/RPC solution is definitely the long term route we wish 
to take for remote control of encoding sessions.  Its really a question 
of time and the notion of doing anything with the PID file would be 
nothing more than a stop-gap solution that we'd rather avoid.  If and 
when we do an XML/RPC solution, it is very likely that we also provide a 
command line switch to submit XML formatted command from the shell 
instead of an HTTP socket.

Thanks for clarifying your need for "pause".  The notion of stopping a 
session is a much easier task.  As you know, producer already listens 
for signals so it looks like you are looking for a more platform 
independent means of stopping a given encoding session.  As you also 
know, I have written a cross-platform cmd line sample app that sends 
signals to producer for stopping a running instance of producer from 
another command line session.  I'll check in the 'prodctrl' sample cmd 
line application.  We'll make this part of the producer distribution and 
since it already compiles on all supported platforms (I need to do some 
porting work for Mac OS X) then that seems better than inventing yet 
another way to stop encoding sessions from another application.

We've actually gotten requests for starting/stopping archives during an 
encoding session from other folks so this is a good confirmation of that 
need. The approach we are taking is a way to do this for producer but 
I'll talk to some of the server folks and let them know this is an 
important feature request. I'll also explore and find out if there is a 
way to get this functionality now but I assume it means some dev work. 
 Its unlikely that this would be implemented based on the server 
listening for changes in the metadata stream.  It is more likely this 
would require and out-of-band communication to the server via something 
like HTTP to force a given archive to roll.

Inserting metadata in the stream is quite feasible using the producer 
C++SDK.  The long term vision for a non-C++ mechanism to do the same 
thing is either ActiveX or XML/RPC or both.

I may take a crack at making the change for updating the timestamp of 
the PID File if I get some time but if someone else wants to make this 
change I'd be happy to help submitting the diffs to our dev team. 
 Off-hand it seems like adding this code to the place where we are 
reporting status is the right thing to do as long as 1) you can get the 
file handle for the PID file at that point in the code and 2) I would 
want to do some testing to make sure the performance impact of 
'touch'ing a file every 1 sec (status gets printed every second) has a 
negligible (e.g. <<1 ms per second - preferably measured in 
microseconds) impact on CPU.

Happy New Year!
Steve McMillen, Program Manager, RealNetworks

Ken Martinek wrote:

> Thanks Steve.
>
> You bring up excellent points about the need for backwards 
> compatibility of the PID file, and of the current 'static' nature of 
> the encoding properties. Given these observations, it sounds more 
> prudent to put all the 'other' params in a separate file, enless folks 
> would be willing to change their wrapper apps to parse out the PID 
> value from all the others in the existing PID file.. And the XML-RPC 
> option is also attractive --it would be be my first vote for 
> development, even though it sacrifices simplicity/compatibility with 
> scripting languages or products that are not XML-RPC savy.
>
> I'd be happy with either of these two methods being implemented (PID 
> file or XML-RPC).
>
> Given all of these considerations, and my lack of expertise and 
> experience with producer's inner workings and how other folks are 
> using it, I offically ask yourself or someone in the community to take 
> ownership of this concept and its devlopment. It looks like you have a 
> solid understanding of what I was trying to convey, though I'd like to 
> elaborate on just a couple topics:
>
>    -the 'run=true' param I gave as an example could ceratainly be used 
> as a 'pause' control, where the producer process would stay in memory, 
> but I envisioned it as more of a way to simply terminate the process 
> (archive/broadcast) altogether, since in my case our scheduling needs 
> for an encode can change mid-stream (so the duration-args to producer 
> dont really help us), and we may also want to terminate a running 
> broadcast and start a new one with a different filename, which I 
> presume an only be done by execing a new process with its filename, 
> etc args?.. -I'm looking for the same effect as sending a CTRL-C to 
> producer from my wrapper ap.
>    -the 'archive=true' param was an example of my need for a way to 
> dynamiclly indicate to the server that the (already running) stream 
> should be archived (or not) without interupting any already-running 
> broadcast (or its alias name), as I would need to do (right?) to 
> change the stream's filename alias to fit into the servers archive 
> name-mask. For example, in a stream that is running 24/7, I might need 
> to archive only the segment of time between 5:30 and 6PM from that 
> encoder. I realize that there currently is no support for this, but 
> its HUGE on my wish list, and I'd hope to see support added to the 
> server sometime really soon. Is there an existsing 'metadata' stream 
> from the encoder to the server that this could be impemented on?
>
> Maybe an evolution of this idea would be to periodically send verbatum 
> all of the data that is contained in the 'PID' file, to the server 
> where a user API could effectively sit between the encoder and server 
> and make decisions based on the server-owners business model (and any 
> meta data that the encoder user chooses to add into his PID file)?
>
> # producer PID file
> pid=1024 //ignored by the server
> time=10123456789 //ignored
> run=true //ignored
> broadcast=true //server will broadcast (if server side user-API code 
> allows, ie. if news affilate paid to have live broadcast)
> archive=true //server will archive (if server side user-API code 
> allows, ie. if news affilate paid to have archiving service)
> title=World News Tonight with Peter Jennings
> X-customerName=KSTP Channel 5
> X-customerGUID=0c601a56e657505b3feb7638311543ac // server-side 
> user-API looks in database to verify customer subscriptions before 
> allowing archive or a given bandwidth broadcast
>
>
> Thanks again, and Happy New Year!!!
>
> -ken
>
>
>
> Steve McMillen wrote:
>
>> Ken,
>>
>> These are great feature requests.  I'm copying the dev mailing lists 
>> in the Helix DNA Producer project for feedback and to look for 
>> volunteers interested in making the changes based on the 
>> specification we eventually come up with.  In fact, if you are 
>> interested, I'd like to explore the option of you making the changes 
>> yourself.
>>
>> Of course, if you choose, you could simply download Helix DNA 
>> Producer source code yourself, under the RCSL (commercial license - 
>> royalty free for Helix Producer) and make the changes you need for 
>> your purposes. Ideally, any changes made would be done under the RPSL 
>> (public source license) so that changes go back into the community 
>> and others can leverage from them or even build on them.
>>
>> I see these ideas as being divided into two parts, both related to 
>> some form of cross-platform interprocess communication:
>>
>>   1. Communication from the producer app to other apps to report
>>      status, most importantly,  the process state (zombie/active).
>>   2. Communication from other apps to the producer app to change
>>      properties dynamically during an encoding session.
>>
>> I'd like to discuss these separately since one is easy and the other 
>> quite hard but first some more background:
>>
>> The PID file (-pid option on the command line) is a means by which 
>> this communication is proposed to occur.  This makes good sense, as 
>> pointed out below, for the following reasons:
>>
>>   1. Most Platform Neutral - Only requires read/write access which is
>>      relatively simple/similar across OSes.
>>   2. Most API friendly - Anything from a scripting language to C++
>>      application can interact with a text file
>>   3. Safe - Solution is not vulnerable to parent-process crashes/errors.
>>
>> The properties you propose can be broken into two general categories 
>> as suggested by the means of communication above:
>>
>>   1. Read-only properties (messages from producer)
>>          * pid=1024          * timestamp=10123456789
>>   2. Read/write properties (messages to producer)
>>          * run=true          * archive=false          * title=World 
>> News Tonight with Peter Jennings
>>
>> I interpreted the later 3 as properties you would like to set on an 
>> encoding session.  Please correct me if I am mistaken.
>>
>> 1.  Extensions to the PID file for reporting status of the encoding 
>> process
>>
>> The most important idea here is the heartbeat..  That is, writing a 
>> timestamp to the PID file on a regular (and probably configurable) 
>> basis.  This is a fantastic idea and would prevent the need for 
>> having code that checks status of a process through OS-specific methods.
>>
>> This provides a simple interprocess communication channel from Helix 
>> Producer to other applications.  The question is how far do we want 
>> to go with this idea.
>>
>> One option is to leave the pid file format the way it is now to avoid 
>> breaking any code out there that assumes a pid file has a single PID 
>> only.  The timestamp on the file can still easily be updated so that 
>> applications can check to make sure producer is still active.
>>
>> The other option is to change the file format and allow more than 
>> just PID in that file.  Then we could add other properties in 
>> addition to timestamp and PID such as some of the info that is 
>> printed on the command line status including:
>>
>>    * status (initializing, encoding, merging)
>>    * encoding duration or %
>>    * total duration
>>
>> The problem with the later is the possibility of breaking any apps 
>> that already depend on the PID file.  There is also a concern about 
>> "feature creep" where folks start to look at this as a means of 
>> getting status from Helix DNA Producer.   However, as explained 
>> below, RealNetworks has specific vision of how to implement  status 
>> on the command line which would be far more supportable in terms of 
>> adding new functionality in the long run.
>>
>> If we do go this route, my vote would be for a simple (not XML) 
>> syntax. This is because the most likely use case is access from 
>> script language and the need for a highly developed XML structure 
>> really does not exist. However, it would be useful to hear back from 
>> folks on which format they prefer. That is:
>> Simple Format:
>> pid=342
>> timestamp=3454534243
>> or XML format:
>> <session>
>>   <pid>342</pid>
>>   <timestamp>343435435</timestamp>
>> </session>
>>
>> I think the first format is much easier to parse and work with.
>>
>> 2. Extensions to the PID file for sending commands to Helix Producer
>>
>> While this is interesting, there are two challenges to this.
>> 1. Helix Producer does not really support dynamically settable 
>> properties as suggested below. For example, one cannot trigger 
>> archiving on the server, either from producer or even from the server 
>> (archiving can only be scheduled to be on or off always and a roll 
>> time/size can be defined before the stream starts). Nor can you set 
>> the encoder in pause mode, though this has been requested.  This is a 
>> useful feature but would take a significant amount of engineering 
>> work and is something that would best be done by the core helix 
>> development team.
>>
>> There are actually very few properties of an encoding session that 
>> can be changed on the fly during an encoding session though this is 
>> something we have considered changing in specific cases.
>>
>> Also, the mechanism for remotely controlling encoding sessions is 
>> something that we feel should be solved in a more general extensible 
>> manor.  The current thinking is to build an XML/RPC based solution 
>> that would enable Helix DNA Producer to listen for commands over an 
>> HTTP connection.  If we go this route, we very likely would consider 
>> adding command line options to submit XML requests via command line 
>> instead of an HTTP socket to enable scripting applications to perform 
>> the same stuff.
>>
>> An XML/RPC based control mechanism is something we are thinking of in 
>> terms of our roadmap but as of yet have not firmly decided to commit 
>> to this feature publicly.
>>
>> Cheers!
>> Steve McMillen
>>
>> Ken Martinek wrote:
>>
>>> Hi Steve, I'm Ken Martinek at Ascent Media (fomerly Liberty Livewire 
>>> Corp).
>>>
>>> You may remember my previous correspondence relating to various 
>>> ideas for improvements to the cmd line producer.
>>>
>>> Previously I'd wrote you asking your opinion on changing producer to 
>>> write some form of datetime indicator to a file for purposes of an 
>>> external 'heartbeat' monitor application. I'd like to extend that 
>>> idea a bit... what if you had producer do that very same thing to 
>>> the PID file every minute or so --not only doing a setModifDate() on 
>>> the file, but perhaps also writing the date into the file itself (as 
>>> a UNIX time integer)? This would allow an external piece of software 
>>> to determine if the process is still active, perhaps with lower 
>>> overhead than than spawning a platform dependent command to see if 
>>> the process is running (ps L/UNIX, tlist Windows).
>>>
>>> But the main benifit of this concept (my biggest need right now) 
>>> would be the following -- not only could you write a time stamp to 
>>> the file, but it would become a (more platform nuetral) method of 
>>> interprocess communication. For example, we could write a String/int 
>>> to an additional line in the PID file which producer reads at some 
>>> interval, which might be labeled 'run' or 'pause'. So if I have 
>>> forked the producer --and no longer have an input stream to send 
>>> Ctrl-c, I could issue the equiv of a TERM signal via that line in 
>>> the PID file ("run=false").
>>>
>>> Suppose that the PID file contained other attribs, like 'archive' 
>>> ---while producer was already running, it could send a flag to the 
>>> server that indicates the straem should be archived --without 
>>> interupting the stream that a live audience is viewing --or perhaps 
>>> another param to dynamically change some characteristic of the video 
>>> encoding. Maybe the steam title?
>>>
>>> # producer PID file
>>> pid=1024
>>> time=10123456789
>>> run=true
>>> archive=false
>>> title=World News Tonight with Peter Jennings
>>>
>>> In the above sample some params would be read-only to outside apps 
>>> by virtue of the fact that producer owns and will always write to 
>>> them (pid, time). The others could be parsed as runtime args whose 
>>> names match existing (or new) command line args. Obviously the 
>>> better approach would be an XML based format.
>>>
>>> Though there may be other/better ways to achive these same benifits, 
>>> I think this one has a few benifts that are really important :
>>>
>>> -Dynamic - one could change the runtime characteristics of the 
>>> stream without have to BOTH maintain AND read write to an 
>>> input/output stream to the Process (if the process that spawned the 
>>> producer crashes, the IO stream is lost until producer is restarted)
>>> -Most platform neutral - one only needs to get handles and 
>>> read/write to the files, which is relatively simple/similar across 
>>> OSs (and identical in Java of course), and not vulnerable to 
>>> parent-process crashes/errors.
>>> -Most API friendly anything from a text editor to a leightweight 
>>> homebrew app can interact with a text file.
>>>
>>>
>>> Please let me know what you think. Aso, I'd be happy to alpha test 
>>> any other software you may have that achieves the same end benifits.
>>>
>>> Thank You,
>>>
>>> ken
>>>
>>>
>>
>