Media summit in Prague 2017
Attendees
- Jacopo Mondi
- Mauro Chehab
- Niklas Söderlund
- Benoit Parrot
- Pavel Machek
- Ricardo Ribalda
- Dimitrios Katasaros
- Laurent Pinchart
- Hans Verkuil
- Gustavo Padovan
- Alexandre Courbot
- Sakari Ailus
- Mike Krufky
- Brad Love
- Tomas Dratva
Meeting notes
CEC status
Latest CEC status is always available here: https://hverkuil.home.xs4all.nl/cec-status.txt
- Side note: should we use the new serial bus (serdev) for the RC serial driver?
Explicit synchronization
In-fences are easy, there is no large issue in the RFC.
Out-fences, however, are subject to buffer reordering issues. A new event has been created to report buffer order to userspace. Fences are created at QBUF time, but can't be used by userspace until an event reports which buffer will be available next.
Not all drivers perform reordering. As an optimization a capability flag could tell userspace that buffers are always ordered and that events are not needed. The capability flag should likely happen at/after setting format, as out-of-order depends on the video format
Reordering is mostly useful for codecs, and can also occur from buffer recycling internally in drivers when capture errors happen. While recycling is allowed by V4L2, we could forbid it when fences are used. In that case all buffers would be returned to userspace in order, and errors be reported with the error flag set.
We may need to make capture queues not reorder buffers as well, so basically if we are using in-fences we don't let the bufffers reorder.
So when there is no reorder we can return the buffer's fence at the QBUF time and not need the OUT_FENCE event for these cases.
If the user doesn't pickup the outstanding OUT_FENCE event before DQBUF just set the fence to -1 at DQBUF time.
Discussion if timestamps are enough for audio/video synchronization... and what kind of timestamps to use. Wall clock timestamps are not usable for synchronisation, monotonic is the way to go. Converting device timestmaps to monotonic in the kernel can be troublesome.
Time stamping with fences --- time stamps related to buffers are only available at dqbuf time (unless SoF events are being used, and they're supported by only some hardware). This is too late in some use cases.
Capture to networking (a.k.a. partial fences) - We need to move from a frame-based API to a slice-based API. The problem isn't limited to fences, the whole V4L2 API is based on frames. Latency could be lowered in this way, but it isn't clear whether frame-based handling is the main source of latency in any use case. To support very low latency (mostly for professional video use cases) we would need to redesign all buffer handling in V4L2. We'll postpone this for now until someone comes with a convincing use case and enough resources to implement it.
Subsystem Maintainance
- Pull requests lost in Patchwork lately, follow up this to see whether it happens still. Patchwork is subscribed to linux-media list.
- V4L2 sub-maintainers
- Helped to solve bottlenecks
- More contributors nowadays
- DRM subsystem has a number of committers (37) who also review each others patches.
- The model switch was done as there were many contributors and bottlenecks due to that.
- This has attracted more contributors.
- Misuse of committer rights will lead to removing the committer rights.
- No commit rights have been removed from committers so far
- Harder for patches to fall through the cracks as there are many reviewers and committers capable of doing the required work.
- The committers will commit work reviewed and approved on the mailing list.
- Core developers in DRM take great care in maintaining the tree.
- Improved core developer skills due to more challenging and responsible work.
- Bad patches getting merged will be reverted, or not getting merged in the first place
- Could V4L2 switch to this model?
- DRM folks have a tool to help the workflow
- checkpatch.pl checks, ensures the branch has the correct baseline and does check for many errors
- Add compilation?
- Currently DRM specific but should be made more generic
- Ensure patch quality checks Mauro is now performing are all being done
- Additional requirements on developer machine security!
- Prevent destructive push
- More automation will be needed in checking pushes
- Jenkins? 0day? Opt-in is needed for 0day. Ask 0day guy: <fengguang.wu@intel.com>
- Jenkins would require more infra on our side whereas 0day already exists.
- Patchwork
- No information in patchwork e-mails on who acted on a patch. This would be important to add.
- Should upgrade to a maintained version
- Patchwork databases incompatible between 1.x and 2.x
- Is there a migration script?
- Or use kernel.org patchwork? This has been troublesome in the past and it is not known it'd be less so now.
Media subsystem documentation
- Videobuf2 has been mostly undocumented but this hasn't been a problem in practice: the API is sensible, and the drivers using videobuf2 are mostly doing the right thing.
- This does not excuse the lack of documentation
- SoC camera removal
- sh-ceu being converted by Jacopo to a proper V4L2 driver
- V4L2 clock used by SoC camera, pxa_camera and 2 i2c sensor drivers only
- A few sensor drivers: one option could be to just remove them; they can be brought back if needed
- videobuf1
- atomisp depends on videobuf1
- plus 10 non-staging drivers
- Push to split kAPI and uAPI documentation
- V4L2 now has them combined in a single document
- Different audiences for the APIs
- V4L2 reference documentation very good
- Howto-type documentation not available for application developers, who have little idea of how to use the API. The reference documentation does not help here.
- KernelDoc in headers vs. in .c files
- Keeping function documentation in .h file
- Increased likelihood for missing documentation update when changing the function
- Extensive documentation fits well for a small header, perhaps not so well to .c file
Request API
- Interest in getting request API to upstream
- Classes of hardware need this
- Cameras supporting Android camera HAL v3
- Stateless video codecs
- How does this work with media devices without a single master driver?
- No good model is known
- The use case is rare but does exist
- Must be addressed independently of requests
- Codecs the first use case
- Ensure that cameras can be supported as well
- Future-proofing the API is vital
- Media graph for a stateless codec
- Video node with two pads, codec as a sub-device with two pads
- The short term solution is to add request ID to the existing V4L2 IOCTL argument structs
- Gets us forward; this has been already proven on codecs
- The fear is that eventually the codec developers stop pushing their code if they cannot get it in
- In the long run the requests will be queued through the media device
- The control framework can be extended to
- Media device, i.e. no longer V4L2 specific
- Generic request command and request command argument transport and storage
- I.e. as controls, but not limited to controls
- Requests essentially are created empty
- As the documentation goes; internal implementation may be different
- Only entities, controls etc. that are changed by the request are part of that request
- Parameters that are not changed by the request are ignored
- E.g. if a value of control a is x while the request is created, its value may well be different from x when the request is executed (assuming the request did not change control a)
- Media device complexes may have devices that are out of scope of requests
- I2c devices
- Slow
- To hit the frame related to the request, needs hitting a narrow time window
- What do you do if you miss?
- Would require additional complex cross-driver mechanisms in kernel
- User space responsible for applying configurations to such devices, at least for now
- Dependent configuration items
- E.g. a hardware device exposing two controls in a single register
- What happens if the controls are changed through a request and an asynchronous s_ctrl at the same time?
- polling
- Implement polling for request file handles
- This allows polling for specific requests
- E.g. get woken up when the last request in a set of requests is done.
- returning request IDs back to user space
- This is problematic, as the file handle related to a request may be different from process to another
- Two approaches
- Add request ID to existing V4L2 IOCTL structs
- struct v4l2_ext_control, struct v4l2_buffer
- This is fairly easy and is known to work in practice
- Stateless codecs need this and they do need this now
- Planned already last year
- In short term gets us forward
- Helps to gather experience on requests to support media device only requests (see below)
- Also usable for complex cameras albeit not very efficient
- Add support for requests to the Media device only, without going through the V4L2 video devices or V4L2 sub-devices
- Long term goal
- Much more efficient than through V4L2 device nodes: instead of ten or more IOCTLs, there will be just one IOCTL to queue a request
- When a request completes, make a copy of the volatile controls (since that's the value at that point of time)
- When dequeuing a buffer, does userspace need to set the request ID in the DQBUF call ? No, as kernelspace knows the buffer/request association already, and userspace might not know which request the next buffer is associated to.
Actions:
Laurent: check the suitability of DRM tooling for V4L2
Mauro: check if kernel.org improved patchwork support.
Hans: tentative (not this year) convert old videobuf documentation to videobuf2.
Alexandre: MC Request API (REQUEST_CMD), with request components through V4L2 devices, request creation and queueing through MC
Hans: control framework API: sync with Alexandre by 2nd week of November at the latest.
Laurent: find a video of a corresponding developer workstation security presentation given in ELC-E
GPG key fingerprints
https://alexcabal.com/creating-the-perfect-gpg-keypair/
https://spin.atomicobject.com/2013/11/24/secure-gpg-keys-guide/