All,
We will be opening up a new campaign called “PIP test” to the wider collector community.
The goal of this campaign is to collect “test data” for the PIP campaign. We use the data that you have collected so far to train visual AI systems, however we also want to know how well our AI systems are performing. To do this, we collect a “test set” which contains activities collected in natural sequences called “scenarios”. We then ask the visual AI system to localize when and where the activities are performed in this video. We will use your videos to predict how well an AI system will perform on new videos, and your test set will give us the metrics we need to predict this performance.
For this campaign, you must collect from a “stabilized security” viewpoint. This means that you cannot have your partner record you while they are holding the device. For this campaign, you must attach your device to a mount and put it on a flat surface up high looking down (e.g. a shelf, a stack of boxes, a ladder). You record your subject performing the scenario, then you use the in-app editor to annotate when and where the activities were performed. In fact, you can do this campaign by yourself as a partner is not required.
Here is a simple example of a device mount you can build yourself at home. This is constructed using a 9" x 3" piece of cardboard, folded and taped into a triangle, with 1" x 0.25" slits. The most important aspect here is that the camera is not moving while it is recording.
Here is a brief tutorial for the editor. To add an activity, select the activity to edit, then press and hold a single finger in the foreground box when the activity occurs. This may overlap slightly with neighboring activities. For more details, please see:
Please note that your goal is to create “scenarios” which are natural sequences of activities that you could imagine occurring in real life. A few notes:
- This requires Android v2.6.6 or iOS v2.6.2, please upgrade if you have not done so already.
- If a video is recorded while holding the device in your hands, it will be rejected and will not be paid out. Please make sure that you begin recording, put the device into the mount, then perform the scenario. The video should be edited after recording to add boxes and activities.
- We require that you annotate all of the activities present, including all instances of “picking up” and “putting down” activities. For example, when you “take off gloves”, and are getting ready to “take off hat”, if you put down your gloves somewhere you need to annotate it. This means that there may be more than one “putting down” activity annotated in a single submission, which is what we want. If there are any missing activities then the video will be rejected and will not be paid out.
- We recommend that you submit your video, then edit it from your video feed rather than editing before submission. Please submit any edits before 8am ET on the following day of your submission. If a video has been rejected due to missing activities, you may edit it and resubmit.
- The annotation of the additional objects are not required, but would be preferred. You only need to include the required object if your subject chooses to perform the associated activity (e.g. gloves are present if your subject does “take off gloves”).
- You may submit at most one of each collection available to you in-app subject to the same rate limits specified in your onboarding email. It is up to you to keep track of which collections you have submitted and the time you submitted them.
The campaign will be available later today.
