PIP Test Campaign

All,

We will be opening up a new campaign called “PIP test” to the wider collector community.

The goal of this campaign is to collect “test data” for the PIP campaign. We use the data that you have collected so far to train visual AI systems, however we also want to know how well our AI systems are performing. To do this, we collect a “test set” which contains activities collected in natural sequences called “scenarios”. We then ask the visual AI system to localize when and where the activities are performed in this video. We will use your videos to predict how well an AI system will perform on new videos, and your test set will give us the metrics we need to predict this performance.

For this campaign, you must collect from a “stabilized security” viewpoint. This means that you cannot have your partner record you while they are holding the device. For this campaign, you must attach your device to a mount and put it on a flat surface up high looking down (e.g. a shelf, a stack of boxes, a ladder). You record your subject performing the scenario, then you use the in-app editor to annotate when and where the activities were performed. In fact, you can do this campaign by yourself as a partner is not required.

Here is a simple example of a device mount you can build yourself at home. This is constructed using a 9" x 3" piece of cardboard, folded and taped into a triangle, with 1" x 0.25" slits. The most important aspect here is that the camera is not moving while it is recording.

Here is a brief tutorial for the editor. To add an activity, select the activity to edit, then press and hold a single finger in the foreground box when the activity occurs. This may overlap slightly with neighboring activities. For more details, please see:

https://visym.com/editor

Please note that your goal is to create “scenarios” which are natural sequences of activities that you could imagine occurring in real life. A few notes:

  • This requires Android v2.6.6 or iOS v2.6.2, please upgrade if you have not done so already.
  • If a video is recorded while holding the device in your hands, it will be rejected and will not be paid out. Please make sure that you begin recording, put the device into the mount, then perform the scenario. The video should be edited after recording to add boxes and activities.
  • We require that you annotate all of the activities present, including all instances of “picking up” and “putting down” activities. For example, when you “take off gloves”, and are getting ready to “take off hat”, if you put down your gloves somewhere you need to annotate it. This means that there may be more than one “putting down” activity annotated in a single submission, which is what we want. If there are any missing activities then the video will be rejected and will not be paid out.
  • We recommend that you submit your video, then edit it from your video feed rather than editing before submission. Please submit any edits before 8am ET on the following day of your submission. If a video has been rejected due to missing activities, you may edit it and resubmit.
  • The annotation of the additional objects are not required, but would be preferred. You only need to include the required object if your subject chooses to perform the associated activity (e.g. gloves are present if your subject does “take off gloves”).
  • You may submit at most one of each collection available to you in-app subject to the same rate limits specified in your onboarding email. It is up to you to keep track of which collections you have submitted and the time you submitted them.

The campaign will be available later today.

11 Likes

Much appreciated

2 Likes

All, this is now live. We are starting slow with just one collection to make sure that everyone understands the new “stabilized security” requirements. We will increase the number of collections once this is confirmed.

5 Likes

Acknowledged :grinning_face_with_smiling_eyes:

4 Likes

Hello, I tried filming and submitting the fist live collection, after submitting it says “upload failed” and “video edited succesfully” then redirected me to the collections panel and the video didn’t reflect on my videos tab.

At exactly 12:00 pm PHT.

4 Likes

@Chan09 We recommend that you submit the video first, then edit it from your video feed. We did see that you tried to submit, and the video failed to upload, but the edited annotations were received. We are working to improve this error handling here.

5 Likes

Thank you :grin::grin:

3 Likes

Me and my partner’s collection does not showing the vedioes
Haroonsayed74@gmail.com my partner

1 Like

@Jasmeen They are available for you. Try to restart the app, or logout and login again.

3 Likes

I did it as mentioned by you
But not showing up vedioes
Thanx

2 Likes

What a doing with my partner is press the record button first then put the phone up on the raised place near a roof, so there is a bit of movement in first 2 seconds before the camera become stabilized, will this affect the ratings in any way?

I have already submitted today’s collection, but there was some shaking just before the camera becomes stabilized for around 2 seconds, please look my videos to see what am trying to say my email festusweyao@gmail.com and tell me if it is acceptable, the rest of the video is stabilized.

@jebyrne I would like to raise a question about this stated guideline a while ago. There may be some devices (like the phone me and my partner were using) cannot apply same activity label at the same scenario. For example, when I was about to perform “take off facemask”, the facemask that I am tending to take off should be placed somewhere, so to complete such normal activity, it must be followed with the “put down an object into the table” or something. However, there may be cases like after performing that one, another activity will occur like the “take object from backpack” and must be followed by same activity such as putting down another object once again to the table (since it will be awkward if you’re going around the area while the object which was taken from backpack still in hand). And same activity cannot be applied at the same time in editing with our devices.

Is there any other way that we can solve this kind of problem?

3 Likes

If it is perform two or three same task in collection what I do is select the button for example “put object on the table”. Play the task and hold the button for all form that task in one time. After that save and all the tag will appear on all activities.

Hope it will be useful.

4 Likes

Hi Jeff, i cant see the collection.

The task refresh for awhile but already appear now in collection. Try refresh.

Hi @jebyrne

I did the collection that was available. I have a couple of observations.

The cartboard mount is not very comfortable to work with, plus, it would need to be angled down, because if you place the cellphone looking forward, you don’t see the subject’s complete body.
I think the best thing would be to have a cellphone tripod that reaches at least 2m in height, so that you can angle it down like a security camera. This would also prevent the cellphone from falling from the cartboard mount (which is not very safe) and getting damaged.

Also, why is it that we have to press start and then mount the cellphone? I think there’s some lost time there. If the cellphone is already mounted, you just press start and don’t have to mount and desmount it every time, that would be better.

Thanks.

1 Like

@Prince_Fessie Yes, that is expected.

1 Like

@jebyrne video uploaded successfully but it is not being shown in the aploaded videos.
Should I make it again???

@Fudgieee In this case, you need to annotate “put object on table” if you put down the face mask. There will be cases where there must be more than one “put object on table” in the video. You choose at least 6/11 activities to perform, perform them in any order you choose, but make sure that if you perform any additional activities (like putting down your face mask on a table) that you annotate it.

2 Likes