2. Descriptions

2.1 Rule of Participation

We will make available a new HDR video dataset (with human subjective quality scores as ground truth) to challenge participants for testing their proposed solutions. Challenge participants can choose to compete in one or both categories below:

(1) Full-Reference VQM - Objective video quality measurement algorithms/models predict video quality scores with reference/access to both the original source video and the distorted video

(2) No-Reference VQM - Objective video quality measurement algorithms/models predict video quality scores with only access to the compressed video (NO access to the original source video)

The participants would be required to submit an easily readable code of their algorithm (preferably in Python) with comments along with a document describing a brief summary and steps of their method. Each solution submission should also contain a demo code which could be used to run the submitted solution using a test video. The VQM solution should output predicted quality score (in the range of 0 to 100) either on a console window/terminal or in a text file referencing the test video.

2.2 Evaluation Criteria

All submissions will be evaluated using the following main criteria:

(1) Spearman’s Rank Order Correlation Coefficient (SROCC).

(2) Pearson Linear Correlation Coefficient (PLCC).

(3) Root Mean Square Error (RMSE).

(4) Runtime Complexity (measured by frames processed per second for a given spatial resolution and bit-depth): all submitted solutions will be evaluated on a standard AWS EC2 instance (p3.2xlarge) with one GPU.

The correlation performance will be be tested with a different reserved video dataset than that provided. The VQM performance of the algorithm will be evaluated separately for the full-reference and no-reference use cases. SROCC will be used to benchmark the result for the final ranking. PLCC, RMSE and runtime complexity will be used for reference but not for the ranking. 

2.3 Dataset Details

The HDR video quality dataset for this challenge was created by the Image and Video Engineering (LIVE) lab at University of Texas Austin (sponsored by Amazon Prime Video). To build the dataset (called HDR-LIVE), 27 UHD (3840x2160) high frame rate (50/60fps) source videos were obtained from public sources. For each of the 27 source videos (SRC), 9 different encoding settings (as shown in the table below) were applied to generate the corresponding processed video sequences (PVS). There are a total of 243 encoded PVS files compressed using X265 in constant bitrate (CBR) mode. During the human subjective video quality study (as detailed in [1]), each PVS was viewed and rated on a 65“ UHD TV under two ambient illumination conditions: (1) lab (dark) and (2) Typical living room lighting. A total of 66 human viewers were divided into 2 groups, each rated the video quality under one illumination condition. The human subjective quality study resulted in ~20,000 Mean Opinion Scores (MOS) using the Absolute Category Rating method.

The MOS scores under bright lighting condition were computed using the Sureal method [2] and released in the training set. Videos are compressed using X265 and released in MP4 format. The naming of the video is “[resolution]_[bitrate]_[video_name].mp4”. The bitrate for the reference video is named “ref”. For example, “1080p_9M_golf1.mp4” is one PVS of the reference video “4k_ref_golf1.mp4”. The metadata and MOS score of each video is in the “train_video_info.csv” file.

4. Prizes 

Amazon (as one of the WACV sponsors) will provide cash prizes for the top 3 competition winners (for the two categories FR and NR).

Cash prize:

• First place: $3,000

• Second place: $2,000

• Third place: $1,000

Final Results:

FR: 

1st Dounia Hammou, joint 2nd and 3rd: Zaixi Shang and Abhinau Venkataramanan

NR: 

1st Chen Feng, 2nd Abhinau Venkataramanan, 3rd Joshua Ebenezer

6. Resources 

Datasets is available for download here: https://live.ece.utexas.edu/research/LIVEHDR/LIVEHDR_index.html

Discussion Forum: https://codalab.lisn.upsaclay.fr/forums/7807/ 

The contact email is: wacv2023-competition@amazon.com 

8. Challenge participation Legal Rule/Terms/Condition 

Challenge Rules

Rules

PLEASE READ THESE OFFICIAL RULES (“Rules”) CAREFULLY. ENTRY INTO THIS CHALLENGE CONSTITUTES YOUR ACCEPTANCE OF THESE OFFICIAL RULES. IF YOU DO NOT AGREE TO ANY PART OF THESE OFFICIAL RULES, PLEASE DO NOT ENTER THIS CHALLENGE. NO PURCHASE IS NECESSARY TO ENTER OR WIN. A PURCHASE OF ANY KIND WILL NOT INCREASE YOUR CHANCES OF WINNING. VOID WHERE PROHIBITED.

1. SPONSORS

This Challenge is organized and sponsored by Amazon.com Services LLC, 410 Terry Ave North, Seattle, Washington 98109, USA (“Sponsor”).

2. SPONSOR ADMINS

“Sponsor Admins” are any companies or organizations authorized by Sponsor to aid it with the administration or execution of this Challenge.

3. CHALLENGE START AND END DATES (“CHALLENGE PERIOD”)

·      Start Date: September 30th, 2022

·      Entry Deadline: December 15th at 00:00:00 UTC

·      Winners Announced: December 23th, 2022

No additional registrations or entries will be accepted after the Entry Deadline. These dates are subject to change at Sponsor’s discretion. 

4. PARTICIPATION ELIGIBILITY REQUIREMENTS

You (“Entrant”) are eligible to enter this Challenge if you (and each member of your Team) meet all of the following requirements as of the time and date of entry:

·      You are an individual;

·      You are 18 years of age or older but in no event less than the age of majority in your place of residence;

·      You have Internet Access, an Email Account, and access to a personal computer; and

·      You have accurately and truthfully completed the participant registration.

You are only permitted to be part of one Team. Any Entrant that is part of more than one Team may be disqualified and their corresponding Teams may be disqualified as well at the sole discretion of Sponsor.

The Challenge is open to residents of the United States and worldwide, except that if you are a resident of the region of Belarus, Crimea, Cuba, Iran, Russia, Syria, North Korea, Sudan, or are subject to U.S. export controls or sanctions, you may not enter the Challenge. This Challenge is void where prohibited or restricted by law. Sponsor reserves the right to limit or restrict participation in the Challenge to any person at any time for any reason.

People who, during the Challenge Period, are directors, officers, employees, interns, and contractors (“Personnel”) of Sponsor, its parents, subsidiaries, affiliates, and their respective advertising, promotion and public relations agencies, representatives, and agents (collectively, “Challenge Entities”), immediate families members of such Personnel (parents, siblings, children, spouses, and life partners of each) and members of the households of such Personnel (whether related or not) are ineligible to win a prize in this Challenge. Sponsor reserves the right to verify eligibility and adjudicate any eligibility dispute at any time.

It is entirely your responsibility to review and understand your employer’s and country’s policies about your eligibility to participate in this Challenge. If you participate in violation of your employer’s or country’s policies, you and your Entry may be disqualified from the Challenge. Sponsor disclaims any and all liability or responsibility with respect to disputes arising between an employer and such employer’s employee or between a country and its resident in relation to this Challenge.

If any Entrant Team receives any third-party funding primarily intended to facilitate its participation in this Challenge, such funding must be disclosed to Sponsor no later than the Entry Deadline, along with any requirements imposed on the Entrant Team in connection with the funding. Entrant Teams may not accept or use any third-party funding if acceptance or use of that funding, or any requirements imposed in connection with that funding, would conflict with these Official Rules.

By participating in the Challenge, all Entrants unconditionally accept and agree to comply with and abide by these Official Rules and the decisions of the Sponsor which will be final and binding including the Sponsor’s right to verify eligibility, to interpret these Official Rules, and to resolve any disputes relating to this Challenge at any time. 

5. IS THE ENTRY AN ELIGIBLE ENTRY?

To be eligible to be considered for a prize, as solely determined by the Sponsor: The Entry MUST:

·      be compatible with the official submission format;

·      be in English;

·      be the Team’s own original work;

·      not have been submitted previously in any promotion of any kind;

·      not contain material or content that: is inappropriate, indecent, obscene, offensive, sexually explicit, pornographic, hateful, tortious, defamatory, or slanderous or libelous; or promotes bigotry, racism, hatred or harm against any group or individual or promotes discrimination based on race, gender, ethnicity, religion, nationality, disability, sexual orientation, or age; or promotes alcohol, illegal drugs, or tobacco; or violates or infringes another’s rights, including but not limited to rights of privacy, publicity, or their intellectual property rights; or is inconsistent with the message, brand, or image of Sponsor, is unlawful; or is in violation of or contrary to the laws or regulations of any jurisdiction in which the Entry is created; and

The Team members MUST:

·      designate one person as the team leader who will be solely responsible for receiving communications from and communicating with Sponsor;

·      ensure the Team has obtained any and all consents, approvals, or licenses required for submission of the Entry;

·      obtain any consents necessary from all members of the Team with respect to the sharing of such member’s personal information as outlined herein;

·      obtain the agreement of all members of the Team to these Rules;

·      not generate the Entry by any means which violate these Rules, the Sponsor’s Terms of Service or the Sponsor’s Privacy Policy;

·      not engage in false, fraudulent, or deceptive acts at any phase during participation in the Challenge; and

·      not tamper or abuse any aspect of this Challenge.

6. DISQUALIFICATION

·      If you, any Team member, or the Entry is found to be ineligible for any reason, including but not limited to conflicts within Teams and noncompliance with Sections 4 and 5 of these Rules, Sponsor and Sponsor Affiliates reserve the right to disqualify the Entry and/or you and/or your Team members from this Challenge and any other contest or promotional activity sponsored or administered in any way by the Sponsor.

·      A participant is not allowed to create more than one account to participate in the challenge. Violating this will result in disqualification from the Challenge.

·      If the Sponsors finds that any team member has not followed the Rules of the competition posted on the challenge website, then the team will be automatically disqualified.

7. HOW MAY THE ENTRY POTENTIALLY BE USED?

The Entry may be used in a few different ways. Sponsor does not claim to own your Team’s Entry, however, by submitting the Entry you and each member of your Team:

·      hereby grants to Sponsor and Sponsor Admins a non-exclusive, irrevocable, royalty-free, world-wide right and license to review and analyze the Entry in relation to this Challenge;

·      agrees that each member will execute any necessary paperwork for Sponsor and Sponsor Admins to use the rights and licenses granted hereunder; acknowledges and agrees that the Team will not be compensated and may not be credited (at Sponsor’s sole discretion) for the use of the Entry as described in these Rules;

·      acknowledges that the Sponsor or Sponsor Admins may have developed or commissioned materials similar to the Entry and waive any claims resulting from any similarities to the Entry;

·      understand that the Entry may be posted on a public website or social media channel and that Sponsor is not responsible for any unauthorized use of the Entry by visitors to such site; and

·      understand and acknowledge that, subject to provision of Prizes, Sponsor are not obligated to use the Entry in any way, even if the Entry is selected as a winning Entry.

Personal data you submit in relation to this Challenge will be used by Sponsor and Sponsor Admins in accordance to Section 14 of these Rules.

8. HOW WILL WINNERS BE SELECTED AND NOTIFIED?

Teams must upload their Entry to the Sponsor Admin’s platform, which will run inference against the test set. Submissions will be evaluated first on a smaller subset of data, and non-meaningful submissions will be discarded without further evaluation and not be posted on the leaderboard. Sponsor maintains right to change the definition of non-meaningful during the Challenge Period based on statistics derived from the highest scoring, previously-submitted entries. All submissions will be evaluated on the same hardware. A Team may make only five submissions per task per 24-hour period. If a Team makes more than five submissions per task per 24-hour period, only the first five submissions per task will be evaluated. If a Team makes more than five submissions per task per 24-hour period on more than one occasion, Sponsor may, at its sole discretion, disqualify the Team.

Potential winners will be contacted via the email through which the Entry was submitted. If a potential winner cannot be contacted, does not respond as directed, refuses the prize, or is found to be ineligible for any reason, such prize may be forfeited and awarded to an alternate winner. Only one alternate winner will be selected per each prize package, after which prizes will remain unawarded.

To the extent that there is any dispute as to the identity of the potential winner, the official account holder of the email address associated with challenge registration and submission will be deemed the official potential winner by Sponsor.

Prize winners represent and warrant that neither they nor their financial institution(s) are subject to sanctions or otherwise designated on any list of prohibited or restricted parties or owned or controlled by such a party, including but not limited to the lists maintained by the United Nations Security Council, the US Government (e.g., the US Department of Treasury's Specially Designated Nationals list and Foreign Sanctions Evaders list and the US Department of Commerce's Entity List), the European Union or its member states, or other applicable government authority.

9. PRIZES

We will award a total of 12,000 USD in cash prizes. Entries will be judged separately across two separate categories, full-reference and no-reference; in each category, the first-place winner will receive $3000, the second-place winner $2000, and the third-place winner $1000.

To receive a prize, the Team must make the submission via the challenge website portal under the Apache 2.0 license, and submit its training code and a detailed document describing its training process. The description must be written in English, with mathematical formulae as necessary. The description must be written at a level sufficient for a practitioner in computer science to reproduce the results obtained by the Team. It must describe substantially the training and tuning process to reproduce results independently. Failure to submit both the code and description within one week of notification will disqualify that entry and additional qualifying entries will be considered for prizes. Sponsor reserves right not to award prizes to any Team whose results cannot be reproduced.

The winning teams for each task will be offered the opportunity to give an oral or poster presentation at the image/video quality workshop. Any winning team that would like to give a workshop presentation must submit a written summary that outlines the presented work (2-4 pages length).

10. WHEN WILL PRIZES BE AWARDED?

The prizes will be awarded within a commercially reasonable time frame. All members of a Team may be required to complete and sign additional documentation, such as non-disclosure agreements, representations and warranties, liability and publicity releases (unless prohibited by applicable law), and tax documents, or other similar documentation in order for the potentially winning team to claim the prize. Sponsor will divide all awards that are payable to Entrant Teams evenly among all Entrant Team members and distribute accordingly. Sponsor will in no way be involved in any dispute with respect to receipt of a prize by any other members of a Team.

Only prizes claimed in accordance to these Rules will be awarded.

11. WINNER LIST

A list of all winners of this Challenge will be posted on Challenge website and may be announced at Sponsor’s discretion via Sponsor’s Twitter, Facebook, Blog, Website, or any other means, or at a Sponsor or Admins sponsored or hosted event.

12. YOUR PERSONAL DATA AND PRIVACY

Sponsor may use cookies and/or collect IP addresses for the purpose of implementing or exercising its rights or obligations under the Rules, for information purposes, identifying your location, including without limitation for the purpose of redirecting you to the appropriate geographic website, if applicable, or for any other lawful purpose in accordance with its Privacy Policy (www.amazon.com/privacy). Sponsor may use the personal data you provide via your participation in this Challenge:

·      to contact you in relation to the Challenge;

·      to confirm the details of your Entry;

·      to administer and execute this Challenge, including sharing it with Sponsor Admins;

·      at Sponsor’s discretion, to credit you and/or your Team for the Entry, identify you and/or your Team as a Winner, or other similar notice; and

·      as otherwise noted in these Rules or as necessary for Sponsor to meet their obligations under these Rules or applicable law.

By participating in this Challenge, Entrants are authorizing the transfer of personal data to the United States for purposes of administering the Challenge, conducting publicity about the Challenge and such additional purposes consistent with Sponsor’s goals or the Challenge goals. By entering the Challenge, Entrants consent to Sponsor’s and Sponsor Admin’s collection, and Sponsor’s use and disclosure of entrants’ personally identifiable information only for the purpose of validation, coordination, and communication of the winning Entries.

13. ADDITIONAL TERMS AND CONDITIONS

If Sponsor determines, in its sole discretion, that any portion of this Challenge is compromised by virus, bugs, unauthorized human intervention, or any other causes beyond its control, that in the sole opinion of Sponsor corrupts, or impairs the administration, security, fairness or proper participation in/of the Challenge, Sponsor reserves the right to (a) cancel the Challenge; (b) pause the Challenge until such time the aforementioned issues may be resolved; or (c) consider only those Entries submitted prior to the when the Challenge was so compromised for the prizes.

All activities relating to Participant’s participation in the Challenge and material submitted are subject to verification and/or auditing for compliance with these Rules and Participants agree to reasonably cooperate with Sponsor concerning verification and/or auditing. In the event that Challenge verification activity or an audit evidences non-compliance with the Rules or official Challenge communications, as determined in Sponsor’s reasonable discretion, a Participant’s continuing participation in any aspect of the Challenge may be suspended or terminated.

14. ADDITIONAL PRIZE CONDITIONS.

Prizes are non-transferable except as directed by Sponsor. No prize substitutions allowed. Except where prohibited by law, all federal, state, provincial or other tax liabilities are the responsibility of the prize winners, the Sponsor will not be responsible for any tax deductions which may be necessary and Sponsor reserves the right to withhold taxes as required by law. Prize winners will be responsible for paying all costs and expenses related to the prize that are not specifically mentioned, including, but not limited to, taxes, withholdings, and any other expenses that might reasonably be incurred by the winner in receiving or using the prize. All prizes awarded will be subject to any taxes Sponsor is required by law to withhold as well as applicable sales, use, gross receipts, goods and service, or similar transaction-based taxes. IF TAXES ARE APPLICABLE TO THE PRIZE(S), IT IS THE RESPONSIBILITY OF THE WINNER TO PAY TO THE APPROPRIATE AUTHORITIES. PAYMENTS TO CHALLENGE WINNERS ARE SUBJECT TO THE EXPRESS REQUIREMENT THAT THE WINNER SUBMIT TO SPONSOR ALL DOCUMENTATION REQUESTED BY SPONSOR (INCLUDING FORMS W-9 OR W-8BEN AS REQUESTED BY SPONSOR) TO PERMIT COMPLIANCE WITH ALL APPLICABLE STATE, FEDERAL, LOCAL AND FOREIGN (INCLUDING PROVINCIAL) TAX REPORTING AND WITHHOLDING REQUIREMENTS. Prize winners are responsible for ensuring that the tax documentation submitted to Sponsor complies with all applicable tax laws and requirements. If a winner fails to provide the documentation or submits incomplete documentation, the prize may be forfeited and Sponsor may, in its sole discretion, select an alternate winner. Designation as a prize winner is subject to Entrant’s proof of compliance with these Official Rules, maintaining compliance with these Official Rules and approval by the Sponsor. All details of prizes not specified herein shall be determined solely by Sponsor. Your use of any other products and services required by these Rules, whether required by these Rules or not, are subject to the terms and conditions associated with such products or services, including the Challenge website and services. In the event any clause or provision of these Rules prove unenforceable, void or incomplete, the validity of the other conditions will remain unaffected.

15. WAIVER, RELEASE AND LIMITATION OF LIABILITY

EACH ENTRANT ACCEPTS THE CONDITIONS STATED IN THESE OFFICIAL RULES, AGREES TO BE BOUND BY THE DECISIONS OF THE SPONSOR, WARRANTS THAT HE OR SHE IS ELIGIBLE TO PARTICIPATE IN THIS CHALLENGE, AND AGREES TO RELEASE, INDEMNIFY, AND HOLD HARMLESS CHALLENGE ENTITIES AND THE PERSONNEL OF EACH FROM AND AGAINST ANY AND ALL CLAIMS, LOSSES, LIABILITY, AND DAMAGES OF ANY KIND (INCLUDING REASONABLE ATTORNEYS’ FEES AND EXPENSES) ASSERTED AGAINST ANY OF THEM, INCURRED OR SUSTAINED IN CONNECTION WITH OR RISING OUT OF ENTRANT’S PARTICIPATION IN THIS CHALLENGE OR ANY TRAVEL OR ACTIVITY RELATED THERETO, OR BREACH OF ANY AGREEMENT OR WARRANTY ASSOCIATED WITH THE CHALLENGE, INCLUDING THESE OFFICIAL RULES. ANY ATTEMPT TO DELIBERATELY DAMAGE ANY WEBSITE OR UNDERMINE THE LEGITIMATE OPERATION OF THE CHALLENGE IS A VIOLATION OF CRIMINAL AND CIVIL LAWS AND, SHOULD SUCH AN ATTEMPT BE MADE, THE CHALLENGE ENTITIES AND EACH OF THEIR LICENSEES RESERVE THE RIGHT TO SEEK ANY AND ALL REMEDIES AVAILABLE FROM ANY SUCH PERSON(S) RESPONSIBLE FOR ANY SUCH ATTEMPT TO THE FULLEST EXTENT PERMITTED BY LAW.

Each Entrant hereby acknowledges and agrees that the relationship between themselves and the Challenge Entities is not a confidential, fiduciary, or other special relationship and that the Entrant’s decision to provide the entry to Sponsor for purposes of the Challenge does not place the Challenge Entities in a position that is any different from the position held by members of the general public with regard to elements of the entry, other than as set forth in these Official Rules. Each Entrant understands and acknowledges that the Challenge Entities have developed their own airborne object detection and tracking tools and that new ideas are constantly being developed by their own employees. Each Entrant also acknowledges that many ideas may be competitive with, similar to, or identical to their submission in theme, idea, format, or other respects. Each Entrant acknowledges and agrees that such Entrant will not be entitled to any compensation as a result of Challenge Entities' use of any such similar or identical material that has or may come to Challenge Entities, or any of them, from other sources. Entrants acknowledge that other Entrants/Entrant Teams may have created ideas and concepts that may have familiarities or similarities to their submission, and that they will not be entitled to any compensation or right to negotiate with the Challenge Entities because of these familiarities or similarities.

Entrants further agree that the Challenge Entities are not responsible for the following: (a) electronic transmissions, entries or notifications that are lost, late, stolen, incomplete, damaged, garbled, destroyed, misdirected or not received by Sponsor or their agents for any reason; (b) any problems or technical malfunctions, errors, omissions, interruptions, deletions, defects, delays in operation or transmission, communication failures and/or human error that may occur in the transmission, shipping errors or delays, receipt or processing of entries or related materials; or for destruction of or unauthorized access to, or alteration of, entries or related material; (c) failed or unavailable hardware, network, software or telephone transmissions, damage to Entrants’ or any person’s computer and/or its contents related to or resulting from participation in this Challenge; (d) causes that jeopardize the administration, security, fairness, integrity, or proper conduct of this Challenge; (e) any entries submitted in a manner that is not expressly allowed under these Official Rules (all such entries will be disqualified); or (f) any printing errors in these Official Rules or in any advertisements or correspondence in connection with this Challenge or the tabulation of scores. Sponsor reserves the right, in its sole discretion, to cancel or suspend this Challenge should virus, bugs, fraud, hacking, or other causes corrupt the administration, security, or proper play of the Challenge, or in the event Sponsor does not receive a minimum of two qualified entries from separate eligible Entrant Teams. Sponsor further reserves the right, in its sole discretion, to cancel or suspend this Challenge or to reschedule or reformat events should Sponsor be prevented, in any manner whatsoever, from holding this Challenge or any event due to any present or future law (whether or not valid); any act of God, earthquake, fire, flood, epidemic (including, without limitation, any pandemic), accident, explosion or casualty; any civil disturbance or armed conflict; or any other cause of any similar nature outside of Sponsor’s control. In all such cases, notice to this effect will be posted on the Challenge Site and prizes to the extent awarded will be awarded as determined by Sponsor prior to cancellation. If, in Sponsor’s opinion, there is any suspected or actual evidence of electronic or non-electronic tampering with any portion of the Challenge or if technical difficulties compromise the integrity of the Challenge, the Sponsor reserves the right to void suspect entries and/or terminate the Challenge and award prizes in its sole discretion. Sponsor reserves the right, in its sole discretion, to disqualify any individual found tampering with the entry process or entry materials or otherwise interfering with the proper administration of the Challenge or violating these Official Rules.

16. DISPUTES: Except where prohibited, you agree that: (1) any and all disputes, claims and causes of action arising out of or connected with this Challenge or any prize awarded shall be resolved individually, without resort to any form of class action; (2) any and all claims, judgments and awards shall be limited to actual out-of-pocket costs incurred, including costs associated with entering this Challenge, but not to exceed $15,000 USD and in no event to include attorneys’ fees; and (3) under no circumstances will you be permitted to obtain awards for, and you hereby waive all rights to claim, indirect, punitive, incidental and consequential damages and any other damages (other than for actual out-of-pocket expenses), and any and all rights to have damages multiplied or otherwise increased. The Sponsors or Sponsor admins are not liable for any mislabeled entry in the dataset offered. All issues and questions concerning the construction, validity, interpretation and enforceability of these Official Rules, or the rights and obligations of the Entrant and Sponsor in connection with the Challenge, shall be governed by and construed in accordance with, the laws of the State of Washington without giving effect to any choice of law or conflict of law rules (whether of the State of Washington or any other jurisdiction), which would cause the application of the laws of any jurisdiction other than the State of Washington, USA.