Reviewer Instructions

Guidelines for the Review Process

Overall Process

Each submitted paper will be reviewed by at least three experts in the field. Submissions will be judged on the following criteria.

  • Suitability: Does the contribution fit to the scope of QoMEX and in particular QoE and UX, as well as to the list of topics of interest?
  • Originality: What new ideas, approaches and/or results are introduced?
  • Significance/contribution: Is the paper significant for QoE and UX research? What is the benefit that others can gain from the contribution?
  • Validity of the work: How confidently can researchers and practitioners use the results? Is the paper technically/methodologically sound? E.g. measurement setup, design of studies, conduction of simulation studies, statistical analysis
  • Related work: Is state-of-the-art relevant to the described work sufficiently addressed? Does the paper refer to peer-reviewed publications?
  • Thoroughness: Is the research carried out in a thorough way? Are the discussions and conclusions detailed enough?
  • Replicability: Are the results and conclusions clearly described and reproducible? Are public data sets or open source code available?
  • Presentation clarity: Is the structure of the text and the presentation of the results clear?

Submitted papers of the general and special session tracks will undergo a double-blind review process. Only the dataset & tools track will follow a single-blind review process as authors need to include a link to the dataset or tool which will be evaluated by the reviewers.

Each paper is assigned a Direct Responsible Individual (DRI) acting who is responsible for a certain paper. The DRI is selected by the TPC chairs and may be a TPC chair, special session organizer, or selected TPC member. The responsibility of the DRI is as follows.

  • Initiate discussions in EDAS after the review submission deadline about diverging reviews. Reviewers can argue for or against the acceptance of a paper.
  • If necessary, provide a meta review summarizing the reviews and the discussions among reviewers.
  • If accepted papers need to be improved, shepherding of the papers is initiated and led by the DRI.

For short and full papers, there will be a paper bidding phase in order to ensure a good match between an individual reviewer’s expertise/interests and her/his review assignments. The TPC meeting for full papers will take place on February 22, 2019 in Berlin, Germany and is organized by the TPC Chairs. It is also possible to participate remotely. Details will be provided later. All TPC Members are invited to participate in the TPC meeting at their own expenses.

For short and demo papers, an online TPC meeting will be hosted on April 11, 2019. All TPC Members are invited to participate in that TPC meeting.

The TPC members are invited by the TPC chairs to cover all topics as mentioned in the Call for Papers. In particular, the QoE and UX domains are reflected in the technical program committee.

For TPC Members and Reviewers

Important information related to the paper assignment and review process:

  • TPC members pick topics (areas) of interest, designating them as “of interest”, “neutral”, “no interest”. These topics are used to limit the number of papers TPC members can “claim” in the paper bidding phase for possible review, as only papers having at least one topic “of interest” and no topics “not of interest” are included.
  • Reviews are assigned to the reviewers, manually, taking into account topic preferences and conflicts of interest.
  • Submissions will be judged on suitability, originality, significance/contribution, validity of the work, related work, thoroughness, replicability, presentation clarity.
  • In the TPC meeting and during the discussion phase, a review with a high score but with a short unsubstantiated review will be discounted.

High quality reviews are of utmost importance to QoMEX 2019 and critical for the advancement of the field. For this reason, please keep the following review guidelines in mind:

  • Avoid being overly negative: try to be critical yet constructive in your reviews, as any research paper has its advantages and drawbacks.
  • Please write a review with the quality that you would expect for your own papers. Do not submit very short, unsubstantiated reviews.
  • Keep a fair and open-minded approach, favoring selection of breakthrough papers that can open new areas and new research directions.
  • If any of the papers are reviewed with the help of graduate students, we expect a TPC member to provide sufficient guidance and oversight, and personally submit the reviews to EDAS.
  • Use the “comments to the TPC chairs” field to share confidential remarks or concerns you may have regarding a submission.
  • Finally, we recognize the efforts and time that TPC members and reviewers contribute to reviewing assigned papers. A best reviewer recognition award will be handed out during the conference.

Full papers

Submitted full papers of the general and special session tracks will undergo a double-blind review process. Submitted full papers must not exceed six (6) pages, including figures, tables and references. Please see the paper submission instructions.

Reviewers should consider the following in their evaluation: suitability, originality, significance/contribution, validity of the work, related work, thoroughness, replicability, presentation clarity. Please note that we also invite to submit position papers as full papers which describe a novel or bold idea; in that case experimental results may not be existing, but the paper should be nevertheless scientifically valid and convincing.

    • Registration: 18.1.2019
    • Paper Bidding: 19.1.2019 –  21.1.2019
    • Submission:   22.1.2019
    • Review phase: 23.1. – 15.2.2019
    • Reviewer discussion phase: until 21.2.2019
  • TPC Meeting:  22.2.2019
  • Author notification: 26.2.2019
  • Camera-ready: 15.3.2019

Short papers

Submitted short papers of the general and special session tracks will undergo a double-blind review process. Submitted papers short must not exceed three (3) pages, including figures, tables and references. Short paper submissions must describe original and unpublished work. Please see the paper submission instructions.

Reviewers should consider the following in their evaluation: suitability, originality, significance/contribution, validity of the work, related work, thoroughness, replicability, presentation clarity. While a short paper is not a shortened long paper, the characteristics of short papers may include one or more of the following:

  • Small, but focused contribution
  • Work-in-progress or late-breaking results
  • A position paper or contribution that is pushing/challenging existing boundaries
  • A counter-example to an existing technique that helps to understand its limitations
  • Extensions or evaluations of existing methods or explorations of new methods
  • An interesting application nugget

The timeline for the short papers is as follows.

  • Short paper submission:   15.3.2019
  • Paper Bidding: 16.3. – 19.3.2019
  • Review deadline: 5.4.2019
  • Short paper TPC meeting:  11.4.2019
  • Short paper notification: 15.4.2019
  • Short paper camera-ready: 22.4.2019

Dataset & Tools track

Full and short paper submissions to the dataset & tools track are single-blind. The authors need to provide a link to the dataset or tool which will be elaborated by the reviewers as part of the submission. The data and tool must be made available to the reviewers upon submission and should eventually be published upon acceptance.

Reviewers should consider the specific features of papers submitted to the dataset & tools track and evaluate the submissions as follows.

  • The dataset or tool must be available online (e.g. on github).
  • Dataset papers must contain all relevant information and describe its content and origin, such that readers can fully understand the experiment environment and execution. Additional information may be provided online in the description of the dataset.
    • its content and origin;
    • the methodology and tools used to obtain it;
    • the schema, structure, or layout to store it;
    • usage scenarios or open analysis challenges for it;
    • relationships to other existing datasets;
    • its current limitations.
  • Tool papers must present either a new tool, a new tool component, or novel extensions to an existing tool. Tool papers should provide a short description of the theoretical foundations, after which emphasis should be on the design and implementation concerns (incl. software architecture and core data structures), closed by the description of experience with realistic case studies.
  • Dataset and Tool papers must clearly describe the novelty and significance and how this dataset/tool advances QoE or UX research.

Demo papers

Demo papers are short papers. Reviewers should evaluate whether the demo format is suited for the submission and whether the demo description is clear. Demo paper contributions must highlight the innovative side of the concept and should be different from normal short papers in that the presentation is significantly improved by the particular demonstration format. Please see the paper submission instructions.

  • Short paper submission:   15.3.2019
  • Review deadline: 5.4.2019
  • Short paper TPC meeting:  11.4.2019
  • Short paper notification: 15.4.2019
  • Short paper camera-ready: 22.4.2019

References

Papers submitted to QoMEX 2019 should refer to peer-reviewed publications. References to non-peer-reviewed publications (including public repositories such as arXiv, Preprints, software, and personal communications) should only be made if there is no peer-reviewed publication available, should be kept to a minimum, and should appear as footnotes in the text (i.e., not in the references list). Reviewers should comment on the suitability of references.

Review Form

  • Familiarity: Your familiarity with the topic of the paper
    • 4:Expert, 3:Familiar, 2:Some knowledge, 1:Novice
  • Suitability: Does the contribution fit to the scope of QoMEX and in particular QoE and UX, as well as to the list of topics of interest?
    • 5: Excellent, 4: Good, 3: Sufficient, 2: Poor, 1: Inadequate
  • Significance/contribution: Is the paper significant for QoE and UX research? Is there a benefit that others can gain from the contribution?
    • 5: Significant contribution, 4: Original work, 3: One step ahead of the pack, 2: Some scientific value, 1: Mix of well-known ideas
  • Presentation: Readability of the paper (organization, comprehensibility …). Is the structure of the text and the presentation of the results clear?
    • 5: Excellent, 4: Good, 3: Readable, 2: Poor, 1: Incomprehensible
  • Related work: State of the work is sufficiently addressed. The paper refers to peer-reviewed publications.
    • 5: Excellent, 4: Good, 3: Sufficient, 2: Poor, 1: Inadequate
  • Reproducibility and scientific credibility: The results are clearly described and reproducible, e.g. public data sets or open source code are available.
    • 3: Fully, 2: Partly, 1: Insufficiently, 0: Not applicable
  • Recommendation: Your overall assessment of the paper
    • 5:Definite accept, 4:Likely accept, 3:Accept if room, 2:Likely Reject, 1:Definite Reject
  • Strengths: Reasons to accept the paper
    • (Free text field)
  • Weaknesses: Reasons to reject the paper
    • (Free text field)