Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision | ||
| apmdd:engseminar [2021/01/15 10:40] – user | apmdd:engseminar [2026/08/29 07:59] (current) – external edit 127.0.0.1 | ||
|---|---|---|---|
| Line 9: | Line 9: | ||
| Assoc. Prof. Anton S. Khritankov, ATP MIPT | Assoc. Prof. Anton S. Khritankov, ATP MIPT | ||
| - | Website: http:// | + | Website: |
| Email: anton.khritankov (at) phystech.edu | Email: anton.khritankov (at) phystech.edu | ||
| Line 15: | Line 15: | ||
| ===== Topics ===== | ===== Topics ===== | ||
| - | {{ :apmdd: | + | {apmdd: |
| * Program synthesis, mining of software repositories and program analysis | * Program synthesis, mining of software repositories and program analysis | ||
| Line 27: | Line 27: | ||
| Preliminary schedule and important dates for Spring 2021: | Preliminary schedule and important dates for Spring 2021: | ||
| - | | + | |
| - | * (Week 2) Topic selection is due | + | |
| - | * (Week 3) SLR short list review | + | * (Week 2) Topic selection is due |
| - | * (Week 5) SLR is due | + | * (Week 3) SLR short list review |
| + | * (Week 5) SLR is due | ||
| * (Week 6) RS study / prototype topic approval | * (Week 6) RS study / prototype topic approval | ||
| - | * (Week 8) RS/ | + | * (Week 8) RS/ |
| * (Week 10) RS/ | * (Week 10) RS/ | ||
| - | * (Week 12-13) Course finals | + | * (Week 12-13) Course finals |
| ===== Grading policy ===== | ===== Grading policy ===== | ||
| Line 42: | Line 43: | ||
| The following activities and goals must be achieved: | The following activities and goals must be achieved: | ||
| + | |||
| - Knowingly choose a topic and propose a goal for the research practice | - Knowingly choose a topic and propose a goal for the research practice | ||
| - Prepare a short systematic review on the selected topic, or | - Prepare a short systematic review on the selected topic, or | ||
| Line 53: | Line 55: | ||
| The overall procedure for SLR is provided in the table below. The results of the SLR include the following: | The overall procedure for SLR is provided in the table below. The results of the SLR include the following: | ||
| + | |||
| - the full list of sources considered (long list) | - the full list of sources considered (long list) | ||
| - a list of relevant sources (short list) | - a list of relevant sources (short list) | ||
| - an SLR report with review statement, golas, research questions, analysis and conclusions | - an SLR report with review statement, golas, research questions, analysis and conclusions | ||
| - | ^# | + | ^ # ^ Activitiesь ^ Outcomes ^ Work products ^ |
| - | |1.|Define research goals | The goal of the reseach along with the topic, a list of specific research questions | A section in the SLR report | | + | | 1. | Define research goals | The goal of the reseach along with the topic, a list of specific research questions | A section in the SLR report | |
| - | |2.|Describe the search protocol | A list of paper sources and scientific search engines, search queries | A separate section in the SLR report | | + | | 2. | Describe the search protocol | A list of paper sources and scientific search engines, search queries | A separate section in the SLR report | |
| - | |3.|Write down inclusion/ | + | | 3. | Write down inclusion/ |
| - | |4.|Search for sources according to the protocol | The full list of sources | A table with titles, links and abstracts in a separate file | | + | | 4. | Search for sources according to the protocol | The full list of sources | A table with titles, links and abstracts in a separate file | |
| - | |5.|Select papers for detailed consideration, | + | | 5. | Select papers for detailed consideration, |
| - | |6.|Collect information relevant to each research question | A short summary of findings for research question with references to the sources | A separate subsection for each research question in the SLR report | | + | | 6. | Collect information relevant to each research question | A short summary of findings for research question with references to the sources | A separate subsection for each research question in the SLR report | |
| - | |7.|Conclude the research: have all RQs been answered, ways to improve, threats to validity | Research analysis and conclusions, | + | | 7. | Conclude the research: have all RQs been answered, ways to improve, threats to validity | Research analysis and conclusions, |
| References: | References: | ||
| - | - Kitchenham, B.A. and S. Charters (2007) Guidelines for performing systematic literature reviews in software engineering, | ||
| - | - Course slides | ||
| + | - Kitchenham, B.A. and S. Charters (2007) Guidelines for performing systematic literature reviews in software engineering, | ||
| + | - Course slides | ||
| ===== Requirements to reproducibility studies (RS) ===== | ===== Requirements to reproducibility studies (RS) ===== | ||
| Line 76: | Line 79: | ||
| The overall procedure for reproducibility study is provided in the table below. Work products of the study include the following: | The overall procedure for reproducibility study is provided in the table below. Work products of the study include the following: | ||
| + | |||
| - A reporductibility study report | - A reporductibility study report | ||
| - Source code, datasets and configuration used for reproduction | - Source code, datasets and configuration used for reproduction | ||
| General requirements to papers selected for reproduction: | General requirements to papers selected for reproduction: | ||
| + | |||
| - source code is available for the paper' | - source code is available for the paper' | ||
| - needed data is available with permissive license, volume of data and computation is appropriate for reproduction | - needed data is available with permissive license, volume of data and computation is appropriate for reproduction | ||
| - paper results represent state-of-the-art now or up to 3 years before | - paper results represent state-of-the-art now or up to 3 years before | ||
| - | ^# | + | ^ # ^ Activities ^ Outcomes ^ Work products ^ |
| - | |1.|Find (or choose from SLR) a paper and get it approved for the study | A paper you are comfortable with | Use email, put rationale for the choise into the RS report | | + | | 1. | Find (or choose from SLR) a paper and get it approved for the study | A paper you are comfortable with | Use email, put rationale for the choise into the RS report | |
| - | |2.|Retrieve code and data, check license | Requirements for reproduction are met | A VCS repo with code and data (refs to data) | | + | | 2. | Retrieve code and data, check license | Requirements for reproduction are met | A VCS repo with code and data (refs to data) | |
| - | |3.|Repeat the experiment as stated in the paper | Any discrepancies in procedure and/or intermediate results, omissions, inaccuracies | A section in the RS report (RS protocol) | | + | | 3. | Repeat the experiment as stated in the paper | Any discrepancies in procedure and/or intermediate results, omissions, inaccuracies | A section in the RS report (RS protocol) | |
| - | |4.|Compare final results with the paper | Your own results from the experiment, discrepancies stated, their sources analysed | A section in the RS report | | + | | 4. | Compare final results with the paper | Your own results from the experiment, discrepancies stated, their sources analysed | A section in the RS report | |
| - | |5.|Make conclusions on reproducibility of the results and think of lessons learned | Reproducibility conclusions, | + | | 5. | Make conclusions on reproducibility of the results and think of lessons learned | Reproducibility conclusions, |
| References | References | ||
| - | | + | |
| + | | ||
| - Course slides | - Course slides | ||
| - reproducibleresearch.net | - reproducibleresearch.net | ||
| - gitlab.com/ | - gitlab.com/ | ||
| - | |||