The thrill and missteps of starting in Year 1 had faded, and the research success that we are starting to see were yet to come. Thus began The Long Road of Year 2.
TLDR; I settled on the a philosophy for improving the Intro to Databases course, gained a critical mass members that have served as the lab’s core, and saw early results on a bunch of interdisciplinary projects that we are continuing to work on today. From the research side, it was a lesson in tenacity and confidence, which I describe in the Research section.
Teaching
My primary teaching goal was to reorganize the Intro to Databases course to fix the mistakes from Year 1.
My hypothesis was that many of the issues stemmed from a feeling of competition in the class. While it can be a motivator for some people, I don’t think having that cloud over every assignment and every exam is productive overall. For one, not all students are great at rigid assignments and timed exams. My challenge was to design changes that also scale to ~150 students.
Extra Credit Opportunities: Since assignments and exams are just to assess that the student has learned something, I added a bunch of extra credit opportunities so students can show that they’ve learned topics in other ways. The primary method was a shared class wiki that students could contribute to. By contributing to the wiki, students could earn 1–5% extra credit.
Students signed up to be be the main scribe for a given lecture, and could flesh out, add examples, fix errors, and edit any page on the wiki. Students added original examples, created midterm study guides, and wrote tutorials on setting up projects and assignments. They added what they did to a contributors page, which I read through and validated at the end of the semester in order to assign the credit. It required some more work to manage and vet, but 38 students contributed and everyone (I hope) benefited.
If you do this, it’s incredibly important to clarify that extra credit does not affect the curve. The possibility that it does really stressed out many students and it took a few classes to clarify.
Phone-a-Friend: I also wanted to encourage students to help each other study for exams. There was a New York Times article that described a “phone-a-friend” idea where, for a specific problem on the exam, the student could write a friend’s ID and receive the friend’s points for the problem if the friend did better.
This was actually an interesting experiment, and I plotted the actual dependency graph, where an arrow points from the test taker to the friend that they wrote down. Darker circles mean they got a higher score.
In midterm 1, although there are fan-outs where a group of students figure out who is likely to do better (say, from the class forum posts), there are a huge number of unconnected nodes that both didn’t write a friend down and were not chosen. Also, the subgraphs are surprisingly local, and perhaps highlights the social or study groups that formed in the class.
In midterm 2, there is clearly a single student that is quite popular, as exhibited by the high fan-in. However, there were more isolated nodes, which was surprising. One goal in future classes is to reduce the number of isolated nodes in the graph.
I will note that grading this is a nightmare! You need to first log the friendship graph, figure out what each student and their friend received, and make a final pass to update the grades appropriately. This easily added an hour to the grading. Based on informal student feedback, it seemed to help relieve some of the competitive pressure, and I would say was worth the grading pain.
Lessons: In my opinion, providing alternative assessment avenues and giving small incentives to collaborate seemed to work well — both in terms of (perceived) student happiness from chatting with students, and in terms of teaching reviews. I’m certainly going to try these approaches in future classes. I’ve started prepping for the fall semester’s iteration of the course, and we are exploring how to further expand the scope of these ideas.
Recruiting and Mentoring
There were no new PhD candidates joining the lab, however the group would grow to nearly 10 members by the end of the year. In the Fall, Fotis, James, Hamed, Zhengjie, and Sharan were continuing from the summer, HaoCi had gone back to China but still worked with us over the school year, and a new undergraduate, Kevin Lin, joined the lab. In the Spring, several undergraduates (Naina, Robert, and Salim), two masters students (Tejas, Gabe), and a postdoc Thibault Sellam also joined! I continued to work with Yifan and Sanjay remotely.
It’s pretty cool that great students can come from places other than a graduate application!
- Kevin and Salim joined the lab through my research questionnaire. To manage research requests, I borrowed Mr. Bailis’ strategy of routing applicants to a research questionnaire, which asks applicants to comment on one of 3 papers from the lab, and talk about another research paper they like. For better or worse, its purpose is to be a bit annoying and gauge how the applicant thinks. I primarily gauge the thoughtfulness of the answers and how long the applicant can be in the lab. Kevin recently graduated, is working at AI2 for a year, and will be joining UC Berkeley as a PhD student in Fall 2019!
- Gabe started a research project to study visualization perception of chart complexity in my Spring seminar on Interactive Data Exploration Systems. He continued to work on it over the next year and a half, and is starting as a Columbia PhD student with my colleague Suman Jana this fall!
- Naina asked to join after taking my fall Intro to DB class, where I tried to mix current research ideas into the curriculum. She helped prototype and test out some of my early chat-bot ideas.
- Robert was referred into the lab by Hamed and would eventually build and lead a mini-research group within the lab. Just like in industry, referrals are so important because students know more about students than I do.
- Tejas just walked into my door one day and said that he read the ActiveClean paper and thought it was cool. He then talked about his independent research generating sign language closed captions for videos. It’s rare to find someone with so much energy and enthusiasm.
- Thibault was completing his PhD in automated advisors for data exploration and looking for post-doc positions. His work approached the topic in a super creative and out-there way, which I really like. Our research interests were very much aligned, and he quickly became the backbone of lab’s organization and culture.
In terms of admitting new PhD applicants in the Spring, it was a mixed bag. More students were applying to me, which was great! However, similar to Year 1, I was very conservative. I also recognized some promising applicants rather late in the admissions process, and by then they had already decided on other schools. Ultimately, none of the admits accepted. C’est la vie.
Lessons: There are several things I learned. All of which are pretty obvious in retrospect, but I didn’t actively think about them until later in the year.
- Startups put a huge amount of effort into recruiting, and it’s one of the most worthwhile efforts, because every single person serves as the foundation of the company. The same thing is true for research labs. After Year 2, I committed to giving more talks and putting in more dedicated time to studying applications.
- Outside of students in your class, there’s not much direct control over who directly applies to the lab. For graduate applications, it appears that fame (doing good work) and marketing (getting the word out) are important, because potential applicants come from all around the world. That’s partly why I started tweeting and writing this blog. For undergraduate researchers, Hamed noted that the word gets around very quickly about which faculty simply give students grunt work, as opposed to treating undergraduates as budding researchers.
- I still struggled with designing small, concrete starter projects for students that are new to databases or research in general. It was clear that some students wanted to see if research was right for them, and early wins help get them excited and confident. I hope to have more to say once I figure this out myself.
Research
We published a “midterm” report on the Data Visualization Management Systems project to CIDR 2017, and the QFix work that we started over a year prior was published at SIGMOD. We had also started lots of projects the summer before Year 2, and submitted initial versions of them throughout the year. The majority of them were rejected and resubmitted as workshop papers. Overall, the keyword for Year 2 was tenacity, as I will briefly illustrate using two undergrad-led research projects.
Hamed had spent the summer working with James on a system to automatically generate writing feedback for social media text. Basically, if you are writing a e.g., review, profile, or description, it might be helpful to provide real-time feedback about how to improve the writing based on the norms of the particular social community. These norms are encoded as features in existing quality prediction models, and we developed techniques to generate feedback by interrogating the models. Although it was outside of our research communities, we were confident that the idea was good, and decided to submit it to WWW… then ICWSM… then VLDB…then CHI (where we completely butchered the positioning). It was repeatedly rejected.
Across the board, the WWW and ICWSM reviewers liked the idea but didn’t like the positioning nor some of the experimental details. This was fair, since we didn’t understand the community norms. Out of desperation, we tried to reframe it as a data cleaning tool for VLDB, and Jiannan generously helped us design experiments and extend the paper. It was an ill-fit reframing, and rightfully rejected.
Honestly, getting repeatedly rejected was very disheartening. It’s a bit easier on me because I know the process and am fighting for the student, but it can be taxing on the student. We kept going partly because the reviewers were actually quite positive (one WWW review even noted that the work could have been a best paper candidate), and because it was clear (to me) that it was solid work. I felt that small wins, such as Hamed’s Collective Intelligence short paper, and presenting to different groups at Cornell Tech, Stanford HCI, Buzzfeed, and others, helped keep our momentum up.
For our ICWSM 18 submission, we got a ton of help in positioning the paper from Lydia Chilton, who had recently join Columbia’s faculty, and the Stanford HCI group, where Hamed interned in 2017. The main point was to stop selling it as a system that developers can use to add automated feedback to their application, because there is an expectation to compare it with alternative systems and show that developers can use it — these are good questions but beside our point. Instead our focus shifted to the promise of leveraging quality prediction models and model analysis techniques to pick good feedback. After this shift, it was finally accepted and presented!
Wait, what does this have to do with Eugene’s actual research? At first glance, this research is pretty far off from the database-centric core of my research. We didn’t know we would end up here when Hamed first started. However, I have interest (even from my thesis work) on providing explanations to end users. In fact, the lab’s data visualization and data lineage research stemmed from wanting to provide this functionality. My feeling is that if the student and I can agree that a project is somehow related to the lab’s interests, and it’s a solid idea, then it’s fine. Particularly for independent undergraduate research!
Kevin joined the group and wanted to work on machine learning. I didn’t have an active project, and didn’t want to apply machine learning to a problem. Over the summer, we had read some deep learning papers that used visualization to understand the model behavior, and it seemed like a problem that could use some automation from a database perspective. However, neither Kevin nor I had expertise in this area, so we basically had to learn the basics on our own.
We spent the entire fall semester going through papers and trying things out to identify a potential problem. Looking back through the notes, we meandered through subgroup discovery, visualization tools, model explanation, and more. Eventually in early Spring, we settled on what we called Deep Neural Inspection, which tries to automate the type of analysis that deep learning visualization tools address. Kevin led the effort that resulted in a NIPS 2017 submission. Although it was rejected, Thibault, Carl, and Ian joined the project and helped publish SysML 2018 paper, as well as our recent VLDB full paper submission!
Are these examples unique? No! Although I focused on two specific examples, many other projects —such as precision interfaces, automated cleaning, applying database concurrency management to UX design, quantitative visualization complexity measures — all worked on less-established problems and faced similar uphill battles to sharpen and clarify the ideas. They’ve also become forces to be reckoned with.
Lessons: I’ve learned a few things from this experience
- It takes long-term commitment to learn the research process and define a problem, but undergraduates can absolutely do great, original research.
- Research that tries to define new problems can take a long time from start to publish. As faculty and as PhD students, it helps to work on side-projects to keep your sanity.
- It’s really important to avoid approaching a community as an outsider. I also experienced this when starting out on the Vis side of my research (where collaborating with Remco Chang helped immensely). I’ve started dragging domain experts into my projects when the first positive signs emerge, and it’s led to great collaborations. Dr. Philip Guo discusses how it can slow research down in his book: The PhD Grind.
Thanks to Agnes Chang, Yifan, Cagatay, Hamed, and Kevin for comments on earlier drafts.
