2021 Engineering Internships — Part II

Curai has been hosting a very successful internship program since its inception, covering both engineering and research positions. We pride ourselves in giving our interns challenging projects leveraging state-of-the-art technology, and with real impact on our product. Our last post, 2021 Engineering Internships — Part I, introduced the work of some of our 2021 interns. In this follow-up, we’re zooming in on the journeys of Alexa, Chris, Laura, Luke, and Niraj, who after successful internships returned to Curai for more… but now as full-time employees!

We sat together for an asynchronous virtual fire-side chat to discuss their experiences, from their goals prior to the internship, to their transition to full-time employees.

Curai offers remote internship opportunities across our teams throughout the year. Please visit our jobs page to see what positions we are currently recruiting for.

What was your trajectory prior to your internship at Curai, and what were your career goals at that point?

[Alexa] Prior to joining Curai, I was pursuing my master’s degree in data science. At that time, I was looking for an internship to help me break into this industry. Since my background was in molecular and cellular biology and I had some knowledge of healthcare, I thought my best bet to get an internship would be for a data science role at one of these two industries. My goal at this time was just to get an internship to add to my resume so my application would be more competitive for full time jobs and I could hopefully afford to be more selective. It just so happened that Curai was searching for an intern at the time I decided to search for internship opportunities, and to my surprise, it was exactly the kind of job I was actually hoping to land at for a full time position! Their mission of making healthcare more accessible to everyone captured my interest first as this is something which strongly resonates with me. After looking at the background of the employees and the size of the company, I realized that I would be able to learn from experienced mentors here, and this made Curai my top choice for an internship.

[Chris] My career goals when I landed the internship at Curai were really just to get that first engineering job, but I also didn’t want to sell myself short by accepting a job at a company that didn’t share the same values I did. I had recently earned a degree studying climate science, and thanks to the pandemic that job market dried up. It was important to me to keep the momentum going, so I retrained my focus on software engineering. I ripped through an immersive full stack bootcamp at the Flatiron School, and I realized that this is what I was meant to be doing all along. The next step was just to find a company whose mission was something I could get behind, and that’s where Curai came in. Healthcare reform is something that I’m passionate about, so I saw this as an opportunity to have my cake, and eat it too!

[Laura] Similar to Chris, I also attended a software engineering bootcamp prior to joining Curai. My degree is in Graphic Design and I spent 5 years designing brands and websites before attending Hackbright Academy to pursue a career in software engineering. When I first joined Curai, I was looking for a mission based company that valued mentorship and had a high ratio of female engineers. Curai fit that bill and was small enough that I knew I would have the opportunity to build up a variety of technical skills.

[Luke] I had recently graduated from my masters program, with a focus in machine learning. Finishing in the middle of the pandemic, I was working primarily online. I was actively looking for ML engineering internships to build my practical experience handling machine learning models in production. Applying to internships and keeping up to date with the latest ML news brought Curai into view for me. Through a podcast interview I learned how Curai was applying ML to help provide healthcare to those who could not afford it. I aspired to work with cutting edge models. Models that excited me for the future of ML and filled me with wonder, such as GPT3 and DALL-E. Additionally, I wanted my work to have a positive impact: Curai seemed like the perfect fit!

[Niraj] Somewhat similar to Luke, I was enrolled in my final semester of my masters in computer science program, with a focus on Machine Learning (ML) at UMass Amherst, when I landed a part-time internship at Curai. With a background in Biological Sciences from my undergraduate degree, I was seeking opportunities to use ML in the healthcare domain. In order to learn ML modeling, I spent the summer of 2020 at BenevolentAI, where I worked on developing algorithms to explain black-box graph embedding models used for target identification in the drug discovery pipeline. After this experience, I wanted to work with product facing use cases for ML and that’s when I found Curai. I really liked the mission of the company and was presented with an opportunity to use ML for securing patient privacy. The complex nature of the project and its potential impact on the company made me very excited and eventually joined Curai.

Did anything take you by surprise during your internship? Do you have any lessons learned that you think would be beneficial to incoming Curai interns?

[Laura] I felt like a full team member right away and that was unexpected. The work that interns do at Curai is not busy work. During my internship, I built a full stack feature to enhance auto charting a patient’s answers to the questions they are asked before chatting with a provider. This project had me working with the product engineering team, the machine learning team, and even the clinical team. The nature of the project meant that I needed at least a foundational knowledge of clinical charting and I loved that I was learning so much more than technical skills. My advice to incoming Curai interns would be to stay curious about what other teams are doing. Even if your project doesn’t require as much cross team collaboration as mine did, set up a lot of 1:1s with people outside your team, you’ll never know what you might learn!

[Luke] I underestimated the magnitude of impact I could have on a company in just 12 weeks. I was used to having strict requirements for internship projects, and those projects did not hold as much weight as I would like. During my internship I was responsible for almost every aspect of StrangeCase (my project). This involved reaching out to many different teams in the organization to gather feature requests and align on the end goal. My decisions were trusted, and I was able to create something that both: felt like a part of me, and better fit the needs of the company. My advice would be to reach out to everyone your project affects. This is especially important in a remote setting. The more you talk with others, the clearer it becomes where your work fits and how to maximize its positive impact.

[Niraj] Among various things that I was pleasantly surprised with, I was particularly impressed with the level of trust my manager had in me, the autonomy that I was provided along with the opportunities for cross-team collaboration. ML in academia focuses a lot on the various modeling approaches, often assuming availability of good quality data. However, in the real-world this assumption need not hold true. Thus, investing in building the datasets and the metrics is critical to creating a baseline and evaluating different modeling approaches. Lack of such scaffolding can lead to lots of modeling efforts that do not always lead to product improvements. This also provides an alternative of using state-of-the-art open-sourced or using proprietary models rather than building in-house models, to make faster progress in time-bounded fashion.

[Alexa] There were many wonderful things that surprised me during my internship! Of them all, I would say the level of impact and agency given to interns was the most surprising however. Many of the projects that former interns had previously worked on had a direct impact on our product as opposed to being simple “intern projects” that would perhaps be forgotten in a few months time. As an example, I was able to show some of my work at an All-Hands, which was both very exciting to present and important to know! As such, it seemed like there was not much distinction between other employees and interns (besides the level of responsibility and knowledge) as the work that was being done was equally important for the company as a whole. This might also be due in part to the friendly, inviting environment that exists at Curai where credit is given where it is due and every opinion is valued. For any incoming interns, I hope you realize how impactful your work will be as a member of the team. There are many projects that intersect with different areas and across teams, and gaining familiarity with these different areas by talking to other people will be beneficial to your growth and understanding.

[Chris] My entire engineering experience up to that point had been remote, so I was very comfortable in that environment. However, this was my first internship ever. So, similar to Luke,I was surprised by the amount of ownership I was allowed to have over the work that I was contributing and just how much it was going to actually be used inside the company. I’m not sure what I was expecting, but I definitely didn’t think I’d be as large a contributor in the projects as I was. To that end, a lesson I learned early in the internship was that your opinion is not only valued, it is sought out. There is too much information for anyone to know everything, so be the expert in your domain and be confident in your knowledge.

Let’s geek out: what about your internship are you proudest of? GO TECHNICAL

[Luke] My main internship project, StrangeCase, is a tool to visualize and explore changes between different versions of our ML system. It’s meant to assist with overall qualitative assessment and error analysis. A key feature of StrangeCase is the generation of interactive charts. These charts allow ML engineers and domain experts to hone in on potential issues in our system.

During development it became clear that the list of metrics included for each model in our system was not guaranteed. For this reason, I had to refactor my working code for chart creation to work for any number of potential metrics. This was particularly tricky because I was using dc.js (a marriage between Crossfilter and D3.js). This library was designed for static web pages, therefore I created a wrapper for use with React.js. The end result of my abstractions is that our ML team can modify what metrics are output from any model in the system and there is zero code change required in StrangeCase to support that new format. They are able to experiment with these outputs with no downstream engineering cost, reducing friction in the model iteration process.

[Niraj] My internship revolved around addressing patient data privacy. At Curai, the patient data is classified into 4 different categories: Red, Orange, Yellow and Green, based on the structure and availability of patient health information (PHIs) in the data. The project I worked, on named “Yellowify”, was aimed at algorithmically identifying PHIs in the free text data (orange data) and pseudonymizing them to get a “Yellow” state. While the project was a little open-ended, I collaborated with the head of privacy and security, and I was able to scope the project to focus on three PHI categories relevant to us (out of 18 per HIPAA).

I made a conscious decision to first explore the efficacy of various available tools/services for deidentification, before investing into building a deidentification model in-house. This required creating an evaluation dataset, most of which came from manual labeling by my mentor Namit, manager François and myself. Using precision and recall as metrics, I was able to compare the performance of four off-the-shelves models/services: bert-deid, AWS Comprehend, AWS Comprehend Medical and PrivateAI. After performing some error analysis that revealed the strengths and weaknesses of each, we found PrivateAI (which is proprietary) to fit right for our use case. With these findings and doing a cost analysis, I convinced the senior leadership to get a license for PrivateAI. As of today, Curai still uses PrivateAI for our deidentification needs

Thus, I am really proud that while working as a part-time intern for ~3 months, I was able to use a data-driven approach and go from an idea to establishing an external collaboration to support patient privacy at Curai. Fast forward 3 months, I joined back as a full-time ML Engineer and I got to deploy the “Yellowify” model and launch this feature company-wide.

[Alexa] I joined Curai as a data science intern and primarily focused on analyzing our clinical data. The key stakeholders for this project consisted of our clinical operations team, who put together a document with several key metrics they would like to measure in our patients. After discussing the prioritization of metrics, I had to search for where our data was being stored in the backend and decide on how the information requested could be gathered. I started writing queries in BigQuery, which I then translated into dbt before pushing out a pull request on Github. Once the code was merged, I then used DataStudio to help visualize the newly created tables.

Through this project, I created a dashboard that could be used by our providers to measure clinical quality metrics in our patients. Though a lot of clinical data had already been collected, it had not yet been analyzed, and as this was the focus of my intern project, I became the first person to analyze a lot of the current clinical metrics. Since joining as an intern, the scope of our clinical metrics expanded to not only include preventive screening information but also BMI, hypertension, HbA1c, and referrals among other focuses. It’s been fulfilling to work on this project since joining Curai as an intern and knowing how impactful my work has been for our clinical team.

[Chris] This is a really tough one for me, because there is so much to geek out on and so much tech that I want to talk about. For some quick context, the ML Infrastructure team uses Docker and docker-compose for our containers, Terraform/Terraform Cloud for our IaC, GitHub Actions for CI/CD and automation, AWS as our primary cloud provider, and a proprietary platform we named opsein (OPerations SEcurity INfrastructure) that stitches everything together.

Around the time that I joined Curai as an intern, the ML Infra team was in the midst of building a self-service deployment infrastructure. New ML applications would be responsible for their own infrastructure and deployments. This architectural pattern would allow the ML Application Engineers to iterate on their work quickly without fear of disrupting other services, but in order to do this we needed to make it modular, easy to implement for the engineers, and future-proof.

With all that said, I think what I’m most proud of is that the majority of the infrastructure and automation that I built during the internship is still being iterated on and used today. One of the first lessons I learned was that infrastructure mistakes are costly and difficult to fix down the road. It gives me great satisfaction to know that I was able to successfully contribute to the vision we had when we first began to implement the new infrastructure design.

[Laura] I joined Curai as part of the Automation team who was responsible for working with our ML team to bring their technology into our product. Being part of the Automation team as an intern was really beneficial because it blends together everything that Curai does. It required gaining a lot of context and learning about all three areas of focus (product, ML, and clinical) required to contribute to the automation team is what I’m most proud of. My project was to add an additional question type to our Question Serving panel (which you can read more about here).

While working on this project I learned about FHIR and how healthcare companies use it as a standard for medical data to allow for interoperability between organizations. I also became familiar with our Knowledge Base and existing medical datasets like SNOMED and LOINC that help power generating relevant questions for our providers based off of patient answers and search parameters. While I was learning all of the medical context, I was also gaining hands-on experience with our tech stack to build out a full stack feature that was production ready.

How does your experience as a full-time staff member differ from your time at Curai as an intern?

[Niraj] During the internship, I was primarily focused on a single project. I had little visibility (and time) to explore other projects that were worked on in the ML team. This changed after joining as full-time, where I had multiple projects going on in parallel and it required frequent context switching to make progress on all. While it was a little difficult for me initially, this made me get better at note-taking, documentation and project management. As a full-time member of the team, I also got opportunities for multiple cross-team collaborations such as working closely with product managers, the clinical innovation team, doctors, and the platform and infrastructure team. This helped me learn team-specific vocabulary and adopt my communication and presentation to the audience.

[Alexa] As an intern, I focused primarily on data analysis for our clinical operations team. This involved monthly meetings with them and my manager to determine what metrics we should measure and how they should be measured. At that time, I was only familiar with a small subset of the data we have, but as I gained more experience, I familiarized myself more with our data. As a full-time employee, I still continue doing the same work I had done, but the scope of my work has expanded. More metrics are now being measured, more meetings have been scheduled, and more responsibilities have been assigned to me. Whereas I primarily focused on clinical data as an intern, I now have a broader focus and provide analysis for other teams as well.

[Chris] Like Niraj, I was siloed a decent amount during my internship. My focus was singular, and although I connected with many members of other teams, my main point of contact was just my manager. As a full-time staff member on the ML Infrastructure team now, that silo no longer exists. The majority of the work I currently contribute has multiple stakeholders on multiple teams. I now balance the work of continuing to iterate on the infrastructure and automation that I developed during my internship with using it to deploy real ML applications that our ML Apps team has built. This required some adjustments in how I communicated as well as how I kept track of the work that I was doing, but I think I’m a much better engineer now for it.

I also think it’s equally important to point out something that hasn’t changed since my internship, the culture. I was always treated like an equal member of the team regardless of my status, and I believe that speaks volumes. Some companies’ people are nice, and some companies’ people are kind. Nice is a façade. Kind is authentic. Everyone at Curai was kind to me since the moment I started my internship, and then they were even more celebratory when I was brought back full-time. Never change Curai, never change lol.

[Laura] The biggest change from internship to full time for me was switching teams. I was part of the Automation team during my internship and switched to the Platform and Infrastructure team so the work I was doing was completely different. Unlike Chris and Niraj, I feel like my projects became much more focused because they no longer required as much collaboration outside of the engineering team. Other than that Curai does such a fantastic job integrating their interns into the team that the transition felt seamless.

[Luke] Yes and no. It is the same in that there exists ample opportunity to bring fresh ideas and to challenge existing practices. One would think as a full-time employee, you feel more connected to your coworkers. In reality, Curai welcomed me as a part of the team even before I came back full time.

One way full-time differs is that my responsibilities have broadened. Just like my colleagues have mentioned, during my internship I was primarily focused on a single project. Now my responsibilities include performing code reviews, collaborating on long term projects, and of course putting out fires 😅. Being a member of a team, and a company, there is a responsibility to support each other; to work towards a common goal. I bring into consideration much more context for my work now that I am full time.

Looking back at your trajectory since joining Curai full-time, where do you think you’ve grown the most, and based on what projects?

[Chris] It’d be easy for me to point to the various technical areas where I’ve grown during my time at Curai. Although it isn’t as fun of an answer, I’d have to say it’s my communication skills. It was easy to keep track of things during the internship. I knew what I was doing, and so did my manager. Shift to full-time and one of my first projects was to build a new internal tool for our labeling team. The tool was just one component in a much larger end-to-end project. At any given time there were five teams working on the same project, and communication was key to development. Early on, there were moments where an unclear communication on my part led to added work and backtracking for me. I am now much more intentional with the questions that I ask and the answers that I give. This also includes self-communication since I take better notes and leave better comments for my future self to refer to.

[Alexa] Throughout my whole experience at Curai, I gained better analytical skills — I became more proficient in SQL and learned how to work with different visualization tools such as DataStudio and Tableau — and, like Chris, I would also say that I gained communication skills. A lot of the work I have done involves knowing clinical terminology, and as someone who is not a healthcare professional, I have sometimes had to backtrack on my work due to an unclear understanding of what should be measured or how something should be measured. From these earlier experiences, I have learned to reach out earlier on in the data analysis process to come to a clear understanding of what different teams would like to be measured versus what can/should be measured.

[Laura] I’ve gained so many technical skills that I had zero experience with before joining Curai. Because I joined the Platform and Infrastructure team, basically everything I was doing was brand new to me. Now I have experience with Terraform, AWS, GraphQL, gRPC, microservices and Datadog. The team is small enough that there are so many areas available to me to discover what areas I enjoy working in and then polish those skills.

[Luke] I agree with Chris, of course I grew technically, but the real impact came in the form of soft skills. Rather than a specific project I would like to refer to our on-call process. During an on-call shift, an engineer is in charge of diagnosing and triaging incoming issues. On-call takes lead on urgent incidents and is the sole DRI (direct responsible individual) for the engineering effort. Having this experience forced me to understand the work of other teams, and the interaction between those teams, on an intimate level. Additionally, this improved my ability to manage individuals and direct team efforts.

[Niraj] I have learnt to collaborate with larger cross-functional teams. Code reviews have helped me in terms of writing efficient codes and adopting good coding practices. Working on some complex projects recently, I have developed better project management skills. Also, in less than a year’s time, I got to co-mentor two interns, which helped me hone my mentorship skills (see the part 1 of the internship blogs). Working on a remote setting, I was initially a little hesitant to reach out for help, but with time, I have improved my skills on asynchronous communication, which is an important skill to have working at a remote first company like Curai Health.

Any last bits of advice?

[Chris] Ask questions. Work hard. Work smarter than harder. Don’t overwork yourself. Ask more questions. Take a walk.

[Laura] Huge plus one to Chris’s advice! Ask tons of questions, no one gets to where they are working alone, and the context that long-timers at the company can provide you with is invaluable.

[Luke] Make your project and experience your own. Have fun with it. Work on things that interest you, that you are passionate about. Reach out and experience as much as Curai has to offer. Find out what brought others to Curai and what they envision for the future. Make connections. I recommend those actions despite the intentions of your internship. If Curai is somewhere you want to stay, establish yourself. Show your value and solidify yourself in impactful projects.

[Alexa] ++ to what everyone else has said! Working hard is important as is taking breaks. Stay curious — be inquisitive and ask questions. Have a dedicated work set up at home and learn how to prioritize your tasks. There may be times when you need to take up multiple roles so it’s important to manage your time properly. Take some time to learn more about the company and the people, and find out if your interests are aligned.

[Niraj] 100% agreed with the responses of my colleagues. I would also add, Curai is a great place to find diverse sets of teams all working towards the same mission. I would encourage discussing your project beyond immediate stakeholders, to get more perspectives and ideas. Be nimble and prepare yourself to wear multiple hats, as there are ample opportunities to grow in and beyond your role. If you like the company mission and are passionate about your growth, join Curai! :)