We pull all of our course evaluation data out of CES through the API at the end of every project to put into a separate database. We occasionally get requests for custom reports that can't be done directly in CES, so the separate database allows us to do this. For awhile now, my colleague who pulls the data has run into issues with the API request/connection timing out. He basically has to babysit the process for hours so when it times out, he can kick it off again until it completes.
Just curious if others pull all of their project data out of CES through the API and if you've also run into timeout issues and if you've come up with any ways around this without having to babysit the process. We are starting to look into doing some reporting immediately at the end of our projects that will be time-sensitive and the timeout issue is going to be a limiting factor in our being able to do that.
We ingestdata via API into our data platform, but I'm not involved in the underlying data architecture so I can't speak to the best technical solution.
It may be worth looking at whether you're pulling the entire dataset each time versus incremental changes. I have not heard of any time out issues for us, and our data model is updated each night.
Hopefully someone with data engineering experience can weigh in with more specific guidance?
I am not directly involved with this process but reached out to my team. This is their response:
"We were for a while. At the beginning of this year, I switched the largest data sets to append loads and now only pull the last 90 days of data for them. I think that resolved the issues we were seeing."
Thanks, Sharleena and Laura! I don't know if incremental data pulls would solve this for us. We're just pulling a single project's worth of data once the evals for that project close. The data isn't changing after they close, so we really just need a one-time pull. My colleague did confirm that he is “pulling the rawdata object which is the most data intensive since it's all the evaluation responses.” Maybe that's why we're hitting the timeout?
I don't know if this would be the same thing, because I am unsure what API is. However, we were needing to download Instructor Means data for several semester projects using the Export function to download to Excel. We were having issues trying to do so. I contacted support and it appears it was a timing issue. I was able to work with them to get the data I needed.
Comments
We ingestdata via API into our data platform, but I'm not involved in the underlying data architecture so I can't speak to the best technical solution.
It may be worth looking at whether you're pulling the entire dataset each time versus incremental changes. I have not heard of any time out issues for us, and our data model is updated each night.
Hopefully someone with data engineering experience can weigh in with more specific guidance?
I am not directly involved with this process but reached out to my team. This is their response:
"We were for a while. At the beginning of this year, I switched the largest data sets to append loads and now only pull the last 90 days of data for them. I think that resolved the issues we were seeing."
Hope it helps.
Thanks, Sharleena and Laura! I don't know if incremental data pulls would solve this for us. We're just pulling a single project's worth of data once the evals for that project close. The data isn't changing after they close, so we really just need a one-time pull. My colleague did confirm that he is “pulling the rawdata object which is the most data intensive since it's all the evaluation responses.” Maybe that's why we're hitting the timeout?
Thanks!
Greg
I don't know if this would be the same thing, because I am unsure what API is. However, we were needing to download Instructor Means data for several semester projects using the Export function to download to Excel. We were having issues trying to do so. I contacted support and it appears it was a timing issue. I was able to work with them to get the data I needed.
Please sign in to leave a comment.