Machine Learning (ML) technologies such as classification models, image recognition, and neural representation learning are finding their ways into the literature on digital humanities. The predictive power of current tools has grown larger and with the digitalization of large humanities corpora, it is expected that we have only seen the early beginning of the use of complex ML models in the field of digital humanities. When ML is being used in research it calls for reflections upon what a field can gain from such technologies. With added predictive power also comes a lack of transparency and with the rise of ML in research methodologies, questions about explainability of these complex computational models follow. In this presentation, I present some operational perspectives on explainability of ML models in digital humanities: How can ML outputs and the field of explainable AI feed into the knowledge tradition of the humanities?