receiving, at a machine learning service of a provider network from a computing device of a user located outside the provider network, a first plurality of unlabeled documents from a domain of the user, a second plurality of labeled documents from the domain of the user, and a request to create a text classifier for the domain;

performing a first training iteration of a language machine learning model on the first plurality of unlabeled documents from the domain of the user to pretrain the language machine learning model for prediction objectives for unlabeled data;

performing a second training iteration of the pretrained language machine learning model on the second plurality of labeled documents to further train the language machine learning model as the text classifier for the domain;

hosting the language machine learning model within the provider network in association with an endpoint;

receiving an inference request at the endpoint for an unlabeled document of the user;

generating, by the language machine learning model, an inference based on the inference request for the unlabeled document of the user; and

transmitting the inference to a client application or to a storage location.