how can i test my model using different dataset in machine learning
i m new in machine learning and i am create a one small project using CountVectorizer model. i am split my data to 80% -20%. 80% for training model and 20% for testing model. my model work properly run on 20% test data but can i used to test my model on different data set that is similar to training data set?
i am using joblib for dump and load my model.
from joblib import dump, load dump(pipe, filename) loaded_model = load('filename') my question is how i directly test my model using different dataset?
Yes, you can use the model to test similar datasets.
However, you must keep in mind the preprocessing step according to the model.
When you trained your model, it was trained on a particular dimension and the size of input would have been AxB matric. When you have a new test sentence or new dataset, you must first do the same preprocessing, otherwise, it will throw dimension mismatch errors.
Example:
suppose you have the following count vectorizer object
cv = CountVectorizer() then you must first fit it on your training dataset, for say
X = dataframe['text_column_name'] X = cv.fit_transform(X) # Fit the Data Once this is done, whenever you have a new sentence, say
test_sentence = "this is a test sentence" then you must use the cv object in the following manner
model_input = cv.transform([test_sentence]).toarray() and then you can make predictions:
model.predict(model_input) This method must be followed even if you wish to test a new dataset which is in a data frame or some other file format.
