technical case study

AI • Machine learning / Classification

AI Credit Score

Machine learning proof of concept for classifying credit scores from financial and behavioral data.

Data preparation, training, model comparison and prediction Completed POC completed
Python Pandas Scikit-learn Jupyter Notebook Machine Learning
executive summary

Problem

Manual credit score analysis is slow and depends on many financial, behavioral and profile variables.

Solution

A machine learning notebook was built to prepare the data, encode categorical variables, train models and compare accuracy between Random Forest and KNN.

Result

With train and test split by customer, Random Forest reached 64.0% on the test set versus 54.9% for KNN. A row-level split raises these to 80.3% and 76.0%, but they are inflated because the same customer appears on both sides.

Current stage

POC completed, suitable for technical demos and future work on pipelines, validation and explainability.

context

Risk classification as an applied AI experiment

The project was created to test a complete classification workflow: import a customer dataset, turn text variables into numbers, split train and test, train two algorithms and apply the best model to new customers.

goals and constraints

Goals

  • Prepare financial and behavioral data for machine learning
  • Turn categorical variables into numeric values
  • Split the data into train and test sets
  • Compare Random Forest and KNN
  • Predict credit scores for new customers

Constraints

  • Notebook POC, with no API or production interface
  • Demo model, not suitable for real financial decisions without an audit
  • Synthetic data: performance needs to be validated on a real dataset
  • Sensitive subject, requiring explainability, governance and human review
solution architecture

Front-end

Notebook with tables, model accuracy, confusion matrix, feature importance and the final prediction for new customers.

Back-end

Python workflow with Pandas and Scikit-learn: Pipelines with OrdinalEncoder (Random Forest) and OneHotEncoder + StandardScaler (KNN), GroupShuffleSplit to split train and test by customer, accuracy_score and classification_report. The best model is saved with joblib.

Database

CSV with 50,000 monthly records from 6,250 customers, generated from the statistics of a real dataset (synthetic data), with occupation, annual salary, accounts, cards, loans, delays, credit history, payment behavior and score.

security and safeguards

Responsible use

The project is treated as a technical proof of concept, not as an autonomous tool for approving or rejecting credit.

Future explainability

Because it deals with credit, a real version would require bias analysis, model explainability and auditable criteria.

Train/test split by customer

Each customer has several months of records. The split is done by customer, so the same customer is never in both train and test, and accuracy measures performance on customers the model has never seen.

process
01

Importing the data

Loading the customer dataset and a first inspection of the columns.

02

Preprocessing

Converting occupations, credit mix and payment behavior into numeric values.

03

Splitting the data

Separating input and output variables, with a 70/30 train/test split by customer.

04

Training

Training Random Forest and KNN to compare classification approaches.

05

Prediction

Applying the chosen model to new customers to predict their score class.

behind the scenes
src/modelo.py Python lines 34–73
def criar_modelos() -> dict[str, Pipeline]:
    #Monta os dois modelos como pipelines
    #Entao a conversão aprendida no treino e exatamente a mesma usada nos novos clientes(nao precisa refazer)

    #Random Forest: arvores nao precisam de escala, basta numerar as categorias
    preparo_floresta = ColumnTransformer(
        [
            (
                "categoricas",
                OrdinalEncoder(handle_unknown="use_encoded_value", unknown_value=-1),
                COLUNAS_CATEGORICAS,
            )
        ],
        remainder="passthrough",
        verbose_feature_names_out=False,
    )

    #KNN: mede as distancias, entao os numeros precisam estar na mesma escala
    #(senao o salario domina tudo) e as categorias viram colunas 0/1
    preparo_knn = ColumnTransformer(
        [
            ("categoricas", OneHotEncoder(handle_unknown="ignore"), COLUNAS_CATEGORICAS),
            ("numericas", StandardScaler(), make_column_selector(dtype_include="number")),
        ]
    )

    return {
        "Random Forest": Pipeline(
            [
                ("preparo", preparo_floresta),
                ("modelo", RandomForestClassifier(random_state=SEMENTE, n_jobs=-1)),
            ]
        ),
        "KNN": Pipeline(
            [
                ("preparo", preparo_knn),
                ("modelo", KNeighborsClassifier()),
            ]
        ),
    }
Each model is a Pipeline: categories become numbers and, for KNN, the scale is standardized. The preparation learned during training is applied the same way to new customers.
src/modelo.py Python lines 24–31
def dividir_por_cliente(x, y, grupos):
    #Divide em treino e teste sem repetir clientes entre os dois
    #Cada cliente aparece em varios meses. Se a divisao for por linha, o mesmo cliente fica no treino e no teste
    #E a IA "aprende" o cliente em vez de aprender o padrao

    divisor = GroupShuffleSplit(n_splits=1, test_size=PROPORCAO_TESTE, random_state=SEMENTE)
    idx_treino, idx_teste = next(divisor.split(x, y, groups=grupos))
    return x.iloc[idx_treino], x.iloc[idx_teste], y.iloc[idx_treino], y.iloc[idx_teste]
Train and test split by customer with GroupShuffleSplit, so the same customer never appears on both sides.
src/treinar.py Python lines 18–46
tabela = carregar_dados(args.dados)
x, y, grupos = separar_x_y(tabela)
x_treino, x_teste, y_treino, y_teste = dividir_por_cliente(x, y, grupos)

print(f"{len(tabela)} linhas de {grupos.nunique()} clientes")
print(f"Treino: {len(x_treino)} linhas | Teste: {len(x_teste)} linhas\n")

modelos = criar_modelos()
acuracias = {}
for nome, modelo in modelos.items():
    modelo.fit(x_treino, y_treino)
    acuracias[nome] = accuracy_score(y_teste, modelo.predict(x_teste))
    print(f"{nome}: {acuracias[nome]:.1%} de acerto")

melhor = max(acuracias, key=acuracias.get)
print(f"\nMelhor modelo: {melhor}\n")
print(classification_report(y_teste, modelos[melhor].predict(x_teste)))

# Depois de escolher, vc pode treinar o melhor modelo com todos os dados antes de salvar
modelo_final = criar_modelos()[melhor]
modelo_final.fit(x, y)

args.saida.parent.mkdir(parents=True, exist_ok=True)
joblib.dump(modelo_final, args.saida)
args.saida.with_name(ARQUIVO_METRICAS.name).write_text(
    json.dumps({"melhor_modelo": melhor, "acuracia_teste": acuracias}, indent=2, ensure_ascii=False),
    encoding="utf-8",
)
print(f"Modelo salvo em {args.saida}")
Trains and compares Random Forest and KNN, retrains the best one on all the data and saves the model together with its metrics.
src/prever.py Python lines 10–36
def prever(modelo, novos_clientes: pd.DataFrame) -> pd.Series:
    colunas_esperadas = list(modelo.feature_names_in_)
    faltando = set(colunas_esperadas) - set(novos_clientes.columns)

    if faltando:
        raise ValueError(f"Colunas faltando no arquivo de novos clientes: {sorted(faltando)}")

    x = novos_clientes[colunas_esperadas]
    return pd.Series(modelo.predict(x), index=novos_clientes.index, name="score_previsto")


def main():
    parser = argparse.ArgumentParser(description="Prevê o score de crédito de novos clientes.")
    parser.add_argument("arquivo", nargs="?", type=Path, default=ARQUIVO_NOVOS_CLIENTES,
                        help="CSV com os novos clientes")
    parser.add_argument("--modelo", type=Path, default=ARQUIVO_MODELO, help="Modelo salvo por src.treinar")
    parser.add_argument("--saida", type=Path, help="CSV para salvar as previsões (opcional)")
    args = parser.parse_args()

    if not args.modelo.exists():
        raise SystemExit(f"Modelo não encontrado em {args.modelo}. Rode antes: python -m src.treinar")

    modelo = joblib.load(args.modelo)
    novos = pd.read_csv(args.arquivo).drop(columns=[COLUNA_ID, COLUNA_ALVO], errors="ignore")

    novos["score_previsto"] = prever(modelo, novos)
    print(novos[["idade", "profissao", "salario_anual", "score_previsto"]].to_string())
Loads the saved model, checks that the file has the expected columns and predicts the score of new customers without retraining.
impact

Results

With the split by customer, Random Forest reached 64.0% test accuracy and KNN 54.9%. Comparing it with the row-level split (80.3%) showed how inflated the evaluation gets when the same customer appears in both train and test.

Learnings

The project reinforced the importance of preprocessing, model comparison and responsibility when presenting AI applied to credit.