Use LLMs for Data Science Tasks

Large Language Models (LLMs) are powerful AI tools. They understand and generate human-like text, which makes them incredibly useful for data science. They help with everything from writing and debugging code to explaining concepts and cleaning data. This makes your workflow more efficient and less prone to errors.

Understanding LLMs in Data Science

LLMs won't replace data scientists. Think of them as capable assistants. They automate repetitive tasks, give quick insights, and act as a sounding board for ideas. If you're a fresher in data science, learning to use these tools effectively can really boost your productivity and learning.

Imagine you need to analyze a new dataset. Normally, this means a lot of manual exploration, coding, and debugging. With an LLM, you can offload some of that work. Ask it to generate SQL queries, explain Python errors, or brainstorm feature engineering ideas. This lets you focus on the more strategic, critical thinking parts of your job.

One common mistake: thinking LLMs are always right. They're not. They can "hallucinate" or give wrong information. Always check their output, especially for critical tasks. Treat their responses as suggestions, not final answers. Many beginners blindly trust AI without checking its output—don't make this mistake.

Practical Applications of LLMs

Let's look at practical ways to use LLMs in your data science toolkit.

1. Code Generation and Debugging

This is a big benefit. If you're stuck on a Python function or need a SQL query, an LLM can often generate the basic structure or even a full solution. It's also great for debugging. Instead of digging through documentation, paste an error message and ask the LLM for an explanation and possible fixes.

Say you're using the ecm_orders and ecm_customers tables from DataStride's e-commerce dataset. You want to find the total orders from 'Mumbai' customers who signed up in 2023.

Prompt:

"Write a SQL query to find the total number of orders placed by customers from 'Mumbai' who signed up in 2023. Use the tables ecm_orders (columns: order_id, customer_id, order_date) and ecm_customers (columns: customer_id, city, signup_date)."

LLM Generated SQL (example output):

SELECT
    COUNT(o.order_id) AS total_orders
FROM
    ecm_orders AS o
JOIN
    ecm_customers AS c ON o.customer_id = c.customer_id
WHERE
    c.city = 'Mumbai'
    AND STRFTIME('%Y', c.signup_date) = '2023';

This query is a good start. Run it in your SQL environment and check the results. If there's an error, paste it back into the LLM for more help.

2. Data Cleaning and Preprocessing Assistance

Messy data is always there for a data scientist. LLMs help you brainstorm cleaning strategies, write regular expressions, or generate Python code for common cleaning tasks. This includes handling missing values, standardising formats, or removing duplicates.

Imagine you have a sales extract, and the product_name column has inconsistent casing and extra spaces. You need to clean it up.

Prompt:

"I have a Pandas DataFrame df with a column product_name. It has inconsistent casing (e.g., 'Laptop', 'laptop', 'LAPTOP') and leading/trailing spaces. Write Python code using Pandas to convert all product names to title case and remove extra spaces."

LLM Generated Python Code (example output):

import pandas as pd

# Assuming 'df' is your DataFrame
# Example DataFrame (replace with your actual data)
data = {'product_name': ['  laptop  ', 'Mouse', 'KEYBOARD ', 'Monitor', 'TABLET']}
df = pd.DataFrame(data)

print("Original DataFrame:")
print(df)

df['product_name'] = df['product_name'].str.strip().str.title()

print("\nCleaned DataFrame:")
print(df)

This code directly solves the problem. Adapt it to your DataFrame and check the transformation.

3. Explaining Concepts and Algorithms

As a fresher, you'll see many new concepts. Instead of just reading definitions, ask an LLM to explain complex algorithms or statistical concepts simply. It can give analogies or generate code examples to show them. This speeds up your learning a lot.

Prompt:

"Explain the concept of 'p-value' in hypothesis testing to a beginner in data science, using a simple, real-world example."

LLM Generated Explanation (example summary):

An LLM would explain that a p-value helps you decide if your data likely happened by chance, assuming a null hypothesis is true. It might use an example like testing if a new fertilizer improves crop yield. A small p-value (e.g., less than 0.05) suggests the improvement probably isn't random, so you might reject the idea that the fertilizer has no effect.

4. Feature Engineering Ideas

When you build models, creating good features is key. Describe your dataset and problem, and an LLM can suggest features or transformations. For example, if you're predicting customer churn, ask for ideas based on signup_date, order_date, and payment_method from the ecm_customers and ecm_orders tables.

Prompt:

"I am trying to predict customer churn using ecm_customers and ecm_orders tables. Given columns like signup_date, order_date, payment_method, and is_prime, what are some good feature engineering ideas to create new features that might be useful for a churn prediction model?"

LLM Generated Ideas (example summary):

An LLM might suggest features like:

• Customer tenure: Days between signup_date and today.
• Recency: Days since the last order_date.
• Frequency: Total number of orders.
• Monetary value: Total spend (if ecm_order_items has it).
• Payment method diversity: Number of unique payment_method used.
• Prime status duration: How long a customer has been is_prime.
• Order frequency per month/quarter: COUNT(order_id) grouped by customer_id and time periods.

These suggestions can spark new ideas and make sure you don't miss obvious feature opportunities.

5. Documenting Code and Generating Explanations

Good documentation is important, but often forgotten. LLMs help by generating docstrings for your Python functions or explaining complex code blocks. This saves you time and makes code easier to read for others (and your future self).

Prompt:

"Generate a Python docstring for the following function that calculates the average order value for a given customer ID from a Pandas DataFrame orders_df containing customer_id, order_id, total_amount columns."

LLM Generated Docstring (example output):

def calculate_average_order_value(df: pd.DataFrame, customer_id: int) -> float:
    """
    Calculates the average order value for a specific customer.

    Args:
        df (pd.DataFrame): A Pandas DataFrame containing order data with
                           'customer_id', 'order_id', and 'total_amount' columns.
        customer_id (int): The ID of the customer for whom to calculate the average.

    Returns:
        float: The average order value for the specified customer.
               Returns 0.0 if the customer has no orders or is not found.
    """
    customer_orders = df[df['customer_id'] == customer_id]
    if not customer_orders.empty:
        return customer_orders['total_amount'].mean()
    return 0.0

This automatically generated docstring clearly explains the function, its arguments, and return types. It's great for keeping code quality high.

Key takeaways

• LLMs are powerful assistants for data scientists, not replacements. They help with tasks from coding to understanding concepts.
• Always check LLM outputs, especially for critical tasks, because they can give wrong or 'hallucinated' information.
• Use LLMs to generate SQL queries, Python code for data cleaning, and debug errors efficiently.
• Ask LLMs to explain complex data science concepts or brainstorm feature engineering ideas for your models.
• Use LLMs in your workflow to automate documentation and make code easier to read. This saves you valuable time.