On this tutorial, we show a completely useful and modular knowledge evaluation pipeline utilizing the Lilac library, with out counting on sign processing. It combines Lilac’s dataset administration capabilities with Python’s useful programming paradigm to create a clear, extensible workflow. From organising a undertaking and producing reasonable pattern knowledge to extracting insights and exporting filtered outputs, the tutorial emphasizes reusable, testable code buildings. Core useful utilities, resembling pipe, map_over, and filter_by, are used to construct a declarative stream, whereas Pandas facilitates detailed knowledge transformations and high quality evaluation.
!pip set up lilac[all] pandas numpyTo get began, we set up the required libraries utilizing the command !pip set up lilac[all] pandas numpy. This ensures now we have the total Lilac suite alongside Pandas and NumPy for easy knowledge dealing with and evaluation. We should always run this in our pocket book earlier than continuing.
import json
import uuid
import pandas as pd
from pathlib import Path
from typing import Listing, Dict, Any, Tuple, Non-compulsory
from functools import cut back, partial
import lilac as llWe import all of the important libraries. These embody json and uuid for dealing with knowledge and producing distinctive undertaking names, pandas for working with knowledge in tabular type, and Path from pathlib for managing directories. We additionally introduce kind hints for improved operate readability and functools for useful composition patterns. Lastly, we import the core Lilac library as ll to handle our datasets.
def pipe(*capabilities):
"""Compose capabilities left to proper (pipe operator)"""
return lambda x: cut back(lambda acc, f: f(acc), capabilities, x)
def map_over(func, iterable):
"""Useful map wrapper"""
return record(map(func, iterable))
def filter_by(predicate, iterable):
"""Useful filter wrapper"""
return record(filter(predicate, iterable))
def create_sample_data() -> Listing[Dict[str, Any]]:
"""Generate reasonable pattern knowledge for evaluation"""
return [
{"id": 1, "text": "What is machine learning?", "category": "tech", "score": 0.9, "tokens": 5},
{"id": 2, "text": "Machine learning is AI subset", "category": "tech", "score": 0.8, "tokens": 6},
{"id": 3, "text": "Contact support for help", "category": "support", "score": 0.7, "tokens": 4},
{"id": 4, "text": "What is machine learning?", "category": "tech", "score": 0.9, "tokens": 5},
{"id": 5, "text": "Deep learning neural networks", "category": "tech", "score": 0.85, "tokens": 4},
{"id": 6, "text": "How to optimize models?", "category": "tech", "score": 0.75, "tokens": 5},
{"id": 7, "text": "Performance tuning guide", "category": "guide", "score": 0.6, "tokens": 3},
{"id": 8, "text": "Advanced optimization techniques", "category": "tech", "score": 0.95, "tokens": 3},
{"id": 9, "text": "Gradient descent algorithm", "category": "tech", "score": 0.88, "tokens": 3},
{"id": 10, "text": "Model evaluation metrics", "category": "tech", "score": 0.82, "tokens": 3},
]On this part, we outline reusable useful utilities. The pipe operate helps us chain transformations clearly, whereas map_over and filter_by enable us to rework or filter iterable knowledge functionally. Then, we create a pattern dataset that mimics real-world data, that includes fields resembling textual content, class, rating, and tokens, which we’ll later use to show Lilac’s knowledge curation capabilities.
def setup_lilac_project(project_name: str) -> str:
"""Initialize Lilac undertaking listing"""
project_dir = f"./{project_name}-{uuid.uuid4().hex[:6]}"
Path(project_dir).mkdir(exist_ok=True)
ll.set_project_dir(project_dir)
return project_dir
def create_dataset_from_data(title: str, knowledge: Listing[Dict]) -> ll.Dataset:
"""Create Lilac dataset from knowledge"""
data_file = f"{title}.jsonl"
with open(data_file, 'w') as f:
for merchandise in knowledge:
f.write(json.dumps(merchandise) + 'n')
config = ll.DatasetConfig(
namespace="tutorial",
title=title,
supply=ll.sources.JSONSource(filepaths=[data_file])
)
return ll.create_dataset(config)With the setup_lilac_project operate, we initialize a singular working listing for our Lilac undertaking and register it utilizing Lilac’s API. Utilizing create_dataset_from_data, we convert our uncooked record of dictionaries right into a .jsonl file and create a Lilac dataset by defining its configuration. This prepares the information for clear and structured evaluation.
def extract_dataframe(dataset: ll.Dataset, fields: Listing[str]) -> pd.DataFrame:
"""Extract knowledge as pandas DataFrame"""
return dataset.to_pandas(fields)
def apply_functional_filters(df: pd.DataFrame) -> Dict[str, pd.DataFrame]:
"""Apply numerous filters and return a number of filtered variations"""
filters = {
'high_score': lambda df: df[df['score'] >= 0.8],
'tech_category': lambda df: df[df['category'] == 'tech'],
'min_tokens': lambda df: df[df['tokens'] >= 4],
'no_duplicates': lambda df: df.drop_duplicates(subset=['text'], maintain='first'),
'combined_quality': lambda df: df[(df['score'] >= 0.8) & (df['tokens'] >= 3) & (df['category'] == 'tech')]
}
return {title: filter_func(df.copy()) for title, filter_func in filters.objects()}We extract the dataset right into a Pandas DataFrame utilizing extract_dataframe, which permits us to work with chosen fields in a well-recognized format. Then, utilizing apply_functional_filters, we outline and apply a set of logical filters, resembling high-score choice, category-based filtering, token rely constraints, duplicate elimination, and composite high quality situations, to generate a number of filtered views of the information.
def analyze_data_quality(df: pd.DataFrame) -> Dict[str, Any]:
"""Analyze knowledge high quality metrics"""
return {
'total_records': len(df),
'unique_texts': df['text'].nunique(),
'duplicate_rate': 1 - (df['text'].nunique() / len(df)),
'avg_score': df['score'].imply(),
'category_distribution': df['category'].value_counts().to_dict(),
'score_distribution': {
'excessive': len(df[df['score'] >= 0.8]),
'medium': len(df[(df['score'] >= 0.6) & (df['score'] < 0.8)]),
'low': len(df[df['score'] < 0.6])
},
'token_stats': {
'imply': df['tokens'].imply(),
'min': df['tokens'].min(),
'max': df['tokens'].max()
}
}
def create_data_transformations() -> Dict[str, callable]:
"""Create numerous knowledge transformation capabilities"""
return {
'normalize_scores': lambda df: df.assign(norm_score=df['score'] / df['score'].max()),
'add_length_category': lambda df: df.assign(
length_cat=pd.reduce(df['tokens'], bins=[0, 3, 5, float('inf')], labels=['short', 'medium', 'long'])
),
'add_quality_tier': lambda df: df.assign(
quality_tier=pd.reduce(df['score'], bins=[0, 0.6, 0.8, 1.0], labels=['low', 'medium', 'high'])
),
'add_category_rank': lambda df: df.assign(
category_rank=df.groupby('class')['score'].rank(ascending=False)
)
}To guage the dataset high quality, we use analyze_data_quality, which helps us measure key metrics like whole and distinctive data, duplicate charges, class breakdowns, and rating/token distributions. This provides us a transparent image of the dataset’s readiness and reliability. We additionally outline transformation capabilities utilizing create_data_transformations, enabling enhancements resembling rating normalization, token-length categorization, high quality tier task, and intra-category rating.
def apply_transformations(df: pd.DataFrame, transform_names: Listing[str]) -> pd.DataFrame:
"""Apply chosen transformations"""
transformations = create_data_transformations()
selected_transforms = [transformations[name] for title in transform_names if title in transformations]
return pipe(*selected_transforms)(df.copy()) if selected_transforms else df
def export_filtered_data(filtered_datasets: Dict[str, pd.DataFrame], output_dir: str) -> None:
"""Export filtered datasets to information"""
Path(output_dir).mkdir(exist_ok=True)
for title, df in filtered_datasets.objects():
output_file = Path(output_dir) / f"{title}_filtered.jsonl"
with open(output_file, 'w') as f:
for _, row in df.iterrows():
f.write(json.dumps(row.to_dict()) + 'n')
print(f"Exported {len(df)} data to {output_file}")Then, by means of apply_transformations, we selectively apply the wanted transformations in a useful chain, making certain our knowledge is enriched and structured. As soon as filtered, we use export_filtered_data to put in writing every dataset variant right into a separate .jsonl file. This allows us to retailer subsets, resembling high-quality entries or non-duplicate data, in an organized format for downstream use.
def main_analysis_pipeline():
"""Major evaluation pipeline demonstrating useful strategy"""
print("🚀 Organising Lilac undertaking...")
project_dir = setup_lilac_project("advanced_tutorial")
print("📊 Creating pattern dataset...")
sample_data = create_sample_data()
dataset = create_dataset_from_data("sample_data", sample_data)
print("📋 Extracting knowledge...")
df = extract_dataframe(dataset, ['id', 'text', 'category', 'score', 'tokens'])
print("🔍 Analyzing knowledge high quality...")
quality_report = analyze_data_quality(df)
print(f"Unique knowledge: {quality_report['total_records']} data")
print(f"Duplicates: {quality_report['duplicate_rate']:.1%}")
print(f"Common rating: {quality_report['avg_score']:.2f}")
print("🔄 Making use of transformations...")
transformed_df = apply_transformations(df, ['normalize_scores', 'add_length_category', 'add_quality_tier'])
print("🎯 Making use of filters...")
filtered_datasets = apply_functional_filters(transformed_df)
print("n📈 Filter Outcomes:")
for title, filtered_df in filtered_datasets.objects():
print(f" {title}: {len(filtered_df)} data")
print("💾 Exporting filtered datasets...")
export_filtered_data(filtered_datasets, f"{project_dir}/exports")
print("n🏆 High High quality Information:")
best_quality = filtered_datasets['combined_quality'].head(3)
for _, row in best_quality.iterrows():
print(f" • {row['text']} (rating: {row['score']}, class: {row['category']})")
return {
'original_data': df,
'transformed_data': transformed_df,
'filtered_data': filtered_datasets,
'quality_report': quality_report
}
if __name__ == "__main__":
outcomes = main_analysis_pipeline()
print("n✅ Evaluation full! Test the exports folder for filtered datasets.")Lastly, within the main_analysis_pipeline, we execute the total workflow, from setup to knowledge export, showcasing how Lilac, mixed with useful programming, permits us to construct modular, scalable, and expressive pipelines. We even print out the top-quality entries as a fast snapshot. This operate represents our full knowledge curation loop, powered by Lilac.
In conclusion, customers may have gained a hands-on understanding of making a reproducible knowledge pipeline that leverages Lilac’s dataset abstractions and useful programming patterns for scalable, clear evaluation. The pipeline covers all important phases, together with dataset creation, transformation, filtering, high quality evaluation, and export, providing flexibility for each experimentation and deployment. It additionally demonstrates how one can embed significant metadata resembling normalized scores, high quality tiers, and size classes, which could be instrumental in downstream duties like modeling or human evaluate.
Try the Codes. All credit score for this analysis goes to the researchers of this undertaking. Additionally, be happy to observe us on Twitter and don’t neglect to affix our 100k+ ML SubReddit and Subscribe to our Publication.
Nikhil is an intern advisor at Marktechpost. He’s pursuing an built-in twin diploma in Supplies on the Indian Institute of Expertise, Kharagpur. Nikhil is an AI/ML fanatic who’s at all times researching purposes in fields like biomaterials and biomedical science. With a powerful background in Materials Science, he’s exploring new developments and creating alternatives to contribute.
