Skip to content

Commit

Permalink
Update ALN file merge.yml
Browse files Browse the repository at this point in the history
remove timestamp on file name
  • Loading branch information
LynnMHouston authored Jan 17, 2025
1 parent cd3bb4b commit b404e91
Showing 1 changed file with 8 additions and 20 deletions.
28 changes: 8 additions & 20 deletions .github/workflows/ALN file merge.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
name: ALN File Merge

on:
workflow_dispatch: # Enables manual trigger
workflow_dispatch:

jobs:
process_and_update:
Expand All @@ -17,24 +17,16 @@ jobs:
python-version: '3.x'

- name: Install Dependencies
run: |
pip install pandas
pip install openpyxl
pip install jsonnet
pip install xkcdpass
sudo apt-get update && sudo apt-get install -y jsonnet
run: pip install pandas openpyxl

- name: Merge, Clean, and Standardize CSV Files
run: |
python -c "
import os
import pandas as pd
from datetime import datetime
import glob
folder = './backend/schemas/source/data/ALNs_raw_downloads'
date_suffix = datetime.now().strftime('%Y%m%d')
output_file = f'./backend/schemas/source/data/cfda-lookup-{date_suffix}.csv'
output_file = './backend/schemas/source/data/cfda-lookup.csv'
print(f'Looking for CSV files in: {folder}')
csv_files = glob.glob(f'{folder}/*.csv')
Expand All @@ -54,9 +46,6 @@ jobs:
all_data.append(df)
combined_data = pd.concat(all_data, ignore_index=True)
all_columns = combined_data.columns.unique()
standardized_data = combined_data.reindex(columns=all_columns, fill_value=None)
column_mapping = {
'Title': 'Program Title',
'Assistance Listings Number': 'Program Number',
Expand All @@ -70,10 +59,9 @@ jobs:
'Types of Assistance': 'Types of Assistance'
}
standardized_data = standardized_data.rename(columns=column_mapping)
print(f'Saving merged and standardized CSV to: {output_file}')
standardized_data.to_csv(output_file, index=False, encoding='utf-8')
print('CSV processing completed successfully.')
combined_data = combined_data.rename(columns=column_mapping)
combined_data.to_csv(output_file, index=False, encoding='utf-8')
print(f'Merged CSV saved to {output_file}')
"
- name: Commit and Push Merged File
Expand All @@ -82,8 +70,8 @@ jobs:
run: |
git config user.name "github-actions[bot]"
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
git add ./backend/schemas/source/data/cfda-lookup-*.csv
git diff --cached --quiet || git commit -m "Add merged and standardized CSV file"
git add ./backend/schemas/source/data/cfda-lookup.csv
git commit -m "Update cfda-lookup.csv file"
git push
- name: Run Make All
Expand Down

0 comments on commit b404e91

Please sign in to comment.