Skip to content

Commit

Permalink
out of the box comparisons
Browse files Browse the repository at this point in the history
  • Loading branch information
RobinL committed Jun 22, 2024
1 parent 379e081 commit 84dc62b
Show file tree
Hide file tree
Showing 6 changed files with 2,180 additions and 994 deletions.
14 changes: 3 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,19 +39,16 @@ and clusters these links to produce an estimated person ID:

## What data does Splink work best with?

Before using Splink, input data should be standardised, with consistent column names and formatting (e.g., lowercased, punctuation cleaned up, etc.).

Splink performs best with input data containing **multiple** columns that are **not highly correlated**. For instance, if the entity type is persons, you may have columns for full name, date of birth, and city. If the entity type is companies, you could have columns for name, turnover, sector, and telephone number.

High correlation occurs when the value of a column is highly constrained (predictable) from the value of another column. For example, a 'city' field is almost perfectly correlated with 'postcode'. Gender is highly correlated with 'first name'. Correlation is particularly problematic if **all** of your input columns are highly correlated.
High correlation occurs when one column is highly predictable from another - for instance, city can be predicted from postcode. Correlation is particularly problematic if **all** of your input columns are highly correlated.

Splink is not designed for linking a single column containing a 'bag of words'. For example, a table with a single 'company name' column, and no other details.

## Documentation

The homepage for the Splink documentation can be found [here](https://moj-analytical-services.github.io/splink/). Interactive demos can be found [here](https://github.com/moj-analytical-services/splink/tree/master/docs/demos), or by clicking the following Binder link:
The homepage for the Splink documentation can be found [here](https://moj-analytical-services.github.io/splink/), including a [tutorial](https://moj-analytical-services.github.io/splink/demos/tutorials/00_Tutorial_Introduction.html) and [examples](https://moj-analytical-services.github.io/splink/demos/examples/examples_index.html) that can be run in the browser.

[![Binder](https://mybinder.org/badge.svg)](https://mybinder.org/v2/gh/moj-analytical-services/splink/binder_branch?labpath=docs%2Fdemos%2Ftutorials%2F00_Tutorial_Introduction.ipynb)

The specification of the Fellegi Sunter statistical model behind `splink` is similar as that used in the R [fastLink package](https://github.com/kosukeimai/fastLink). Accompanying the fastLink package is an [academic paper](http://imai.fas.harvard.edu/research/files/linkage.pdf) that describes this model. The [Splink documentation site](https://moj-analytical-services.github.io/splink/topic_guides/fellegi_sunter.html) and a [series of interactive articles](https://www.robinlinacre.com/probabilistic_linkage/) also explores the theory behind Splink.

Expand Down Expand Up @@ -164,22 +161,17 @@ df_clusters = clusters.as_pandas_dataframe(limit=5)
- [A introductory presentation on Splink](https://www.youtube.com/watch?v=msz3T741KQI)
- [An introduction to the Splink Comparison Viewer dashboard](https://www.youtube.com/watch?v=DNvCMqjipis)

## Charts Gallery

You can see all of the interactive charts provided in Splink by checking out the [Charts Gallery](https://moj-analytical-services.github.io/splink/charts/index.html).

## Support

To find the best place to ask a question, report a bug or get general advice, please refer to our [Contributing Guide](./CONTRIBUTING.md).
To find the best place to ask a question, report a bug or get general advice, please refer to our [Guide](./CONTRIBUTING.md).

## Use Cases

To see how users are using Splink in the wild, check out the [Use Cases](https://moj-analytical-services.github.io/splink/#use-cases) section of the docs.

## Awards

❓ Future of Government Awards 2023: Open Source Creation - [Shortlisted, result to be announced shortly](https://futureofgovernment.com/en)

🥈 Civil Service Awards 2023: Best Use of Data, Science, and Technology - [Runner up](https://www.civilserviceawards.com/best-use-of-data-science-and-technology-award-2/)

🥇 Analysis in Government Awards 2022: People's Choice Award - [Winner](https://analysisfunction.civilservice.gov.uk/news/announcing-the-winner-of-the-first-analysis-in-government-peoples-choice-award/)
Expand Down
2,404 changes: 1,978 additions & 426 deletions docs/demos/examples/sqlite/dashboards/50k_cluster.html

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion docs/demos/tutorials/04_Estimating_model_parameters.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@
"| Rob | Jane | All other | bad match |\n",
"| Rob | Robert | All other | bad match, this comparison has no notion of nicknames |\n",
"\n",
"More information about comparisons can be found [here](https://moj-analytical-services.github.io/splink/comparison.html).\n",
"More information about specifying comparisons can be found [here](../../topic_guides//comparisons/customising_comparisons.ipynb) and [here](../../topic_guides//comparisons/comparisons_and_comparison_levels.md).\n",
"\n",
"We will now use these concepts to build a data linking model.\n"
]
Expand Down
Loading

0 comments on commit 84dc62b

Please sign in to comment.