Noureddine RAMDI / linkedin_scraper: async Playwright-powered LinkedIn scraping with typed data models

Created Sat, 23 May 2026 20:41:14 +0000 Modified Sat, 23 May 2026 20:41:27 +0000
813 Words
github-stars python playwright web-scraping asyncio linkedin pydantic

joeyism/linkedin_scraper

LinkedIn’s data is valuable but notoriously tricky to scrape reliably due to dynamic content and strict bot detection. linkedin_scraper tackles this by rewriting its core from Selenium to Playwright in version 3.0.0, embracing Python’s async/await paradigm and introducing strong typing with Pydantic models. This shift makes it a robust example of modern scraping architecture balancing performance, structure, and developer experience.

what linkedin_scraper does and its architecture

linkedin_scraper is a Python library designed to scrape multiple LinkedIn entities: individual profiles, companies, jobs, and company posts. It automates browser interaction using Playwright, a newer alternative to Selenium, which provides better concurrency and reliability. The library supports authenticated sessions, enabling reuse of login cookies to bypass LinkedIn’s login walls.

Under the hood, linkedin_scraper launches a Chromium browser instance controlled asynchronously. It exposes scrapers for different LinkedIn entities — PersonScraper, CompanyScraper, JobSearchScraper — each encapsulating logic to navigate pages, wait for dynamic content, and extract structured data.

A key architectural choice is returning all scraped data as Pydantic models. This means every profile, company, or job scraped has a predictable, typed Python object structure, which improves downstream data handling and validation.

The library supports both headless and visible browser modes and allows manual or programmatic login flows. It also offers progress callbacks so you can monitor scraping progress in real time.

The tech stack is Python 3.7+, Playwright (with Chromium), asyncio for concurrency, and Pydantic for data models.

technical strengths and design tradeoffs

The move from Selenium to Playwright is the most significant technical improvement. Playwright’s native async API fits Python’s async/await ecosystem perfectly, enabling concurrent scraping operations without blocking. This is a solid architecture decision for a scraping library that often waits on network and rendering.

Using Pydantic models to represent scraped entities is another strong point. It enforces a clear schema for scraped data, making it easier to catch changes when LinkedIn updates its page structure and simplifying integration with downstream pipelines.

Session management is handled thoughtfully. linkedin_scraper can load and save session cookies to JSON files, allowing authenticated scraping sessions to persist across runs. This avoids the overhead and friction of logging in repeatedly.

Progress callbacks provide hooks for users to track scraping state or update UIs, improving developer experience.

The tradeoff is the inherent fragility of scraping LinkedIn’s dynamic and frequently changing UI. Even with Playwright’s resilience, selectors and scraping logic require maintenance over time.

Another limitation is the dependency on installing Playwright’s Chromium browser separately, which can add a setup step and impact portability.

The codebase itself is reasonably clean, with async functions throughout, clear separation of concerns (browser management, scraping logic, data models), and documented usage patterns. However, the need to manage browser sessions and cookies adds complexity for newcomers.

quick start with linkedin_scraper

Installation is straightforward from PyPI:

pip install linkedin-scraper

Then you need to install Playwright’s Chromium browser:

playwright install chromium

Here’s a minimal async example scraping a LinkedIn profile:

import asyncio
from linkedin_scraper import BrowserManager, PersonScraper

async def main():
    # Initialize browser
    async with BrowserManager(headless=False) as browser:
        # Load authenticated session
        await browser.load_session("session.json")
        
        # Create scraper
        scraper = PersonScraper(browser.page)
        
        # Scrape a profile
        person = await scraper.scrape("https://linkedin.com/in/williamhgates/")
        
        # Access data
        print(f"Name: {person.name}")
        print(f"Headline: {person.headline}")
        print(f"Location: {person.location}")
        print(f"Experiences: {len(person.experiences)}")
        print(f"Education: {len(person.educations)}")

asyncio.run(main())

There are similar scrapers for companies and jobs with analogous usage patterns. The library’s async nature means you can run multiple scrapes concurrently if needed.

verdict

linkedin_scraper is a practical, modern Python library for LinkedIn scraping that demonstrates a clean async architecture with Playwright and typed data models. It’s well suited for developers needing structured LinkedIn data extraction in their Python projects.

The tradeoff is the usual fragility of scraping complex dynamic sites like LinkedIn and the setup overhead of Playwright’s browser dependencies. Managing authenticated sessions requires some manual steps but helps bypass LinkedIn’s login hurdles.

If you want a robust starting point for LinkedIn scraping with modern Python async patterns and strong data validation, linkedin_scraper is worth evaluating. Its design choices reflect thoughtful tradeoffs between reliability, developer experience, and maintainability.

→ GitHub Repo: joeyism/linkedin_scraper ⭐ 4,146 · Python