Fun with Data: Unveiling the Quirks of a Local News Website
0:00:19 Introduction: Alex, inspired by mirror mining, undertakes a hobby project to analyze the local news website HL-Live, which serves the city of Lübeck, Germany.
0:01:54 Motivation: After moving to Lübeck in 2022, Alex sought to understand the local culture and community through their online interactions on HL-Live.
0:02:24 Website Overview: HL-Live, despite its outdated design, is a popular platform for local news and discussions in Lübeck.
0:02:57 No Paywalls: The site operates without paywalls, relying on advertising for revenue, including Google Ads and Outbrain recommendations.
0:04:21 News Content: The platform publishes a mix of local news, press releases, event announcements, and occasional "heavyweight" articles.
0:05:51 AFD Press Releases: HL-Live publishes press releases from the AFD (Alternative for Germany) political party, drawing criticism for providing them a platform.
0:06:46 Commenting System: Users can comment on articles, but comments are moderated and often deleted. The system allows for anonymous comments, although users can choose to display their email addresses.
0:08:27 Data Collection: Alex collects articles and comments from HL-Live, noting that comments are deleted after two weeks and articles after three years.
0:08:43 Limited Commenting: Comments are only allowed on weekdays, with the commenting feature disabled on weekends.
0:09:26 Project Development: Alex creates a system to scrape and store articles and comments from HL-Live, using a graph database (Neo4j) to analyze the data.
0:11:03 ID-Based System: The website's use of sequential IDs for articles enables easy scraping by incrementing the ID number.
0:12:05 Data Inconsistencies: Alex encounters numerous errors and inconsistencies in the website's data, including incorrect dates and missing titles.
0:14:51 Keyword Extraction: Despite the website's broken JSON implementation, Alex successfully extracts keywords associated with articles, which serve as categories.
0:17:08 "Functionally Broken Award": As a humorous response to the website's flaws, Alex creates a mock award for HL-Live.
0:17:27 Data Analysis: Alex analyzes the collected data, finding that most articles are about Lübeck, with a significant portion originating from press releases.
0:18:13 Content Analysis: An attempt to correlate negative sentiment with engagement yields no significant results.
0:18:59 Low-Information Article: Using a formula to calculate information content, Alex identifies an article with minimal informative value.
0:20:19 Top Commented Topics: Alex identifies the most commented-on topics for the current and previous years, revealing recurring themes like traffic issues, construction, and political debates.
0:23:30 User Analysis: Although users can choose different names, some consistently use the same name. One user, "Martin," stands out for having 839 comments.
0:24:28 Bot Detection: Cluster analysis does not reveal any clear evidence of bot activity in the comments.
0:25:03 User "Dissident": A user initially critical of immigration later apologizes and accuses someone else of impersonating him, leading to an analysis of his commenting patterns.
0:27:36 User Profiling: Based on his comments, "Dissident" is profiled as likely being older, conservative, and critical of progressive ideas.
0:28:38 AI-Generated Profile: ChatGPT is used to create a profile of "Dissident," which aligns with the manual analysis.
0:29:19 AFD Engagement: Analysis shows that the AFD has minimal engagement on HL-Live compared to other political parties.
0:30:13 Category Analysis: The website has 38,000 categories, many of which are nonsensical or automatically generated.
0:32:34 Weekend Comment Workaround: Alex creates a website that allows users to comment on HL-Live articles even on weekends when commenting is normally disabled.
0:32:50 Comment Game: A game is developed where users guess which article a given comment belongs to, highlighting the often-absurd nature of the comments.
0:35:55 Future Article Prediction: Alex discovers that unpublished articles are accessible by incrementing the ID, allowing for a glimpse into future news. A dedicated website, hl-future.de, is created to display these upcoming articles with a countdown.
0:37:45 Call to Action: Alex encourages viewers to engage in constructive online discussions, attend demonstrations, and support journalism.
0:39:28 Q&A: The talk concludes with a Q&A session, addressing topics like contacting HL-Live, translating comments, and the possibility of bot activity.
0:41:10 Other Newspapers: Alex is asked if he has tried this with other newspapers and mentions the article from 2055 is likely a typo. He is open to suggestions for other newspapers to try this on.
0:48:43 Commenting Patterns: They discuss patterns observed, such as articles published before the weekend receiving fewer comments. The commenting window is typically around 2.5 days, but it can be shorter if an article attracts too many comments deemed "bot-like."
0:50:02 Weekend Hackers: Alex is asked if there are users who post comments during the weekend, indicating they may have found a workaround. Alex believes it's more likely that these users started their comments before the cutoff time and submitted them later.