Bookmarks (1511)

Streaming Amazon DynamoDB Data into a Centralized Data Lake — The DynamoDB document database works great for serverless apps but what about when business teams want to plug in their BI tools and do analysis? You can use Kinesis Data Streams and Kinesis Data Firehose to pipe the data to S3 for use elsewhere.

How Databases Handle 10 Million Devices in High-Cardinality Benchmarks — When you have lots of sensors (or similar IoT devices) pumping out information on a continuous basis, you end up with data sets with huge cardinality, even if the underlying data is quite small (e.g. a temperature and a time). This post focuses on how the QuestDB benchmark their database against such use cases.

How to Build a Graph Application with Amazon Neptune and AWS Amplify — The coming years are only going to see even more ‘stitching together’ of various data services to meet data processing needs and this is a neat practical example of doing that with AWS’s managed graph database and Amplify for adding a GraphQL API to the mix.

Joining CSV and JSON Data with an In-Memory SQLite Database — sqlite-utils is a collection of Python utility functions for working with SQLite databases and the latest version lets you import CSV and JSON data into an in-memory database, combine and query it, then output the results as you wish.

Start Building with a SQL API in the Cloud — CockroachCloud is a transactional database available as a PostgreSQL API. It’s simple and ops-free. Just connect and instantly start building. Oh, and there is a forever-free plan. No credit cards, no commitments.

Graph Databases: Why Are They Suddenly Popular? — Off the back of last week’s huge round of funding for Neo4j comes this piece (curiously from a director at Oracle) making a high level case for graph databases. I’m not too sure about the suddenly.. this trend has been growing nicely over the past several years. Could I resist leading with an item from Oracle in...

Hosted Monitoring: Evaluating InfluxDB Cloud and Grafana Cloud — Experiences from someone who’s worked on self-hosted Grafana and InfluxDB setups but has explored the potential of managed cloud solutions.

Leveraging BigQuery Public US Boundaries Datasets for Geospatial Analytics — A demonstration of joining first party data onto the BigQuery Public Boundaries Datasets for comprehensive geospatial analytics.

▶  The Rise of 'Data Services' with Patrick McFadin — Developers increasingly want to think less about ‘databases’ and more about data being handled by managed, immediately available cloud services.

Do You Need Redis? Postgres Does Queuing, Locking, and Pub/Sub.. — I’m a huge Redis fan and while it’s common to have a mainstream database (like Postgres) and Redis running side by side, you could potentially just use Postgres for things like job queues or locks.

Neo4j Announces $325 Million Series F Investment — Representing what they claim is the “largest investment in a private database company” ever, the company maintaining the popular Neo4j graph database system is now valued at over $2bn.

Scale Low Latency MySQL Apps, Globally — Distributed just like a CDN, PolyScale intelligently distributes and caches MySQL data to the edge. Implement in minutes without writing code or deploying servers.

rqlite 6.0: The Evolution of a Distributed Database Design — rqlite is a database system that uses SQLite under the hood. This announcement focuses on a new leader-finding approach for nodes and the options considered for the problem.

Exploring Serverless Patterns for Amazon DynamoDB — Learn about some different DynamoDB patterns used in serverless applications and how to lean on the Serverless Patterns Collection to configure DynamoDB as an event source for AWS Lambda functions.

Scheduling Amazon RDS Stops and Starts with AWS Lambda — Amazon Relational Database Service (Amazon RDS) makes it easy to set up, operate, and scale a relational database in the cloud and in situations where you might only want an instance running on a schedule (e.g. a test setup during work hours) Lambda can help out.

▶  On Postgres's Optimizer Methodology — EDB’s Robert Haas, a long time major contributor to Postgres, gave an hour long talk about two key parts of Postgres’s query optimizer: join planning and statistics. This is technical stuff, but Robert does a great job of making reasonably opaque processes accessible and knowing this stuff will really up your Postgres game.

🏆  Select Star SQL: An Interactive SQL Tutorial — Want to get a bit better at SQL? This interactive online guide will give you some practical experience and is oriented, perhaps a bit morbidly, around querying a dataset of death row inmates..

Exploring Performance Differences Between Amazon Aurora and 'Vanilla' MySQL — Aurora is a managed database service provided by AWS in both MySQL and Postgres flavors but with a totally different performance and scale oriented architecture under the hood.

🤔What’s Time-Series Data Compression and Why Does It Matter? — Learn how compression algorithms work, how to use TimescaleDB to store 100TB of data in 10TB of space, and more 🔥

Avoiding Integer Overflows with Zero Downtime at Buildkite — Learn how Buildkite migrated over 2 billion rows in one of our largest tables.

Superintendent.app: Load CSV Files and Perform SQL Queries on Them — The only 'news' item in this issue, but it fits so well.. Superintendent is a new desktop app (for Windows, macOS and Linux) for pulling in CSV files to then query using SQL. The results can then be saved back to CSV, if you wish.

Datasette: An Open Source 'Multi-Tool' for Exploring and Publishing Data — A tool aimed at anyone who has data they wish to share with the world (e.g. data journalists, curators, archivists..) This has become a mature and popular tool over the years and is well worth exploring. Simon's Personal Data Warehouses talk is also worth a watch.

Fauna - A Flexible, Developer-Friendly, Serverless Database — Fauna is the “Data API” for modern applications. Whether you’re building new microservices or augmenting existing services applications, Fauna lets you simplify code, reduce costs and ship faster. Learn more...

sqlfmt: An Opinionated Online SQL Formatter — Provide sqlfmt with some SQL, specify a target line width, and get back some better formatted SQL that takes SQL’s syntax into account. Ideal for blog posts or documentation.

Unix Command Line Tricks For Data Scientists — OK, we’re kinda cheating here, but Unix favorites like sort, split, head, grep, sed and awk make up a serious part of my own data processing arsenal at least..

Mockaroo: A Random Data Generator — Specify your schema and get back ‘realistic’ test data in CSV, JSON, SQL or Excel format. You get up to 1000 rows of generation for free. GenerateData.com is a similar tool in this space.

SQLSmith: A Random SQL Query Generator — It hasn’t been updated in a few years, but this tool went beyond merely being interesting to actually discovering bugs in real database systems. SQLancer is another, newer tool in this space.

dadbod.vim: A Modern Database Interface for Vim — A Vim plugin for interacting with numerous databases, including Postgres.

PostgREST: Serve a RESTful API from a Postgres Database — We’ve covered this popular tool numerous times in Postgres Weekly but never here.. It’s a standalone app that turns a PostgreSQL database into a RESTful API. Prefer GraphQL? PostGraphile does similar things in that area.

Firebird 4.0 Released — Firebird 4.0 (release notes) has been in the pipeline so long that even two years ago they were giving talks about the road to v5.0. But that’s one of the great things about it - it’s steady, solid, open source, and mature, having originated as a fork of InterBase 6 back in 2000 (but broadly rewritten since).

SQLite Data Starter Packs: Public Data Packaged Up in SQLite Databases — A collection of public datasets conveniently packaged as SQLite databases to practice on.

pgSCV: A Metrics Exporter for Postgres — A new tool that attempts to go beyond what postgres_exporter offers by making it easy to collect ‘almost all metrics’ from Postgres with as minimal configuration as possible. GitHub repo.

Aurora vs. RDS: An Engineer's Guide to Choosing a Database — Learn about the difference between two approaches for running relational databases in AWS.

12 Common Misconceptions About DynamoDB — DynamoDB’s approach is different enough to most other databases that it’s not surprising there are a variety of common misconceptions.

PostGIS at 20 — PostGIS, the popular set of geospatial extensions for Postgres, has turned twenty years old. This post covers a little of the early history.

Free CKAD Study Guide Excerpt — The CKAD exam is not multiple choice. You need to demonstrate your k8s skills in sample exercises. This study guide prepares you for those exercises.

PolarDB: Another Postgres-Based Database — Out of Chinese ecommerce giant Alibaba comes a new Postgres-based database that aims to provide global data consistency, ACID across nodes, and to be more distributed generally. Think something like Aurora but open source. It’s rather early days, but there’s a roadmap if you want to see where they’re headed with it.

Virtual NoSQL Event for Java Database Developers — Java is one of the most popular programming languages in the world. Join our free virtual event to learn how to bring the full power of Java and NoSQL together. Secure your spot.

Lakehouses Prevent Data Swamps, Bill Inmon Says — Bill Inmon coined the term ‘data warehouse’ back in 1992 so if anyone’s going to opine on lakehouses, data swamps, and whatever other metaphors are doing the rounds these days, let him be the one to do so! He also explains why he’s not a fan of ELT compared to ETL.

Amazon Redshift ML Now Generally Available — Redshift is Amazon’s cloud data warehouse that lets you use SQL to query and combine potentially exabytes (if you have the $!) of structured and semi-structured data.. and the ML extension goes a step further in training machine learning models to generate insights from data in your data warehouse. Even better, you can do all of this...

ClangQL: Query C++ Codebases using SQLite — An interesting experiment. clangd is a language server for parsing and processing C++ code and ClangQL then adds SQLite as an intermediary so you can use SQL to ‘query’ the codebase in question.

An Early Look at Postgres 14: Performance and Monitoring Improvements — Last week we featured the Postgres 14 beta 1 release, and now people are starting to chew over it and see where the underlying improvements are. Small, steady, and reliable improvements seem to be the order of the day.

immudb v1.0: A 'Tamper-Evident' Database with Immutable History — SQL support has been added and so the creators now boast immudb is the ‘first immutable database with relational capabilities’. Written in Go and Apache licensed.

SQLite fdw 2.0.0 Released for PostgreSQL — A foreign data wrapper to work with SQLite from Postgres 9.6, 10, 11, 12, and 13. GitHub repo.

A Look at a Variety of 'Free' Cloud Database Services — In reality, some of these are enterprise-y options that have free developer or trial tiers, but still, it’s a reasonable roundup.

Why Are Time Series Databases Important Now? — Learn what time series is, what makes it different from stream processing, full-text search and other solutions.

SQL Query Optimization: Understanding Some Key Principles — An interesting benchmark-led guide to how different SQL approaches scale upon different sized tables.

Rethinking Postgres in a Post-Server World — The increasing popularity of ‘serverless’ solutions is having an impact on databases with entire paradigms (e.g. the DynamoDB model) and new services (e.g. Fauna) being created or extended to support serverless use cases. Michael Rispoli shares his thoughts on how databases created in the pre-serverless era can fit into this new world.

Free eBook: Finding the Root Cause of Slow Postgres Queries Using EXPLAIN — An in-depth guide on how to use Postgres EXPLAIN to find and improve problematic query plans and how to use auto_explain.

SQLFluff: A SQL Linter and Auto-Formatter — A ‘dialect-flexible’ SQL linter that aims to automatically fix most linting errors. Written in Python.