github

Custom SQL query returning 15 rows (hide)

Query parameters

This data as json, CSV

html_urlissue_urlidnode_idusercreated_atupdated_atauthor_associationbodyreactionsissueperformed_via_github_app
https://github.com/simonw/datasette/issues/670#issuecomment-1816642044 https://api.github.com/repos/simonw/datasette/issues/670 1816642044 IC_kwDOBm6k_c5sR8H8 16142258 2023-11-17T15:32:20Z 2023-11-17T15:32:20Z NONE Any progress on this? It would be very helpful on my end as well. Thanks! 
{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 564833696  
https://github.com/simonw/datasette/issues/670#issuecomment-1637293044 https://api.github.com/repos/simonw/datasette/issues/670 1637293044 IC_kwDOBm6k_c5hlxv0 4863782 2023-07-17T02:23:32Z 2023-07-17T02:23:32Z NONE Is there any working version of datasette/postgresql ? 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 564833696  
https://github.com/simonw/datasette/issues/670#issuecomment-849022714 https://api.github.com/repos/simonw/datasette/issues/670 849022714 MDEyOklzc3VlQ29tbWVudDg0OTAyMjcxNA== 9599 2021-05-26T18:33:47Z 2021-05-26T18:33:58Z OWNER Worth mentioning here: I've been doing a tun of research around running Datasette-like functionality against PostgreSQL in my https://github.com/simonw/django-sql-dashboard project - which will definitely inform the Datasette implementation. 
{
    "total_count": 3,
    "+1": 3,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 564833696  
https://github.com/simonw/datasette/issues/670#issuecomment-848425056 https://api.github.com/repos/simonw/datasette/issues/670 848425056 MDEyOklzc3VlQ29tbWVudDg0ODQyNTA1Ng== 52261150 2021-05-26T03:22:32Z 2021-05-26T03:22:32Z NONE I've also been investigating serving postgresql databases over postgrest. I like the idea of hosting some static html + js on github, but having it backed by datasets I can update and control on the database server. I started from SQLite + datasette but would like to host larger datasets (with smaller materialized views exposed publicly). I think the postgrest model where all the authorization and ownership is defined in database role grants is really powerful. But I really miss being able to define an ad-hoc query in sql, then instantly link to a json representation of it like datasette does. P.S.: I've been sort of following along as you pop up in hacker news here and there. It's been great! Thanks for doing this all out in the open! 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 564833696  
https://github.com/simonw/datasette/issues/670#issuecomment-797158641 https://api.github.com/repos/simonw/datasette/issues/670 797158641 MDEyOklzc3VlQ29tbWVudDc5NzE1ODY0MQ== 9599 2021-03-12T00:59:49Z 2021-03-12T00:59:49Z OWNER > Challenge: what's the equivalent for PostgreSQL of opening a database in read only mode? Will I have to talk users through creating read only credentials? It looks like the answer to this is yes - I'll need users to setup read-only credentials. Here's a TIL about that: https://til.simonwillison.net/postgresql/read-only-postgresql-user 
{
    "total_count": 1,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 1,
    "rocket": 0,
    "eyes": 0
}
 564833696  
https://github.com/simonw/datasette/issues/670#issuecomment-696163452 https://api.github.com/repos/simonw/datasette/issues/670 696163452 MDEyOklzc3VlQ29tbWVudDY5NjE2MzQ1Mg== 652285 2020-09-21T14:46:10Z 2020-09-21T14:46:10Z NONE I'm currently using PostgREST to serve OpenAPI APIs off Postgresql databases. I would like to try out datasette once this becomes available on Postgres. 
{
    "total_count": 2,
    "+1": 2,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 564833696  
https://github.com/simonw/datasette/issues/670#issuecomment-585962849 https://api.github.com/repos/simonw/datasette/issues/670 585962849 MDEyOklzc3VlQ29tbWVudDU4NTk2Mjg0OQ== 9599 2020-02-13T20:44:18Z 2020-02-13T20:46:53Z OWNER Got the database page working! It lists tables, their columns and their row count. Got the table page partially working! It can list rows. It can't apply filters yet (because PostgreSQL `$1` parameters differ from SQLite `:blah` parameters) and faceting doesn't work because PostgreSQL requires that subqueries have an alias: ``` subquery in FROM must have an alias HINT: For example, FROM (SELECT ...) [AS] foo. ``` Still a pretty promising start though! 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 564833696  
https://github.com/simonw/datasette/issues/670#issuecomment-585903240 https://api.github.com/repos/simonw/datasette/issues/670 585903240 MDEyOklzc3VlQ29tbWVudDU4NTkwMzI0MA== 9599 2020-02-13T18:28:42Z 2020-02-13T18:29:14Z OWNER Challenge: what's the equivalent for PostgreSQL of opening a database in read only mode? Will I have to talk users through creating read only credentials? Can I do this at runtime somehow? Can I detect if the connection has write permission and disable the arbitrary query feature? 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 564833696  
https://github.com/simonw/datasette/issues/670#issuecomment-585885812 https://api.github.com/repos/simonw/datasette/issues/670 585885812 MDEyOklzc3VlQ29tbWVudDU4NTg4NTgxMg== 9599 2020-02-13T17:49:33Z 2020-02-13T17:49:33Z OWNER ``` In [12]: await conn.fetch("""SELECT a.attname, format_type(a.atttypid, a.atttypmod) AS data_type ...: FROM pg_index i ...: JOIN pg_attribute a ON a.attrelid = i.indrelid ...: AND a.attnum = ANY(i.indkey) ...: WHERE i.indrelid = 'blog_blogmark'::regclass ...: AND i.indisprimary;""") Out[12]: [<Record attname='id' data_type='integer'>] ``` 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 564833696  
https://github.com/simonw/datasette/issues/670#issuecomment-585885242 https://api.github.com/repos/simonw/datasette/issues/670 585885242 MDEyOklzc3VlQ29tbWVudDU4NTg4NTI0Mg== 9599 2020-02-13T17:48:27Z 2020-02-13T17:48:27Z OWNER Finding out the primary keys for a table: https://wiki.postgresql.org/wiki/Retrieve_primary_key_columns 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 564833696  
https://github.com/simonw/datasette/issues/670#issuecomment-585883109 https://api.github.com/repos/simonw/datasette/issues/670 585883109 MDEyOklzc3VlQ29tbWVudDU4NTg4MzEwOQ== 9599 2020-02-13T17:44:08Z 2020-02-13T17:44:08Z OWNER Introspecting columns in PostgreSQL: ``` In [3]: sql = """SELECT * ...: FROM information_schema.columns ...: WHERE table_schema = 'public' ...: AND table_name = 'blog_blogmark' ...: ;""" ``` Each column looks like this: ```pythton {'table_catalog': 'simonwillisonblog', 'table_schema': 'public', 'table_name': 'blog_blogmark', 'column_name': 'id', 'ordinal_position': 1, 'column_default': "nextval('blog_blogmark_id_seq'::regclass)", 'is_nullable': 'NO', 'data_type': 'integer', 'character_maximum_length': None, 'character_octet_length': None, 'numeric_precision': 32, 'numeric_precision_radix': 2, 'numeric_scale': 0, 'datetime_precision': None, 'interval_type': None, 'interval_precision': None, 'character_set_catalog': None, 'character_set_schema': None, 'character_set_name': None, 'collation_catalog': None, 'collation_schema': None, 'collation_name': None, 'domain_catalog': None, 'domain_schema': None, 'domain_name': None, 'udt_catalog': 'simonwillisonblog', 'udt_schema': 'pg_catalog', 'udt_name': 'int4', 'scope_catalog': None, 'scope_schema': None, 'scope_name': None, 'maximum_cardinality': None, 'dtd_identifier': '1', 'is_self_referencing': 'NO', 'is_identity': 'NO', 'identity_generation': None, 'identity_start': None, 'identity_increment': None, 'identity_maximum': None, 'identity_minimum': None, 'identity_cycle': 'NO', 'is_generated': 'NEVER', 'generation_expression': None, 'is_updatable': 'YES'} ``` 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 564833696  
https://github.com/simonw/datasette/issues/670#issuecomment-585874699 https://api.github.com/repos/simonw/datasette/issues/670 585874699 MDEyOklzc3VlQ29tbWVudDU4NTg3NDY5OQ== 9599 2020-02-13T17:26:58Z 2020-02-13T17:27:13Z OWNER First page to get working: the database view page, which shows a list of tables. In the code this is entirely implemented with calls to the `Database` class, which is a good starting point - I can drop in a new version of that class that talks to PostgreSQL instead: https://github.com/simonw/datasette/blob/0091dfe3e5a3db94af8881038d3f1b8312bb857d/datasette/views/database.py#L25-L66 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 564833696  
https://github.com/simonw/datasette/issues/670#issuecomment-585873401 https://api.github.com/repos/simonw/datasette/issues/670 585873401 MDEyOklzc3VlQ29tbWVudDU4NTg3MzQwMQ== 9599 2020-02-13T17:24:38Z 2020-02-13T17:24:38Z OWNER The biggest difference between the two will be around introspection. I searched the codebase for potential introspection queries, defined as `select ... from sqlite_master` and `pragma ...` statements. - 7 results for sqlite_master in database.py - 3 results in utils/__init__.py - (Ignoring the 4 in inspect.py) - 1 result for “pragma table_info” in app.py - 2 in utils/__init__.py - 2 results for “pragma foreign_key_list” in utils/__init__.py 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 564833696  
https://github.com/simonw/datasette/issues/670#issuecomment-585872538 https://api.github.com/repos/simonw/datasette/issues/670 585872538 MDEyOklzc3VlQ29tbWVudDU4NTg3MjUzOA== 9599 2020-02-13T17:22:54Z 2020-02-13T17:22:54Z OWNER A couple of things I'd like to support: - The same datasette instance can have both PostgreSQL and SQLite databases attached to it, and both types will be listed on the homepage. - The full test suite runs against both SQLite and PostgreSQL, with as few changes as possible (maybe a few `skipIf()` decorators for features not available on both). Probably easiest do do this with some kind of flag - e.g. `pytest --database=sqlite` v.s. `pytest --database=postgresql` I can implement that with this in `conftest.py`: ```python def pytest_addoption(parser): parser.addoption("--database", action="store", default="sqlite") ``` See https://stackoverflow.com/questions/40880259/how-to-pass-arguments-in-pytest-by-command-line 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 564833696  
https://github.com/simonw/datasette/issues/670#issuecomment-585870836 https://api.github.com/repos/simonw/datasette/issues/670 585870836 MDEyOklzc3VlQ29tbWVudDU4NTg3MDgzNg== 9599 2020-02-13T17:19:23Z 2020-02-13T17:19:37Z OWNER I'm excited about https://github.com/MagicStack/asyncpg for this - it's a true async PostgreSQL library (my SQLite queries run in a threadpool right now) with extremely impressive performance benchmarks, from the team behind `uvloop`. 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 564833696