home / github

Menu
  • Search all tables
  • GraphQL API

issue_comments

Table actions
  • GraphQL API for issue_comments

16 rows where "created_at" is on date 2022-11-01 and "updated_at" is on date 2022-11-01 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: issue_url, reactions, created_at (date), updated_at (date)

issue 8

  • Make it easier to fix URL proxy problems 5
  • Ensure insert API has good tests for rowid and compound primark key tables 3
  • Make `cursor.rowcount` accessible (wontfix) 2
  • SQL query should wrap on SQL interrupted screen 2
  • Create a new table from one or more records, `sqlite-utils` style 1
  • API explorer tool 1
  • conn.execute: UnicodeEncodeError: 'utf-8' codec can't encode character 1
  • /db/table/-/upsert API 1

user 2

  • simonw 15
  • chapmanjacobd 1

author_association 2

  • OWNER 15
  • CONTRIBUTOR 1
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions issue performed_via_github_app
1299349741 https://github.com/simonw/datasette/issues/1871#issuecomment-1299349741 https://api.github.com/repos/simonw/datasette/issues/1871 IC_kwDOBm6k_c5NcoDt simonw 9599 2022-11-01T23:22:55Z 2022-11-01T23:22:55Z OWNER

It's weird that the API explorer only lets you explore POST APIs. It should probably also let you explore GET APIs, or be renamed.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
API explorer tool 1427293909  
1299098458 https://github.com/simonw/datasette/issues/1879#issuecomment-1299098458 https://api.github.com/repos/simonw/datasette/issues/1879 IC_kwDOBm6k_c5Nbqta simonw 9599 2022-11-01T20:27:40Z 2022-11-01T20:33:52Z OWNER

https://github.com/simonw/datasette-x-forwarded-host/blob/main/datasette_x_forwarded_host/init.py could happen in core controlled by:

--setting trust_forwarded_host 1

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Make it easier to fix URL proxy problems 1432037325  
1299102108 https://github.com/simonw/datasette/issues/1879#issuecomment-1299102108 https://api.github.com/repos/simonw/datasette/issues/1879 IC_kwDOBm6k_c5Nbrmc simonw 9599 2022-11-01T20:30:54Z 2022-11-01T20:33:06Z OWNER

One idea: add a /-/debug page (or /-/tips or /-/checks) which shows the incoming requests headers and could even detect if there's an x-forwarded-host header that isn't being repeated and show a tip on how to fix that.

{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Make it easier to fix URL proxy problems 1432037325  
1299102755 https://github.com/simonw/datasette/issues/1879#issuecomment-1299102755 https://api.github.com/repos/simonw/datasette/issues/1879 IC_kwDOBm6k_c5Nbrwj simonw 9599 2022-11-01T20:31:37Z 2022-11-01T20:31:37Z OWNER

And some JavaScript that can spot if Datasette thinks it is being served over HTTP when it's actually being served over HTTPS.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Make it easier to fix URL proxy problems 1432037325  
1299096850 https://github.com/simonw/datasette/issues/1879#issuecomment-1299096850 https://api.github.com/repos/simonw/datasette/issues/1879 IC_kwDOBm6k_c5NbqUS simonw 9599 2022-11-01T20:26:12Z 2022-11-01T20:26:12Z OWNER

The other relevant plugin here is https://datasette.io/plugins/datasette-x-forwarded-host

Maybe that should be rolled into core too?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Make it easier to fix URL proxy problems 1432037325  
1299090678 https://github.com/simonw/datasette/issues/1879#issuecomment-1299090678 https://api.github.com/repos/simonw/datasette/issues/1879 IC_kwDOBm6k_c5Nboz2 simonw 9599 2022-11-01T20:20:28Z 2022-11-01T20:20:28Z OWNER

My first step in debugging these is to install https://datasette.io/plugins/datasette-debug-asgi - but now I'm thinking maybe something like that should be part of core.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Make it easier to fix URL proxy problems 1432037325  
1299073433 https://github.com/simonw/datasette/issues/1862#issuecomment-1299073433 https://api.github.com/repos/simonw/datasette/issues/1862 IC_kwDOBm6k_c5NbkmZ simonw 9599 2022-11-01T20:04:31Z 2022-11-01T20:04:31Z OWNER

It really feels like this should be accompanied by a /db/-/create API for creating tables. I had to add that to sqlite-utils eventually (initially it only supported creating by passing in an example document):

https://sqlite-utils.datasette.io/en/stable/cli.html#cli-create-table

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Create a new table from one or more records, `sqlite-utils` style 1425011030  
1299071456 https://github.com/simonw/datasette/issues/1878#issuecomment-1299071456 https://api.github.com/repos/simonw/datasette/issues/1878 IC_kwDOBm6k_c5NbkHg simonw 9599 2022-11-01T20:02:43Z 2022-11-01T20:02:43Z OWNER

Note that "update" is partially covered by the replace option to /-/insert, added here: - https://github.com/simonw/datasette/issues/1873#issuecomment-1298885451

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
/db/table/-/upsert API 1432013704  
1298919552 https://github.com/simonw/datasette/issues/1873#issuecomment-1298919552 https://api.github.com/repos/simonw/datasette/issues/1873 IC_kwDOBm6k_c5Na_CA simonw 9599 2022-11-01T18:11:27Z 2022-11-01T18:11:27Z OWNER

I forgot to document ignore and replace. Also I need to add tests that cover:

  • Forgetting to include a primary key on a non-autoincrement table
  • Compound primary keys
  • Rowid only tables with and without rowid specified

I think my validation logic here will get caught out by the fact that rowid does not show up as a valid column name: https://github.com/simonw/datasette/blob/9bec7c38eb93cde5afb16df9bdd96aea2a5b0459/datasette/views/table.py#L1151-L1160

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Ensure insert API has good tests for rowid and compound primark key tables 1428630253  
1298905135 https://github.com/simonw/datasette/issues/1873#issuecomment-1298905135 https://api.github.com/repos/simonw/datasette/issues/1873 IC_kwDOBm6k_c5Na7gv simonw 9599 2022-11-01T17:59:59Z 2022-11-01T17:59:59Z OWNER

It's a bit surprising that you can send "ignore": true, "return_rows": true and the returned "inserted" key will list rows that were NOT inserted (since they were ignored).

Three options:

  1. Ignore that and document it
  2. Fix it so "inserted" only returns rows that were actually inserted (bit tricky)
  3. Change the name of "inserted" to something else

I'm picking 3 - I'm going to change it to be called "rows" instead.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Ensure insert API has good tests for rowid and compound primark key tables 1428630253  
1298885451 https://github.com/simonw/datasette/issues/1873#issuecomment-1298885451 https://api.github.com/repos/simonw/datasette/issues/1873 IC_kwDOBm6k_c5Na2tL simonw 9599 2022-11-01T17:42:20Z 2022-11-01T17:42:20Z OWNER

Design decision: json { "rows": [{"id": 1, "title": "The title"}], "ignore": true } Or "replace": true.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Ensure insert API has good tests for rowid and compound primark key tables 1428630253  
1298879701 https://github.com/simonw/sqlite-utils/issues/506#issuecomment-1298879701 https://api.github.com/repos/simonw/sqlite-utils/issues/506 IC_kwDOCGYnMM5Na1TV simonw 9599 2022-11-01T17:37:13Z 2022-11-01T17:37:13Z OWNER

The question I was originally trying to answer here was this: how many rows were actually inserted by that call to .insert_all()?

I don't know that .rowcount would ever be useful here, since the "correct" answer depends on other factors - had I determined to ignore or replace records with a primary key that matches an existing record for example?

So I think if people need rowcount they can get it by using a cursor directly.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Make `cursor.rowcount` accessible (wontfix) 1429029604  
1298877872 https://github.com/simonw/sqlite-utils/issues/506#issuecomment-1298877872 https://api.github.com/repos/simonw/sqlite-utils/issues/506 IC_kwDOCGYnMM5Na02w simonw 9599 2022-11-01T17:35:30Z 2022-11-01T17:35:30Z OWNER

This may not make sense.

First, .last_rowid is a property on table - but that doesn't make sense for rowcount since it should clearly be a property on the database itself (you can run a query directly using db.execute() without going through a Table object).

So I tried this prototype:

`diff diff --git a/docs/python-api.rst b/docs/python-api.rst index 206e5e6..78d3a8d 100644 --- a/docs/python-api.rst +++ b/docs/python-api.rst @@ -186,6 +186,15 @@ Thedb.query(sql)`` function executes a SQL query and returns an iterator over # {'name': 'Cleo'} # {'name': 'Pancakes'}

+After executing a query the db.rowcount property on that database instance will reflect the number of rows affected by any insert, update or delete operations performed by that query: + +.. code-block:: python + + db = Database(memory=True) + db["dogs"].insert_all([{"name": "Cleo"}, {"name": "Pancakes"}]) + print(db.rowcount) + # Outputs: 2 + .. _python_api_execute:

db.execute(sql, params) diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py index a06f4b7..c19c2dd 100644 --- a/sqlite_utils/db.py +++ b/sqlite_utils/db.py @@ -294,6 +294,8 @@ class Database:

 _counts_table_name = "_counts"
 use_counts_table = False
  • Number of rows inserted, updated or deleted

  • rowcount: Optional[int] = None

    def init( self, @@ -480,9 +482,11 @@ class Database: if self._tracer: self._tracer(sql, parameters) if parameters is not None: - return self.conn.execute(sql, parameters) + cursor = self.conn.execute(sql, parameters) else: - return self.conn.execute(sql) + cursor = self.conn.execute(sql) + self.rowcount = cursor.rowcount + return cursor

    def executescript(self, sql: str) -> sqlite3.Cursor: """ But this happens:pycon

    from sqlite_utils import Database db = Database(memory=True) db["dogs"].insert_all([{"name": "Cleo"}, {"name": "Pancakes"}])

<Table dogs (name)> >>> db.rowcount -1 ``` Turning on query tracing demonstrates why: ```pycon >>> db = Database(memory=True, tracer=print) PRAGMA recursive_triggers=on; None >>> db["dogs"].insert_all([{"name": "Cleo"}, {"name": "Pancakes"}]) select name from sqlite_master where type = 'view' None select name from sqlite_master where type = 'table' None select name from sqlite_master where type = 'view' None CREATE TABLE [dogs] ( [name] TEXT ); None select name from sqlite_master where type = 'view' None INSERT INTO [dogs] ([name]) VALUES (?), (?); ['Cleo', 'Pancakes'] select name from sqlite_master where type = 'table' None select name from sqlite_master where type = 'table' None PRAGMA table_info([dogs]) None <Table dogs (name)> >>> ``` The `.insert_all()` function does a bunch of other queries too, so `.rowcount` is quickly over-ridden by the same result from extra queries that it executed.
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Make `cursor.rowcount` accessible (wontfix) 1429029604  
1298856054 https://github.com/simonw/datasette/issues/1876#issuecomment-1298856054 https://api.github.com/repos/simonw/datasette/issues/1876 IC_kwDOBm6k_c5Navh2 simonw 9599 2022-11-01T17:16:01Z 2022-11-01T17:16:01Z OWNER

ta.style.height = ta.scrollHeight + 'px' is an easy way to do that.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
SQL query should wrap on SQL interrupted screen 1431786951  
1298854321 https://github.com/simonw/datasette/issues/1876#issuecomment-1298854321 https://api.github.com/repos/simonw/datasette/issues/1876 IC_kwDOBm6k_c5NavGx simonw 9599 2022-11-01T17:14:33Z 2022-11-01T17:14:33Z OWNER

I could use a textarea here (would need to figure out a neat pattern to expand it to fit the query):

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
SQL query should wrap on SQL interrupted screen 1431786951  
1297859539 https://github.com/simonw/sqlite-utils/issues/507#issuecomment-1297859539 https://api.github.com/repos/simonw/sqlite-utils/issues/507 IC_kwDOCGYnMM5NW8PT chapmanjacobd 7908073 2022-11-01T00:40:16Z 2022-11-01T00:40:16Z CONTRIBUTOR

Ideally people could fix their data if they run into this issue.

If you are using filenames try convmv

convmv --preserve-mtimes -f utf8 -t utf8 --notest -i -r .

maybe this script will also help:

```py import argparse, shutil from pathlib import Path

import ftfy

from xklb import utils from xklb.utils import log

def parse_args() -> argparse.Namespace: parser = argparse.ArgumentParser() parser.add_argument("paths", nargs='*') parser.add_argument("--verbose", "-v", action="count", default=0) args = parser.parse_args()

log.info(utils.dict_filter_bool(args.__dict__))
return args

def rename_invalid_paths() -> None: args = parse_args()

for path in args.paths:
    log.info(path)
    for p in sorted([str(p) for p in Path(path).rglob("*")], key=len):
        fixed = ftfy.fix_text(p, uncurl_quotes=False).replace("\r\n", "\n").replace("\r", "\n").replace("\n", "")
        if p != fixed:
            try:
                shutil.move(p, fixed)
            except FileNotFoundError:
                log.warning("FileNotFound. %s", p)
            else:
                log.info(fixed)

if name == "main": rename_invalid_paths() ```

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
conn.execute: UnicodeEncodeError: 'utf-8' codec can't encode character 1430325103  

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options:

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
, [performed_via_github_app] TEXT);
CREATE INDEX [idx_issue_comments_issue]
                ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
                ON [issue_comments] ([user]);
Powered by Datasette · Queries took 911.137ms · About: github-to-sqlite