home / github

Menu
  • Search all tables
  • GraphQL API

issue_comments

Table actions
  • GraphQL API for issue_comments

12 rows where issue = 1005891028 sorted by updated_at descending

✖
✖

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date), updated_at (date)

user 1

  • simonw 12

issue 1

  • Rethink approach to [ and ] in column names (currently throws error) · 12 ✖

author_association 1

  • OWNER 12
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions issue performed_via_github_app
968470212 https://github.com/simonw/sqlite-utils/issues/329#issuecomment-968470212 https://api.github.com/repos/simonw/sqlite-utils/issues/329 IC_kwDOCGYnMM45ua7E simonw 9599 2021-11-15T02:49:28Z 2021-11-15T02:49:28Z OWNER

I was going to replace all of the validate_column_names() bits with something that fixed them instead, but I think I have a better idea: I'm only going to apply the fix for the various '.insert()` methods that create the initial tables.

I'll keep the validate_column_names() where they are at the moment. Once you've inserted the data and created the tables it will be up to you to use the new, correct column names.

This avoids the whole issue of needing to rewrite parameters, and solves the immediate problem which is consuming CSV files with bad column names.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Rethink approach to [ and ] in column names (currently throws error) 1005891028  
968458837 https://github.com/simonw/sqlite-utils/issues/329#issuecomment-968458837 https://api.github.com/repos/simonw/sqlite-utils/issues/329 IC_kwDOCGYnMM45uYJV simonw 9599 2021-11-15T02:21:15Z 2021-11-15T02:21:15Z OWNER

I'm not going to implement a fix that rewrites the pk and column_order and other parameters - at least not yet. The main thing I'm trying to fix here is what happens when you attempt to import a CSV file with [ ] in the column names, which should be unaffected by that second challenge.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Rethink approach to [ and ] in column names (currently throws error) 1005891028  
968453129 https://github.com/simonw/sqlite-utils/issues/329#issuecomment-968453129 https://api.github.com/repos/simonw/sqlite-utils/issues/329 IC_kwDOCGYnMM45uWwJ simonw 9599 2021-11-15T02:07:46Z 2021-11-15T02:07:46Z OWNER

If I replace validate_column_names(row.keys()) with fix_column_names(row) I need to decide what to do about things like pk= and column_order=.

What should the following do?

python table.insert({"foo[bar]": 4}, pk="foo[bar]", column_order=["foo[bar]"]) Should it spot the old column names in the pk= and column_order= parameters and pretend that foo_bar_ was passed instead?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Rethink approach to [ and ] in column names (currently throws error) 1005891028  
968451954 https://github.com/simonw/sqlite-utils/issues/329#issuecomment-968451954 https://api.github.com/repos/simonw/sqlite-utils/issues/329 IC_kwDOCGYnMM45uWdy simonw 9599 2021-11-15T02:05:29Z 2021-11-15T02:05:29Z OWNER

I could even have those replacement characters be properties of the Database class, so uses can sub-class and change them.

I'm not going to do this, it's unnecessary extra complexity and it means the function that fixes the column names needs to have access to the current Database instance.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Rethink approach to [ and ] in column names (currently throws error) 1005891028  
926208819 https://github.com/simonw/sqlite-utils/issues/329#issuecomment-926208819 https://api.github.com/repos/simonw/sqlite-utils/issues/329 IC_kwDOCGYnMM43NNMz simonw 9599 2021-09-23T22:26:54Z 2021-09-23T22:26:54Z OWNER

I could even have those replacement characters be properties of the Database class, so uses can sub-class and change them.

python class Database: left_brace_replace = "_" right_brace_replace = "_" ...

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Rethink approach to [ and ] in column names (currently throws error) 1005891028  
926207719 https://github.com/simonw/sqlite-utils/issues/329#issuecomment-926207719 https://api.github.com/repos/simonw/sqlite-utils/issues/329 IC_kwDOCGYnMM43NM7n simonw 9599 2021-09-23T22:24:09Z 2021-09-23T22:24:09Z OWNER

I think I like the underscore option best. I don't like the idea of injecting surprise ( ) parenthesis, and having them vanish entirely could result in things like item[price] becoming itemprice which feels confusing.

item_price_ is a little ugly but I think I can live with it.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Rethink approach to [ and ] in column names (currently throws error) 1005891028  
926207246 https://github.com/simonw/sqlite-utils/issues/329#issuecomment-926207246 https://api.github.com/repos/simonw/sqlite-utils/issues/329 IC_kwDOCGYnMM43NM0O simonw 9599 2021-09-23T22:23:09Z 2021-09-23T22:23:09Z OWNER

What are my options for replacing those characters?

  • [ becomes ( and ] becomes )
  • [ and ] are removed entirely
  • [ and ] both become _
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Rethink approach to [ and ] in column names (currently throws error) 1005891028  
926206705 https://github.com/simonw/sqlite-utils/issues/329#issuecomment-926206705 https://api.github.com/repos/simonw/sqlite-utils/issues/329 IC_kwDOCGYnMM43NMrx simonw 9599 2021-09-23T22:21:58Z 2021-09-23T22:21:58Z OWNER

I'm inclined to just fix them and not have an option for opting-out of fixing them, since it adds quite a bit of cruft to the overall API design for an option that maybe no-one will ever use.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Rethink approach to [ and ] in column names (currently throws error) 1005891028  
926206220 https://github.com/simonw/sqlite-utils/issues/329#issuecomment-926206220 https://api.github.com/repos/simonw/sqlite-utils/issues/329 IC_kwDOCGYnMM43NMkM simonw 9599 2021-09-23T22:20:55Z 2021-09-23T22:21:19Z OWNER

If I add a new parameter for opting in and out of fixing these, what should it be called? A few options:

  • fix_columns=False
  • rename_bad_columns=False
  • fix_column_names=False
  • error_on_invalid_column_names=True
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Rethink approach to [ and ] in column names (currently throws error) 1005891028  
926205047 https://github.com/simonw/sqlite-utils/issues/329#issuecomment-926205047 https://api.github.com/repos/simonw/sqlite-utils/issues/329 IC_kwDOCGYnMM43NMR3 simonw 9599 2021-09-23T22:18:34Z 2021-09-23T22:19:38Z OWNER

Here's the code where this happens: https://github.com/simonw/sqlite-utils/blob/54191d4dc114d7dc21e849b48a4d5ae4f9e601ca/sqlite_utils/db.py#L2943-L2948

It's called from three different methods in db.py: create_table_sql(), update() and insert_all().

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Rethink approach to [ and ] in column names (currently throws error) 1005891028  
926204046 https://github.com/simonw/sqlite-utils/issues/329#issuecomment-926204046 https://api.github.com/repos/simonw/sqlite-utils/issues/329 IC_kwDOCGYnMM43NMCO simonw 9599 2021-09-23T22:16:18Z 2021-09-23T22:16:18Z OWNER

So either I do the automatic replacement, or I let the user request automatic replacement, or a third option: I do automatic replacement but let the user opt to receive an error instead.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Rethink approach to [ and ] in column names (currently throws error) 1005891028  
926203501 https://github.com/simonw/sqlite-utils/issues/329#issuecomment-926203501 https://api.github.com/repos/simonw/sqlite-utils/issues/329 IC_kwDOCGYnMM43NL5t simonw 9599 2021-09-23T22:15:07Z 2021-09-23T22:15:30Z OWNER

Quoting https://github.com/simonw/sqlite-utils/issues/86#issuecomment-586676856 in full:

I'm not sure what to do about this one.

I can't fix it: this bug in Python's sqlite3 module means that even if I write a database out with column names that include [] I won't be able to read them back again.

So... I could do one of the following:

  • Throw an error if a column name includes those characters. That's my preferred option I think.
  • Automatically replace [ in column names with ( and ] with )
  • Do the automatic replacement but show a user-visible warning when I do it
  • Throw an error, but give the user an option to run with e.g. --fix-column-names which applies that automatic fix.

Since this is likely to be an incredibly rare edge-case I think I'd rather minimize the amount of code that deals with it, so my preferred option is to just throw that error and stop.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Rethink approach to [ and ] in column names (currently throws error) 1005891028  

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options:

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
, [performed_via_github_app] TEXT);
CREATE INDEX [idx_issue_comments_issue]
                ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
                ON [issue_comments] ([user]);
Powered by Datasette · Queries took 22.354ms · About: github-to-sqlite
  • Sort ascending
  • Sort descending
  • Facet by this
  • Hide this column
  • Show all columns
  • Show not-blank rows