Yale Semantic Parsing and Text-to-SQL Challenge (2024)

Rank Model Dev Test

1

Nov 2, 2023 MiniSeek

Anonymous

Code and paper coming soon 80.3 81.5

1

Sep 13, 2022 Graphix-3B + PICARD (DB content used)

Alibaba DAMO & HKU STAR & SIAT

(Li et al., AAAI'2023) code 77.1 74.0

2

Sep 14, 2022 CatSQL + GraPPa (DB content used)

Anonymous

78.6 73.9

3

Sep 1, 2022 SHiP + PICARD (DB content used)

AWS AI Labs

(Zhao et al.,'22) 77.2 73.1

4

May 23, 2022 G³R + LGESQL + ELECTRA (DB content used)

Southeast University & Tencent Cloud Xiaowei

(Xiang et al., ACL-Findings '23) 78.1 72.9

6

Aug 12, 2022 RESDSQL+T5-1.1-lm100k-xl (DB content used)

Anonymous

78.1 72.4

6

May 8, 2022 T5-SR (DB content used)

Anonymous

77.2 72.4

7

Dec 15, 2022 N-best List Rerankers + PICARD (DB content used)

Alexa AI

(Zeng et al., IEEE SLT 2023) 76.4 72.2

8

Sep 1, 2021 S²SQL + ELECTRA (DB content used)

Alibaba DAMO

(Hui et al., ACL-Findings '22) code 76.4 72.1

9

Feb 7, 2023 RESDSQL-3B + NatSQL (DB content used)

Renmin University of China

(Li et al., AAAI'23) code 80.5 72.0

10

Jun 1, 2021 LGESQL + ELECTRA (DB content used)

SJTU X-LANCE Lab & AISpeech

(Cao et al., ACL'21) code 75.1 72.0

11

Jul 14, 2021 T5-3B+PICARD (DB content used)

Element AI, a ServiceNow company

(Scholak et al., EMNLP'21) code 75.5 71.9

12

Aug 12, 2022 RESDSQL+T5-1.1-lm100k-large (DB content used)

Anonymous

76.6 71.1

13

Jun 4, 2022 RASAT+PICARD (DB content used)

SJTU LUMIA & Netmind.AI

(Qi et al., EMNLP'22) code 75.3 70.9

14

Nov 19, 2020 DT-Fixup SQL-SP + RoBERTa (DB content used)

Borealis AI

(Xu et al., ACL'21) code 75.0 70.9

15

Nov 19, 2020 RAT-SQL + GraPPa + Adv (DB content used)

Anonymous

75.5 70.5

16

Oct 18, 2021 RATSQL++ + ELECTRA (DB content used)

Anonymous

75.7 70.3

17

Nov 19, 2020 SADGA + GAP (DB content used)

DMIR Lab

(Cai and Yuan et al., NeurIPS'21) code 73.1 70.1

18

Dec 25, 2020 RATSQL + GraPPa + GP (DB content used)

OCFT Gamma Big Data Lab

(Zhao et al.,'21) 72.8 69.8

19

Sep 08, 2020 RATSQL + GAP (DB content used)

University of Waterloo & AWS AI Labs

(Shi et al., AAAI'21) code 71.8 69.7

20

Apr 4, 2023 RASAT + NatSQL + Reranker (DB content used)

Anonymous

Paper coming soon 73.6 69.6

20

Aug 18, 2020 RATSQL + GraPPa (DB content used)

Yale & Salesforce Research

(Yu et al., ICLR'21) code 73.4 69.6

22

Mar 10, 2021 SmBoP + GraPPa (DB content used)

Tel-Aviv University & Allen Institute for AI

(Rubin and Berant, NAACL'21) code 74.7 69.5

23

Aug 05, 2021 RaSaP + ELECTRA (DB content used)

Ant Group, ZhiXiaoBao & Ada

(Huang et al.,'21) 74.7 69.0

24

May 4, 2021 RATSQL+GAP+NatSQL (DB content used)

Queen Mary University of London

(Gan et al., EMNLP Findings'21) code - 68.7

25

Nov 20, 2020 RAT-SQL + STRUG (DB content used)

Microsoft Research & OSU

(Deng et al., NAACL '21) 72.6 68.4

26

Jun 1, 2021 LGESQL + BERT (DB content used)

SJTU X-LANCE Lab & AISpeech

(Cao et al., ACL'21) code 74.1 68.3

27

Jan 16, 2021 COMBINE (DB content used)

Novelis.io Research

(Youssef et al.,'21) 71.4 67.7

28

Nov 24, 2020 BRIDGE v2 + BERT(ensemble) (DB content used)

Salesforce Research

(Lin et al., EMNLP-Findings '20) code 71.1 67.5

29

Sep. 8, 2020 ShadowGNN + RoBERTa (DB content used)

SJTU X-LANCE Lab & AISpeech

(Chen et al., NAACL'21) 72.3 66.1

30

Jul. 22, 2022 T5QL-Base (DB content used)

Anonymous

69.3 65.9

31

May 02, 2020 RATSQL v3 + BERT (DB content used)

Microsoft Research

(Wang and Shin et al., ACL '20) code 69.7 65.6

32

Dec. 07, 2020 DuoRAT + BERT (DB content used)

Anonymous

- 65.4

33

Sep. 8, 2020 YCSQL + BERT (DB content used)

Anonymous

- 65.3

34

Jan. 29, 2021 ETA + BERT (DB content used)

Microsoft Research Asia

(Liu et al., ACL-Findings '21) 70.8 65.3

35

Nov 24, 2020 BRIDGE v2 + BERT (DB content used)

Salesforce Research

(Lin et al., EMNLP-Findings '20) code 70.0 65.0

36

October 17, 2023 DPG-SQL + GPT-4 + Self-Correction

Anonymous

Code and paper coming soon - 64.7

37

Sep. 8, 2020 GP-RATSQL + BERT (DB content used)

Anonymous

- 64.5

38

Nov. 25, 2020 RATSQL-HPFT + BERT (DB content used)

Anonymous

- 64.4

39

Feb 2, 2021 LGESQL + GLOVE (DB content used)

SJTU X-LANCE Lab & AISpeech

(Cao et al., ACL'21) code 67.6 62.8

40

May 31, 2020 AuxNet + BART (DB content used)

Anonymous

70.0 61.9

41

Dec 13, 2019 RATSQL v2 + BERT (DB content used)

Microsoft Research

(Wang and Shin et al., ACL '20) code 65.8 61.9

42

May 31, 2020 AuxNet + BART

Anonymous

68.0 61.3

43

Feb 18, 2020 RYANSQL v2 + BERT

Kakao Enterprise

(Choi et al., '20) 70.6 60.6

44

Oct 19, 2020 SmBoP + BART

Tel-Aviv University & Allen Institute for AI

(Rubin and Berant '20) 66.0 60.5

45

Dec 18, 2019 IRNet++ + XLNet (DB content used)

Anonymous

65.5 60.1

46

Apr 21, 2023 DIN-SQL + GPT-4

University of Alberta

(Pourreza et al.,'2023) code 60.1 60

47

May 30, 2020 BRIDGE + BERT (DB content used)

Salesforce Research

(Lin et al., EMNLP-Findings '20) code 65.5 59.2

48

Nov 12, 2019 RYANSQL + BERT

Kakao Enterprise

(Choi et al., '20) 66.6 58.2

49

Dec 13, 2019 RATSQL v2 (DB content used)

Microsoft Research

(Wang and Shin et al., ACL '20) code 62.7 57.2

50

Apr 21, 2023 DIN-SQL + CodeX

University of Alberta

(Pourreza et al.,'2023) code 57.2 57

51

Dec 13, 2019 SLSQL + BERT + Data Annotation

National University of Singapore

(Lei and Wang et al., EMNLP '20) code 60.8 55.7

52

Dec 13, 2019 EditSQL+LSL + BERT

Anonymous

57.9 55.2

53

June 24, 2019 IRNet v2 + BERT

Microsoft Research Asia

63.9 55.0

54

Sep 20, 2019 GIRN + BERT

Anonymous

60.2 54.8

55

May 19, 2019 IRNet + BERT

Microsoft Research Asia

(Guo and Zhan et al., ACL '19) code 61.9 54.7

56

Nov 4, 2019 GNN + Bertrand-DR

Got It R&D

(Kelkar et al., '20) code 57.9 54.6

57

Apr 8, 2020 CNSQL

Anonymous

58.0 54.0

58

Sep 19, 2019 RATSQL

Anonymous

60.6 53.7

59

Sep 1, 2019 EditSQL + BERT

Yale University & Salesforce Research

(Zhang et al., EMNLP '19) code 57.6 53.4

60

May 21, 2020 GAZP + BERT

University of Washington & Facebook AI Research

(Zhong et al., EMNLP '20) - 53.3

61

May 21, 2020 NatSQL v3

Anonymous

- 53.2

62

May 28, 2020 IRNET+ GNN

Anonymous

- 49.6

63

June 24, 2019 IRNet v2

Microsoft Research Asia

55.4 48.5

64

Aug 30, 2019 Global-GNN (DB content used)

Tel-Aviv University & Allen Institute for AI

(Bogin et al., EMNLP '19) code 52.7 47.4

65

Dec 13, 2019 LSL

Anonymous

56.8 47.0

66

Apr 5, 2020 GraphSQL

Anonymous

52.8 46.9

67

May 19, 2019 IRNet

Microsoft Research Asia

(Guo and Zhan et al., ACL '19) code 53.2 46.7

68

Mar 17, 2020 SG-IRNet

Anonymous

- 46.6

69

Dec 13, 2019 NatSQL v2

Anonymous

52.0 46.4

70

June 11, 2019 HSRNet

Anonymous

51.5 45.6

71

June 12, 2019 CFGN

Anonymous

48.7 44.1

72

Aug 31, 2019 NatSQL

Anonymous

52.9 42.5

73

May 16, 2019 GNN

Tel-Aviv University & Allen Institute for AI

(Bogin et al., ACL '19) code 40.7 39.4

74

Feb 25, 2019 SASeq

Anonymous

40.8 37.4

75

May 30, 2019 GrammarSQL

Allen Institute for AI

(Lin et al., '19) 34.8 33.8

76

Sep 1, 2019 EditSQL

Yale University & Salesforce Research

(Zhang et al., EMNLP '19) code 36.4 32.9

77

Dec 13, 2019 GuideSQL

Anonymous

36.8 31.5

78

Sep 20, 2018 SyntaxSQLNet + augment

Yale University

(Yu et al., EMNLP '18) code 24.8 27.2

79

April 18, 2019 RCSQL

SAP Labs Korea

(Lee, EMNLP'19) 28.5 24.3

80

Sep 20, 2018 SyntaxSQLNet

Yale University

(Yu et al., EMNLP '18) code 18.9 19.7

81

Sep 20, 2018 SQLNet

Shanghai Jiao Tong University (modified by Yale)

(Xu et al., '18) code 10.9 12.4

82

Sep 20, 2018 TypeSQL

Yale University

(Yu et al., NAACL '18) code 8.0 8.2

83

Sep 20, 2018 Seq2Seq + attention

University of Edinburgh (modified by Yale)

(Dong and Lapata, ACL '16) code 1.8 4.8